Everything on the Quest 3 is a performance budget
AR Drone was a group project for Augmented Reality - Interaction, done with Lund University School of Aviation: fly a real drone through a virtual obstacle course seen through a Meta Quest 3. The Quest 3 has to run passthrough, render the scene, and do inference at the same time, and that ceiling shaped every decision we made.
The first YOLO model we tried was simply too resource intensive for the headset, which is how we ended up training a nano-sized one ourselves. A fair number of the drone-detection resources on HuggingFace are published as training sets rather than finished models. It was trained on drones in general. Training one on the specific drone we had borrowed would have been better, and there was never time for it.
Near the end of the project Meta shipped a Building Block for object detection. Our model would not work with it, because we had the non-max suppression layer inside the model graph and the block expects to run NMS as a separate post-processing pass. We dropped ours for a finished model off HuggingFace that matched what the block expected. Splitting it that way meant NMS could run on the GPU while the model ran on the CPU, so the GPU was not left running the model, handling NMS, and rendering passthrough at once. With quantization on top, we got detection confidence of 0.85 to 0.90 at a reasonable latency.
The other thing worth writing down is how you turn a 2D detection into a position in the room, because we ended up trying both ways. Raycasting sends a ray from the detection into the scene and gives you world coordinates, so the marker is rooted in the room and stays on the drone through head movement. Meta’s block instead uses a depth texture, which is faster and more accurate, but returns coordinates in local screen space. During fast movement the marker stayed put on the display rather than on the drone, and only re-attached at the next detection.
The marker is not just an indicator, it is a collider. The box follows the drone’s world position, and a collision fires when it intersects one of the obstacles, which turns that obstacle red and plays a crash sound. Where we left it: the block’s object detection can be combined with the earlier raycasting version, so tracking holds through head movement on world coordinates instead of local screen coordinates.