Skip to content

AI Object Detection — Passthrough

AI Agent

Download Latest Version

Overview

AI Object Detection brings automatic object identification to XR/AR studies by combining real-time YOLOX object detection with SightLab gaze/head tracking data collection. During a session, the participant wears a passthrough-enabled headset and the headset viewpoint is cast to a desktop window. For example, Meta Quest Developer Hub casting can show the Quest passthrough view in a window on the PC. This script captures that cast/mirror window, runs inference on each captured frame, and creates invisible 3D collision volumes at the approximate positions of detected objects. These volumes are registered as SightLab sceneObjects with gaze=True, so dwell time, view count, and other attention metrics are collected automatically per object — no manual scene object setup required.

The key requirement is that YOLOX sees the same passthrough viewpoint the participant sees. The normal SightLab desktop mirror does not include the headset passthrough camera feed, so use an external cast or mirror source that does include passthrough, such as Meta Quest Developer Hub casting.

With an eye-tracked headset, SightLab records gaze-based object attention. With a headset that does not provide eye tracking, such as Meta Quest 3, the same workflow can still run, but object attention is based on the headset/view direction rather than true eye gaze.

This allows researchers to run mixed-reality studies where participants look around a real passthrough scene and the system automatically records which detected objects they looked toward, for how long, and how many times — all without having to pre-label every object in the scene.

Desktop testing: You can also test the detection pipeline by selecting any visible desktop window as the capture source. This is useful for validating YOLO settings, but it does not replace headset passthrough casting for a real mixed-reality run.

Architecture

Passthrough-enabled headset shows real-world view
        |
        v
External cast/mirror window on the PC
  e.g. Meta Quest Developer Hub cast
        |
        v
ARWindowCapture (captures selected window, can be occluded)
        |
        v
YOLOXObjectDetector (background thread, YOLOX)
        |
        v
VolumeManager
   - Creates vizshape.addBox() collision volumes
   - Registers each as sightlab.addSceneObject(key, node, gaze=True)
   - Matches persistent detections across frames with ByteTrack
        |
        v
SightLab collects gaze/dwell data on each tracked object
        - Eye-tracked headset: eye gaze drives object hits
        - Non-eye-tracked headset: headset/view direction drives object hits

Requirements

Software

Requirement Notes
Vizard 8 WorldViz Vizard with Python 3.x
SightLab sightlab_utils must be on the Python path

Hardware / Capture Source

Requirement Notes
Passthrough-enabled headset Required for mixed-reality object detection. Examples include Meta Quest Pro, Meta Quest 3, and other OpenXR passthrough-capable headsets
Desktop cast/mirror window Required capture source for YOLOX. Use a window that shows the headset passthrough viewpoint, such as Meta Quest Developer Hub casting. This window can be occluded or background, but NOT minimized.
Eye tracking Optional. Eye-tracked headsets produce gaze-based metrics; non-eye-tracked headsets use headset/view direction instead

Config

See AI Object Detection.

How to Run

  1. Start headset passthrough and cast the headset view to the PC.
  2. For Quest headsets, using Meta Quest Developer Hub casting is the recommended option because it can show the passthrough viewpoint that YOLOX needs to analyze. Note that for the Quest Pro, you must open PC Link before beginning casting.
  3. Ensure the casting window is not minimized (it can be occluded or in the background)
  4. Set the detector configs to your choosing (see AI Object Detection).
  5. Open AI_ObjectDetection_MR.py in Vizard and press F5 (or use the "Run WinViz on Current File" task).
  6. Select your hardware config and the casting window title.
  7. Press Spacebar to start the trial — YOLOX detection begins on the captured cast window, and appears in the HMD.\
  8. The participant looks around the passthrough scene; detected objects appear as semi-transparent volumes with labels.
  9. Can press the i key to toggle the participant seeing the debug volumes, or can set this to never show the volumes to the participant by setting SHOW_VOLUMES_IN_HMD = False. This will then only show the debug volumes on the mirrored view.
  10. Press Spacebar again to end the trial.
  11. SightLab saves per-object attention data (dwell time, view count, etc.) to the data/ folder.

Runtime Keyboard Controls

Key Action
b Toggle 2D bounding boxes on mirror window
m Toggle collision volumes in mirror window
i Toggle collision volumes in HMD
o Toggle origin marker
r Reset position
p Toggle SightLab gaze point visibility

How Dwell Time Collection Works

SightLab tracks the active gaze/head ray on registered scene objects automatically. For dwell data to accumulate on a YOLOX-detected object:

  1. The object must persist with the same key across multiple frames (e.g. yolo_chair_3 stays yolo_chair_3)
  2. The object must survive long enough for the user's gaze/head ray to exceed SightLab's dwell threshold (default 500ms)

Output Data

SightLab saves standard experiment data to the data/ folder, including per-object:

  • Dwell time — total time gaze/head direction rested on each detected object
  • View count — number of times gaze/head direction entered each object
  • Average dwell time — mean gaze duration per view
  • First view time — when the user first looked at each object
  • Gaze/head timeline — temporal sequence of object attention events

Each YOLO-detected object appears in the data with its key (e.g. yolo_chair_3, yolo_laptop_7). When using a headset without eye tracking, label exported metrics and study notes accordingly so they are not interpreted as eye-gaze measurements.