AI Object Detection - Documentation

Click here to Download the Latest Version

Overview

AI Object Detection brings automatic object identification to VR eye tracking studies by combining real-time YOLOX object detection with SightLab gaze data collection. During a session, the script captures the Vizard render MainScene, runs YOLO inference on an interval to identify objects in the scene, and overlays detections at the positions of objects. Gaze intersection with detections is automatically registered with Sightlab, so dwell time, view count, and other gaze metrics are collected automatically per object— no manual scene object setup required.

This allows researchers to run VR eye tracking studies where participants look around a virtual environment and the system automatically records what they looked at, for how long, and how many times, all without having to pre-label every object in the scene. See also this page for a version that can track using AR Passthrough.

Desktop testing: The script also works in desktop mode without a headset, which is useful for testing and validating detection settings before running a full VR session.

Requirements

Software

Requirement	Notes
Vizard 8	WorldViz Vizard with Python 3.x
SightLab	`sightlab_utils` must be on the Python path
YOLOX/requirements.txt	Python pypi dependencies

Installing Dependencies in Vizard

Use Vizard's built-in Package Manager (Tools → Package Manager → cmd) and run pip install -r "path\to\Sightlab2\ExampleScripts\AI_ObjectDetection\YOLOX\requirements.txt" or run pip directly from Vizard's Python:

"C:\Program Files\WorldViz\Vizard8\bin\python.exe" -m pip install -r "path\to\Sightlab2\ExampleScripts\AI_ObjectDetection\YOLOX\requirements.txt"

Files

File	Purpose
`AI_ObjectDetection_Config.py`	All tunable settings (model, thresholds, visuals, capture, environment)
`AI_ObjectDetection.py`	Main script — run this in Vizard (can change to run with the GUI or without the GUI in the Config file)

How to Run

Open AI_ObjectDetection_Config.py and verify settings (model, environment, confidence, etc.)
USE_GUI - Choose whether to run with the SightLab GUI or code
Open AI_ObjectDetection.py in Vizard and press F5 (or use the "Run WinViz on Current File" task)
The script loads the configured environment (default: homeOffice.osgb)
Click or Trigger to start the trial — YOLO detection begins automatically
The participant looks around the VR scene; detected objects appear as semi-transparent boxes with labels
Press Spacebar to end the trial
SightLab saves gaze data (dwell time, view count, etc.) per detected object to the data/ folder

Output Data

SightLab saves standard experiment data to the data/ folder, including for each detection:

Dwell time — total time gaze rested on each detected object
View count — number of times gaze entered each object
Average dwell time — mean gaze duration per view
First view time — when the user first looked at each object
Gaze/Dwell timeline — temporal sequence of gaze events

Each YOLO-detected object appears in the data with its key (e.g. yolo_chair_3, yolo_laptop_7). Instances of each class are differentiated and tracked using ByteTrack via supervision (e.g., yolo_cup_1 and yolo_cup_2 are two different tracked objects).

Runtime Keyboard Controls

Key	Action
`b`	Toggle 2D bounding boxes on mirror window
`m`	Toggle collision volumes in mirror window
`i`	Toggle collision volumes in HMD
`o`	Toggle origin marker
`r`	Reset position
`p`	Toggle SightLab gaze point visibility

Configuration Reference (`AI_ObjectDetection_Config.py`)

YOLO Detection

Setting	Default	Description
`CONFIDENCE`	`0.6`	Minimum confidence threshold (0.0–1.0). Lower = more detections but more false positives
`DETECTION_INTERVAL`	`0.1`	Seconds between YOLO inference runs

3D Collision Volumes

Setting	Default	Description
`USE_OBJECT_MANAGER`	`True`	Create 3D collision volumes in the scene that can be used to automatically log gaze per object
`ENABLE_GAZE_TRACKING`	`True`	Enable gaze tracking on detected objects' collision volumes, using SightLab registered Gaze Objects. If False and USE_OBJECT_MANAGER is true, the collision volumes are only for visual purposes, and log no data
`MAX_TRACKED_VOLUMES`	`15`	Maximum tracked 3D collision volumes
`SHOW_VOLUMES_IN_HMD`	`True`	Show collision volumes in HMD view
`SHOW_VOLUMES_IN_MIRROR`	`True`	Show collision volumes in desktop mirror
`DEBUG_VOLUME_ALPHA`	`0.25`	Default collision volume alpha (opacity)
`LOG_GAZE_IN_CONSOLE`	`True`	Print when user gazes at a detected object's collision volume

2D Bounding Box Overlays

Setting	Default	Description
`USE_BOUNDING_BOXES`	`True`	Overlay YOLO bounding boxes on the mirrored view (can not be shown in HMD), not used for gaze logging
`SHOW_BOUNDING_BOXES`	`True`	Show 2D bounding boxes on the mirror window (classic YOLO-style outlines). These are screen-space overlays that appear on the researcher's desktop view on top of the 3D scene
`BOUNDING_BOX_COLOR`	`[0, 1, 0]`	RGB bounding box color
`SHOW_LABELS`	`True`	Show text labels above detected objects (and collision volumes if enabled)

Sightlab Configs

Setting	Default	Description
`ENVIRONMENT_MODEL`	`'sightlab_resources/environments/homeOffice.osgb'`	Default environment model
`INSTRUCTION_MESSAGE`	`AI Object Detection\nLook around the scene to automatically detect objects\n\nPress Trigger or Click to Continue`	Instructions shown at the start of the experiment
`USE_GUI`	`True`	Use Sightlab GUI to configure experiment settings
`SCREEN_RECORD_WINDOW`	`False`	Sightlab recording parameter