AI Object Detection — Documentation

AI Agent

Overview

AI Object Detection brings automatic object identification to VR eye tracking studies by combining real-time YOLOv8 object detection with SightLab gaze data collection. During a session, the script captures the Vizard render window, runs YOLO on each frame to identify objects in the scene, and overlays detections at the approximate positions of detected objects. Gaze intersection with detections is automatically registered with SightLab, so dwell time, view count, and other gaze metrics are collected automatically per object — no manual scene object setup required.

This allows researchers to run VR eye tracking studies where participants look around a virtual environment and the system automatically records what they looked at, for how long, and how many times — all without having to pre-label every object in the scene. See also this page for a version that can track using AR Passthrough.

Desktop testing: The script also works in desktop mode without a headset, which is useful for testing and validating detection settings before running a full VR session.

Requirements

Software

Requirement	Notes
Vizard 8	WorldViz Vizard with Python 3.x
SightLab	`sightlab_utils` must be on the Python path
ultralytics	YOLOv8 — `pip install ultralytics`
opencv-python	Image processing — `pip install opencv-python`
numpy	Array handling — `pip install numpy`
pywin32	Window capture — `pip install pywin32`

Installing Dependencies in Vizard

Use Vizard's built-in Package Manager (Tools → Package Manager) or run pip directly from Vizard's Python:

"C:\Program Files\WorldViz\Vizard8\bin\python.exe" -m pip install ultralytics opencv-python numpy pywin32

Note: The first time ultralytics runs, it will download the YOLOv8 model file (~6 MB for yolov8n.pt). This requires an internet connection.

Files

File	Purpose
`AI_ObjectDetection_Config.py`	All tunable settings (model, thresholds, visuals, capture, environment)
`AI_ObjectDetection.py`	Main script — run this in Vizard (can change to run with the GUI or without the GUI in the Config file)

How to Run

Open AI_ObjectDetection_Config.py and verify settings (model, environment, confidence, etc.)
CAPTURE_WINDOW_TITLE must match the Vizard render window title (default: "AI_ObjectDetection"). Set to None to be prompted with a window picker at startup
Default depth (meters) at which to place detected objects in 3D space (Default 2 meters)
USE_GUI - Choose whether to run with the SightLab GUI or code
Open AI_ObjectDetection.py in Vizard and press F5 (or use the "Run WinViz on Current File" task)
The script loads the configured environment (default: homeOffice.osgb)
Press Spacebar to start the trial (and trigger or left mouse button to click past the instructions)— YOLO detection begins automatically
The participant looks around the VR scene; detected objects appear as semi-transparent boxes with labels
Press Spacebar again to end the trial
SightLab saves gaze data (dwell time, view count, etc.) per detected object to the data/ folder

Output Data

SightLab saves standard experiment data to the data/ folder, including for each detection:

Dwell time — total time gaze rested on each detected object
View count — number of times gaze entered each object
Average dwell time — mean gaze duration per view
First view time — when the user first looked at each object
Gaze/Dwell timeline — temporal sequence of gaze events

Each YOLO-detected object appears in the data with its key (e.g. yolo_chair_3, yolo_laptop_7) if the experimental DetectedObjectManager class is used, otherwise they are labeled by class.

Runtime Keyboard Controls

Key	Action
`Space`	Start / stop trial
`d`	Toggle debug bounding boxes on/off
`i`	Toggle YOLO overlays in HMD (keeps them on desktop mirror for researcher)
`o`	Toggle origin direction arrow
`r`	Reset viewpoint position
`p`	Toggle SightLab gaze point visibility

Configuration Reference (`AI_ObjectDetection_Config.py`)

YOLO Detection

Setting	Default	Description
`YOLO_MODEL`	`'yolov8n.pt'`	Model size. Options: `yolov8n.pt` (nano, fastest), `yolov8s.pt` (small), `yolov8m.pt` (medium, most accurate)
`YOLO_CONFIDENCE`	`0.6`	Minimum confidence threshold (0.0–1.0). Lower = more detections but more false positives
`YOLO_DETECTION_INTERVAL`	`0.1`	Seconds between YOLO inference runs. Lower = more responsive, higher = less CPU
`YOLO_CLASSES`	`None`	COCO class IDs to detect. `None` = all classes. Example: `[56, 62, 63]` for chair, tv, laptop

Visualization

Setting	Default	Description
`SHOW_DEBUG_VOLUMES`	`True`	Show green semi-transparent collision volumes over detected objects (only takes effect when `USE_OBJECT_MANAGER` is `True`)
`DEBUG_VOLUME_ALPHA`	`0.25`	Opacity of debug collision volumes (0.0–1.0) (only takes effect when `USE_OBJECT_MANAGER` is `True`)
`SHOW_LABELS`	`True`	Show text labels (class name + confidence) above each object
`SHOW_VOLUMES_IN_HMD`	`True`	Whether detection volumes render in the HMD at startup. Toggle with `i` key at runtime. When off, detection volumes still appear on the desktop mirror (only takes effect when `USE_OBJECT_MANAGER` is `True`)

Gaze Tracking

Setting	Default	Description
`ENABLE_GAZE_TRACKING`	`False`	Register detected objects' collision volumes as SightLab gaze targets
`GAZE_INTERSECTION_DETECTION_INVERAL`	`0.0111`	Seconds between gaze intersection detections. Lower = more precise timing measurement (this should not exceed your HMD's update rate), higher = less CPU
`USE_GAZE_BASED_ID`	`True`	Print console messages and show labels when gaze dwells on an object

Experimental DetectedObjectManager Class Configuration

Setting	Default	Description
`USE_OBJECT_MANAGER`	`True`	Enable creation of 3D collision volumes for detected objects
`COLLISION_VOLUME_SIZE`	`[0.2, 0.2, 0.15]`	Default size of the collision volume for each detected object (meters)
`DEFAULT_VOLUME_DEPTH`	`0.5`	Default depth (meters) at which to place detected objects' collision volumes in 3D space
`MATCHING_DISTANCE_THRESHOLD`	`0.5`	How close (normalised 0-1 screen space) a new detection must be to an existing tracked volume of the same class to be considered the SAME object. Higher = more forgiving when the user moves their head.
`VOLUME_PERSISTENCE_TIME`	`0.5`	How long (seconds) a volume persists after its object is no longer detected.
`MAX_TRACKED_VOLUMES`	`15`	Maximum simultaneous tracked objects and their collision volumes

Window Capture

Setting	Default	Description
`CAPTURE_WINDOW_TITLE`	`"AI_ObjectDetection"`	Window title to capture. Must match the Vizard window title. Set to `None` to be prompted with a window picker at startup
`CAPTURE_FLIP`	`None`	Flip captured frame: `0` = vertical, `1` = horizontal, `-1` = both, `None` = no flip

Other

Setting	Default	Description
`SCREEN_RECORD`	`True`	Enable SightLab's built-in screen recording
`ENVIRONMENT_MODEL`	`'sightlab_resources/environments/homeOffice.osgb'`	3D environment to load
`INSTRUCTION_MESSAGE`	(see config)	Text shown at trial start

Common COCO Class IDs

For use with YOLO_CLASSES:

ID	Class	ID	Class	ID	Class
0	person	56	chair	66	keyboard
39	bottle	57	couch	67	cell phone
41	cup	58	potted plant	73	book
46	banana	59	bed	74	clock
47	apple	60	dining table	75	vase
49	orange	62	tv/monitor	76	scissors
51	carrot	63	laptop	77	teddy bear
55	cake	64	mouse

Full list: COCO dataset classes

Architecture

flowchart TD
    A["`Vizard renders a 3D scene
    (e.g. homeOffice)`"] --> B["`Mirror the HMD view as a desktop window`"]
    B --> C["`**YOLOScreenDetector** class
    (two daemon threads, ultralytics YOLOv8)`"]
    C --> C1["`WindowCapture
    - finds window by title from config
    - converts captured window into NumPy rgb frame to run inference on`"]
    C1 --> D["`**YOLO inference loop** 
    (daemon thread)
    - Detections set every YOLO_DETECTION_INTERVAL seconds
    - Posts async INFERENCE_EVENT to main thread to overlay detections on mirrored window
    - Detections gazed >= 0.5s (default SightLab dwell time) are logged as dwelled.
    Detections are represented using the two methods below.`"]
    D --> Branch{"ONE/BOTH of"}
    Branch --> E["`**Gaze intersection loop** 
    (daemon thread)
    - Checks if gaze point (x,y) falls in any detected bounding boxes
    - Sets currently gazed class every GAZE_INTERSECTION_INTERVAL seconds
    - Compares to last gazed class to detect change
    - Appears in mirror view ONLY`"]
    Branch --> F["`**CollisionVolumeManager** (separate class)
    - 3D collision volumes registered as SightLab scene objects for each detection
    - Volumes persist across frames as unique scene objects, differentiating same-class objects (e.g. cup_1, cup_2)
    - Appears in mirror view, HMD view, or both`"]
    E --> G["`**YOLOScreenDetector** collects gaze/dwell data on each detected class`"]
    F --> H["`SightLab collects gaze/dwell data on each tracked object`"]

How Dwell Time Collection Works

The YOLOScreenDetector tracks gaze on detections automatically. For dwell data to accumulate on a detection:

  1. The object must persist across multiple frames
  2. The object must survive long enough for the user's gaze to exceed SightLab's dwell threshold (default 500ms)

Gaze Intersection Detection Methods

There are two ways to detect gaze intersection with YOLO detections:

  1. The 2D BoundingBoxOverlay class (two-dimensional flat bounding boxes from YOLO)
  2. The 3D DetectedObjectManager class (three-dimensional collision volumes created in SightLab)

The BoundingBoxOverlay class manages overlaying bounding boxes in the mirrored experiment window using the detections and their coordinates returned by the YOLOScreenDetector class. Since the mirrored HMD view is the captured window for YOLO inference, the bounding boxes are returned in the captured window's coordinate system. The DetectedObjectManager class also uses the detections' coordinates to create 3D collision volumes as gaze-registered SightLab scene objects in the mirrored experiment window.

The BoundingBoxOverlay method is the default object detection method. Since the bounding boxes aren't created as SightLab objects, gaze intersection is implemented in the YOLOScreenDetector class itself in a second daemon thread.

The DetectedObjectManager is an experimental feature that has object persistence capabilities. For example, if a cup is detected in the captured window, a collision volume is created as a SightLab object in the approximate location of the cup with a unique id for that specific instance of the cup class. Other cups detected will have different unique ids, and the SightLab gaze tracking will recognize the difference between the two cups as the two SightLab objects transform along with the HMD POV changes.

Default YOLO bounding boxes are created frame by frame, and the difference between two objects of the same class is not tracked when using the BoundingBoxOverlay class as the detection method.

Troubleshooting

Issue	Solution
No detections appearing	Check that `CAPTURE_WINDOW_TITLE` in the config matches the Vizard window title. Try setting it to `None` to use the window picker
`DeleteDC failed` error	The script already calls `screen_capture.stop_capture()` to prevent this. If it still occurs, ensure only one capture source is active
`ultralytics not installed` warning	Install with: `pip install ultralytics` using Vizard's Python
Vizard autocomplete spamming errors	The script uses `__import__()` for third-party packages to avoid this. If it persists, ensure no standard `import ultralytics` lines exist
Boxes appear but no gaze data	Verify `ENABLE_GAZE_TRACKING = True` and that the collision box depth is sufficient (≥ 0.1m)
Low frame rate	Increase `DETECTION_INTERVAL`, use `yolov8n.pt` (nano model), or reduce `MAX_TRACKED_OBJECTS`