Skip to content

External Data Recorder - Overview

External Data Replay with AI-detected objects, including chairs, bottles, a clock, and a person, outlined in a painted café scene

Beta

This page covers the External Data Recorder 2.0 beta. Contact support@worldviz.com for beta access. For version 1.0, see External Application Data Recorder.

The External Data Recorder records, saves, and synchronizes eye tracking and physiological data while participants use external applications, including SteamVR games, Unity and Unreal applications, Meta apps, web-based VR, standalone headset apps, and desktop applications such as first-person games. After a session, AI object detection measures which objects participants looked at, and you can replay the recording with gaze and object overlays synchronized with the data.

What's New in 2.0

  • AI object detection: after each session, every frame of the recording is scanned for everyday objects (COCO classes), or for exactly the objects you describe with open-vocabulary detection (see AI Object Detection)
  • Gaze data on objects: dwell time, fixations, and dwell counts for each detected object and class, saved to CSV files
  • A real calibration stage: follow a short sequence of dots in the headset, with no need to manually line up a virtual screen (see the Eye Calibration Guide)
  • Angular accuracy reported every session: a validation round measures accuracy in degrees and prompts you to recalibrate if it isn't good enough
  • An audit trail of gaze quality: each participant's calibration and validation results are saved to a JSON file
  • Headset slippage check: an optional validation at the end of the session measures how much accuracy drifted during the test
  • Simpler, more stable recording: the video recording methods from 1.0 are replaced by OBS Studio, and the old "may crash after about 10 minutes" caveat is gone
  • Cropped recordings: for setups that use the center of the view as the gaze point, crop the recording to the region you care about
  • Pimax Dream Air support, with a step-by-step guide
  • Reprocess old sessions: run postprocessing again on any recorded session with different detection settings (see the Postprocessing Guide)

At a Glance

  • Platforms: Unreal, Unity, and SteamVR/OpenXR PC apps; desktop apps; web; standalone headset apps via casting
  • Headsets with eye tracking: Vive Focus Vision, Vive Focus 3, Vive Pro Eye, Meta Quest Pro, Pimax Dream Air, Varjo XR-3/XR-4, HP Omnicept, and generic OpenXR headsets (results may vary)
  • Without eye tracking: Meta Quest 3, 3S, and 2 use the center of the headset view as the gaze point; desktop apps use the mouse or the center of the screen
  • Outputs: CSV files (gaze, fixations and saccades, events, custom markers, face tracking, gaze on detected objects, and more), videos with AI object detection and gaze overlays, Biopac AcqKnowledge markers, and a replay with scan paths, fixation spheres, and heatmaps. See Common Metrics for the full list.

Setup

Install prerequisites

  • In the Vizard Package Manager, open the CMD tab and install the requirements.txt file from the External Data Recorder folder: pip install -r "path\to\Sightlab2\ExampleScripts\External Data Recorder\requirements.txt"
  • The requirements install PyTorch built for CUDA 13.2 (tested with 2.13.0+cu132). Some graphics cards need a different PyTorch build.
  • For Vive Focus Vision, Focus 3, and Pro Eye: the SRanipal driver, installed with Vive Console for SteamVR
  • OBS Studio

OBS setup

The External Data Recorder records your sessions by controlling OBS over WebSockets, so the OBS WebSocket server must be turned on:

  1. Open OBS. If it is your first time running OBS, keep the default settings and optimize for recording.
  2. In the menu bar, open Tools > WebSocket Server Settings.
  3. Check Enable WebSocket server.
  4. Uncheck Enable Authentication, and make sure Server Port is set to 4455. (Advanced users can change the script to use authentication.)
  5. Click OK.

OBS WebSocket Server Settings with Enable WebSocket server checked, Server Port set to 4455, and Enable Authentication unchecked

If OBS isn't running when a session starts, the recorder tries to open it from its default install location. For each session, the recorder creates an OBS scene called "External Data Recorder", adds the window you chose to it, and deletes the scene when it's done, so your existing OBS scenes aren't affected. It does change the output folder for all OBS recordings, though. All OBS requests are in the record_screen_obs() function in External_Data_Recorder.py; see the OBS WebSocket protocol for what each one does.

Quick Start

This walkthrough uses a Vive Focus Vision recording an Unreal app through SteamVR's VR View. The steps are the same for other hardware; see the hardware guides for headset-specific setup.

OBS WebSocket Server Settings with Enable WebSocket server checked, Server Port set to 4455, and Enable Authentication unchecked 1. Complete setup. Install the requirements and OBS Studio, and turn on the OBS WebSocket server (see Setup).
SteamVR's VR View window mirroring the headset view on the desktop 2. Connect your headset and open its mirror window. The recorder records the headset's view through a mirror window on the desktop:
Vive: right-click the SteamVR Status window and choose Display VR View.
Meta Quest Pro: open the Oculus Mirror.
Pimax Dream Air: turn on Screen Mirror (Right Eye).
For desktop apps, skip this step.
The External Data Recorder in the SightLab Dashboard, with Run: External Data Recorder and Replay buttons 3. Start the External Data Recorder. In the SightLab Dashboard, go to Tools/Features > External Data Recorder and click Run: External Data Recorder. You can also open External_Data_Recorder.py in Vizard and click the green run arrow.
Prompt to select the window to record, with VR View selected 4. Choose the window to record. For a headset, choose its mirror window (here, VR View). For a desktop app, choose the app's window. The window can be covered by other windows, but it must not be minimized at any point.
The External Data Recorder console window showing status messages 5. Keep an eye on the console. A console window opens with the recorder. Press Alt+Tab to switch to it at any time to see progress, calibration results, and errors. Set DEBUG = True in Data_Recorder_Config.py for more detail. The console output is also saved to the logs folder.
Prompt to select a hardware configuration, with Vive Focus Vision Recorder selected 6. Choose your hardware. Pick the configuration that matches your headset or desktop setup (see Choosing a Hardware Configuration).
Eye calibration instructions: fixate each red dot, press Space or the trigger, and keep fixating until the dot moves 7. Calibrate the eye tracker. When asked for a calibration file, choose New. In the headset, look at the center of each red dot, press the trigger or Spacebar, and keep looking until the dot moves. Keep your head still. Hardware without eye tracking skips this step. See the Eye Calibration Guide.
Console message reporting that the validation error was above the threshold 8. Check the validation. After 9 calibration dots, 5 validation dots measure the accuracy in degrees. If the error is too high, the console says so: press the trigger or Spacebar to calibrate again, or A to accept the result. After three failed rounds, the most accurate one is kept.
An Unreal app running in the headset and showing in the VR View window 9. Open your external application. Start the app you want to record and make sure it shows in the mirror window. With OpenXR runtimes, you'll be asked to choose the app, and it's brought into focus when the trial starts.
Participant Information form with Last Name, First Name, and ID fields 10. Set the recording length and participant. Enter how many seconds to record (10 by default), then optionally enter the participant's name and ID and click Submit.
OBS recording the External Data Recorder scene, with Stop Recording active and the recording timer running 11. Record. Press Spacebar to start. OBS records the window in its own External Data Recorder scene, and a red dot appears on the OBS icon in the taskbar while it records. A beep plays when the trial ends. If Biopac is connected, AcqKnowledge acquisition starts with the recording.
Message: Postprocessing video, don't close this window. Check the console window for updates. 12. Wait for postprocessing. The recording is analyzed with AI object detection and gaze data. Don't close the recorder window; Alt+Tab to the console to follow progress. Closing SteamVR, Unreal, or Unity first makes this much faster. For Desktop First Person Game and Meta Quest 3 Recorder, you'll be asked to crop the recording and adjust the gaze point. See the Postprocessing Guide.
Message: Session finished, you may now close this window, with the location of the log file 13. Find your data. When the session is finished, close the window. Videos are in recordings, data files in data, calibration results in calibration_data, and console logs in logs (see What Gets Recorded).
External Data Replay prompt to choose between the transcoded and overlay videos 14. Replay the session. Click Replay in the Dashboard, or run External_Data_Replay.py. Choose the session, the calibration (Same as video if you calibrated in that session), and the overlay or transcoded video (see Session Replay).

Choosing a Hardware Configuration

Hardware option Gaze point Calibration Guide
Vive Focus Vision Recorder, Vive Focus 3 Recorder, Vive Pro Eye Headset eye tracking (SRanipal) Yes Vive Headsets
Meta Quest Pro Headset eye tracking over Meta Horizon Link Yes Meta Headsets
Pimax Dream Air Headset eye tracking Yes Pimax Dream Air
Meta Quest 3 Recorder Center of the headset view No Meta Headsets, or Meta Standalone Apps for apps that run on the headset
Desktop First Person Game Center of the screen No Desktop and Desktop First Person
Desktop Mouse cursor No Desktop and Desktop First Person
Omnicept Recorder, Varjo, OpenXR Recorder Headset eye tracking Yes Guide coming soon

How It Works

flowchart LR
    A[Choose the window<br>and hardware] --> B[Calibrate and<br>validate gaze]
    B --> C[Record the window<br>with OBS]
    C --> D[Postprocess: detect objects<br>and match gaze]
    D --> E[Data files, videos,<br>and replay]
  • Why record the mirror window: calibration maps the eye tracker's gaze onto the pixels of the window you choose, so the same window, at the same size, has to be used from calibration through recording. Even without eye tracking, the mirror window matches what the participant actually sees more closely than the app's own desktop window.
  • Calibration: Eye_Calibration.py opens in the headset, shows a grid of dots, finds each dot in the mirror window, and fits a mapping from gaze direction to window pixels. Validation then tests that mapping on separate dots. Resizing the mirror window or headset slippage after calibration reduces accuracy.
  • Recording: OBS records the chosen window while SightLab records eye tracking, physiological, and event data on the same timeline.
  • Postprocessing: Postprocessor.py runs AI object detection on the recording, matches the gaze point to detected objects, writes the object CSV files, and creates the overlay and transcoded videos. It runs automatically after each trial when AUTO_POSTPROCESS = True.

AI Object Detection

Postprocessing runs an AI object detector over every frame of the session recording. It draws a box around each object it finds, gives each object an ID that follows it from frame to frame (for example #44 clock), checks which box the gaze point falls in, and records dwell time and fixations on each object and each class of object.

Two detection models are available:

RT-DETRv2 (default) OmDet-Turbo (open vocabulary)
What it detects The 80 COCO classes Whatever you describe in words
Speed About 2–3 times faster Slower, since it's a larger model
Use it when Your scene contains everyday objects such as people, vehicles, furniture, bottles, or screens You need specific objects, such as "coffee mug" or "yellow car", or objects that aren't COCO classes

RT-DETR (Real-Time DEtection TRansformer) is a fast, transformer-based object detector. The version used here, RT-DETRv2, is trained on COCO.

COCO (Common Objects in Context) is a widely used image dataset with 80 classes of everyday objects, such as person, bicycle, car, bus, traffic light, chair, couch, bed, dining table, bottle, cup, laptop, TV, cell phone, book, and clock. A model trained on COCO can only label objects as one of those 80 classes. If you know some classes can't appear in your scene, list them in BANNED_CLASSES so objects aren't mislabeled as them.

Open-vocabulary detection isn't limited to a fixed list. You write the classes you want as plain phrases, and the model finds only those. You can also give it a natural-language instruction:

# Postprocess_Config.py
USE_OBJECT_DETECTION = True
USE_OVD = True
OVD_CLASSES = ["coffee mug", "yellow car", "computer monitor"]
OVD_TASK = "Detect anything that looks like a coffee mug, a yellow car, or a computer monitor."

How gaze is matched to objects

  • By default, each box is padded by the eye tracker's measured accuracy from validation (USE_ANGULAR_THRESHOLD), so small gaze errors still count as looking at the object.
  • If the gaze point isn't in any box, the nearest "near miss" within that accuracy margin is used (CLOSEST_EDGE).
  • A dwell starts once gaze stays on an object for DWELL_THRESHOLD seconds (0.5 by default), and short dropouts of up to DWELL_GRACE_S don't break it.

Tips

  • Object detection needs a CUDA-capable graphics card with a matching PyTorch build.
  • Close VRAM-heavy programs (SteamVR, Unreal, Unity) before postprocessing, or set USE_FP16 = True, to speed it up.
  • Try different models and thresholds on a recorded session by running Postprocessor.py on its own. See the Postprocessing Guide for all settings.

Session Replay

External Data Replay showing detected objects and the Replay Preferences panel

  1. Click Replay in the SightLab Dashboard, or run External_Data_Replay.py.
  2. Choose the session. Sessions are named by the date and time they were recorded and the participant ID.
  3. Choose the calibration file. Select Same as video if you calibrated in the same session; otherwise, find the calibration you reused in the list.
  4. Choose the video: the overlay video shows detected objects and the gaze point, and the transcoded video has no overlays.
  5. Make the External_Data_Replay window full screen.

In the replay you can:

  • Scrub: drag the slider, or step with B/N or C/V.
  • Follow the participant's view: first-person view is on by default (FOLLOW_ON).
  • Sync with AcqKnowledge: when AcqKnowledge is connected, scrubbing the replay moves AcqKnowledge to the same time (see Synchronizing Session Replay with AcqKnowledge).

See Session Replay for all replay features and controls.

What Gets Recorded

Files are named with the session's <date>_<participant ID>, so files from the same session are easy to match.

Videos (recordings/)

  1. <id>_overlay_trial_<n>: the recording with object detections and the gaze point drawn on it
  2. <id>_transcoded_trial_<n>: the recording converted for use in the replay, without overlays
  3. <id>_experiment_data_trial_<n>: the original OBS recording

The overlay and transcoded videos can both be used in the replay.

Console logs (logs/)

Each session's console output, saved when you close the recorder.

SightLab data files (data/<id>_experiment_data/)

SightLab records the available eye tracking, head tracking, physiological, and other data into its standard CSV files (see Data Analysis Overview and Common Metrics). The trial_timeline_dwell file doesn't contain useful object data for external applications, since there is no SightLab 3D scene; object dwell data is written to the AI object detection files below instead.

Calibration files (calibration_data/)

Each calibration is saved as a JSON file with the hardware configuration, the window it was calibrated against, the calibration settings, every calibration point, and every validation attempt with its accuracy.

Calibration file fields
  • datetime: when the file was written
  • hardware_config: the selected hardware configuration
  • window_title: the title of the window calibrated against, such as SteamVR's "VR View"
  • window_size_px: the window's width and height in pixels
  • mirror_eye: the eye the mirror window shows, and that gaze data is collected from
  • dwell_seconds: how long gaze is sampled on each calibration dot
  • settle_seconds: how long gaze is allowed to settle on a dot before sampling
  • dot_depth_m: the distance of the calibration dots, in meters
  • dot_extent_deg: half the angular size of the calibration grid in the headset view
  • grid: the columns and rows of the calibration grid
  • dot_jitter_frac: the small offset applied so no three dots are in a straight line
  • validation_attempts: each validation attempt, with:
    • attempt: the attempt number
    • n_calibration_points: the number of calibration points
    • passed: whether the validation passed
    • reason: a summary of the errors if it passed, or why it failed
    • summary: n_points, mean_deg (mean angular error), rms_deg (root mean square angular error), max_deg (largest angular error), rms_px and max_px (pixel errors), rms_pct_diag (pixel error as a percentage of the window diagonal, for comparing different window sizes), mean_offset_px and offset_magnitude_px (average offset between gaze and targets), offset_noise_floor_px (the offset expected from random error alone), and mean_precision_deg
  • saved_attempt: which validation attempt was kept
  • points: each calibration point, with its dot_index, target_px (the dot's position in the window), gaze_ab (the sampled gaze direction), dot_world (the dot's 3D position), n_red_target_frames, n_gaze_samples, and precision_deg

AI object detection files (data/<id>_experiment_data/trial_data/)

When object detection is on, three CSV files are written for each trial.

Object detection timeline (obj_det_timeline_trial_<n>.csv), one row per video frame:

  • Frame PTS: the frame's presentation timestamp, in seconds
  • All Detected Classes, All Detected Objects, All Bounding Boxes, All Confidences: everything detected in the frame
  • Gazed Object and Gazed Object Confidence: the object being looked at
  • Gaze Flag: whether the gaze was a fixation or a saccade
  • Object Dwell: Gaze Started when the gaze point enters an object's box, Dwell Registered once gaze has stayed for DWELL_THRESHOLD seconds, and Dwelling while it continues
  • Class Dwell: the same as Object Dwell, for the object's class
  • Gaze Screen X Position and Gaze Screen Y Position: the gaze point in pixels of the recorded window
  • Near Misses Angular Margins and Near Misses Objects: boxes the gaze point fell just outside of, within the accuracy margin, and how far outside

Object detection summary (obj_det_summary_trial_<n>.csv), one row per detected object:

  • Tracker ID: the ID assigned to the object (by ByteTrack)
  • Total Dwell, Mean Dwell, and Dwell Count
  • Total Fixation Time and Total Saccade Time while dwelling on the object
  • First Dwell Time and First Frame
  • Mean Confidence: the average detection confidence while dwelling
  • Majority Class: the class with the most dwell time, since a tracked object's label can occasionally change

Class detection summary (class_det_summary_trial_<n>.csv): the same measures as the object summary, per class instead of per object.

Biopac Integration

With BIOPAC_ON = True in Data_Recorder_Config.py, the recorder connects to AcqKnowledge:

  • Acquisition starts with the session recording, and a video start marker is inserted when recording begins
  • Press t (NETWORK_SYNC_KEY), or send the triggerPress event (NETWORK_SYNC_EVENT), to insert a sync event marker at any point
  • In the replay, scrubbing moves AcqKnowledge to the same time

See Biopac Integration for setting up AcqKnowledge, and Sending Events to Biopac for custom markers.

Lab Streaming Layer

With SAVE_LSL_DATA = True, the recorder connects to the first Lab Streaming Layer stream it finds on the network when a trial starts, and saves each sample and its timestamp to the LSL Data and LSL Timestamp columns of the trial data. If no stream is found, the columns say so. See Lab Streaming Layer for sending data from other devices.

Network Events with External Applications

The External Data Recorder can be controlled from an external application, for example starting the trial when the app sends a signal. It can also send triggers to the app, and exchange other information through network events. External apps send UTF-8 JSON over UDP, such as {"event": "start_trial"} or {"event": "sync"}, to NETWORK_HOST and NETWORK_PORT. See External Data Networking for details and Unity and Unreal examples.

Configuration

The External Data Recorder's settings are in three places:

File What it controls Reference
Data_Recorder_Config.py Recording, session control, Biopac, network events, LSL, live preview, replay, and hardware options The table below
Data_Recorder_Config.py Eye calibration and validation Eye Calibration Guide
Postprocess_Config.py Object detection, dwell measurement, cropping, gaze point adjustment, and overlay drawing Postprocessing Guide

Recorder and replay settings (Data_Recorder_Config.py)

Setting Default Description
DEBUG True Print diagnostic messages to the console
PID True Ask for a participant ID and optional name, and save them to the data
DEFAULT_PARTICIPANT_ID "0" ID used when the participant ID is left blank
BIOPAC_ON True Communicate with Biopac AcqKnowledge
LOCK_TRANSPORT True Lock the transport
NETWORK_SYNC_KEY "t" Key that sends an event marker to AcqKnowledge
NETWORK_SYNC_EVENT "triggerPress" Event that sends an event marker to AcqKnowledge
USE_NETWORK_EVENT False Send network events to the external app
NETWORK_START False Start session recording with a network event instead of START_END_SESSION_KEY
NETWORK_HOST "localhost" IP address of the network host
NETWORK_PORT 4950 Port on NETWORK_HOST to listen and send on
NETWORK_START_EVENT_NAME "start_trial" Value of the event field that starts session recording
TRIAL_CONDITION "A" Trial label
SAVE_LSL_DATA True Save data from a Lab Streaming Layer stream (see Lab Streaming Layer)
RECORD_VIDEO True Record the chosen window with OBS. Required for postprocessing.
RECORD_FACE_TRACKER_DATA False Save facial expression data
AUTO_POSTPROCESS True Run postprocessing automatically after each trial
TARGET_FPS 60 Maximum frame rate for the OBS recording
USE_TIMER True End the trial with a timer instead of a key press
USE_TIMER_DROPDOWN True Show a dropdown to choose the timer length
DEFAULT_TIMER_LENGTH 10 Default timer length, in seconds
START_END_SESSION_KEY " " Key to start and stop the session
PLAY_END_SOUND True Play a sound at the end of each trial
SET_NUMBER_OF_TRIALS 1 Number of trials in the session
REAL_TIME_STREAMING True Show a live copy of the chosen window with the gaze point drawn on it. Turned off automatically for Desktop First Person Game, Meta Quest 3 Recorder, and OpenXR Recorder.
PREVIEW_SCALE 0.5 Size of the live copy, relative to the chosen window
RECORD_VIDEO_OF_PLAYBACK False Allow recording the replay as a video with the 4 and 5 keys
HIDE_REPLAY_GUI False Hide SightLab's replay GUI
FOLLOW_ON True Use the first-person view in the replay
HARDWARE_CONFIGS See Choosing a Hardware Configuration Maps each hardware option to its vizconnect file

Additional Features

  • Rating/Likert scales and surveys: collect participant feedback before or after the external session, with custom scale labels, saved in the data exports
  • Inputs and demographics: gather participant data, such as age, ID, or gender, before the session starts
  • Labels and conditions: tag sessions with experimental conditions for sorting and analysis
  • Flags, network events, and button clicks: log custom triggers, such as key presses or network signals, on the session timeline
  • Speech recording (optional): record microphone input for later analysis or transcription
  • Transcriptions: combine microphone recordings with transcription tools to create searchable dialogue data
  • Instructions: show instructions on the desktop before launching the external app
  • Plotly for data analysis: visualize gaze, movement, and behavioral metrics with the built-in Plotly tools
  • Face tracking and expression analysis: capture facial expressions with supported headsets, such as the Meta Quest Pro, when enabled in the config
  • Baseline: record a short resting or neutral task before launching the external app to establish baseline physiological readings
  • Biofeedback Ball: display a 3D object that responds to physiological data
  • Lab Streaming Layer: connect to additional devices (see Lab Streaming Layer)

Tips and Troubleshooting

  • Don't minimize the mirror window or the external application window, or calibration and recording won't work. Covering them with other windows is fine.
  • Use Alt+Tab to check the console whenever something seems stuck. It shows calibration results, postprocessing progress, and errors, and is saved to the logs folder.
  • Speed up postprocessing by closing SteamVR, Unreal, Unity, Pimax Play, and other RAM- or VRAM-heavy programs after the trial. Task Manager shows which processes use the most.
  • Match the runtime to your headset, for example Pimax OpenXR for the Pimax Dream Air.
  • Check the eye tracker before a session by running eyeTrackerTest.py in the External Data Recorder folder. A green gaze ball should follow your eyes in the headset.
  • Red calibration dots aren't found: increase FOCUS_SIZE, or decrease RED_MIN_PIXELS, in Data_Recorder_Config.py.
  • Validation keeps failing: a threshold stricter than the headset's own accuracy can't be met, however many times you retry. Typical accuracy is about 0.5–1.1° for the Vive Pro Eye, 1.5° for the Meta Quest Pro, and 1.0° for Varjo; raise VALIDATION_MAX_MEAN_DEG if needed.
  • Pimax gaze unavailable: run Pimax_Gaze_Probe.py to check the Pimax eye tracking connection.
  • OBS warning about multiple instances: close the extra OBS instances, which can come from other signed-in Windows users, so the recorder can control the right one.
  • PyTorch errors: you may need a different PyTorch build for your graphics card's CUDA version. The recorder was tested with 2.13.0+cu132.
  • Face tracking data is saved in the data folder, and can be visualized with facial_expressions_over_time.py from the ExampleScripts/Face_Tracking_Data folder.

Limitations

  • Headset applications must be recorded through a mirror window, because eye tracking is calibrated against a window that stays open across applications.
  • Depth isn't recorded: all spatial information is two-dimensional.
  • Head position and rotation aren't recorded.