Skip to content

External Application Data Recorder

Overview

The External Application Data Recorder enables you to record, save, and synchronize sensor data while running external VR applications, including SteamVR games, Oculus apps, or web-based VR experiences. It integrates with desktop video recorders (e.g., Nvidia Shadow Play, Windows Game Bar, or Honeycam), allowing you to capture video alongside your data. These video recordings can later be reviewed and synchronized with the corresponding sensor data.

Data

List of some of the data that can be recorded (will be saved to the data folder, unless a different directory is chosen):

  • Facial Expression values (Meta Quest Pro, HTC Face Tracker)
  • Eye gaze rotation
  • Fixations  and Saccades
  • Head orientation (needs to be manually aligned if not at same origin point)
  • Pupil Diameter (Varjo, Omnicept, Vive Pro Eye, Pupil Labs)
  • Eye Openness Value (Varjo, Omnicept, Vive Pro Eye, Vive Focus 3, Vive Focus Vision)
  • Heart Rate (Omnicept)
  • Cognitive Load (Omnicept)
  • Interactions using the chosen event trigger
  • Time Stamp
  • Gaze mat of eye rotation
  • Video Recording

  • All Biopac Acqknowledge physiological data if connecting to Biopac 

    • Video recording can be imported to Acqknowledge to see how it matches up with physiological data
    • Gaze point overlay on video recording (currently may take some manual alignment depending on the field of view of the headset)

Limitations

  • Sensor data from headset will only will work with PC based applications (android applications running on stand alone can still record video using casting and synchronize with any PC connected hardware and physiological data from BIOPAC)
  • Gaze point position is not going to be accurate in  the Z axis, as it's only hitting a collide box but not objects in the scene a user is actually looking at
  • Head orientation (Meta Quest Pro) would need to be manually aligned if not using the same origin starting point
  • Not able to tag regions or objects of interest
  • For headsets that go through SteamVR (Omnicept, Vive Focus 3, Vive Pro Eye) you will not get head orientation (but will show in the replay with the virtual screen)
  • To better synchronize head position if using a controller for navigation, will have to modify the SightLab vizconnect to match the external application's navigation controls

Setup

Required Software

To record video, ensure the following Python libraries are installed:

  • pyautogui
  • pywin32

You can install these via the Package Manager. You will also need to install Tkinter from the Vizard installer.

Video Recording Setup

By default, the Windows Game Bar is used to trigger video recording. Here's how to set it up:

  1. Configure Keyboard Shortcuts:

    • Use Windows Key + G to open the Game Bar.
    • Configure Ctrl + R to start and stop recording by going to the settings (gear icon), selecting "Shortcuts," and pressing Ctrl + R to set the shortcut.

  2. Set the External Application Window Title:

    • Set this in the Data_Recorder_Config.py file. If unsure of the window title for the external application, use the list_open_window_titles.py script to retrieve it.

  3. Troubleshooting:

    • If you receive an error such as "A supported game is required" with Nvidia Shadow Play, ensure you are using your dedicated graphics card instead of the onboard one.

  4. AcqKnowledge Configuration for multiple trials:

    • If using AcqKnowledge for physiological data with multiple trials, set it to save each trial's data to disk. For guidance, see the Sending Events to Biopac page.

How to Run

  1. Start the Data_Recorder_External_Application script.
  2. Open the desired VR application on your PC.
  3. Make Data Recorder screen active and press spacebar
  4. Click on the Data Recorder window to make it active and press Spacebar to begin recording.
  5. Ensure the extWindow Title is set correctly in the configuration file. SightLab will automatically select this window to start recording video and sensor data.
  6. Alt + Tab back to the Data Recorder window and press Spacebar to stop recording (alternatively, set a timer or other event to end the trial). Note: To change the GUI version to use time or use a keypress, press "Modify", then click the "Condition" dropdown for start or end. For a timed trial, add 2 extra seconds to accommodate time to start the recorder.
  7. Quit External Application
  8. Video files are saved in the recordings folder, while data files are stored in the data folder.
  9. Open the recorded video in AcqKnowledge to see the synchronization with physiological data. See here for how to do that.
  10. To view a replay, run DataRecorderReplay.py, which will display gaze point data and other visualizations on the virtual screen. Note: The eye tracking may need to be first calibrated on the device, and then can also calibrate the replay screen by focusing on a point in the scene and moving the REPLAY_SCREEN_CALIBRATION values until the point matches up.
  11. Can also leverage the Plotly visualization example to view a walk path and gaze path (requires the Plotly Python library)
  12. Face tracking data can be found in the data folder and visualized using the facial_expressions_over_time.py script.

Configuration Settings

# Key that can set when certain events happen, saves to data file and sends marker to Acqknowledge
NETWORK_SYNC_KEY = 't'

#Event, such as trigger press to send markers to Acqknowledge
NETWORK_SYNC_EVENT = 'triggerPress'

# Choose if wanting to send a network based event
USE_NETWORK_EVENT = False

# Option to save all eye tracking gaze mat data to the data file
RECORD_GAZE_MAT = True

# Save facial expression data (Meta Quest Pro or Vive Face Tracker)
RECORD_FACE_TRACKER_DATA = False

# Start and stop a video recording using your chosen Video Recorder (Windows Game Recorder, Nvidia Shadow Play, etc.)
RECORD_VIDEO = True

# How long to wait for the video to start recording when the session starts
TIMER_TO_START_VIDEO_RECORDING = 2

# Keys to open Video Recorder. For Windows Game Bar default is 'winleft' and 'g', ShadowPlay 'alt','z'
OPEN_RECORDER_KEY_ONE = 'winleft'
OPEN_RECORDER_KEY_TWO = 'g'

# Keys to Start and Stop Recording. Windows Game Bar 'ctrl' and 'r', ShadowPlay 'alt', 'f9'
START_RECORDING_KEY_ONE = 'ctrl'
START_RECORDING_KEY_TWO = 'r'

#Title of other application window
WINDOW_TITLE = 'Roblox'

#Input values to move the screen to match with calibration point
REPLAY_SCREEN_CALIBRATION = [-0.28,0.0,0.0]

|