Skip to content

External Application Data Recorder

Overview

The External Application Data Recorder allows you to record, save and synchronize sensor data while running external VR applications from any source (SteamVR game, Oculus, Web application, etc.). This can use desktop video recorders, such as the Nvidia Shadow Play, Windows Game Bar, or screen recorders like Honeycam to have the option of recording and saving a video that can then be viewed back and matched up with the data recording.

Data

List of some of the data that can be recorded (will be saved to the data folder, unless a different directory is chosen):

  • Facial Expression values (Meta Quest Pro, HTC Face Tracker)
  • Eye gaze rotation
  • Fixations  and Saccades
  • Head orientation (needs to be manually aligned if not at same origin point)
  • Pupil Diameter (Omnicept, Vive Pro Eye, Pupil Labs)
  • Eye Openness Value (Omnicept, Vive Pro Eye, Vive Focus)
  • Heart Rate (Omnicept)
  • Cognitive Load (Omnicept)
  • Interactions using the chosen event trigger
  • TIme Stamp
  • Gaze mat of eye rotation
  • Video Recording

  • All Biopac Acqknowledge physiological data if connecting to Biopac 

    • Video recording can be imported to Acqknowledge to see how it matches up with physiological data
    • Gaze point overlay on video recording (currently may take some manual alignment depending on the field of view of the headset)

Limitations

  • Sensor data from headset will only will work with PC based applications (android applications running on stand alone can still record video using casting and synchronize with any PC connected hardware and physiological data from BIOPAC)
  • Gaze point position is not going to be accurate in  the Z axis, as it's only hitting a collide box but not objects in the scene a user is actually looking at
  • Head orientation (Meta Quest Pro) would need to be manually aligned if not using the same origin starting point
  • Not able to tag regions or objects of interest
  • For headsets that go through SteamVR (Omnicept, Vive Focus 3, Vive Pro Eye) you will not get head orientation (but will show in the replay with the virtual screen)
  • To better synchronize head position if using a controller for navigation, will have to modify the SightLab vizconnect to match the external application's navigation controls

Setup

Note to record videos you need to install the pyautogui python library using the Package Manager, as well as install Tkinter from the Vizard installer

If getting error " A supported game is required" message with Nvidia Shadow Play, make sure you are using your dedicated graphics card and not the onboard one

How to Run

  1. Start Data_Recorder_External_Application script
  2. Start External Application
  3. Make Data Recorder screen active and press spacebar
  4. After timer value that is set, the video recording  and data recording will start. Alt-tab and go back to the external application within this time (if AUTO_ALT_TAB = True then this can happen automatically, can also set how many Alt+TABS to use)
  5. Press spacebar to end recording
  6. Quit External Application
  7. Video recording will be saved in your UserName- Videos folder (or wherever your chosen screen recorder saves videos) and data will be in 'data'
  8. Can also leverage the Plotly visualization example to view a walk path and gaze path (requires the Plotly Python library)
  9. Face Tracking Data can be viewed in the data folder and visualized using the facial_expressions_over_time script
  10. If using Biopac Acqknowledge can open the video to see the synchronization with the physiological data. See here for how to do that
  11. To view a replay run DataRecorderReplay.py that will show the gaze point and visualizations on a virtual screen that is tied to the user's head

Configuration Settings

# Key that can set when certain events happen, saves to data file and sends marker to Acqknowledge
NETWORK_SYNC_KEY = 't'

# Choose if wanting to send a network based event
USE_NETWORK_EVENT = False

# Option to save all eye tracking gaze mat data to the data file
RECORD_GAZE_MAT = True

# Save facial expression data (Meta Quest Pro or Vive Face Tracker)
RECORD_FACE_TRACKER_DATA = False

# Start and stop a video recording using your chosen Video Recorder (Windows Game Recorder, Nvidia Shadow Play, etc.)
RECORD_VIDEO = True

# How long to wait for the video to start recording when the session starts
TIMER_TO_START_VIDEO_RECORDING = 2

# Set path to video recording for replay
PATH_TO_VIDEO = ''

# Keys to open Video Recorder. For Windows Game Bar default is 'winleft' and 'g', ShadowPlay 'alt','z'
OPEN_RECORDER_KEY_ONE = 'winleft'
OPEN_RECORDER_KEY_TWO = 'g'

# Keys to Start and Stop Recording. Windows Game Bar 'ctrl' and 'r', ShadowPlay 'alt', 'f9'
START_RECORDING_KEY_ONE = 'ctrl'
START_RECORDING_KEY_TWO = 'r'

# Enable Auto Alt Tab to other window
AUTO_ALT_TAB = True

# Number of times to Alt Tab
NUM_ALT_TABS = 1

|