External Data Recorder - Vive Guide

This guide details Vive headset usage with the External Data Recorder and relevant implementation details. Please read the External Data Recorder - Overview documentation to complete setup and familiarize yourself with the External Data Recorder.

Specifically, this guide uses a Vive Focus Vision HMD.

Step-by-step

HMD setup guides can be found here. The following setup section is for the Vive Focus Vision.
Ensure that you have the SRAnipal driver installed, which can be downloaded through the Vive Console installer

Turn on your Vive Focus Vision and connect a USB-C to USB 3.0 cable from the HMD to your desktop.

Then, open Vive Business Streaming on your desktop. It should display a USB connection icon:

Now, in the HMD, open the Vive Business Streaming app. This should change the Vive Business Streaming window on the desktop to look like this:

And SteamVR should open with it too.

If the HMD is unable to pick up on the USB connection, try removing the cable and re-plugging it in, or swap to a different cable. Restarting the HMD can help too.

Right-click the SteamVR Status window, and choose Display VR View. This should open up an HMD mirror window called VR View on the desktop:

In the top left corner, click Menu -> Both Eyes - Right Dominant. Next, right-click the Steam VR Status window and click Settings. Navigate to the OpenXR tab, and verify your Current OpenXR Runtime is set to SteamVR:

Once you've verified this, you can close the SteamVR Settings window.

To verify that everything is ready, run the eyeTrackerTest.py script found at Sightlab2/ExampleScripts/External Data Recorder/eyeTrackerTest.py, which will verify that the SRAnipal eye tracker driver is working.

You can double click the file, and it should bring up an empty white room with a green gaze ball in your HMD:

If you look around in the HMD, you should see the green gaze ball moving around. This verifies that eye tracking data is being recorded correctly.

Once you've verified this, you can close the eyeTrackerTest window, which will bring you back to the SteamVR home.

Next, open the External Data Recorder script. You can open it through any of:

The SightLab Dashboard, at Tools/Features -> External Data Recorder
In the Vizard IDE, by clicking Edit Code in the External Data Recorder's Sightlab tab
Directly, by double clicking it in the file explorer at Sightlab2\ExampleScripts\External_Data_Recorder\External_Data_Recorder.py

Either way, the usage is the same.

Click the green arrow in the utilities bar at the top to start the External Data Recorder. This is equivalent to clicking the Run: Exernal Data Recorder button in the Sightlab dashboard:

This will bring up a prompt asking you to choose which window to record:

Choose VR View, and make sure it is not minimized.

You will also see a console window open. This will display any console print outputs from the External Data Recorder.

You will see more messages if DEBUG = True in Data_Recorder_Config.py. Right now the only message shown is that we are using real time streaming, which will open a mirror window showing the live gaze point.

Next, pick your appropriate hardware config. For this tutorial, we are using the Vive Focus Vision, so we'll choose the Vive Focus Vision Recorder:

Once selected, a window titled External_Data_Recorder will open along with a prompt asking you to select an old calibration file for eye tracking, or to perform a new calibration. Select New, and this will open a new window titled Eye_Calibration:

Press trigger on the controller or spacebar on the keyboard to start calibrating. You should see a single red dot like this on the VR View window. The VR View window can be occluded, but it can NOT be minimized:

The script grabs calibration dot positions from the HMD mirror window (VR View) and not the Eye_Calibration window. However, the HMD mirrow window doesn't need to be in the foreground for data to be captured, since the script uses Windows Graphics Capture. This means the window only needs to not be minimized.

Following the instructions from the start screen, fixate each dot before pressing trigger/space and continue fixation until it moves to its next position. Repeat this for each dot.

If VR View is minimized, you will see the following output in the console when you try to calibrate a dot:

The error will keep showing if you retry and the window is still minimized. The final line indicates that the first calibration dot was successfuly calibrated after un-minimizing the VR View window.

After 9 dots of calibration, an additional 5 validation dots will appear. If the gaze produced for these validation dots exceeds an error threshold, you will see an error message in the console window:

You may press the "A" key to accept a faulty calibration, or trigger/space to restart calibration. Note that the gaze mappings are likely to be inaccurate if you accept a faulty calibration. If all three rounds fail validation, the best in terms of mean angular error (degrees) will be kept.

After finishing Eye_Calibration.py, the External_Data_Recorder window will flash a message notifying that calibration has finished:

If your external application was closed by the calibration script, re-open it now.

This tutorial will demo an Unreal Engine application, but any application that can be run through SteamVR and seen in the VR View window work. This includes SteamVR games and Unity apps.

Unreal Engine App

After opening your Unreal project, make sure the play button has a glasses icon under it:

Once you click this, it should bring your Unreal App into the VR View window (and your HMD). Unreal will take your focus to its own preview window, so Alt+Tab to switch focus to the VR View window. Note that we have to record through the VR View window because the calibration uses its screen dimensions. Ensure that VR View is showing your app and is not minimized:

Now return to the External_Data_Recorder window where you'll be prompted to enter a timer in seconds to record for, with default being 10. Enter whatever value you want and click OK.

Finally, you'll be prompted to enter Participant Information:

For an actual study, enter appropriate values. For testing, you can leave the fields blank and click Submit. This can be skipped in the config (see Overview).

Before starting the session, ensure the VR View window is NOT minimized.

After this, you can start the session recording by pressing the spacebar. You should see a red dot appear next to the OBS icon in the task bar once you press spacebar, indicating that OBS is recording.

At the end of your session, a beep will play to signify that the trial has ended.

The OBS dot will disappear to indicate it has finished recording.

If you switch back to the External_Data_Recorder window, you should see a message saying the script is being postprocessed, which entails transcoding and overlaying the video with RT-DETR object detection and gaze information:

The console window provides postprocessing progress updates:

Note that closing Vive Business Streaming (and subsequently SteamVR and Unreal) can drastically speed up postprocessing times (60+ FPS postprocessing speed). Monitoring expensive processes with Task Manager can help you identify RAM/VRAM heavy processes to close.

Once postprocessing is finished, the External_Data_Recorder window will let you know:

It is now safe to close the External_Data_Recorder window. This will save the console window outputs to a log file, listed in the message.

The recordings produced from this session (original, transcoded, and overlay) can be found in the recordings\ folder.

The csv files and other data files from this session can be found in the data\ folder inside an experiment_data folder corresponding to the datetime and participant id recorded during the session.

Console window content can be found in the logs\ folder.

Replay

To operate the replay, you can open it from the Vizard IDE, directly, or the External Data Recorder tab in the SightLab Dashboard:

This will open a window named External_Data_Replay and prompt you to select a session. Sessions are named by the datetime they were recorded, and the participant id:

Next, it will ask you to select a calibration file to use with the selected session:

If you ran a calibration in the same session you ran the recording (the flow described in this guide), select Same as video. If you reused an old calibration from another session, you can search for it in the dropdown.

Next, select a video to use in the replay. You can choose from either the overlay video, which has the RT-DETR object detections overlaid on the video, or the transcoded video, which has no postprocessing overlays.

See Session Replay and the full Session Replay documentation for more instructions on how to use the replay. Make sure to full-screen the External_Data_Replay window.

Also note that RT-DETR object detection capabilities can be swapped out for the OVD OmDetTurbo model, which can be prompted to look for specific objects instead of being limited to the COCO dataset classes. See the Postprocessing Guide and AI Object Detection for more details.