External Data Recorder - Pimax Dream Air Guide

This guide details Pimax Dream Air usage with the External Data Recorder and relevant implementation details. Please read the External Data Recorder - Overview documentation to complete setup and familiarize yourself with the External Data Recorder.

Step-by-step

Connect your Pimax Dream Air to your laptop via the USB 3.0 and DisplayPort connections.

Ensure the DisplayPort is connected directly to the graphics card's port.

Also ensure that your runtime is set to Pimax OpenXR.

To enable the screen mirror, open the Screen Mirror tab on the left, enable Screen Mirror, and select Right Eye.

To verify everything is ready, open the Pimax Mirror window and verify its view matches what you see in the HMD.

Next, open the External Data Recorder script. You can open it through any of:

The SightLab Dashboard, at Tools/Features -> External Data Recorder
In the Vizard IDE, by clicking Edit Code in the External Data Recorder's Sightlab tab
Directly, by double clicking it in the file explorer at Sightlab2\ExampleScripts\External_Data_Recorder\External_Data_Recorder.py

Either way, the usage is the same.

Click the green arrow in the utilities bar at the top to start the External Data Recorder. This is equivalent to clicking the Run: Exernal Data Recorder button in the Sightlab dashboard:

This will bring up a prompt asking you to choose which window to record:

For the Pimax Dream Air, select the window Pimax Mirror - Right (the screen mirror window we just opened in Pimax EVO) and click OK.

There will also be a console window opened that shows any printed console messages of the External Data Recorder:

You will see more messages if DEBUG = True in Data_Recorder_Config.py. Right now the only message shown is that we are using real time streaming, which will open a mirror window showing the live gaze point.

Next, pick Pimax Dream Air as the hardware config:

Once selected, a window titled External_Data_Recorder will open along with a prompt asking you to select an old calibration file for eye tracking, or to perform a new calibration. Select New, and this will open a new window titled Eye_Calibration:

Press trigger on the controller or spacebar on the keyboard to start calibrating. You should see a single red dot like this on Pimax Mirror - Right. The Pimax Mirror - Right window can be occluded, but make sure NOT to minimze it:

The script grabs calibration dot positions from the HMD mirror window (Pimax Mirror - Right) and not the Eye_Calibration window. However, the HMD mirrow window doesn't need to be in the foreground for data to be captured, since the script uses Windows Graphics Capture. This means the window only needs to not be minimized.

Following the instructions from the start screen, fixate each dot before pressing trigger/space and continue fixation until it moves to its next position. Repeat this for each dot.

If Pimax Mirror - Right is minimized, you will see the following output in the console when you try to calibrate a dot:

The error will keep showing if you retry and the window is still minimized. The final line indicates that the first calibration dot was successfuly calibrated after un-minimizing the Pimax Mirror - Right window.

After 9 dots of calibration, an additional 5 validation dots will appear. If the gaze produced for these validation dots exceeds an error threshold, you will see an error message in the console window:

You may press the "A" key to accept a faulty calibration, or trigger/space to restart calibration. Note that the gaze mappings are likely to be inaccurate if you accept a faulty calibration. If all three rounds fail validation, the best in terms of mean angular error (degrees) will be kept.

After finishing Eye_Calibration.py, the External_Data_Recorder window will flash a message notifying that calibration has finished:

Now you can open the external application you want to record.

Note that it must be opened AFTER calibration, because the Pimax runtime can only support one OpenXR application at once, and Vizard's Eye_Calibration would boot your external application off the runtime and take its place.

This tutorial will demo a Unity VR application. Before clicking "OK", open your external application.

After opening the external application, lets ensure it shows up on the HMD mirror:

Looks good!

Now we can switch back to the External_Data_Recorder window.

Click OK on the message window.

You'll be prompted to enter a timer in seconds to record for, with default being 10. Enter whatever value you want and click OK.

Next, you'll be prompted for the actual external application you're recording through the HMD mirror (in our case, Pimax Mirror - Right). Choose your external application and click OK. For this tutorial, we choose the Unity project:

This will automatically switch desktop focus to your external application so it shows up in the Pimax Mirror - Right window and your HMD, since the External Data Recorder briefly takes its place before the session starts.

Finally, you'll be prompted to enter Participant Information:

For an actual study, enter appropriate values. For testing, you can leave the fields blank and click Submit. This can be skipped in the config (see Overview).

Before starting the session, ensure the Pimax Mirror - Right window is NOT minimized.

After this, you can start the session recording by pressing the spacebar. You should see a red dot appear next to the OBS icon in the task bar once you press spacebar, indicating that OBS is recording.

At the end of your session, a beep will play to signify that the trial has ended.

The OBS dot will disappear to indicate it has finished recording.

If you switch back to the External_Data_Recorder window, you should see a message saying the script is being postprocessed, which entails transcoding and overlaying the video with RT-DETR object detection and gaze information:

After this, you can start the session recording by pressing the spacebar. You should see a red dot appear next to the OBS icon in the task bar once you press spacebar, indicating that OBS is recording.

At the end of your session, a beep will play to signify that the trial has ended.

The OBS dot will disappear to indicate it has finished recording.

If you switch back to the External_Data_Recorder window, you should see a message saying the script is being postprocessed, which entails transcoding and overlaying the video with RT-DETR object detection and gaze information:

The console window provides postprocessing progress updates:

Note that closing Pimax from the system tray can drastically speed up postprocessing times (60+ FPS postprocessing speed). Monitoring expensive processes with Task Manager can help you identify RAM/VRAM heavy processes to close.

Once postprocessing is finished, the External_Data_Recorder window will let you know:

It is now safe to close the External_Data_Recorder window. This will save the console window outputs to a log file, listed in the message.

The recordings produced from this session (original, transcoded, and overlay) can be found in the recordings\ folder.

The csv files and other data files from this session can be found in the data\ folder inside an experiment_data folder corresponding to the datetime and participant id recorded during the session.

Console window content can be found in the logs\ folder.

Replay

To operate the replay, you can open it from the Vizard IDE, directly, or the External Data Recorder tab in the SightLab Dashboard:

This will open a window named External_Data_Replay and prompt you to select a session. Sessions are named by the datetime they were recorded, and the participant id:

Next, it will ask you to select a calibration file to use with the selected session:

If you ran a calibration in the same session you ran the recording (the flow described in this guide), select Same as video. If you reused an old calibration from another session, you can search for it in the dropdown.

Next, select a video to use in the replay. You can choose from either the overlay video, which has the RT-DETR object detections overlaid on the video, or the transcoded video, which has no postprocessing overlays.

See Session Replay and the full Session Replay documentation for more instructions on how to use the replay. Make sure to full-screen the External_Data_Replay window.

Also note that RT-DETR object detection capabilities can be swapped out for the OVD OmDetTurbo model, which can be prompted to look for specific objects instead of being limited to the COCO dataset classes. See the Postprocessing Guide and AI Object Detection for more details.

Implementation Detail questions

Tobii XR5

Analytical use?

polling callback every frame