Skip to content

External Data Recorder - Desktop & Desktop First Person Guide

This guide details the usage of the External Data Recorder with the Desktop and Desktop First Person configs. These configs treat the mouse as the gaze point, with the center of the screen being the gaze for Desktop First Person.

Please read the External Data Recorder - Overview documentation to complete setup and familiarize yourself with the External Data Recorder.

The Desktop First Person hardware config is meant for desktop first-person games like Minecraft, Valorant, CS:GO, etc.

The Desktop hardware config is meant for desktop apps where the mouse can move freely. An example might be browsing on Google Chrome, or a game like Terraria.

Step-by-step

The usage for both configs is the same, except for the post-processing stage.

This guide will use the Desktop First Person config on a browser-based first person shooter game called Krunker.io.

Open the External Data Recorder script. You can open it through any of:

  1. The SightLab Dashboard, at Tools/Features -> External Data Recorder

  2. In the Vizard IDE, by clicking Edit Code in the External Data Recorder's Sightlab tab

  3. Directly, by double clicking it in the file explorer at Sightlab2\ExampleScripts\External_Data_Recorder\External_Data_Recorder.py

Either way, the usage is the same.

Click the green arrow in the utilities bar at the top to start the External Data Recorder. This is equivalent to clicking the Run: Exernal Data Recorder button in the Sightlab dashboard:

This will bring up a prompt asking you to choose which window to record:

Choose your desired window. We will choose the Krunker window.

You will also see a console window open. This will display any console print outputs from the External Data Recorder.

You will see more messages if DEBUG = True in Data_Recorder_Config.py. Right now the only message shown is that we are using real time streaming, which will open a mirror window showing the live gaze point.

Next, pick your appropriate hardware config. For this tutorial, we are recording Krunker.io, so we'll choose the Desktop First Person Game option:

Note that unlike many other hardware configs, there is no calibration stage, since we just use the mouse or the center of the screen as the gaze point.

You will be prompted to enter the number of seconds to record for. Enter a value and click Ok.

Finally, you'll be prompted to enter Participant Information:

For an actual study, enter appropriate values. For testing, you can leave the fields blank and click Submit. This can be skipped in the config (see Overview).

After this, you can start the session recording by pressing the spacebar. This will automatically switch the foreground window to the window you selected, for us it will switch to Chrome and the Krunker.io tab.

You should see a red dot appear next to the OBS icon in the task bar once you press spacebar, indicating that OBS is recording.

At the end of your session, a beep will play to signify that the trial has ended.

The OBS dot will disappear to indicate it has finished recording.

If you switch back to the External_Data_Recorder window, you should see a message saying the script is being postprocessed, which entails transcoding and overlaying the video with RT-DETR object detection and gaze information:

After this, you can start the session recording by pressing the spacebar. You should see a red dot appear next to the OBS icon in the task bar once you press spacebar, indicating that OBS is recording.

At the end of your session, a beep will play to signify that the trial has ended.

The OBS dot will disappear to indicate it has finished recording.

If you switch back to the External_Data_Recorder window, you should see a message saying the script is being postprocessed, which entails transcoding and overlaying the video with RT-DETR object detection and gaze information:

The console window provides postprocessing progress updates, and it will prompt you to crop the window region and adjust the centerpoint (if you chose Desktop First Person Game) if you wish to. See the postprocessing guide for more instructions. Updates on object detection and other postprocessing is shown in the console window:

Note that if you chose Desktop as your hardware config, you will not be asked to adjust a center point since your mouse is treated as the gaze point.

Closing RAM/VRAM heavy processes can drastically speed up postprocessing times (60+ FPS postprocessing speed). Monitoring expensive processes with Task Manager can help you identify RAM/VRAM heavy processes to close.

Once postprocessing is finished, the External_Data_Recorder window will let you know:

It is now safe to close the External_Data_Recorder window. This will save the console window outputs to a log file, listed in the message.

The recordings produced from this session (original, transcoded, and overlay) can be found in the recordings\ folder.

The csv files and other data files from this session can be found in the data\ folder inside an experiment_data folder corresponding to the datetime and participant id recorded during the session.

Console window content can be found in the logs\ folder.

Replay

To operate the replay, you can open it from the Vizard IDE, directly, or the External Data Recorder tab in the SightLab Dashboard:

This will open a window named External_Data_Replay and prompt you to select a session. Sessions are named by the datetime they were recorded, and the participant id:

Next, it will ask you to select a calibration file to use with the selected session:

For Desktop and Desktop First Person runs, always choose Same as video, since you didn't run an actual calibration. A fake one is used to map the mouse onto the screen as gaze.

Next, select a video to use in the replay. You can choose from either the overlay video, which has the RT-DETR object detections overlaid on the video, or the transcoded video, which has no postprocessing overlays.

See Session Replay and the full Session Replay documentation for more instructions on how to use the replay. Make sure to full-screen the External_Data_Replay window.

Also note that RT-DETR object detection capabilities can be swapped out for the OVD OmDetTurbo model, which can be prompted to look for specific objects instead of being limited to the COCO dataset classes. See the Postprocessing Guide and AI Object Detection for more details.