Session Replay

To view the tracking data after a simulation is run, start the Session Replay file (also referred to as "SightLab Replay"). You will be presented with a list of options for session playback. Note: You must first run an experiment session for the session replay to show any data.

For 360 video/photo playback, the viewpoint will be from a fixed position.

Spacebar to start/stop.
R to reset.
Use the slider along the bottom to scrub through the playback

Session playback can be viewed in either Desktop Mode or in a headset if you choose "Meta," "SteamVR," or "OpenXR."

Note: Session Replay uses a "Free Camera" mode, meaning you can use the WASD and ZX keys to move the camera around, or use the VR controls or walk freely if viewing in a headset.

Features and Options

Trial Number - Select any trial from your session to replay.

Open Data Folder - Opens the folder containing data files associated with the session.

Avatar Visualization - Toggle the display of the avatar (or avatars in multi-user mode). Head and hands are shown separately.

Gaze Visualizations - Toggle: - Gaze Point: The intersection of gaze and objects. - Gaze Ray: The line of sight from the user. - Dwell Spheres: A sphere related to objects of interest that will change size and color based on the time spent gazing at them, once the dwell time threshold is met.

Fixation Spheres - Toggle visualization of fixation spheres. Fixations are calculated based on: - Angular distance (<1 degree). - Duration (>100 ms, default; configurable). - Click Static to view all fixations at once.

Heat Map Visualization - Provides a dynamic representation of gaze data: - Dynamic Mode: Displays gaze over time during playback. - Static Mode: Aggregates all gaze data into a single heatmap. - Occlusion Setting: Enables heatmaps that respect physical barriers (e.g., walls). - Adjust parameters using sliders for intensity, decay rate, and transparency.

Scan Path - Visualize gaze paths dynamically or statically. - Options: - Gaze Path: Tracks gaze movement. - Dispersion Path: Tracks areas of focus.

Console - Toggle text overlays for additional session information.

Views - Toggle between: - First-person view: From the user's perspective. - Third-person view: Scene-wide view. - For 360 media: Toggle whether the head position is locked or free.

Reset to Default - Resets all settings to their default values.

Controls

If using a headset the default controls apply, except in the case of the Vive Pro/2/Vive Pro Eye, where the system button (one above trackpad) on the right needs to be held down and use the left trackpad to move, and the left system button is used for the arc teleport. For the other headsets, the navigation is their default

Action	Key
Start/Stop Playback	`Spacebar`
Reset Playback	`R`
Scrub Playback	`C`, `V` (or use slider)
Frame-by-Frame	`B`, `N`
Move Camera	`WASD`, `Arrow Keys`
Move Up/Down	`X`, `Z`
Rotate Camera	`Q`, `E`, or Mouse
Hide Menu	`.`, `F1`
Take Screenshot	`/`
Video Capture (uncompressed)	`Ctrl + F12`
Full Screen Toggle	`F2`
Rendering Mode Toggle	`F3`
Performance Data Toggle	`F4`
Show Instructions	`I`
Exit	`Escape`

Advanced Features

Heatmap Aggregation

Found in ExampleScripts/HeatMap_Visualizer

Create an aggregated heatmap for:

All session data.
Specific conditions.
Individual trials.

Walk Path

This is in preview mode and may affect loading time. This will be found in the ExampleScripts/Walk_Path folder. It can be toggled on or off with the 'p' key.

Noise Filter

Smooth eye tracking data by averaging the gaze points over a specified window of time. This helps to reduce the impact of random noise or fluctuations in the eye tracking data, resulting in more stable and reliable gaze points. The moving average algorithm works by maintaining a rolling window of the most recent gaze points and calculating the average for each new point as it becomes available.

replay = Replay.SightLabReplay(noiseFilter=True)

Synchronizing Traditional Video

For synchronizing 2D video playback with the replay (note: first need to define and add video in replay as well):

def reset():
  video.setTime(0)
  video.play()

def play():
  video.play()

def pause():
  video.pause()

def scrub(time):
  video.setTime(time)
viz.callback(REPLAY_RESET, reset)
viz.callback(REPLAY_PLAYING, play)
viz.callback(REPLAY_PAUSED, pause)
viz.callback(REPLAY_SCRUB, scrub)

Custom Replay Flags and Events

Set custom flags during an experiment session and visualize them in the replay (for instance to keep track of the state of an object like color change and play that back in the replay at the correct time).

Example:

# During Experiment
def changeColor():
    target_color = random.choice([viz.RED, viz.BLUE, viz.GREEN, viz.YELLOW])
    basketball.color(target_color)
    sightlab.setCustomReplayFlag("color", target_color)

vizact.onkeydown("a", changeColor)

# During Replay
def applyReplayFlag():
    basketball = replay.sceneObjects[GAZE_OBJECTS]["basketball"]
    flag_data = replay.getCurrFlag()
    if "color" in flag_data:
        basketball.color(flag_data["color"])

vizact.onupdate(0, applyReplayFlag)

Data Directory

If you have a custom location for your data files give the path to your custom data folder:

replay = Replay.SightLabReplay(dataDirectory='data')

Advanced Configuration: Scan Path Settings

Modify the appearance of scan paths:

replay.replayVisualisations.scanPathConfig = {
    POINT_COLOR: viz.GREEN,
    POINT_SIZE: 2,
    LINE_COLOR: viz.BLUE,
    LINE_WIDTH: 2
}

Environment Access

Access the environment object during replay:

replay.transportNode.setPosition([x, y, z])
env = replay.getEnvironmentObject()

Scene Objects

Access specific scene objects during replay:

screen = replay.sceneObjects[GAZE_OBJECTS]["screen"]

Setting Starting Position of Viewpoint

replay.transportNode.setPosition([x, y, z])

Loading Older Replay Files

Steps for replaying data from older SightLab versions (1.x):

Copy the blank project folder.
Move older data files to a folder named data.
Place environments or media in the appropriate Resources subfolders.
Use this code to enable compatibility:

replay = Replay.SightLabReplay(sightlab_1_9=True)

Trial Loaded Event

Perform an action when trials are changed

def ontrialchanged():
    trial_number = int(replay.currTrial)
    print("change: ", trial_number)       
viz.callback(TRIAL_LOADED_EVENT, ontrialchanged)