Skip to content

Session Replay

To view the tracking data after a simulation is run, click "Replay" in the Dashboard details page or start the Session Replay file (also referred to as "SightLab Replay"). You will be presented with a list of options for session playback. Note: You must first run an experiment session for the session replay to show any data.

For 360 video/photo playback, the viewpoint will be from a fixed position, but can look around if uncheck "Follow Mode" (see Replaying 360 Sessions)

  • Spacebar to start/stop.
  • R to reset.
  • Use the slider along the bottom to scrub through the playback

Session playback can be viewed in either Desktop Mode or in a headset if you choose "Meta," "SteamVR," or "OpenXR."

Note: Session Replay uses a "Free Camera" mode, meaning you can use the WASD and ZX keys to move the camera around, or use the VR controls or walk freely if viewing in a headset. To follow the participant instead, use Follow Mode or Third Person (see Views below).

Features and Options

Trial Number
- Select any trial from your session to replay.

**Client ID **(Multi- User Only)

  • Show Visualizations for only the selected client

Open Data Folder

  • Opens the folder containing data files associated with the session.

Avatar Visualization
- Toggle the display of the avatar (or avatars in multi-user mode). Head and hands are shown separately.

Gaze Visualizations
- Toggle:
- Gaze Ray: The line of sight from the user (individual or per eye).
- Gaze Point: The intersection of gaze and objects (individual or per eye).
- Dwell Spheres: A sphere related to objects of interest that will change size and color based on the time spent gazing at them, once the dwell time threshold is met.

Fixation Spheres
- Toggle visualization of fixation spheres. Fixations are calculated based on:
- Angular distance (<1 degree).
- Duration (>100 ms, default; configurable).
- Click Static to view all fixations at once.
- First: Shows the time and location of the first fixation
- All: Will show a numbered sequence of all the fixations

Heat Map Visualization
- Provides a dynamic representation of gaze data:
- Dynamic Mode: Displays gaze over time during playback.
- Static Mode: Aggregates all gaze data into a single heatmap.
- Occlusion Setting: Enables heatmaps that respect physical barriers (e.g., walls).
- Note: requires setting fastHeatmap = False and can affect frame rate.
- Gradient: A heatmap that fades over a specified time.
- Adjust parameters using sliders for intensity, decay rate, and transparency.
- Note: If running a trial longer than 2 or 3 minutes you can switch to the faster heatmap if loading times increase or frame rate drops (replay = Replay.SightLabReplay(fastHeatmap = True) )

Scan Path

  • Visualize gaze paths dynamically or statically.
  • Options:
  • Gaze Path: Tracks gaze movement.
  • Dispersion Path: Tracks areas of focus.

Walk Path

  • The walk path will show arrows in the direction that a user walked and will also show velocity based on the spacing between arrows. Walk Path can be shown as static or dynamic.

Console

  • Toggle text overlays for additional session information such as Pupil Diameter and real time views
  • ROIs: Shows the regions of interest for 360 media. Moving regions follow the paths they had during the trial (see 360 Moving Regions of Interest).
  • Tags: Shows the names of the ROIs

Views
- Follow Mode: The camera moves with the participant's head, so you see the session from their point of view (first person). Uncheck it to return to the free camera.
- For 360 media: Toggle whether the head position is locked or free.
- Third Person: The camera follows the participant from just behind and above their head, so you can see their avatar head along with what they were looking at. Checking it also turns on Follow Mode, and unchecking it returns to the first-person view.
- The camera turns with the direction the participant faces but ignores head tilt, so the view stays level.
- Hold the right mouse button and drag to orbit the camera around the participant's head (left/right to circle around, up/down to swing below or above). When you let go, the camera eases back behind them.
- The avatar head is shown if Head is checked under Avatar.
- Available for 3D model scenes only, and turns off when you change trials.
- To change where the camera sits, see Third Person Camera Offset.
- For a third-person view during the experiment itself, see Third Person.

Reset to Default

  • Resets all settings to their default values.

Controls

If using a headset the default controls apply, except in the case of the Vive Pro/2/Vive Pro Eye, where the system button (one above trackpad) on the right needs to be held down and use the left trackpad to move, and the left system button is used for the arc teleport. For the other headsets, the navigation is their default 

Action Key
Start/Stop Playback Spacebar
Reset Playback R
Scrub Playback C, V (or use slider)
Frame-by-Frame B, N
Move Camera WASD, Arrow Keys
Move Up/Down X, Z
Rotate Camera Q, E, O, L, or Mouse
Orbit Camera (Third Person) Hold right mouse button and drag
Hide Menu ., F1
Take Screenshot /
Video Capture (uncompressed) Ctrl + F12
Full Screen Toggle F2
Rendering Mode Toggle F3
Performance Data Toggle F4
Show Instructions i
Connect to Biopac 1
Connect to Biopac Graph 2
Save Equirectangular Image 3
Reset Viewpoint 4
Exit Escape

Advanced Features

Heatmap Aggregation

See where people looked across many recordings of the same scene, whether from different participants or from one participant over several trials and sessions. The Group Heatmap Replay example merges every recording of a scene in a data folder into one session that plays back like a multi-user session, with each recording as its own client and one heatmap that adds up all the gaze. It works for 3D environments and 360 media.


Download Group Heatmap Replay

Group Heatmap Replay of a 360 video: four participants' gaze points and the combined heatmap

To use it

  1. Download the example and extract it anywhere, for example into your ExampleScripts folder.
  2. Run merge_sessions.py and select the data folder of the project you recorded with: the folder that holds the <date>_experiment_data folders (for example MyProject\data). Every scene recorded at least twice becomes one trial of the group session.
  3. Run group_heatmap_replay.py and pick the session ending in _GROUP.

Group sessions use the same controls as the regular Session Replay:

  • The Heat Map settings (Static, Intensity, Radius, Rate) apply to the combined heatmap. Static shows the whole trial at once.
  • Choose a client under Client ID to show only that recording's gaze and heatmap. Choose All to see them combined again.
  • Take a screenshot with / or save an equirectangular image with 3.

Scan paths, walk paths, fixation spheres, dwell spheres and the console are turned off when a group trial loads, because drawing them for many recordings slows playback. Turn them back on with their checkboxes.

Settings (in group_heatmap_config.py)

Setting What it does
SOURCE_DATA_FOLDER Data folder to merge, so merge_sessions.py doesn't ask each time
GROUP_NAME Name shown after the date in the session list, such as PilotA (no underscores)
TRIAL_NUMBERS Merge only these trials, such as [1, 3]
TRIAL_LABELS Merge only trials with these labels (conditions), such as ['A']
MIN_RECORDINGS How many recordings a scene needs to become a group trial (default 2)
MIN_DURATION Skip recordings shorter than this many seconds (default 1.0)
HEATMAP_NORMALIZATION 'mean' shows average dwell per recording, so groups of different sizes can be compared directly. 'sum' shows total dwell across all recordings
HIDE_PARTICIPANT_OVERLAYS Set to False to keep scan paths and the other per-client overlays on

Notes

  • Each merge creates a new session, named by date and time, in the example's data folder, so earlier merges are kept. The group_manifest.csv file in each session lists which recording each client came from. Delete a session's folder when you no longer need it.
  • Your original data files are only read, never changed.
  • Recordings are lined up from the start of each trial on a shared 60 Hz timeline, so each recording counts equally whatever headset it was made with. The longest recording drives the scene (environment, moving objects and 360 video), so this suits studies where everyone saw the same content.
  • Merged sessions point to your project's environment and media files. If you move or rename the project, run the merge again.
  • Loading takes longer with more recordings (around 20 seconds for 20 recordings).
  • Normal sessions also play in group_heatmap_replay.py, exactly as in the Session Replay.

Noise Filter

Smooth eye tracking data by averaging the gaze points over a specified window of time. This helps to reduce the impact of random noise or fluctuations in the eye tracking data, resulting in more stable and reliable gaze points. The moving average algorithm works by maintaining a rolling window of the most recent gaze points and calculating the average for each new point as it becomes available. 

replay = Replay.SightLabReplay(noiseFilter=True)

Synchronizing Traditional Video

For synchronizing 2D video playback with the replay (note: first need to define and add video in replay as well):

def reset():
  video.setTime(0)
  video.play()

def play():
  video.play()

def pause():
  video.pause()

def scrub(time):
  video.setTime(time)
viz.callback(REPLAY_RESET, reset)
viz.callback(REPLAY_PLAYING, play)
viz.callback(REPLAY_PAUSED, pause)
viz.callback(REPLAY_SCRUB, scrub)

Custom Replay Flags and Events

Set custom flags during an experiment session and visualize them in the replay (for instance to keep track of the state of an object like color change and play that back in the replay at the correct time). See also ExampleScripts\Replay_Events

Example:

# During Experiment with GUI
def changeColor():
    color_list = [viz.RED, viz.BLUE, viz.GREEN, viz.YELLOW]
    target_color = random.choice(color_list)
    basketball.color(target_color)  
vizact.onkeydown('a', changeColor)

def getColor(): 
    sightlab.setCustomReplayFlag('color', basketball.getColor())

#run sightlab experiment
def sightLabExperiment():
    while True:
        yield viztask.waitKeyDown(' ')
        yield sightlab.startTrial()

        #Add update here for GUI
        vizact.onupdate(0,getColor)

        yield viztask.waitKeyDown(' ')
        yield sightlab.endTrial()

# During Replay
def changeColor():
    basketball = replay.sceneObjects[GAZE_OBJECTS]['basketball']
    flag_data = replay.getCurrFlag()
    if "color" in flag_data:
        print(flag_data)
        basketball.color(flag_data["color"])    
import vizact
vizact.onupdate(0,changeColor)

#Visibility
def changeVisibility():
    Les_Paul.visible(viz.TOGGLE)
vizact.onkeydown("a", changeVisibility)

def getVisibility():
    visibilityFlag = Les_Paul.getVisible()
    sightlab.setCustomReplayFlag("visible", visibilityFlag)
vizact.onupdate(0, getVisibility)

#Replay Side 
def applyReplayFlag():
    Les_Paul = replay.sceneObjects[GAZE_OBJECTS]["Les_Paul"]
    flag_data = replay.getCurrFlag()
    if "visible" in flag_data:
        print(flag_data)
        Les_Paul.visible(flag_data["visible"])
vizact.onupdate(0, applyReplayFlag)

Data Directory

If you have a custom location for your data files give the path to your custom data folder:

replay = Replay.SightLabReplay(dataDirectory='data')

Advanced Configuration: Scan Path Settings

Modify the appearance of scan paths:

replay.replayVisualisations.scanPathConfig = {
    POINT_COLOR: viz.GREEN,
    POINT_SIZE: 2,
    LINE_COLOR: viz.BLUE,
    LINE_WIDTH: 2
}

Environment Access

Access the environment object during replay:

replay.transportNode.setPosition([x, y, z])
env = replay.getEnvironmentObject()

Scene Objects

Access specific scene objects during replay:

screen = replay.sceneObjects[GAZE_OBJECTS]["screen"]

Setting Starting Position of Viewpoint

replay.transportNode.setPosition([x, y, z])

Third Person Camera Offset

Set where the Third Person camera sits relative to the participant's head. offset is [x, y, z] in meters, measured in the direction the participant faces (+y is up, -z is behind them). pitch is how many degrees the camera tilts down toward them. The default is [0, 0.5, -1.5] with a pitch of 15. Leave out either argument to keep its current value. If Third Person is already on, the change applies immediately.

replay = Replay.SightLabReplay()
replay.setThirdPersonOffset([0, 1.5, -4], pitch=20)

When the participant is in a vehicle, a larger offset keeps the whole vehicle in view. The Driving example uses:

replay.setThirdPersonOffset([0.42, 0.97, -3.54], pitch=15)

To start every trial in Third Person, turn it on each time a trial loads:

def startInThirdPerson():
    replay.replayGUI.thirdPersonON()
viz.callback(TRIAL_LOADED_EVENT, startInThirdPerson)
startInThirdPerson()   # the first trial has already loaded

Loading Older Replay Files

Steps for replaying data from older SightLab versions (1.x):

  1. Copy the blank project folder.

  2. Move older data files to a folder named data.

  3. Place environments or media in the appropriate Resources subfolders.

  4. Use this code to enable compatibility:

replay = Replay.SightLabReplay(sightlab_1_9=True)

Trial Loaded Event

Perform an action when trials are changed

def ontrialchanged():
    trial_number = int(replay.currTrial)
    print("change: ", trial_number)       
viz.callback(TRIAL_LOADED_EVENT, ontrialchanged)