360 Moving Regions of Interest

This example is included with SightLab. For the most up-to-date version, or if a feature described on this page is missing from your copy, download the latest version:

Download Latest Version

Overview

Regions of interest (ROIs) added in the SightLab GUI stay in one place. When the thing you want to measure moves through a 360 video, such as a person, a vehicle or a ball, use this example to make ROIs that follow it.

You trace each moving object once with the mouse in the 360 ROI Path Tracer, on the desktop, with no headset or eye tracker needed. During the experiment, SightLab's ROIs follow the traced paths in time with the video. Gaze, dwell time and fixations on them are recorded under each region's name, just like static ROIs, and the regions move in the Session Replay.

For setting up 360 media and adding regions that stay in place, see 360 Media.

The example uses the built-in warehouse video, with two regions that follow the forklift riders. Open it from the SightLab Dashboard under Tools/Features, where buttons run each part, or find it in ExampleScripts/360 Moving Regions of Interest.

File	What it's for
`360_ROI_Path_Tracer.py`	Trace the regions on a 360 video (run this first)
`360_moving_RegionsOfInterest.py`	Run the experiment with the moving regions
`Replay.py`	Replay a session
`moving_rois/`	The traced paths, one file per video (`<video name>_regions.csv`)
`moving_rois.py`	Shared code that places and moves the regions (no need to edit it)

Workflow at a Glance

Run the tracer (Run: 360 ROI Path Tracer in the Dashboard, or 360_ROI_Path_Tracer.py) and drag your 360 video onto the window.
Pick which trial should play the video.
Add a region for each moving object and trace it while the video plays.
Press P to preview the regions exactly as the experiment will show them.
Run the experiment (360 moving RegionsOfInterest in the Dashboard, or 360_moving_RegionsOfInterest.py) and collect data as usual.
Review the session with Replay (Replay.py).

The tracer saves your paths and adds the regions of interest to the SightLab configuration automatically. You don't need to add them in the SightLab GUI.

1. Open a Video in the Tracer

In the Dashboard, click Run: 360 ROI Path Tracer (or run 360_ROI_Path_Tracer.py). Drag a 360 video onto the window, or press O to browse.

When you open a video, the tracer sets it up for the experiment:

Copies it into the project. A video from outside the project, such as SightLab's built-in media folder, is copied into resources/media. The top bar shows the progress, and you can start tracing while it copies.
Detects the video format. The format comes from a SightLab configuration that already plays the video, or from the frame shape: a square frame is top/bottom stereo. Otherwise you're asked to choose Monoscopic, Stereoscopic T/B or Stereoscopic L/R. Press F to change it later.
Asks which trial should play it. If no trial plays the video yet, choose a trial from the list. That trial's video and format are updated (and its end time, if the trial ends on Media End), and its old regions of interest are replaced by this video's regions. Choose Don't use it in the experiment yet to skip this, and press C when you're ready.

Which configuration is changed?

The tracer updates the configuration SightLab opens when it starts, usually sightlab_configs/recent_config.stlb. The top bar names it, for example Experiment: recent_config.stlb (opens on start) trial 1 plays this video. A backup copy (.bak) is saved the first time the tracer changes a configuration in each session.

2. Trace Each Moving Object

Press N to add a region.
Press F2 to give it a meaningful name, such as Rider1. This name becomes the region of interest's name in SightLab, so it's what appears in your data files.
Find the object: hold the right mouse button and drag to look around. Ctrl + mouse wheel zooms the view.
Press Space to play the video, then hold the left mouse button on the object and follow it.
Scroll the mouse wheel to make the region grow or shrink as the object gets nearer or farther. Shift + wheel makes it taller or wider. The colored outline around the mouse shows the region's size and shape.
Repeat for each object. Tab or the number keys switch regions.
To delete all the current regions and start over can just delete the .csv file or files in moving_rois

Objects that stand still, or stop and start

A region appears at its first keyframe and stays at its last keyframe until the end of the video, so:

An object that doesn't move: pause on the frame where it appears (press Home for the start of the video) and click it once. It's tracked for the rest of the video.
An object that stands still, then moves: click it while it's still, then play the video and start tracing as it starts to move. The region stays at your click until your trace begins, follows it, and stays where it stops. This works in either order, so you can also add the click after tracing the movement.
When the object leaves the scene: press E to end the region there. If it comes back, click or trace it again.

Fast-moving objects

Press the Down arrow to slow the video to 0.5x, 0.25x or 0.1x while you trace. The path is recorded in video time, so the region still moves at the right speed in the experiment. Press Up to return to normal speed.

Fixing a path

You don't need to trace the whole video in one pass:

Retrace part of it: play from just before the mistake and trace again. Only the time you trace over is replaced.
Erase: hold Shift and drag while the video plays to hide the region for that time, for example while the object is behind something. Press E to end the region at the current time.
Frame by frame: pause, step with , and ., and click to place the region on each frame. The tracer fills in the frames between your clicks. While paused, scrolling resizes the region at that frame, and Q and W tilt it.
Undo: press Z. Backspace deletes the keyframe at the current frame, and Delete removes the selected region.

Region shapes and tilt

Press B to switch the selected region to the next shape: sphere, box or triangle.

A box covers a rectangle of the view instead of a circle, which fits doors, screens, signs and vehicles better.
A triangle points up and fills the same width and height a box of that size would. It fits warning signs, roofs, tents and sails.

Use Shift + wheel to make any shape taller or wider, for example a tall box for a standing person. A triangle is as tall as it is wide until you change it; make it a little flatter (an aspect of about 0.87) for one with equal sides.

Press Q or W to tilt the selected region left or right, 5° per press (1° with Shift), for objects that lean or run at an angle, such as a sloping path or a tilted sign. Tilting also turns a triangle to point another way. Like the size, the tilt can change over time: while paused, it changes the region at that frame; while tracing, it applies to what you trace next.

A box or triangle is a thin panel that always faces the participant, so it covers exactly the area you see in the tracer. 360 video has no depth, so a deeper shape would only stick out toward the participant and look bigger than the object. Gaze counts only on the shape itself: a glance at the empty corners beside a triangle isn't on the region.

The SightLab GUI, the experiment and the replay all use the same shape. SightLab creates every region of interest as a sphere, so 360_moving_RegionsOfInterest.py swaps in a box or triangle for those regions, both in the SightLab GUI and at the start of each trial, and Replay.py draws them in their shapes. Other replay scripts, such as the Group Heatmap Replay, draw a box or triangle region as a flat ellipse of the same size and tilt (to draw it in its shape, see Using Moving Regions in Your Own Script). Gaze data is recorded under the region's name either way.

Note

The replay draws each region with the shape it has in the region file now, so choose shapes before collecting data.

The tracer window

Regions: each region is drawn as the sphere, box or triangle the experiment uses. The selected region has an outline, its name and its path.
Top bar:
- the video, its format, the time and the playback speed
- the selected region and its size at the current time
- what's under the mouse
- where the regions were saved
- which configuration and trial play the video
Timeline: along the bottom, one colored row per region shows when that region is in the scene. Click or drag the timeline to seek.
Text size: the text grows with the window. Press [ and ] to change it, and F1 to show the controls.

3. Check the Regions

Press P to preview. The regions are drawn exactly where and how large the experiment will draw them. The mouse acts like the participant's gaze: the top bar shows which region it's on, using the same test SightLab uses for gaze. Play the video, and move the mouse over each object to check that its region covers it. Press P again to keep editing.

Everything is saved as you work:

The paths go to moving_rois/<video name>_regions.csv.
The regions of interest: each trial that plays the video gets one for every traced region, with the same name. Renaming or deleting a region in the tracer updates them. Regions of interest you added by hand in the SightLab GUI are left alone.

Press Escape to close the tracer when you're done.

4. Run the Experiment

In the Dashboard, click 360 moving RegionsOfInterest (or run 360_moving_RegionsOfInterest.py), and run the session as usual. In the SightLab GUI, the trial you chose plays the video and lists a region of interest for each traced region, under the names you gave them.

To preview the regions in the SightLab GUI, press Space to play the trial's video. The regions follow their paths, sized, tilted and hidden as the participant will see them. Press Space again to pause; the regions stay where they are at that point in the video. Before you first play the video, each region shows where its path starts, with box and triangle regions already in their shapes.

During each trial, every region of interest whose name matches a traced region follows that region's path, timed to the video. While a region is hidden (before its first keyframe, after you end it with E, or where you erased it), it can't be gazed at.

To hide the regions from the participant while still recording gaze on them, drag each region's opacity slider (the second slider next to its name) all the way to the left in the GUI. You can also hide them all from code by setting SHOW_REGIONS = False at the top of 360_moving_RegionsOfInterest.py.

Note

A moving region's position, size and shape come from the tracer, so the size slider (the first one) and the GUI's fine-tune keys (i, j, k, l, u, o) have no effect on it in the experiment. To change them, edit the region in the tracer.

Check the trial's end condition

If the trial ends on a Timer that is shorter than the video, the regions stop when the trial ends. Use Media End as the end condition in the SightLab GUI to play the whole video.

5. Review the Data

Gaze data for the moving regions is recorded in the same files as for other regions of interest, under the names you gave them. For example, the experiment summary from one warehouse session:

ROI/Object	Dwell Visits	Fixation Count	Total Dwell Time (secs)	Time to First Fixation (secs)
Rider1	1	11	4.42	3.536
Rider2	3	18	7.007	7.672

The trial timeline (trial_timeline_dwell) lists when each dwell on each rider started and ended.

Click Replay in the Dashboard (or run Replay.py) to watch the session. The regions move as they did during the trial. Toggle them with ROIs and their names with Tags under Console.

See Session Replay for the other visualizations.

Controls

Action	Control
Open a video	Drag it onto the window, or O
Trace the selected region	Hold the left mouse button while the video plays
Set a keyframe at this frame	Click while paused
Erase (hide the region while held)	Shift + left drag
Region size	Mouse wheel
Taller / wider	Shift + mouse wheel
Look around	Right mouse button + drag
Zoom the view	Ctrl + mouse wheel, or - and =
Seek	Click or drag the timeline
Play / pause	Space
Back / forward 1 s (Shift: 5 s)	Left / Right
Previous / next frame	, / .
Playback speed	Up / Down
Go to start	Home
New region	N
Select region	Tab, 1–9
Look at the selected region	G
Shape: sphere, box or triangle	B
Tilt left / right (Shift: 1°)	Q / W
Rename region	F2
End the region here	E
Delete the keyframe at this frame	Backspace
Delete the selected region	Delete
Undo	Z
Preview as the experiment shows it	P
Show / hide paths	V
Use this video in the experiment (choose a trial)	C
Change the video format	F
Smaller / larger text	[ / ]
Show / hide the controls	F1
Close the tracer	Escape

Tracer Settings

These are at the top of 360_ROI_Path_Tracer.py:

Setting	Default	What it does
`MEDIA_FILE`	`None`	A video to open on start, for example `'resources/media/01_Warehouse.avi'`
`MEDIA_FOLDER`	`resources/media`	Where videos from outside the project are copied
`CONFIG_FOLDER`	`sightlab_configs`	Where the SightLab configurations are
`SAMPLE_INTERVAL`	`0.1`	Seconds of video between traced keyframes
`EDIT_ALPHA` / `PREVIEW_ALPHA`	`0.3` / `0.5`	Region opacity while editing and in preview
`START_FOV`	`80`	Starting field of view, in degrees
`WINDOW_SIZE`	`0.75`	Window size as a fraction of the screen (`None` for Vizard's default)
`TEXT_SCALE`	`1.0`	Text size (it also grows with the window)

The Region File

Each video's paths are saved in moving_rois/<video name>_regions.csv, one row per keyframe. You can open it in Excel, but close it before tracing again, or the tracer can't save.

Column	Meaning
`Region`	Region name (matches the region of interest name in SightLab)
`Time`	Seconds into the video
`Azimuth`	Degrees left or right of the video's front (positive is right)
`Elevation`	Degrees above or below the horizon (positive is up)
`Radius`	Size of the region, in degrees of visual angle from its center to its edge (half its width)
`Aspect`	Height divided by width (1 is a circle, a square, or a triangle as tall as it is wide)
`Roll`	Tilt in degrees as the participant sees it, positive is clockwise (files without this column have no tilt)
`Visible`	`1` shows the region; `0` hides it until the next keyframe
`Shape`	`sphere`, `box` or `triangle` (the same on every row of a region; files without this column use spheres)

A region appears at its first keyframe and stays at its last keyframe until the end of the video, unless a keyframe with Visible set to 0 hides it. Between keyframes, the region moves smoothly along the sphere and its size changes gradually. The exception is a traced stretch that starts more than half a second after the keyframe before it: the region stays at that keyframe until the stretch starts. Because positions and sizes are angles, a region covers the same part of the video wherever the participant's head is.

Using Moving Regions in Your Own Script

To add moving regions to another 360 experiment, copy moving_rois.py and the moving_rois folder into your project. Then start and stop the regions with each trial:

import sightlab_utils.sightlab as sl
from sightlab_utils.settings import *
import moving_rois

sightlab = sl.SightLab()
regions = moving_rois.MovingRegions(sightlab, show=True)

def sightLabExperiment():
    yield viztask.waitEvent(EXPERIMENT_START)
    for trial in range(sightlab.getTrialCount()):
        yield viztask.waitEvent(TRIAL_START)
        regions.start()
        yield viztask.waitEvent(TRIAL_END)
        regions.stop()

viztask.schedule(sightlab.runExperiment)
viztask.schedule(sightLabExperiment)

Each trial loads the region file for the video it plays, so trials with different videos each get their own regions. MovingRegions also previews the regions in the SightLab GUI (see Run the Experiment), with no extra code.

If any regions are boxes or triangles, add this to your replay script so the replay draws them in their shapes. It finds each trial's region file from the video the trial played, so the replay script doesn't have to be in the project folder:

import moving_rois

replay = Replay.SightLabReplay()
moving_rois.show_shapes_in_replay(replay)

Objects with Complex Shapes

A region covers a circle, an ellipse, a square, a rectangle or a triangle of the view. Most objects fit one of these closely enough:

Stretch the shape: Shift + wheel makes a sphere into an ellipse, a box into a rectangle, or a triangle taller or flatter, such as a tall box for a standing person or a wide one for a car.
Tilt the shape: Q and W turn it to match an object that leans or runs at an angle, such as a sloping path, or to point a triangle another way.
Change the size and tilt over time: scroll or tilt while tracing, or pause and adjust at any frame, as the object turns or changes size.
Cover one object with several regions: for an object with distinct parts, trace a region for each part, for example Rider1_body and Rider1_head, and add their dwell times together in your analysis.

In 3D model scenes, a region of interest can be any shape, since you can tag any object or draw a region in Inspector.

Troubleshooting

A region doesn't move during the experiment. Check the console for [moving_rois] messages. Region "X" has no region of interest with that name means the trial playing that video has no region of interest called X. Open the video in the tracer and press C, or add one with that name in the SightLab GUI.
The regions don't line up with the video. Check the video format in the tracer's top bar, and press F to change it. A top/bottom stereo video opened as Monoscopic shows the two eye images stacked, and regions traced that way won't match the experiment.
The playback speed doesn't change. Videos wider than 4K use a different player that only plays at normal speed. Trace frame by frame instead.
"NOT SAVED" in the top bar. The region file is open in another program, such as Excel. Close it and make another edit to save.
Stereo video in the tracer. The tracer shows the left eye's image on the desktop. Regions apply to both eyes in the headset.

Earlier versions of this example

Earlier versions recorded keyframes by pressing keys 1–9 in a replay of an eye-tracking session (360_moving_RegionsOfInterest_Tagging.py and moving_regions.csv), with a separate multiple_moving_regions folder for more than one region. The Path Tracer replaces all of these. Trace the regions again in the tracer.