Skip to content

Shopping Visual Search

Not in included Example Scripts, but see Download button below to get the latest version


Download Latest Version
This experiment simulates a visual search task within a virtual supermarket aisle. Participants are instructed to locate and grab a target item from a dense array of shelf-placed distractor items using gaze and controller-based interaction.

Overview

Participants are placed in a VR environment and presented with instructions to locate a specific food item (e.g., Breakfast Tuna). Items are randomly placed across a left and right shelf grid, with a single target placed randomly among them. One random seed draws the whole layout, so a session's layout can be repeated exactly.

When the target is found (grabbed), feedback is presented (image quad and sound), and the time taken to find it is recorded to the experiment summary, along with where the target was. The trial continues until the experimenter presses Space, or until an optional time limit runs out. A trial in which the target was never grabbed is recorded as not found.

Core Features

  • Random placement of 3D food models on two opposing shelves, drawn from one seed that is saved with the data.
  • Trial logic with instructions and fade-in/out transitions.
  • Grab detection with optional visual feedback. Every grab and release of any item is recorded in the trial data.
  • Timing from the start of the search to the first grab of the target, logged as "Time to Find Target".
  • The target's side, column, shelf and position recorded in the experiment summary.
  • Space ends the trial; an optional time limit can end it too.
  • Image-based instructions head-locked to the participant's view.
  • Optional audio and environment constraints.
  • A multi-user version (server and client), described below.

Configurable Parameters

Variable Description Default Value
TARGET_OBJECT The internal name of the target item to find "BreakfastTuna"
TARGET_FILE The filename of the .osgb model for the target "f_breakfastjerky_7A.osgb"
SHOW_ITEM_INFO Whether to display a quad image above found object True
TIME_LIMIT Seconds the participant has to search. None: no limit, and only Space ends the trial None
RANDOM_SEED Seed for the shelf layout. None draws a new seed for each session; a number repeats a session's layout exactly None
left_origin / right_origin Grid positions for shelf start locations Custom 3D coords
cols, rows Grid layout dimensions for shelf item placement 20 cols x 5 rows

The seed is printed when the script starts and saved in the experiment summary's Random Seed column. To show the same layout again, set RANDOM_SEED to that number.

Components

ShelfGrid System

The ShelfGrid class (grid_placer.py) provides methods to position items in a grid with optional rotation and depth offset.

Modify:

  • Grid size and orientation
  • Sampling behavior (use .sample_positions(n) if desired)

Scene Object Management

Each item is added to the scene with SightLab tracking. The target is a gaze object, so the usual dwell and fixation measures cover it, and every item is grabbable:

sightlab.addSceneObject(item_name, item, gaze=enable_gaze, grab=True)

The target is named BreakfastTuna (TARGET_OBJECT). The other items are named by shelf and slot: food_90_<n> on the left shelf and food_-90_<n> on the right. The names are the same in every session, so the data from different sessions can be compared item by item.

All added objects are also tracked in a scene_items list for batch visibility control:

for item in scene_items:
    item.visible(viz.OFF)  # Hide all items for example

Instructions Panel

Image-based instructions are locked to the participant's head using the following logic:

headRoot = viz.addGroup()
viz.link(mainViewNode, headRoot, mask=viz.LINK_ALL).preTrans([0, -2.0, 1.2])
sightlab.instructionsImageQuad.setParent(headRoot)

You can scale the quad to suit portrait-style instructions:

sightlab.instructionsImageQuad.setScale([1.2, 1.8, 1])  # Taller portrait

For landscape instructions, consider:

sightlab.instructionsImageQuad.setScale([1.8, 1.2, 1])  # Wider landscape

Grabbing and Timing

The search starts when the trial starts, after the instructions. The first time the participant grabs the target in a trial:

  • ✅ Sound feedback is played.
  • ✅ A floating image quad can be displayed over the item.
  • ✅ The time from the start of the search to the grab is measured and saved to the experiment summary:
search['time'] = round(sightlab.hud.getCurrentTime() - search['start'], 3)
sightlab.setExperimentSummaryData("Target Found", True)
sightlab.setExperimentSummaryData("Time to Find Target", search['time'])

The trial does not end at the grab. It ends when the experimenter presses Space or, if TIME_LIMIT is set, when the time runs out.

The script's grab handler is registered once, before the trial loop, with vizact.addCallback:

vizact.addCallback(grabber.GRAB_EVENT, onGrab)

Use vizact.addCallback for your own grab handlers too, not viz.callback. viz.callback keeps one handler per event, so it replaces SightLab's own grab handler: SightLab then stops recording grabs and, in the multi-user version, stops sending grabbed items' movements to the server.

Data Recorded

Each trial's row for ENTIRE_SCENE in the experiment summary has these columns:

Column What it records
Target Found True if the participant grabbed the target, False if the trial ended first
Time to Find Target Seconds from the start of the search to the first grab of the target. Blank when the target wasn't found.
Target Side left or right
Target Column Which column of the shelf. Column 1 is 2.5 m in front of the starting point, and each next column is 0.22 m further back.
Target Shelf 1 (bottom) to 5 (top)
Target Position The target's position (x, y, z) in meters
Random Seed The seed that drew the layout (see RANDOM_SEED)
Trial End space key or time limit

The trial data also flags every grab and release, of any item, as grabbed: <name> and released: <name>, so a participant's wrong grabs can be counted from it.

Running the Experiment

To run the full experiment:

  1. Run Main_Shopping_Search.py
  2. Choose options such as if running BIOPAC, eye tracking thresholds and click Continue
  3. Choose hardware from the dropdown (note: this list can be modified)
  4. Enter Participant ID and click Submit
  5. Put on the headset, read the instructions and press the trigger to begin the task
  6. Look for the intended target, use the trigger buttons to grab items (target is identified when grabbed). On Desktop, the left mouse button grabs what the hand is touching.
  7. Additional controls here
  8. Press Spacebar to end the trial, whether or not the item was found, and the data will be saved (might take a few moments). If TIME_LIMIT is set, the trial also ends when the time runs out.
  9. Please wait until it says "Experiment Complete" for the replay to work
  10. Run SightLabVR_Replay to see a replay of the session with visualizations
  11. View the data in the data folder

Multi-User Version

Main_Shopping_Search_Server.py runs on the experimenter's machine, and Main_Shopping_Search_Client.py runs on each participant's machine.

  • Same shelves everywhere. The server and every client build the shelves from RANDOM_SEED, so they all show the same layout with the same item names. The seed must be the same number in both scripts (default 1). For a different layout, change it in both.
  • Finding the target. Each client times its participant's search from the moment they close the instructions, so reading time is not included. When the participant grabs the target, the client sends that time to the server. The server records the first client to find it in Found By, together with Target Found and Time to Find Target, and plays the feedback.
  • Ending the trial. Space on the server ends the trial. TIME_LIMIT is set in the server script. There it counts from the start of the trial, so it includes the time a participant spends on the instructions.
  • Grabbed items. A grabbed item's movement is sent to the server, so the experimenter sees it move.

Number of items: every frame, a client sends the server the position of every scene object in one network message, which must stay under about 64 KB. The default of 20 columns per side (200 items) fits. 25 columns (250 items) does not: the messages never arrive, and the server shows no head, hands or grabbed items for that client. If you add items or other scene objects to the multi-user version, keep the total under about 200.

Possible Modifications

  • Place new .osgb model files in resources/foods/

  • Set gui=False when initializing SightLab() if not using the GUI

  • Configure vizconnect for your headset, controller, and eye-tracker

  • Item Sampling
    Replace:

grid.all_positions()

with:

grid.sample_positions(n)

to reduce the number of distractor items.

  • ROI Tracking
    Use the Scene Editor (Inspector) to place RegionOfInterest.osgb objects and register them with:
sightlab.addSceneObject("roi_name", roi_object, gaze=True)
  • Audio Feedback
    Replace sound files:
backgroundAudio = viz.addAudio('Resources/audio/your_custom_loop.wav')
foundAudio = viz.addAudio('Resources/audio/your_success_sound.wav')
  • Dynamic Instructions
    You can show multiple instruction slides:
yield sightlab.showInstructions(image='sightlab_resources/instruction1.jpg')
yield viztask.waitEvent('triggerPress')
yield sightlab.showInstructions(image='sightlab_resources/instruction2.jpg')

Assets Structure

resources/
├── audio/
│   ├── supermarket.wav
│   └── tada.wav
├── foods/
│   ├── f_breakfastjerky_7A.osgb
│   ├── ... (other distractor items)
└── images/
    └── item_found.png

Notes

  • Use sightlab.getEnvironment().visible(viz.OFF) to hide the environment at startup.
  • Use optimized or low-poly .osgb models for better performance.
  • Be sure to add all objects using sightlab.addSceneObject(...) (with gaze=True and/or grab=True as needed) to ensure proper data collection.
  • Give each scene object its own name: SightLab keeps one object per name for gaze data and replay.

Dependencies

  • sightlab_utils
  • viztask, vizconnect, vizproximity
  • Custom tools:
  • grid_placer.py
  • tools/grabber.py

Authoring Tips

  • Use Inspector to place and align 3D models on shelves.

  • Adjust object orientation with .getEuler() and .setEuler().

  • Enable proximity debug with:

manager.setDebug(viz.ON)

Additional Options

This experiment can be extended in a variety of ways depending on your research goals:

  • Multi-user mode for collaborative or competitive tasks, building on the server and client scripts
  • Biopac integration for synchronized physiological data collection
  • Real-time biofeedback to visualize participant state (e.g., heart rate, EDA)
  • Dynamic cutoffs that change the time limit (TIME_LIMIT) per trial or condition
  • User surveys and self-report (ratings, open-text input)
  • Custom event tracking (e.g., time in ROI, distance walked)
  • Condition randomization or block-based designs
  • Gaze zone and ROI analysis using tagged scene regions
  • Environmental changes like lighting, sound, or object density per trial
  • GUI integration for managing trial parameters, objects, and logging visually

These features can be enabled or combined depending on your experimental design needs

Support

🔗 SightLab Documentation