Skip to content

Virtual Screencasting into SightLab

SightLab supports capturing and displaying real-time content from other applications or browser windows directly into your VR/AR environment. This is useful for a range of research and presentation purposes.

See ExampleScripts/ScreenCasting_Examples or for Multi-User in Multi User ExamplesScripts\Virtual_Screencast

🔧 Setup Requirements

If casting browser based content, before starting, make sure you disable hardware acceleration in your browser (if using browser-based sources):

Python Libraries

Requires installing these additional python libraries (via. Tools-Package Manager in the Vizard IDE)

pip install opencv-python
pip install pillow
pip install pywin32

Disable Hardware Acceleration in Chrome:

  1. Open chrome://settings
  2. Scroll to System
  3. Toggle Use hardware acceleration when available → OFF
  4. Restart Chrome

🎬 What You Can Do

  • Display live application windows (e.g., Psychopy, PowerPoint, browsers) inside your VR/AR scene
  • Record these streams automatically for replay or analysis
  • Capture specific windows or the entire desktop
  • Scale, reposition, and rotate the virtual screen in VR
  • Enable pass-through for mixed reality setups
  • Automatically save AVI and MP4 video versions for each trial

📁 Required Modules

The main functionality is provided by the following modules:

  • WindowCapture – Captures a user-selected window.
  • ScreenCapture – Captures the entire desktop.
  • convert_avi_to_mp4() – Transcodes recorded trials to MP4.
  • get_experiment_video_filename() – Auto-generates unique filenames.

Trial Integration in SightLab

Select Window to Capture

screen_capture = WindowCapture(ask_user=True, flip_code=1, record_screen=True)

Or use a specific title:

screen_capture = WindowCapture(window_title="My App", flip_code=1, record_screen=True)

screen = viz.add('video_screen_object/3D_Video_Screen.osgb')
screen.texture(screen_capture.get_texture())

Optional: Enable Pass-through (for XR devices)

passthrough.setEnabled(True)
viz.clearcolor(viz.BLACK, 0.0)

Control Scale of Virtual Screen

vizact.onkeydown('m', scaleVideoUp)
vizact.onkeydown('n', scaleVideoDown)

Optional: View Screen Properties

vizact.onkeydown('t', getScreenPosition)

⏺️ Recording per Trial

SightLab automatically starts/stops screen recording per trial:

screen_capture.start_recording()
screen_capture.stop_recording()

MP4 is created automatically at the end of the trial using the correct naming scheme: <timestamp>_<participantID>_experiment_data_trial_<trialNumber>.mp4

💼 Use Cases

  • Run web-based experiments or surveys within VR
  • Project video calls or webcams into immersive space
  • Conduct slide presentations or lecture playback
  • Show live data dashboards or coding IDEs
  • Cast a phone screen to a virtual phone

🔁 Replay Support

Captured videos are compatible with SightLab’s replay tools and visualizations. Recordings can be reviewed with:

  • Heatmaps
  • Scan paths
  • Gaze point overlays
  • Fixation spheres

📂 File Output

Recordings are saved into a recordings folder, and are accessible automatically via the replay system or file browser.

Tips

  • Always scale or reposition the screen for optimal placement in your scene or attach to a 3D model of a monitor for instance
  • For multi-user setups, use a screenshare meeting application such as Zoom or GoogleMeet

📱 Casting from Mobile Devices

SightLab supports casting a phone screen into VR, which is ideal for experiments simulating smartphone use, live mobile camera feeds, or interacting with mobile content inside a 3D environment.

✅ Android – Using Scrcpy

Scrcpy is a free, open-source tool to mirror your Android screen to your PC over USB.

🔧 Requirements

  • Android phone with Developer Mode and USB Debugging enabled
  • USB cable
  • Windows PC

🛠 Setup Steps

  1. Enable USB Debugging on your Android:

  2. Settings > About phone > Tap “Build number” 7 times

  3. Then go to System > Developer options > Enable USB debugging

  4. Download and install Scrcpy:

  5. Download from GitHub

  6. Extract the ZIP file (e.g., scrcpy-win64.zip)

  7. Connect your phone via USB

  8. Approve any on-screen prompts to allow debugging

  9. Launch Scrcpy

  10. Double-click scrcpy.exe

  11. Your phone screen appears in a new window

  12. Use with SightLab

  13. Pass "scrcpy" as the window title in WindowCapture

    screen_capture = WindowCapture(window_title="scrcpy", flip_code=1, record_screen=True)

🍎 iPhone – Using AirPlay (LonelyScreen)

  1. Install LonelyScreen on your PC

  2. Open the app

  3. On your iPhone, open Control Center

  4. Tap Screen Mirroring → Select LonelyScreen

  5. Use:

screen_capture = WindowCapture(window_title="LonelyScreen", flip_code=1, record_screen=True)

🧪 Example Script

A full working example is available in:

ExampleScripts/Virtual_Screencast/Virtual_screencast_Phone.py

This script shows how to:

  • Mirror a phone screen into VR
  • Attach the video texture to a phone-shaped model
  • Record the session trial-by-trial

📱 3D Smartphone Model

To enhance immersion, use the included model:

Resources/Virtual_Screencast/smartphone.osgb

Example placement in the scene:

phone_model = vizfx.addChild('Resources/Virtual_Screencast/smartphone.osgb')
phone_model.texture(screen_capture.get_texture())
phone_model.setPosition([0, 1.5, 1.2])

You can rescale, rotate, or parent this to a hand/controller as needed.