Skip to content

Virtual Screencasting into SightLab

SightLab supports capturing and displaying real-time content from other applications or browser windows directly into your VR/AR environment. This is useful for a range of research and presentation purposes.

See ExampleScripts/ScreenCasting_Examples or for Multi-User in Multi User ExamplesScripts\Virtual_Screencast

Note: Using the full desktop casting will slow down frame rate if your desktop resolution is high. Casting a specific window that is resized to smaller than your desktop is preferred for performance.

🔧 Setup Requirements

If casting browser based content, before starting, make sure you disable hardware acceleration in your browser (if using browser-based sources):

Python Libraries

Requires installing these additional python libraries (via. Tools-Package Manager in the Vizard IDE). These are now included with SightLab, so should not be necessary to install manually. 

pip install opencv-python
pip install pillow
pip install pywin32

Disable Hardware Acceleration in Chrome:

  1. Open chrome://settings
  2. Scroll to System
  3. Toggle Use hardware acceleration when available → OFF
  4. Restart Chrome

🎬 What You Can Do

  • Display live application windows (e.g., Psychopy, PowerPoint, browsers) inside your VR/AR scene
  • Record these streams automatically for replay or analysis
  • Capture specific windows or the entire desktop
  • Scale, reposition, and rotate the virtual screen in VR
  • Enable pass-through for mixed reality setups
  • Automatically save AVI and MP4 video versions for each trial

📁 Required Modules

The main functionality is provided by the following modules:

  • WindowCapture – Captures a user-selected window.
  • ScreenCapture – Captures the entire desktop.
  • convert_avi_to_mp4() – Transcodes recorded trials to MP4.
  • get_experiment_video_filename() – Auto-generates unique filenames.

Trial Integration in SightLab

Select Window to Capture

screen_capture = WindowCapture(ask_user=True, flip_code=1, record_screen=True)

Or use a specific title:

screen_capture = WindowCapture(window_title="My App", flip_code=1, record_screen=True)

Note: If using on another project, you will also need to copy the blank.jpg file from the example project. 

screen = viz.add('video_screen_object/3D_Video_Screen.osgb')
screen.texture(screen_capture.get_texture())

Optional: Enable Pass-through (for XR devices)

passthrough.setEnabled(True)
viz.clearcolor(viz.BLACK, 0.0)

Control Scale of Virtual Screen

Easiest method is to scale the virtual screen object you are using in the Inspector 3d model editor, or if using the example script can use these keys to dynamically change the scale in the scene. 

vizact.onkeydown('m', scaleVideoUp)
vizact.onkeydown('n', scaleVideoDown)

Optional: View Screen Properties

vizact.onkeydown('t', getScreenPosition)

⏺️ Recording per Trial

If record_screen is set to "True", SightLab automatically starts/stops screen recording per trial:

screen_capture.start_recording()
screen_capture.stop_recording()

MP4 is created automatically at the end of the trial using the correct naming scheme:
<timestamp>_<participantID>_experiment_data_trial_<trialNumber>.mp4

This is useful for the replay to be able to view heatmaps, fixations on top of the content that was viewed.

💼 Use Cases

  • Run web-based experiments, Psychopy experiments or surveys within VR
  • Project video calls or webcams into immersive space
  • Conduct slide presentations or lecture playback
  • Show live data dashboards or coding IDEs
  • Cast a phone screen to a virtual phone

🔁 Replay Support

Captured videos are compatible with SightLab’s replay tools and visualizations. Recordings can be reviewed with:

  • Heatmaps
  • Scan paths
  • Gaze point overlays
  • Fixation spheres

📂 File Output

Recordings are saved into a recordings folder, and are accessible automatically via the replay system or file browser.

Tips

  • Always scale or reposition the screen for optimal placement in your scene or attach to a 3D model of a monitor for instance
  • For multi-user setups, use a screenshare meeting application such as Zoom or GoogleMeet
  • Cast from a specific window and resize this window to a smaller size to improve performance
  • Playing YouTube content may require to keep the YouTube window not in the background (might need to play side by side with your mirrored VR window)

📱 Casting from Mobile Devices

SightLab supports casting a phone screen into VR, which is ideal for experiments simulating smartphone use, live mobile camera feeds, or interacting with mobile content inside a 3D environment. To do this, use the following tools:

✅ Android – Using Scrcpy

Scrcpy is a free, open-source tool to mirror your Android screen to your PC over USB.

🔧 Requirements

  • Android phone with Developer Mode and USB Debugging enabled
  • USB cable
  • Windows PC

🛠 Setup Steps

  1. Enable USB Debugging on your Android:

  2. Settings > About phone > Tap “Build number” 7 times

  3. Then go to System > Developer options > Enable USB debugging

  4. Download and install Scrcpy:

  5. Download from GitHub

  6. Extract the ZIP file (e.g., scrcpy-win64.zip)

  7. Connect your phone via USB

  8. Approve any on-screen prompts to allow debugging

  9. Launch Scrcpy

  10. Double-click scrcpy.exe

  11. Your phone screen appears in a new window

  12. Use with SightLab

  13. Pass "scrcpy" as the window title in WindowCapture

    screen_capture = WindowCapture(window_title="scrcpy", flip_code=1, record_screen=True)

🍎 iPhone – Using AirPlay (LonelyScreen)

  1. Install LonelyScreen on your PC

  2. Open the app

  3. On your iPhone, open Control Center

  4. Tap Screen Mirroring → Select LonelyScreen

  5. Use:

screen_capture = WindowCapture(window_title="LonelyScreen", flip_code=1, record_screen=True)

🧪 Example Script

A full working example is available in:

ExampleScripts/Virtual_Screencast/Virtual_screencast_Phone.py

This script shows how to:

  • Mirror a phone screen into VR
  • Attach the video texture to a phone-shaped model
  • Record the session trial-by-trial

📱 3D Smartphone Model

To enhance immersion, use the included model:

Resources/Virtual_Screencast/smartphone.osgb

Example placement in the scene:

phone_model = vizfx.addChild('Resources/Virtual_Screencast/smartphone.osgb')
phone_model.texture(screen_capture.get_texture())
phone_model.setPosition([0, 1.5, 1.2])

You can rescale, rotate, or parent this to a hand/controller as needed.