Skip to content

Rotoscoped Avatars

Adding Rotoscoped Avatars in SightLab

Rotoscoped avatars are videos with transparent backgrounds, enabling the seamless integration of live-action actors into a virtual environment. This guide outlines the steps for creating, processing, and incorporating rotoscoped avatars in your SightLab session. Find this example in ExampleScripts\Rotoscoped_Avatars (if not in Example Scripts use download link below). To playback the alpha channel videos install the K-Lite Codec Pack.

Can also download example here

Step 1: Create the Rotoscoped Video

  1. Capture High-Quality Video

  2. Use a tripod to stabilize your camera.

  3. Ensure good lighting to avoid shadows and improve contrast.
  4. Avoid busy backgrounds to help with the removal process.

  5. If not using an AI background remover such as RunwayML then you would need to use a greenscreen.

  6. Remove the Background in Runway

  7. Use RunwayML’s advanced background remover to process videos without requiring a green screen. Even with a green screen this method will provide the cleanest output.

    1. Upload your video to RunwayML.
    2. Use the "Remove Background" tool.
    3. Select points on the main subject, and Runway’s algorithm will isolate the actor from the background.
    4. Export the video
    5. See this tutorial for more information
    6. If the ProRes option is not available, then you can use another video editing program, such as Adobe Premiere to crop out the actor.

  8. Use Chroma Key in Adobe Premiere and Export with alpha channel

  9. Use a video editing program to remove background and export with an alpha channel. Here are the steps for Premiere.

    1. Import your clip
    2. Apply the Ultra Key effect to your clip and select the green background with the eyedropper tool.
    3. Adjust settings as needed to refine the background removal.
    4. Export as a sequence of PNG images with an alpha channel.
    5. Export audio separately

Step 2: Combine PNGs into Video with FFMPEG

  1. Prepare PNG Files

  2. Ensure your PNGs are sequentially numbered (e.g., rotoscoped001.png, rotoscoped002.png).

  3. Install FFMPEG

  4. Download and install FFMPEG from ffmpeg.org.

  5. Add FFMPEG to your system's PATH environment variable.

  6. Run the Command

  7. Open a Command Prompt or Terminal, navigate to the folder containing your PNGs, and run the following command:

    ffmpeg -start_number 0 -framerate 30 -i "rotoscoped%03d.png" -i "rotoscoped.wav" -c:v png -pix_fmt rgba -c:a pcm_s16le output_with_alpha.avi

  8. Replace rotoscoped.wav with the name of your audio file if applicable.

  9. Replace "rotoscoped" with the naming sequence of your png files.

Step 3: Prepare Your Virtual Environment

  1. Set Up Lighting

  2. Ensure your virtual environment contains adequate lighting. You can enable the headlight or add directional lights

  3. Load and Position the Video

  4. Use the provided SightLab script template in 

    ExampleScripts\Rotoscoped_Avatars

    or add the following code to your script:

    vid1 = viz.add('output_with_alpha.avi')  # Replace with your video file path
quad = viz.addTexQuad(texture=vid1)
quad.setScale(1.8, 3.5, 1.8)  # Adjust size to match your actor's scale
quad.setEuler(0, 0, 0)        # Set orientation
quad.setPosition([0.032, 1.5, 3.3])  # Set position in the environment

Step 4: Fine-Tuning and Integration

  1. Positioning in SightLab GUI
  2. Using standinAvatar:
    1. Add the standinAvatar object from sightlab_resources to your environment.
    2. Use the Move tools to adjust its position and replace it with the rotoscoped avatar.
    3. With the example script it will automatically move the rotoscoped avatar to this position
  3. For scripting, ensure the position matches your desired placement.
  4. Preview and Debug
  5. Test the playback to confirm the avatar integrates well into the scene.
  6. Adjust lighting, scaling, or positioning as necessary.
  7. Session Replay
  8. Afterwards you can view a replay of the scene to see where a participant focused with visualizations and view the data files.

Step 5: Final Integration

  • Save your project and run the experiment to verify the rotoscoped avatar functions as expected.

Tips

  • Keep backup copies of your video and PNG files in case re-editing is needed.
  • Refer to the "ExampleScripts" folder for more examples and templates.

By following these steps, you can effectively integrate rotoscoped avatars into your SightLab experiments, creating immersive virtual environments with live-action elements.