Skip to content

Scripted Avatar Agent

Overview

The scripted avatar agent example is a template for adding avatars that can read text saved in text files. The lines can be read back by any event and there are examples for using both keypresses and a timer.

Can be found in ExampleScripts > Scripted Avatar Agent

Key Features

  • Use text to speech engines to read from text files (set up for either OpenAI or ElevenLabs)
  • Modify avatar appearance, animations, environment, and more. Works with most avatar libraries (Avaturn, ReadyPlayerMe, Mixamo, Rocketbox, Reallusion, etc.).
  • Works with all features of SightLab, including data collection, visualizations, and transcript saving.
  • Easily add to any SightLab script.
  • Randomize avatar choice per trial and measure changes in user response

Instructions

1. Installation

  • Ensure you have the required libraries installed using the Vizard Package Manager. These include:
  • openai (for OpenAI text-to-speech)
  • elevenlabs (for ElevenLabs text-to-speech)
  • python-vlc
  • Install VLC Player (for Open AI TTS). Minimum version required: 3.0.20.
  • Download VLC 3.0.20 for Windows (64-bit)
  • For ElevenLabs, you may need to install ffmpeg. Download ffmpeg here.

Note: Requires an active internet connection if using OpenAI or ElevenLabs.

2. API Keys

  • Obtain API keys from OpenAI and ElevenLabs (if using ElevenLabs TTS).
  • Create a folder named keys in your SightLab root directory and place these text files inside:
  • key.txt: Contains your OpenAI API key.
  • elevenlabs_key.txt: Contains your ElevenLabs API key (if using ElevenLabs).
  • ffmpeg_path.txt: Contains the path to the ffmpeg bin folder.

3. Configuration

  • Modify or add new text files for the avatars to read in the scripts folder
  • Customize additional avatar options in the config files
  • Modify global variables in the modules/Config_Global.py file
  • For STIM File method (used to randomize or change avatar per trial), modify stim_files/stim_file_avatar.csv

4. Running the Script

  1. Starting and Stopping Trials
  2. Use the spacebar to start and stop trials (default setting). This can be configured to other keys or events in (see SightLab documentation)
  3. Script Overview
  4. Scripted_Avatar_Agent_GUI.py: A modifiable script for experiments using the GUI.
  5. Scripted_Avatar_Agent_Rating.py: Includes a rating scale and end-of-trial questions.
  6. Scripted_Avatar_Agent_Rating_Timed.py: Reads text files sequentially, using a timer set in Config_Global.py (TIME_BETWEEN_SPEECH = 4, default 4 seconds).
  7. Scripted_Avatar_Agent_Rating_STIM.py: Randomizes avatar choice per trial from a .csv file.
  8. Loading Text Files
  9. Unless using the timed method, load the three text files with keys 1, 2, and 3. Then press c (or the key defined in START_SPEAKING_KEY).
  10. Choosing Avatars
  11. Enable the avatar chooser dropdown by setting USE_AVATAR_CHOOSER = True in Config_Global.py.
  12. Alternatively, avatars can be set manually or randomized.
  13. Session Replay
  14. Use Scripted_Avatar_Agent_Replay.py for an interactive session replay. View the session summary and trial data files in the data folder.
  15. Data Collection
  16. Data files are saved in the experiment's data folder. The rating scale script logs answers, avatar choices, and summary data per trial.
  17. Customization Tips
  18. Adjust keys, timers, and file paths in Config_Global.py.
  19. To configure events (e.g., start or stop conditions), refer to the rest of the SightLab documentation

Modifying Environment and Avatars

Refer to this page for instructions on obtaining assets. Place new assets in resources/environments or resources/avatar/full_body.

Change avatar starting point by adding the standin-Avatar object and using the move and rotate tools in the 3D editor (in the GUI click modify then "Edit" to open the 3D editor). Change the user starting position by adding or moving the SIGHTLAB_START_POINT object.

Obtaining API Keys

To use certain features of the AI Agent, you'll need to obtain API keys from the following services:

  • OpenAI (for ChatGPT and Open AI Text to Speech):
  • Visit the OpenAI website: https://platform.openai.com
  • Sign up or log in.
  • Navigate to the API section (API Keys).
  • Click "Create a new secret key" and copy the key.
  • Paste the copied key into a text file named key.txt and place it in your root SightLab folder.
  • Set a usage limit on your account if needed: OpenAI Usage.
  • Eleven Labs (for ElevenLabs Text-to-Speech):
  • Log in to your ElevenLabs account: https://elevenlabs.io
  • Go to your profile, locate the "API Key" field, and copy it.
  • Paste the key into a file named elevenlabs_key.txt in your root SightLab folder.

Global Configuration Options

  • USE_AVATAR_CHOOSER: Use to select avatars from a dropdown.

  • Default: 'Rocketbox_Male'

  • DEFAULT_AVATAR: Designate the manual avatar to use if not chosen another way.

  • Default: True

  • USE_SPATIAL_AUDIO: Enables spatial audio for immersive sound experiences.

  • Default: True

  • USE_PASSTHROUGH: Toggles passthrough mode for the environment.

  • Default: False

  • DISABLE_LIGHTING_AVATAR: Disables lighting effects for the avatar.

  • Default: True

  • TEXT_FILES: Dictionary listing text files for text to speech.

  • Default: 

    {
    'file1': 'scripts/script1.txt',
    'file2': 'scripts/script2.txt',
    'file3': 'scripts/script3.txt'
}

  • TIME_BETWEEN_SPEECH: Time (in seconds) between reading text files.

  • Default: 4

  • START_SPEAKING_KEY: Key used to initiate text-to-speech.

  • Default: 'c'

  • SPEECH_MODEL: Selects the speech synthesis model.

  • Options: 'Eleven Labs', 'Open AI TTS'

  • Default: 'Eleven Labs'

  • OPEN_AI_TTS_MODEL: Model name for Open AI TTS.

  • Default: "tts-1"

  • ELEVEN_LABS_MODEL: Model name for Eleven Labs TTS.

  • Default: "eleven_monolingual_v1"

  • ELEVEN_LABS_STREAM_AUDIO: Streams audio output directly.

  • Default: True

  • BIOPAC_ON: Connects to Biopac Acqknowledge for experimental data.

  • Default: False

  • SET_CONTINUE_KEY: Key to start or stop the experiment.

  • Default: ' ' (Spacebar)

  • STARTING_TEXT: Message displayed at the beginning of the experiment.

  • Default: "Press Spacebar to Start"

Avatar Config Options

  • ELEVEN_LABS_VOICE: Voice choice for Eleven Labs Text-to-Speech.
  • Options: 'Chris', 'Charlie', 'Serena', 'Charlotte', 'Jessica'

  • Default: "Serena"

  • OPEN_AI_VOICE: Voice choice for Open AI Text-to-Speech.

  • Options: Male voices: 'onyx', 'echo', 'fable'; Female voices: 'nova', 'shimmer', 'alloy'
  • Documentation: OpenAI TTS

  • Default: "nova"

  • ENVIRONMENT_MODEL: Path to the environment model file.

  • Default: 'Resources/environments/homeOffice.osgb'

  • AVATAR_MODEL: Path to the avatar model file.

  • Default: 'Resources/avatars/RocketBox_Female1.osgb'

#Avatar Options

  • TALK_ANIMATION: 7

  • IDLE_ANIMATION: 1

  • AVATAR_POSITION: [0, 0.1, 1.2]

  • AVATAR_EULER: [180, 0, 0]

  • NECK_BONE: 'Bip01 Neck'

  • HEAD_BONE: 'Bip01 Head'

  • SPINE_BONE: 'Bip01 Spine1'

  • TURN_NECK: False

  • NECK_TWIST_VALUES: [180, -80, 0]

  • USE_MOUTH_MORPH: True

  • MOUTH_OPEN_ID: 4

  • MOUTH_OPEN_AMOUNT: 0.4

  • BLINKING: False

  • BLINK_ID: 8

  • DISABLE_LIGHTING_AVATAR: False

  • ATTACH_FACE_LIGHT: False

  • FACE_LIGHT_BONE: 'FacialBone'

  • MORPH_DURATION_ADJUSTMENT: 1