Skip to content

AI Agent

AI Agent

Download Latest Version

This document describes how to use the AI Agent in SightLab, an interactive, intelligent AI agent that can be connected to various large language models like Chat GPT, Claude, Gemini, Offline Ollama models and more. You can customize the agent's personality, use speech recognition, and leverage high-quality text-to-speech models.

Key Features
Multiple LLM Models OpenAI, Anthropic, Gemini, and hundreds of offline models via Ollama (deepseek, Gemma, Llama, Mistral, etc.) — including vision capabilities
Custom Avatars Modify appearance, animations, and environment. Works with Avaturn, ReadyPlayerMe, Mixamo, Rocketbox, Reallusion, and more
Personality & Context Customize personality, contextual awareness, emotional state, and interactions. Save your creations as custom agents
Voice Interaction Speech recognition to converse using your voice, or text-based input
High-Quality TTS Edge-TTS, Kokoro, Piper, GPT-Realtime, OpenAI TTS, ElevenLabs (requires API), or PYTTSX3
Interactive Events Avatars can trigger custom events — facial expressions, animations, emotional tone (TTS Dependent) and scene interactions based on conversation context
40+ Languages Text to speech models automatically adjust to the selected language
SightLab Integration Data collection, visualizations, transcript saving, and easy addition to any SightLab script
Structured Trials Automatically swap prompts per trial, branch by condition using contextual prompts or fully scripted responses, end a trial after a set number of exchanges, and speak a scripted closing line — ideal for research studies with multiple conditions
Conversation Transcripts Save a full, timestamped transcript of every conversation for review or research records
Custom Knowledge Base Feed the agent background info from text or PDF files so it can draw on that knowledge naturally in conversation
Adaptive Learning The agent adapts using conversation history and interactions

Quick Start

  1. Run AI_Agent_GUI.py to start with the GUI or AI_Agent.py without a GUI
  2. Select your environment, hardware, and options in the GUI and press Continue
  3. Hold the 'c' key or RH grip button to speak; release to let the agent respond
  4. Press 'h' to send a screenshot and ask about what the agent sees (or can just ask "What are we looking at?")

💡 Tip: Run multi_agent_interaction.py for a multi-agent conversation demo.

Setup & Installation

Required Libraries

Ensure you have the required libraries installed using the Vizard Package Manager:

Category Libraries
LLM Providers openai, anthropic, google, google-generativeai, ollama
Speech Recognition SpeechRecognition, faster_whisper, sounddevice
Text-to-Speech edge-tts (samples), kokoro-onnx, soundfile, piper-tts==1.3.0 (samples), elevenlabs, pyttsx3
Other numpy

⚠️ Important: Use piper-tts==1.3.0 — do not update to 1.4.0.

Kokoro has very high quality, low latency and runs offline, but requires downloading the models from here (download and extract into the main/root AI Agent folder)

ElevenLabs Additional Setup

  • For elevenlabs you may need to Install mpv and add to environment variable path (also requires FFMPEG, but this comes with SightLab):
    • Download mpv
    • Unzip and move to C:\Program Files\
    • In Windows search, type powershell, run as administrator
    • Run: setx /M PATH "$($env:PATH);C:\Program Files\mpv-x86_64-20250812-git-211c9cb"
    • Restart Vizard

Using More Voices (Voice Library & Voice Cloning)

The ELEVEN_LABS_VOICE setting in your avatar config (e.g. configs/MyAvatar.py) only recognizes voices already present in your ElevenLabs account. To use a voice beyond the handful of default ones:

  1. Sign in at elevenlabs.io
  2. Browse the Voice Library and click Add to my voices on any voice you like — or use Voice Cloning to create a new voice from your own audio sample
  3. Set ELEVEN_LABS_VOICE in your avatar config to that voice's exact name, e.g.:
    ELEVEN_LABS_VOICE = "Dan"

⚠️ Paid plan required for Voice Library / cloned voices: ElevenLabs' free tier can preview these voices in the web app, but blocks them entirely through the API — calls fail with a 402 payment_required error ("Free users cannot use library voices via the API"). This applies to any voice added from the Voice Library as well as cloned voices. To use them from SightLab, upgrade to a paid ElevenLabs plan (the Starter tier is sufficient). The default voices that ship with every account work on the free tier without any upgrade.

Offline Models (Ollama)

Ollama comes with the SightLab installer, but if you didn't install it with SightLab, Install Ollama from here, then open a command prompt and type ollama run followed by the model name (e.g., ollama run gemma). See the full model list. The first run may take longer as the model loads.

Note: Requires an active internet connection unless running offline models via Ollama.

API Keys

API keys are needed for online models (OpenAI, Anthropic, Gemini) and optional services (ElevenLabs TTS). For offline models via Ollama, no key is needed.

Setting API Keys (recommended method)

Open a command prompt (type "cmd" in Windows search) and run:

setx OPENAI_API_KEY your-api-key
setx ANTHROPIC_API_KEY your-api-key
setx GEMINI_API_KEY your-api-key
setx ELEVENLABS_API_KEY your-api-key

Restart Vizard after setting keys. With this method your API keys are accessible from any folder.

Obtaining Keys

Service Where to Get Key
OpenAI platform.openai.com/api-keys — Set a usage limit if needed
Anthropic console.anthropic.com — Sign up, verify email, generate key
Gemini aistudio.google.com/app/apikey — Also install google and google-generativeai via Package Manager. See Python Quickstart
ElevenLabs elevenlabs.io/app/developers/api-keys — Enable unrestricted access to voices

Customizing Your Agent

Follow these steps to create a custom AI agent with your own personality, avatar, and voice.

Step 1 – Create a Prompt

  1. In the prompts folder, make a copy of default.txt and rename it (e.g., MyAgent.txt)
  2. Write your role-playing instructions for the agent's personality, context, and behavior
  3. You can use an LLM to help create a detailed role-playing prompt

💡 Tip: You can use default.txt as-is if you don't need a custom personality.

SightLab GUI avatar workflow settings

Step 2 – Create an Avatar Config

  1. In the configs folder, make a copy of Template_Male.py or Template_Female.py and give it a name (e.g., MyAvatar.py). It is best to use those templates as the other included ones such as RocketBoxMale.py would need to have USE_SCENE_AVATAR = True in their config in order to replace the avatar.
  2. Find an avatar — either from the Avatars tab in the SightLab Dashboard Assets library, or drag in from any source (for auto-detection, use RocketBox or Mixamo avatars (requires adding .cfg to the name in Inspector))
  3. Open the avatar in Inspector by double clicking or choosing Open with - Inspector and check the Animations panel (bottom left) — note which animation state number you want for idle and which for talking
  4. In your config file, set:
  5. TALK_ANIMATION — the state number for the talking animation
  6. IDLE_ANIMATION — the state number for the idle animation
  7. Optionally change voices, language, and other settings in the config file (see Avatar Configuration Options for the full list)

SightLab GUI avatar workflow settings

Step 3 – Add the Avatar to Your Scene

The simplest way to add an avatar is to drag and drop it directly into your environment in Inspector. The AI Agent will automatically detect and use the first avatar found in the scene (controlled by USE_SCENE_AVATAR = True, which is the default).

Using the GUI (simplest)

  1. Run AI_Agent_GUI.py
  2. Choose the template config you made (or Template_Male.py or Template_Female.py). If you choose RocketBoxMale.py or one of the other ones you will have two avatars if you add one.
  3. Choose an included environment or drag in a new one
  4. Click Edit in the GUI to open the environment in Inspector
  5. Drag and drop your avatar model into the scene
  6. Position the avatar where you want it using the translate and rotate tools (see Adding Avatar Agents for placement tips)
  7. If the avatar's name doesn't end in .cfg, right-click it in Inspector and choose Rename to add .cfg to the end

Avatar animation and morph settings in the GUI

  1. Go to File → Save in Inspector, then close it
  2. Back in the SightLab GUI, you will see the avatar you added — set the State for the starting idle animation (should match the IDLE_ANIMATION in your config file)
  3. Run the scene (note: you will need to run it again if you want to change the prompt or config)

SightLab GUI avatar workflow settings

Without the GUI

  1. Add the avatar to your environment in Inspector as above (steps 4–7)
  2. Set the avatar name in Config_Global.py:
    SCENE_AVATAR_NAME = 'CC2_m001_hipoly_A0_v2.cfg'
  3. If running from the SightLab Dashboard, click the config action button to directly open the config file
  4. Run AI_Agent.py

Reallusion Avatars

Reallusion avatars need to be loaded by path rather than from the scene. Make a copy of the x_ReallusionMale.py config file, set the path to the avatar directly (this can be done for any avatar that isn't automatically detected as well):

AVATAR_MODEL = 'Resources/avatars/Reallusion_ActorCore_Male.osgb'

Getting Assets

Refer to this page for instructions on obtaining environments and avatars. Drag new assets into the SightLab Dashboard asset library and then can drag into your project.

For more details on avatar placement, animations, and data collection, see Adding Avatar Agents.

Interaction Controls

  • Hold 'c' key or RH grip button to speak; release to let the agent respond
  • If USE_SPEECH_RECOGNITION is False, press 'c' to type a question
  • To stop the conversation, type "q" and click "OK" in text chat
  • If HOLD_KEY_TO_SPEAK is False, you only need to speak and when there is a pause of over 0.8 seconds, the agent will respond, then wait for you to speak again (requires wearing headphones)
  • Press 'h' to send a screenshot as a prompt and ask questions about what the AI agent is seeing (in the sub-window), or can just ask "What are we looking at?"

Configuration

Most settings (AI model, voice, language, speech model, etc.) are available as dropdown selections in the GUI. For advanced customization, open Config_Global.py directly — or click the config action button from the SightLab Dashboard.

Passthrough / Augmented Reality

To enable AR passthrough mode, set this in Config_Global.py:

USE_PASSTHROUGH = True

This works with Meta Quest Pro, Meta Quest 3, and Varjo headsets.

Text-to-Speech Models & Latency

Not all TTS options respond at the same speed. Testing each engine under identical conditions (same local Ollama model, same prompts) showed a clear pattern:

TTS latency comparison

Why the gap: Piper and Kokoro run entirely on your machine, so there's no network request involved — the avatar starts speaking as soon as the text is ready. Edge TTS, ElevenLabs, OpenAI TTS, and GPT Realtime are all cloud services, so every reply pays a network round-trip on top of synthesis, typically adding a second or more before audio starts regardless of how short the reply is. Piper is the fastest overall; Edge TTS and Kokoro are good middle-ground choices when voice quality or multilingual support matters more than shaving off that extra second.

Ollama vs. online LLM APIs: the same local-vs-cloud trade-off applies to the LLM itself, separate from the TTS engine:

  • A local Ollama model, once loaded into memory, typically starts generating a response in well under a second — there's no network call involved at all.
  • ChatGPT, Claude, and Gemini are cloud APIs, so each reply pays a network round-trip on top of generation time — in testing this generally added roughly 1–2 seconds compared to a warmed-up local model, on top of whichever TTS latency above.

For the most responsive setup, pair Piper with an Ollama model. For the best voice quality, at the cost of a second or two extra per reply, Edge TTS, Eleven Labs, or a cloud LLM are all reasonable trade-offs.

Choosing a TTS Engine

Engine Description
Piper Fully offline, fast, no API key needed. A solid choice for privacy or no-internet use, though the voices sound noticeably less natural than the other options here. Samples
Edge TTS Free, high-quality Microsoft neural voices with a large library, no API key required — but needs an internet connection. Samples
Kokoro The best-sounding fully offline option, free, and runs well even on CPU-only machines. Fixed set of ~54 voices, English is best supported. Samples
OpenAI TTS Convenient if you're already using OpenAI for the LLM. Quality ranges from good to very natural depending on the model selected; requires an API key. Samples
GPT Realtime Its standout feature is genuine emotional expressiveness and natural intonation — it can laugh, shift tone mid-sentence, and follow delivery instructions like "sound excited" or "speak sadly" (this project supports it via built-in tone tags). Samples (Realtime-exclusive Marin/Cedar voices not included)
Eleven Labs The most natural, expressive voice quality of the group, with a Flash model built for very low latency, and the only option with true voice cloning. Also supports tone tags via a slower model reserved for emotionally-tagged lines. Requires an API key and extra setup. Samples (note: There are many more voice options for ElevenLabs that require a paid account)

Speech Recognition & Response-Timing Settings

A few Config_Global.py settings beyond what's exposed in the GUI dropdowns are worth knowing about, particularly for tuning the trade-off between accuracy and speed.

Whisper accuracy vs. speed

WHISPER_MODEL = "base"      # tiny, base, small, medium, large
WHISPER_BEAM_SIZE = 5       # 1 = fastest/lowest quality, 5 = best quality (Whisper's default)
Setting Faster More accurate
WHISPER_MODEL "tiny" "large" (default: "base")
WHISPER_BEAM_SIZE 1 5

"base" with beam size 5 is a good default for most use cases. Drop to "tiny" / beam size 1 for the fastest possible turnaround in demos, or move up to "small" / "medium" for professional transcription accuracy.

Avatar Configuration Options

Each avatar uses a config file in the configs folder that controls animations, voice, and behavior. Key settings include TALK_ANIMATION, IDLE_ANIMATION, voice selection, mouth morphing, blinking, and bone/joint references. Make a copy of Template_Male.py or Template_Female.py and adjust as needed — see the comments in the template for all available options. If the avatar doesn't apply things like the head turning to follow you, facial expressions, etc. then you may need to edit these parameters and refer to the avatar in Inspector to see the bone names, etc.

AI Agent Event System

The AI agent supports event-driven interactions, allowing the AI to trigger custom actions during conversations for more expressive and interactive experiences.

The event system allows the AI to execute custom actions (like facial expressions, animations, or any other callback) by including special "event" keywords in its responses. These events are automatically detected, executed, and removed from the text shown to the user.

AI Agent

How It Works

  1. Event Detection: The AI includes event: <event_name> on a line in its response
  2. Event Execution: The system detects this line, triggers the corresponding handler
  3. Text Cleaning: The event line is removed before displaying text to the user
  4. Action: The custom action (e.g., facial expression, processing a screenshot, etc.) is performed

Event Configuration

Global Settings (Config_Global.py)

# Event System Settings
USE_EVENT_SYSTEM = True  # Enable/disable event system
EVENT_KEYWORD = "event:"  # Keyword that triggers events

# Morph Target Indices for Facial Expressions
SMILE_MORPH_ID = 3   # Avatar-specific morph index for smile
SAD_MORPH_ID = 2     # Avatar-specific morph index for sad
EXPRESSION_MORPH_AMOUNT = 0.7  # Intensity (0.0 to 1.0)
EXPRESSION_DURATION = 1.2      # Duration in seconds

Avatar-Specific Settings (configs/RocketBoxMale.py, etc.)

Each avatar config can override the default morph indices:

# Event System - Facial Expression Morph Targets
SMILE_MORPH_ID = 3   # RocketBox smile morph
SAD_MORPH_ID = 2     # RocketBox sad morph
EXPRESSION_MORPH_AMOUNT = 0.7
EXPRESSION_DURATION = 1.2

Built-in Events

Facial Expressions

  • smile: Makes avatar smile (uses SMILE_MORPH_ID)
  • sad: Makes avatar look sad (uses SAD_MORPH_ID)
  • neutral: Returns avatar to neutral expression (resets both morphs)

Actions

nod: Nod head in agreement

shake head: shake head side to side in disagreement

Vision

  • Capture and process screenshot of scene: When asked things such as "What do you see" the agent can capture and process a screenshot to give an understanding of its surroundings.

Placeholder Events (for future implementation)

  • wave: Wave animation placeholder

Finding Morph Target Indices for Your Avatar

Different avatars have different morph target indices. To find the correct indices:

  1. Load your avatar in Inspector
  2. Click on the avatar name in the scene graph and view the morph IDs on the right side Properties pane under "Morphs"
  3. Update your avatar config file with the correct indices

Creating Custom Events

Step 1: Create the Event Handler Function

In AI_Agent_Avatar.py, after the existing event handlers:

def event_my_custom_action():
    """Description of what this event does"""
    try:
        # Your custom code here
        # Examples:
        # - Trigger animations: avatar.state(MY_ANIMATION)
        # - Move objects: object.setPosition([x, y, z])
        # - Play sounds: viz.playSound('sound.wav')
        # - Change lighting: viz.clearcolor(viz.RED)
        print("Custom action executed!")
    except Exception as e:
        print(f"Error in custom event: {e}")

Step 2: Register the Event

In the register_default_events() function:

def register_default_events():
    """Register all built-in event handlers"""
    if USE_EVENT_SYSTEM:
        EVENT_REGISTRY.register("smile", event_smile)
        EVENT_REGISTRY.register("sad", event_sad)
        EVENT_REGISTRY.register("neutral", event_neutral)
        EVENT_REGISTRY.register("my_custom_action", event_my_custom_action)  # Add this
        # ... rest of the events

Step 3: Update AI Prompt

Add your custom event to the prompt file so the AI knows about it:

Available events you can trigger:
- smile: Makes you smile
- sad: Makes you look sad  
- event: screenshot - Take and analyze a screenshot of what you're seeing
- my_custom_action: Description of what it does

Usage Example

User: "I just won the lottery!"

AI Response (raw):

event: smile
That's incredible! Congratulations! You must be so excited!

What happens:

  1. Avatar smiles (smile morph applied)
  2. User sees: "That's incredible! Congratulations! You must be so excited!"

Emotional Tone Control

For TTS engines that support expressive delivery, the AI can tag part of its response with tone:<name> (e.g. tone:excited, tone:sad, tone:angry, tone:warm, tone:calm) to control how that line is spoken. Like event tags, the tone tag is automatically parsed out before the text is displayed or spoken — the user never sees or hears the tag itself, just its effect on delivery.

Two TTS engines support this: GPT Realtime and Eleven Labs. Each is controlled by its own flag, since the latency cost differs between them:

# Config_Global.py
USE_EMOTIONAL_TONE_GPT_REALTIME = True   # little to no added latency
USE_EMOTIONAL_TONE_ELEVENLABS = True     # switches tagged lines to a slower model - see below

GPT Realtime: the tone is applied as a delivery instruction on the same request that reads the line aloud, so there's little to no added latency.

Eleven Labs: only the eleven_v3 model interprets emotional delivery cues, so a tagged line is automatically prepended with a [tone] audio cue and switched to ELEVEN_LABS_TONE_MODEL (defaults to eleven_v3) just for that line — every other line still uses your fast, normal ELEVEN_LABS_MODEL. eleven_v3 is noticeably slower, so if that latency isn't worth it for your use case, set USE_EMOTIONAL_TONE_ELEVENLABS = False to disable tone tags for Eleven Labs while keeping them enabled for GPT Realtime (or the other way around).

# Config_Global.py
ELEVEN_LABS_TONE_MODEL = "eleven_v3"
ELEVEN_LABS_STREAMING_LATENCY = 0  # 0-4, latency optimization for normal (non-toned) lines only

💡 You can also write a literal tag directly into a prompt file, e.g. [angry] Get out of here!, and Eleven Labs will honor it the same way as an AI-chosen tone: tag — useful for a scripted response or closing line (LAST_MESSAGE) that always needs a specific delivery.

The AI is only told about the tone: convention (via a short addition automatically appended to its prompt) when the active SPEECH_MODEL is GPT Realtime or Eleven Labs and that engine's tone flag is enabled — so it won't try to use tone tags with an engine that doesn't support them.

Structured Trials & Automatic Conversation Flow

When running the AI Agent inside a SightLab experiment (AI_Agent_GUI.py or AI_Agent.py), you can automate the conversation across trials — useful for research studies with multiple conditions, structured interviews, or timed interactions.

Per-Trial Prompts

Assign a different prompt file to specific trial numbers. When that trial starts, the agent's context prompt and conversation history reset to the file listed for that trial number. Trials with no entry in the map keep whatever prompt is currently loaded — so, for example, leaving trial 1 out of the map lets it keep the prompt chosen from the startup dropdown, while trial 2 onward follows the script.

# Config_Global.py
USE_PER_TRIAL_PROMPTS = True

TRIAL_PROMPTS = {
    2: "prompts/Job_Interview.txt",
    3: "prompts/Patient_Surgery.txt",
}

Session Conditions

Both scripted responses and the contextual prompt (below) can branch based on a "condition" for the session — e.g. the avatar behaving "appropriately" vs. "inappropriately" in a behavioral study. A condition is resolved once — either randomly from a list or pinned to a specific value — and reused for the rest of the session.

# Config_Global.py
SCRIPT_CONDITIONS = ['appropriate', 'inappropriate']  # leave as [] if you don't need condition branching

# Pin a specific condition instead of randomizing (useful for testing a specific condition). None = randomize.
FORCE_CONDITION = 'inappropriate'

💡 FORCE_CONDITION takes effect on its own — you don't need to populate SCRIPT_CONDITIONS just to force a specific value.

The resolved condition can also be read from your own code — for example from a custom event handler reacting to something that happens in the trial:

condition = AI_Agent_Avatar._resolve_condition()

Contextual Prompt

Automatically choose the agent's system prompt based on the resolved session condition (above), instead of picking one from the startup dropdown. Useful when different conditions need entirely different agent personalities or behavior, not just different scripted lines.

# Config_Global.py
CONTEXTUAL_PROMPT = True

CONTEXTUAL_PROMPT_FILES = {
    'appropriate':   "prompts/Prompt-Condition2.txt",
    'inappropriate': "prompts/Prompt-Condition1.txt",
}

When enabled, the startup prompt-file dropdown is skipped entirely — the prompt is loaded based on the resolved condition before the experiment starts.

The prompt can also be swapped mid-session — for example from a custom event handler reacting to something that happens in the trial. This also clears the conversation history so the agent starts fresh under the new prompt:

AI_Agent_Avatar._load_contextual_prompt('condition_name')

Scripted Responses

Skip the live AI model entirely and speak pre-written lines from a text file instead — useful for demos, pilot testing, and controlled trials where the agent's exact wording needs to be fixed in advance. Scripted lines still go through the normal event and tone parsing, so they can trigger avatar actions (event: smile) or tone tags (tone:excited) just like a live AI response can.

# Config_Global.py
SCRIPTED_RESPONSES = True

# Maps trial number (1-based) to a script file, relative to the script folder.
# Without condition branching (SCRIPT_CONDITIONS = []):
SCRIPTED_RESPONSE_FILES = {
    1: "Scripted_Responses/trial1.txt",
    2: "Scripted_Responses/trial2.txt",
}

# With SCRIPT_CONDITIONS branching, map trial number to a {condition: path} dict instead:
SCRIPTED_RESPONSE_FILES = {
    1: {
        'appropriate':   "Scripted_Responses/trial1_appropriate.txt",
        'inappropriate': "Scripted_Responses/trial1_inappropriate.txt",
    },
    2: "Scripted_Responses/trial2.txt",
}

Each line of the script file is spoken in order, one per turn; blank lines and lines starting with # are ignored. Once the last line is reached, the agent keeps repeating it for any further turns in that trial.

Trials without an entry in SCRIPTED_RESPONSE_FILES keep whatever script is already loaded — the same behavior as TRIAL_PROMPTS.

Auto-Ending a Trial After N Exchanges

Automatically end a trial once a set number of back-and-forth exchanges (one user message + one agent response) has happened, instead of only ending when the continue key is pressed. The trial ends only after the agent's final response finishes playing — not the instant the response text is generated — so the agent is never cut off mid-reply.

# Config_Global.py
SET_NUMBER_OF_INTERACTIONS = True
NUMBER_OF_INTERACTIONS = 3

💡 If USE_SPEECH_RECOGNITION is False (typed chat), the chat window closes itself automatically once the limit is reached — the same as typing "q" — and reopens on its own at the start of trial 2 onward.

Scripted Closing Line

Have the agent speak a fixed closing line right before the trial ends — a natural way to wrap up the conversation instead of cutting it off abruptly. Only takes effect together with SET_NUMBER_OF_INTERACTIONS; if the trial is ended manually with the continue key instead, the scripted line is skipped.

# Config_Global.py
SET_LAST_MESSAGE = True
LAST_MESSAGE = "Thank you for your time today, now I must end this conversation."

Transcript Logging

Save a full, timestamped record of the conversation — every user message and every agent response — for review or research records.

# Config_Global.py
SAVE_TRANSCRIPT = True

A new file is created the first time something is logged each run, named with that session's start date and time, e.g. Transcripts/transcript_2026-07-29_14-32-05.txt.

Knowledge Base

Give the agent background information to draw on during conversation — product info, character backstory, FAQs, policies, and more — without writing it all directly into the prompt file.

  1. Create a Knowledge folder next to AI_Agent_Avatar.py
  2. Add .txt and/or .pdf files with the information you want the agent to know
  3. Enable it in Config_Global.py:
USE_KNOWLEDGE = True

Every file in the folder is read at startup and appended to the agent's system prompt, so it can reference that information naturally in conversation rather than reciting it verbatim.

⚠️ Note: Reading .pdf files requires the pypdf package (install via the Vizard Package Manager or pip install pypdf). .txt files work with no extra install; if pypdf isn't installed, PDFs are skipped with a console warning rather than stopping the script. Local Ollama models have smaller context windows than the hosted APIs, so keep knowledge files concise if you're using an offline model.

Multi Agent Interactions

AI Agent

See the script multi_agent_interaction.py to see how multiple agents can interact and communicate with each other. You can modify the individual agents by calling the AIAgent class and setting parameters such as config_path, name, and prompt_path.

Conversation Modes

The conversation loop can operate in two modes:

  • Scripted Dialogue: Pre-written lines that agents speak in sequence
  • AI-Generated: Dynamic conversation based on a leading question

User Interaction

You can interrupt and speak to either agent during their conversation:

  • Press 'c' to speak to Agent1 (left avatar)
  • Press 'v' to speak to Agent2 (right avatar)
  • Press 'r' to restart the conversation loop

When you interrupt, the agent will turn toward you and stop any current speech. The conversation loop pauses while you interact.

Note that for multi agent interactions, the AI model and voices are defined in the respective config file for each agent (see the configs folder).

Example

from AI_Agents import AIAgent

agents = [
    AIAgent(config_path='configs/x_Multi_RocketBoxFemale.py', name='Agent1', prompt_path="prompts/Susan_Home Office.txt"),
    AIAgent(config_path='configs/x_Multi_RocketBoxMale.py', name='Agent2', prompt_path="prompts/Tom_Home_Office.txt"),
]

Adding to Existing Scripts

  1. Copy the "configs", "keys", and "prompts" folders, as well as AI_Agent_Avatar.py.
  2. Import with:
    from configs.AI_Agent_Config import *
import AI_Agent_Avatar
  3. Add avatar:
    avatar = AI_Agent_Avatar.avatar
sightlab.addSceneObject('avatar', avatar, avatar=True)

#Or if using the avatar you place in your scene
USE_SCENE_AVATAR = AI_Agent_Avatar.USE_SCENE_AVATAR
  4. Add these lines to enable passthrough augmented reality:
    if USE_PASSTHROUGH:
    import openxr
    xr = openxr.getClient()

    if sightlab.getConfig() in ["Meta Quest Pro", "Meta Quest 3"]:
        passthrough = xr.getPassthroughFB()
    elif sightlab.getConfig() == "Varjo":
        passthrough = xr.getPassthroughVarjo()

    viz.clearcolor(viz.BLACK, 0.0)
    if passthrough:
        passthrough.setEnabled(True)

Publishing as an Executable

To publish an application with the AI Agent (or to just publish the standard included template) add this code to the top of the script (before importing SightLab):

import viz

# Filter out problematic packages that cause issues in published EXE
viz.res.addPublishFilter('*google_generativeai*')
viz.res.addPublishFilter('*-nspkg.pth')
viz.res.addPublishFilter('*.pth')

# Add publish directories - ensures correct package versions are bundled
publish_directories = [
    'data',
    viz.res.getVizardPath() + 'bin/lib/site-packages/sightlab_utils/',
    viz.res.getVizardPath() + 'bin/lib/site-packages/deepdiff',
    viz.res.getVizardPath() + 'bin/lib/site-packages/numpy',
    viz.res.getVizardPath() + 'bin/lib/site-packages/pandas',
    viz.res.getVizardPath() + 'bin/lib/site-packages/pydantic',
    viz.res.getVizardPath() + 'bin/lib/site-packages/pydantic_core',
    viz.res.getVizardPath() + 'bin/lib/site-packages/openai',
    viz.res.getVizardPath() + 'bin/lib/site-packages/httpx',
    viz.res.getVizardPath() + 'bin/lib/site-packages/httpcore',
    viz.res.getVizardPath() + 'bin/lib/site-packages/anyio',
    viz.res.getVizardPath() + 'bin/lib/site-packages/sniffio',
    viz.res.getVizardPath() + 'bin/lib/site-packages/annotated_types',
    viz.res.getVizardPath() + 'bin/lib/site-packages/certifi',
    viz.res.getVizardPath() + 'bin/lib/site-packages/speech_recognition',
    viz.res.getVizardPath() + 'bin/lib/site-packages/jiter',
]

for directory in publish_directories:
    viz.res.addPublishDirectory(directory)

import sightlab_utils.sightlab as sl
from sightlab_utils.settings import *

Tips & Troubleshooting

Tips

  • Environment Awareness: Press 'h' to take a screenshot that is sent to the agent, or simply ask it "What do you see" or "What are we looking at", etc.
  • Prompts: For configuring the agent, use "I am..." for OpenAI and "You are..." for Anthropic without quotes.
  • Event Trigger for Speech Button: Modify vizconnect to add an event for speaking button hold. Open settings.py in sightlab_utils/vizconnect_configs, and modify mappings for triggerDown and triggerUp or create new ones if needed. More Info on Vizconnect Events
  • If getting an error with Gemini "out of quota", try using a model with more quota or enable billing for much higher limits.

Common Issues

Issue Fix
Microphone/headset conflicts Check that microphone settings match between the VR headset and output device
ElevenLabs character limit Free tier limits to 10,000 characters; paid accounts get more
ElevenLabs 402 payment_required / "Free users cannot use library voices via the API" The selected voice was added from the Voice Library (or is a cloned voice) — this requires a paid ElevenLabs plan to use via the API. See Using More Voices

ElevenLabs Documentation: ElevenLabs GitHub

Features

Interact Models Avatar
Interact and converse with custom AI Large Language Models in a VR or XR simulation in real time. Choose from various LLM Models. Requires API Key. Modify avatar appearance, animations, environment, and more. Works with most avatar libraries
Personality Speech Voices
Customize personality of agent, contextual awareness, emotional state, interactions, and more. Save as custom agents. Use speech recognition to converse using your voice or text-based input. Choose from high-quality voices from Eleven Labs and other libraries (requires API) or customize and create your own.
Training Features
Train the agent as it adapts using a history of the conversation and its interactions. Works with all features of SightLab, such as data collection and visualizations, transcript saving, and more.

There is also available a version of this that just runs as an education based tool, where you can select objects in a scene and get information and labels on that item (such as paintings in an art gallery). See this page for that version. Additionally, there are teaching based AI Agent avatars in the E-Learning Lab tool.