Skip to content

AI Intelligent Agent

AI Agent

Overview

The AI Intelligent Agent is an interactive tool within E-Learning Lab that enables real-time conversation with large language models like ChatGPT, Claude, Gemini, and offline Ollama models. You can customize the agent's personality, leverage speech recognition, and use high-quality text-to-speech.

How to Run

Basic Interaction

  • Toggle AI Agent Assistance and Speech Recognition in the Options tab.
  • Hold the 2 key (desktop) or RH B button (VR) to speak; release to stop.
  • If INTERRUPT_AI is enabled, press 2 or RH B to interrupt and speak again.
  • Without speech recognition, press 2 to open a text chat window.
  • Type q and click OK to end conversations.
  • Use the laser pointer to click objects for information.

Placing in Scene

There are three ways to position the AI Agent in a scene, in order of simplicity:

  1. Drag a custom avatar into the Scene Editor — Drag any avatar into the Scene Editor preview and position it with the Move / Rotate gizmo. Then in Options > AI Agent, check Use Scene Avatar and choose that avatar from the dropdown. This sets the AI Agent's position, rotation, and appearance in one step; AVATAR_MODEL (Male/Female) still determines the voice/TTS.
  2. Drag the start point into the Scene Editor — Drag the START_POINT_AI_AGENT marker into the Scene Editor preview and position it with the Move / Rotate gizmo. This only sets where the built-in avatar (chosen via AVATAR_MODEL) spawns, without changing its appearance.
  3. Use Inspector — Open a slide in Inspector, use File > Add to place START_POINT_AI_AGENT, then position it with the move and rotate tools. This is the least simple option, but bakes the start point permanently into the scene file.

Key Features

Real-time conversation feature

Real-time conversation feature

Multiple AI model support

  • Real-time conversation with custom AI models in VR/XR simulations
  • Support for OpenAI, Anthropic, Gemini, and offline Ollama models
  • Customizable avatar appearance, animations, and environment
  • Personality and contextual awareness customization
  • Speech recognition for voice interaction, or text-based input
  • High-quality voices from OpenAI TTS, Eleven Labs, Edge TTS, or offline Kokoro, Pocket TTS and Piper
  • Adaptive learning through conversation history
  • Multi-language support via Edge TTS and Piper language packs (see Setting the Language)

Configuration

Access configuration under the Options Tab > AI Agent section:

  • AI_MODEL — Choose CHAT_GPT, CLAUDE, GEMINI, or Offline Ollama
  • AVATAR_MODEL — Select Male or Female; also sets the voice/TTS used even when Use Scene Avatar is enabled
  • Use Scene Avatar — Use an avatar dragged into the scene instead of the built-in model; pick which one from the dropdown (see Placing in Scene)
  • SPEECH_MODEL — OpenAI TTS, Eleven Labs, Edge TTS, Kokoro, Pocket TTS, Piper, or Pyttsx3
  • AI Agent Assistance — Toggle agent availability
  • Speech Recognition — Toggle between voice or text input

Pocket TTS (offline voice)

Pocket TTS is a free, fully offline voice that runs on the CPU and needs no API key. The SightLab installer installs it as the Pocket TTS (offline AI voice) component, checked by default. To add it later, run installer\AI-Agent-Pocket-TTS.exe in the E-Learning Lab folder. If it isn't installed, Edge TTS is used instead.

  • Voice: choosing Male or Female for AVATAR_MODEL picks the voice bill_boerst or eve. To use one of the 17 other voices, or a cloned voice, add it as described in Pocket TTS Setup. Then set POCKET_TTS_VOICE in AI_Enabled\AI_Agent\configs\AI_Agent_Config_Education.py; do this after choosing the avatar, since changing the avatar resets it.

Adding Custom Knowledge Sources

From Windows Explorer

Name files with _knowledge in the filename:

  • my_notes_knowledge.txt
  • Heart_anatomy_knowledge.pdf
  • lesson_knowledge.md

Drag files onto the preview panel; they save to collections/<project>/Knowledge/<scene_name>_knowledge.<ext>.

From Within the App

Drag .txt files with _knowledge in the Prompts tab onto the preview panel to route them to the Knowledge folder.

Adding Custom Prompts

  1. Select the Prompts tab.
  2. Add new text files or select existing prompts.
  3. Right-click and select Add to Scene, or drag into the preview window.

Prompts appear on the Scene tab in the Projects panel.

API Key Setup

Obtaining API Keys

OpenAI (ChatGPT and Text-to-Speech)

  1. Visit openai.com.
  2. Sign up or log in.
  3. Navigate to the API section.
  4. Click Create a new secret key and copy it.
  5. Purchase credits (roughly $5 minimum recommended).
  6. Adjust usage limits in the Usage section.

Eleven Labs (Text-to-Speech)

  1. Log in to elevenlabs.io.
  2. Click the profile icon (top-right).
  3. Click the eye icon next to the API Key field.
  4. Copy the API key.
  5. Save it to elevenlabs_key.txt in the root SightLab folder.

Anthropic

  1. Visit console.anthropic.com/login.
  2. Sign up or log in.
  3. Complete the sign-up form with email and use-case details.
  4. Verify your account via the confirmation email.
  5. Log in and navigate to the API section.

Gemini and Gemini Ultra

  1. Visit aistudio.google.com/app/apikey.
  2. Install google-generativeai: install -q -U google-generativeai
  3. Reference the Google AI Python Quickstart for setup details.

Setting API Keys in Windows

Open Command Prompt and enter:

setx OPENAI_API_KEY "your-api-key"
setx GEMINI_API_KEY "your-api-key"
setx ELEVENLABS_API_KEY "your-api-key"
setx ANTHROPIC_API_KEY "your-api-key"

Troubleshooting

  • Microphone/audio issues — Ensure microphone and sound output device settings are consistent (both headset or both computer).
  • Eleven Labs character limit — Free accounts have a 10,000 character limit; paid accounts offer higher quotas.
  • [Pocket TTS] Not installed in the console — Pocket TTS isn't installed on this computer, so Edge TTS is used. Run installer\AI-Agent-Pocket-TTS.exe in the E-Learning Lab folder, or the SightLab installer with the Pocket TTS component.
  • ffplay error — Install ffmpeg and add it to the Vizard environment path (gyan.dev/ffmpeg/builds).
  • mpv player error — Install mpv and add it to the Vizard environment path (mpv.io/installation).
  • Gemini quota error — Switch to gemini-1.5-flash-latest or enable billing for higher limits.

Tips

Environment awareness — You can ask "What are we looking at" or press h to capture a screenshot and send it to the AI agent for environmental context.