Skip to content

AI Intelligent Agent

AI Agent

Overview

The AI Intelligent Agent is an interactive tool within E-Learning Lab that enables real-time conversation with large language models like ChatGPT, Claude, Gemini, and offline Ollama models. You can customize the agent's personality, leverage speech recognition, and use high-quality text-to-speech.

How to Run

Basic Interaction

  • Toggle AI Agent Assistance and Speech Recognition in the Options tab.
  • Hold the 2 key (desktop) or RH B button (VR) to speak; release to stop.
  • If INTERRUPT_AI is enabled, press 2 or RH B to interrupt and speak again.
  • Without speech recognition, press 2 to open a text chat window.
  • Type q and click OK to end conversations.
  • Use the laser pointer to click objects for information.

Placing in Scene

Open a slide in Inspector, use File > Add to place START_POINT_AI_AGENT, then position it with the move and rotate tools.

Key Features

Real-time conversation feature

Multiple AI model support

  • Real-time conversation with custom AI models in VR/XR simulations
  • Support for OpenAI, Anthropic, Gemini, and offline Ollama models
  • Customizable avatar appearance, animations, and environment
  • Personality and contextual awareness customization
  • Speech recognition for voice interaction, or text-based input
  • High-quality voices from OpenAI TTS, Eleven Labs, or offline Piper
  • Adaptive learning through conversation history
  • Multi-language support via Piper language packs (see Setting the Language)

Configuration

Access configuration under the Options Tab > AI Agent section:

  • AI_MODEL — Choose CHAT_GPT, CLAUDE, GEMINI, or Offline Ollama
  • AVATAR_MODEL — Select Male/Female or a custom avatar
  • SPEECH_MODEL — OpenAI TTS, Eleven Labs, Piper, or Pyttsx3
  • AI Agent Assistance — Toggle agent availability
  • Speech Recognition — Toggle between voice or text input

Adding Custom Knowledge Sources

From Windows Explorer

Name files with _knowledge in the filename:

  • my_notes_knowledge.txt
  • Heart_anatomy_knowledge.pdf
  • lesson_knowledge.md

Drag files onto the preview panel; they save to collections/<project>/Knowledge/<scene_name>_knowledge.<ext>.

From Within the App

Drag .txt files with _knowledge in the Prompts tab onto the preview panel to route them to the Knowledge folder.

Adding Custom Prompts

  1. Select the Prompts tab.
  2. Add new text files or select existing prompts.
  3. Right-click and select Add to Scene, or drag into the preview window.

Prompts appear on the Scene tab in the Projects panel.

API Key Setup

Obtaining API Keys

OpenAI (ChatGPT and Text-to-Speech)

  1. Visit openai.com.
  2. Sign up or log in.
  3. Navigate to the API section.
  4. Click Create a new secret key and copy it.
  5. Purchase credits (roughly $5 minimum recommended).
  6. Adjust usage limits in the Usage section.

Eleven Labs (Text-to-Speech)

  1. Log in to elevenlabs.io.
  2. Click the profile icon (top-right).
  3. Click the eye icon next to the API Key field.
  4. Copy the API key.
  5. Save it to elevenlabs_key.txt in the root SightLab folder.

Anthropic

  1. Visit console.anthropic.com/login.
  2. Sign up or log in.
  3. Complete the sign-up form with email and use-case details.
  4. Verify your account via the confirmation email.
  5. Log in and navigate to the API section.

Gemini and Gemini Ultra

  1. Visit aistudio.google.com/app/apikey.
  2. Install google-generativeai: install -q -U google-generativeai
  3. Reference the Google AI Python Quickstart for setup details.

Setting API Keys in Windows

Open Command Prompt and enter:

setx OPENAI_API_KEY "your-api-key"
setx GEMINI_API_KEY "your-api-key"
setx ELEVENLABS_API_KEY "your-api-key"
setx ANTHROPIC_API_KEY "your-api-key"

Troubleshooting

  • Microphone/audio issues — Ensure microphone and sound output device settings are consistent (both headset or both computer).
  • Eleven Labs character limit — Free accounts have a 10,000 character limit; paid accounts offer higher quotas.
  • ffplay error — Install ffmpeg and add it to the Vizard environment path (gyan.dev/ffmpeg/builds).
  • mpv player error — Install mpv and add it to the Vizard environment path (mpv.io/installation).
  • Gemini quota error — Switch to gemini-1.5-flash-latest or enable billing for higher limits.

Tips

Environment awareness — You can ask "What are we looking at" or press h to capture a screenshot and send it to the AI agent for environmental context.