Skip to content

Educational Interaction Tool- AI


Download Latest Version

This document describes how to use the Educational Interaction Tool in SightLab. This can be connected to an interactive, intelligent AI agent powered by various large language models like Chat GPT. You can customize the agent's personality, use speech recognition, and leverage high-quality text-to-speech models. You can also record your own annotations that work with these features, connecting them to a virtual avatar or just using voice-over.

Tagged objects can display 3D text annotations and trigger audio explanations. With AI integration, users can ask follow-up questions. Any 3D scene object should automatically be taggable for interactions and conversational information.

Location: ExampleScripts > Education_Application_AI

Key Features

  • Interact and converse with custom AI Large Language Models in real-time VR or XR simulations.
  • Customize the agent's personality, contextual awareness, emotional state, interactions, and more.
  • Use speech recognition for voice or text-based interaction.
  • Select high-quality voices from OpenAI TTS (or can modify to connect others like Eleven Labs and Piper).
  • Train the agent to adapt using conversation history and interactions.
  • Works seamlessly with all SightLab features, including data collection, visualizations, and transcript saving.
  • Automatically tag objects in scenes to prompt questions and present information.

Instructions

1. Installation

  • Install the required libraries using the

Vizard Package Manager

. These include:

  • openai (for OpenAI GPT agents, or the specific library you need if adding another LLM)
  • SpeechRecognition
  • sounddevice (pyaudio for older versions)

  • numpy (included in SightLab)

  • Note: An active internet connection is required.

2. API Keys

  • OpenAI:
  • Visit OpenAI.

  • Sign up/log in and navigate to the API section.

  • In windows search type "cmd" enter setx OPENAI_API_KEY "your-api-key", setx GEMINI_API_KEY "your-api-key",

  • setx ELEVENLABS_API_KEY "your-api-key", setx ANTHROPIC_API_KEY "your-api-key"
  • Restart Vizard
  • With this method you don't need to keep the keys in a folder in your project and your api keys can be accessed from any folder.

3. Running the Script

Run AI_Agent_Education.py to start.

4. Interaction

  • Press and hold the 'c' key or RH grip button to speak. Release to stop and let the AI respond.
  • Use the mouse or RH trigger to select objects and prompt information.
  • Right click or use the RH grip to bring up the highlighter (should be on by default)

Modifying Environment and Avatars

  • Place environment models in resources/environments
  • Use the SightLab VR GUI to select which objects in the scene will be interactive.

Issues and Troubleshooting

  • Microphone Settings: Errors may occur if the microphone source conflicts between the VR headset and the system output.
  • Character Limits: Eleven Labs' free tier limits output to 10,000 characters (paid plans offer higher limits).
  • ffmpeg/mpv Errors: Ensure ffmpeg and mpv are installed and their paths are added to Vizard's environment path.

Tips

  • Environment Awareness: Take screenshots in SightLab (/ key), upload them to ChatGPT, and use the descriptions in prompts.
  • Custom Event Mapping: Modify vizconnect settings in settings.py under sightlab_utils/vizconnect_configs for speaking button events. See Vizconnect Events for more.
  • You can connect "Assistants" through the openai API, but not custom GPTs.