Studio: 11labs VPS Agent
Project where an 11labs conversational agent is integrated into niantic studio and can talk about and interact with the scene.
Project where an 11labs conversational agent is integrated into niantic studio and can talk about and interact with the scene.
11labs based AI voice agent answering questions about vps' that are augmented with additional data. The agent can retrieve this info by calling a function called get_spatial_data as well as additional POIs he can highlight with highlight_poi function. This creates an engaging and interactive way for the user to learn infos about a specific location.
Code explanation:
11labs-agent.ts Component
Schema fields:
- html: Entity ID for HTML loader
- highlightEntity: Entity ID for highlighting objects
- agentId: String for the agent identifier on 11labs
Key functions:
- init(agentId): Sets up conversation controls and UI elements
- startConversation(): Initiates voice conversation with the agent
- stopConversation(): Ends the current conversation session
- highlightFunc: Handles highlighting POIs in 3D space
- Listens for 'vps:found' events to update spatial data
vps-data.ts Component
Schema fields:
- agent: Entity ID for the agent
- name: String identifier for the location
- vpsdata: String containing asset path for spatial data information
Key functions:
- Listens for two global events:
- 'reality.locationlost': Logs when a location is no longer detected
- 'reality.locationfound': When a location is found, dispatches 'vps:found' event if the location name matches
JSON Format for vps-data
json { "name": "string", "data": { "info": "string", "poi": [ { "descr": "string", "position": x, y, z, "info": "string" } ] } }
Where:
- name: Identifies the location/artwork
- data.info: Contains general information about the artwork
- data.poi: Array of Points of Interest, each containing:
- descr: Description of the point
- position: 3D coordinates x, y, z
- info: Additional information about the point
The two example JSON files describing different VPS':
- 298b.json: "Junge Naturforscher" (Young Nature Researchers) statue
- b351.json: "Lesendes Mädchen" (Reading Girl) statue by Heinz Mamat from 1958
These components work together to provide location-aware information about artworks, with the ability to highlight specific points of interest in an AR environment.
Other Work by VRketingTeam