broswer-automation/docs/MULTI_USER_CHROME_LIVEKIT_INTEGRATION.md

Multi-User Chrome Extension to LiveKit Agent Integration

This document explains how the system automatically spawns LiveKit agents for each Chrome extension user connection, creating a seamless multi-user voice automation experience.

Overview

The system now automatically creates a dedicated LiveKit agent for each user who installs and connects the Chrome extension. Each user gets:

• Unique Random User ID - Generated by Chrome extension and consistent across all components
• Dedicated LiveKit Agent - Automatically started for each user with the same user ID
• Isolated Voice Room - Each user gets their own LiveKit room (mcp-chrome-user-{userId})
• Session-Based Routing - Voice commands routed to correct user's Chrome extension
• Complete User ID Consistency - Same user ID flows through Chrome → Server → Agent → Back to Chrome

Architecture Flow

Chrome Extension (User 1) → Random User ID → LiveKit Agent (Room: mcp-chrome-user-{userId})
Chrome Extension (User 2) → Random User ID → LiveKit Agent (Room: mcp-chrome-user-{userId})
Chrome Extension (User 3) → Random User ID → LiveKit Agent (Room: mcp-chrome-user-{userId})
                    ↓
            Remote Server (Session Manager)
                    ↓
            Connection Router & LiveKit Agent Manager

How It Works

1. Chrome Extension Connection

When a user installs and connects the Chrome extension:

// Chrome extension generates unique user ID
const userId = `user_${Date.now()}_${Math.random().toString(36).substring(2, 15)}`;

// Chrome extension connects to ws://localhost:3001/chrome with user ID
const connectionInfo = {
  type: 'connection_info',
  userId: userId, // Chrome extension provides its own user ID
  userAgent: navigator.userAgent,
  timestamp: Date.now(),
  extensionId: chrome.runtime.id,
};

// Remote server receives and uses the Chrome extension's user ID
// Session created with user-provided ID: session_user_1703123456_abc123

2. Manual LiveKit Agent Management

LiveKit agents are no longer started automatically. They should be started manually when needed:

// LiveKit Agent Manager can spawn agent process with user ID when requested
const roomName = `mcp-chrome-user-${userId}`;
const agentProcess = spawn('python', ['livekit_agent.py', 'start'], {
  env: {
    ...process.env,
    CHROME_USER_ID: userId, // Pass user ID to LiveKit agent
    LIVEKIT_URL: this.liveKitConfig.livekit_url,
    LIVEKIT_API_KEY: this.liveKitConfig.api_key,
    LIVEKIT_API_SECRET: this.liveKitConfig.api_secret,
    MCP_SERVER_URL: 'http://localhost:3001/mcp',
  },
});

3. User-Specific Voice Room

Each user gets their own LiveKit room:

User 1 → Room: mcp-chrome-user-user_1703123456_abc123
User 2 → Room: mcp-chrome-user-user_1703123457_def456
User 3 → Room: mcp-chrome-user-user_1703123458_ghi789

4. Session-Based Command Routing with User ID

Voice commands are routed to the correct Chrome extension using user ID:

# LiveKit agent includes user ID in MCP requests
async def search_google(context: RunContext, query: str):
    # MCP client automatically includes user ID in headers
    result = await self.mcp_client._search_google_mcp(query)
    return result

// Remote server routes commands based on user ID
const result = await this.sendToExtensions(message, sessionId, userId);
// Connection router finds the correct Chrome extension by user ID

LiveKit Agent (User 1) → [User ID: user_123_abc] → Remote Server → Chrome Extension (User 1)
LiveKit Agent (User 2) → [User ID: user_456_def] → Remote Server → Chrome Extension (User 2)
LiveKit Agent (User 3) → [User ID: user_789_ghi] → Remote Server → Chrome Extension (User 3)

Key Components

LiveKitAgentManager

Location: app/remote-server/src/server/livekit-agent-manager.ts

Features:

• Automatic agent spawning on Chrome connection
• Process management and monitoring
• Agent cleanup on disconnection
• Room name generation based on user ID

Enhanced ChromeTools

Location: app/remote-server/src/server/chrome-tools.ts

Features:

• Integrated LiveKit agent management
• Automatic agent startup in registerExtension()
• Automatic agent shutdown in unregisterExtension()
• Session-based routing with LiveKit context

Enhanced LiveKit Agent

Location: agent-livekit/livekit_agent.py

Features:

• Room name parsing to extract Chrome user ID
• Chrome user session creation
• User-specific console logging
• Command line room name support

Console Logging

When Chrome Extension Connects:

🔗 Chrome extension connected - User: user_1703123456_abc123, Session: session_user_1703123456_abc123
🚀 Starting LiveKit agent for user: user_1703123456_abc123
✅ LiveKit agent started successfully for user user_1703123456_abc123

When LiveKit Agent Starts:

============================================================
🔗 NEW USER SESSION CONNECTED
============================================================
👤 User ID: user_1703123456_abc123
🆔 Session ID: session_user_1703123456_abc123
🏠 Room Name: mcp-chrome-user-user_1703123456_abc123
🎭 Participant: chrome_user_user_1703123456_abc123
⏰ Connected At: 1703123456.789
📊 Total Active Sessions: 1
============================================================

🔗 Detected Chrome user ID from room name: user_1703123456_abc123
✅ LiveKit agent connected to Chrome user: user_1703123456_abc123

When User Issues Voice Commands:

🌐 [Session: session_user_1703123456_abc123] Navigation to: https://google.com
✅ [Session: session_user_1703123456_abc123] Navigation completed

🔍 [Session: session_user_1703123456_abc123] Google search: 'python programming'
✅ [Session: session_user_1703123456_abc123] Search completed

When Chrome Extension Disconnects:

🔌 Chrome extension disconnected
🛑 Stopping LiveKit agent for user: user_1703123456_abc123
✅ LiveKit agent stopped for user user_1703123456_abc123

Setup Instructions

1. Start Remote Server

cd app/remote-server
npm start

2. Install Chrome Extensions (Multiple Users)

Each user:

1. Open Chrome → Extensions → Developer mode ON
2. Click "Load unpacked"
3. Select: app/chrome-extension/.output/chrome-mv3/
4. Extension automatically connects and gets unique user ID

3. Configure Cherry Studio (Each User)

Each user adds to their Cherry Studio:

{
  "mcpServers": {
    "chrome-mcp-remote-server": {
      "type": "streamableHttp",
      "url": "http://localhost:3001/mcp"
    }
  }
}

4. Join LiveKit Rooms (Each User)

Each user joins their specific room:

• User 1: mcp-chrome-user-user_1703123456_abc123
• User 2: mcp-chrome-user-user_1703123457_def456
• User 3: mcp-chrome-user-user_1703123458_ghi789

Testing

Test Multi-User Integration:

cd app/remote-server
node test-multi-user-livekit.js

This test:

1. Simulates multiple Chrome extension connections
2. Verifies unique user ID generation
3. Checks LiveKit agent spawning
4. Tests session isolation
5. Validates room naming

Expected Test Output:

👤 User 1: Chrome extension connected
📋 User 1: Received session info: { userId: "user_...", sessionId: "session_..." }
🚀 User 1: LiveKit agent should be starting for room: mcp-chrome-user-user_...

👤 User 2: Chrome extension connected
📋 User 2: Received session info: { userId: "user_...", sessionId: "session_..." }
🚀 User 2: LiveKit agent should be starting for room: mcp-chrome-user-user_...

✅ Session isolation: PASS
✅ User ID isolation: PASS
✅ Room isolation: PASS

Benefits

1. Zero Configuration

• Users just install Chrome extension
• LiveKit agents start automatically
• No manual room setup required

2. Complete Isolation

• Each user has dedicated agent
• Separate voice rooms
• Independent command processing

3. Scalable Architecture

• Supports unlimited concurrent users
• Automatic resource management
• Process cleanup on disconnect

4. Session Persistence

• User sessions tracked across connections
• Automatic reconnection handling
• State management per user

Monitoring

Agent Statistics:

// Get LiveKit agent stats
const stats = chromeTools.getLiveKitAgentStats();
console.log(stats);
// Output: { totalAgents: 3, runningAgents: 3, startingAgents: 0, ... }

Active Agents:

// Get all active agents
const agents = chromeTools.getAllActiveLiveKitAgents();
agents.forEach((agent) => {
  console.log(`User: ${agent.userId}, Room: ${agent.roomName}, Status: ${agent.status}`);
});

Troubleshooting

Common Issues:

1. LiveKit Agent Not Starting

   • Check Python environment in agent-livekit/
   • Verify LiveKit server is running
   • Check agent process logs

2. Multiple Agents for Same User

   • Check user ID generation uniqueness
   • Verify session cleanup on disconnect

3. Voice Commands Not Working

   • Verify user is in correct LiveKit room
   • Check session routing in logs
   • Confirm Chrome extension connection

Debug Commands:

# Check agent processes
ps aux | grep livekit_agent

# Monitor remote server logs
cd app/remote-server && npm start

# Test Chrome connection
node test-multi-user-livekit.js

The system now provides a complete multi-user voice automation experience where each Chrome extension user automatically gets their own dedicated LiveKit agent! 🎉