Voice Assistant MCP Server

A powerful Model Context Protocol (MCP) server that integrates Twilio Voice, Deepgram AI, and OpenAI to create intelligent voice-based HR automation tools. This system enables AI assistants like Claude to conduct phone interviews, deliver notifications, and manage HR communications through natural voice conversations.

Demo video

https://github.com/user-attachments/assets/6c47a8b7-7428-4711-bf26-00d5951ec66f

Features

Core Capabilities

- AI-Powered Voice Interviews: Conduct professional phone interviews with candidates using AI
- Interview Result Notifications: Automatically call candidates with interview results and feedback
- Job Opportunity Outreach: Reach out to potential candidates about new positions
- Real-time Voice Processing: Powered by Deepgram’s Voice Agent API
- MCP Integration: Seamless integration with Claude and other MCP-compatible AI assistants

Technical Features

- WebSocket-based Media Streaming: Real-time audio processing with Twilio Media Streams
- Dynamic Prompt Injection: Contextual AI prompts based on call purpose
- Function Call Handling: Advanced AI function calling for call management
- Comprehensive Logging: Detailed logging for debugging and monitoring
- Environment Configuration: Secure credential management

Architecture

sequenceDiagram
    participant Claude as MCP Client(Claude App)
    participant MCP as MCP Server
    participant Twilio as Twilio Voice
    participant Deepgram as Deepgram
    participant AI as OpenAI
%% Step 1: Initiate interaction

    Claude->>MCP: Initiate voice action (e.g., interview, notification)

    MCP->>Twilio: Setup voice call

    Twilio-->>MCP: Call status updates
%% Step 2: Real-time audio processing

    Twilio->>Deepgram: Start audio stream

    Deepgram-->>AI: Transcribed text

    AI->>Deepgram: LLM response

    Deepgram->>Twilio: Stream audio

Prerequisites

Before setting up the project, ensure you have:

1. Node.js (v22 or higher)
2. Twilio Account with:
- Account SID
- Auth Token
- Phone Number (for outbound calls)
3. Deepgram Account with API Key
4. Public URL (ngrok or production server) for webhooks

Installation

1. Clone the repository:

   git clone https://github.com/prakharbhardwaj/voice-agent-mcp-server.git
   cd voice-agent-mcp-server
   

2. Install dependencies:

   npm install
   

3. Environment Configuration:
Create a .env file in the root directory:

   # Server Configuration
   PORT=3000
   SERVER_URL=your_ngrok_url_or_server_url
# Twilio Credentials

   TWILIO_ACCOUNT_SID=your_twilio_account_sid

   TWILIO_AUTH_TOKEN=your_twilio_auth_token

   TWILIO_PHONE_NUMBER=+your_twilio_phone_number
# Deepgram API Key

   DEEPGRAM_API_KEY=your_deepgram_api_key

   

4. Configure MCP Server:
Update mcp-config.json with your actual paths and credentials:

   {

     "mcpServers": {

       "voice-agent-mcp-server": {

         "type": "stdio",

         "command": "node",

         "args": ["/path/to/your/voice-agent-mcp-server/mcp-server.js"],

         "env": {

           "NODE_ENV": "production",

           "SERVER_URL": "your_ngrok_url_or_server_url",

           "TWILIO_ACCOUNT_SID": "your_twilio_account_sid",

           "TWILIO_AUTH_TOKEN": "your_twilio_auth_token",

           "TWILIO_PHONE_NUMBER": "+your_twilio_phone_number"

         }

       }

     }

   }

   

Usage

Start the Web Server

npm run dev

Using with Claude Desktop

1. Add the MCP server configuration to Claude Desktop's settings:
- For macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
- For Windows: %APPDATA%\Claude\claude_desktop_config.json
2. Restart Claude Desktop
3. Use the available tools in your conversation with Claude

Available MCP Tools

1. conduct_interview

Initiate a voice call to conduct a professional interview with a candidate.

Parameters:

- candidatePhone (string): Phone number in E.164 format
- candidateName (string): Name of the candidate
- position (string): Position they're applying for
- interviewQuestions (array): List of questions to ask

Example:

Conduct an interview with John Doe at +1234567890 for the Software Engineer position.
Ask about their experience with React, their problem-solving approach, and their career goals.

2. notify_interview_result

Call a candidate to inform them about their interview results.

Parameters:

- candidatePhone (string): Phone number in E.164 format
- candidateName (string): Name of the candidate
- position (string): Position they interviewed for
- result (enum): "accepted", "rejected", or "next_round"
- message (string): Additional feedback message

Example:

Call Jane Smith at +1234567890 to let her know she's been accepted for the Product Manager role.

3. discuss_job_opening

Reach out to potential candidates about job opportunities.

Parameters:

- candidatePhone (string): Phone number in E.164 format
- candidateName (string): Name of the potential candidate
- position (string): Job position to discuss
- companyInfo (string): Brief company and role information
- nextSteps (string): Next steps if interested

4. get_call_status

Get the status of active voice calls and system health.

5. check_twilio_config

Verify Twilio configuration and service readiness.

Project Structure

├── index.js                   # Main Fastify server
├── mcp-server.js              # MCP server implementation
├── mcp-config.json            # MCP configuration
├── package.json               # Dependencies and scripts
└── src/
    ├── SettingsConfiguration.js   # Deepgram agent settings
    ├── config/
    │   └── dotenv.js              # Environment configuration
    ├── mcp/
    │   ├── logger.js              # Logging utilities
    │   ├── prompts.js             # AI prompt generators
    │   ├── server.js              # MCP server logic
    │   └── tools.js               # MCP tool definitions
    ├── routes/
    │   └── twilioRoute.js         # Twilio webhook handlers
    ├── services/
    │   ├── functionCallHandler.js # AI function call processing
    │   └── twilioService.js       # Twilio API wrapper
    └── websockets/
        └── mediaStreamHandler.js  # WebSocket media processing

Configuration Details

Deepgram Agent Settings

The system uses Deepgram's Voice Agent with:

- Speech Recognition: Nova-3 model
- Text-to-Speech: Aura-2 Asteria voice
- LLM Integration: OpenAI GPT-4o-mini
- Audio Format: μ-law encoding at 8kHz (Twilio compatible)

Twilio Integration

- Media Streams: Real-time audio streaming via WebSocket
- TwiML: Dynamic call routing with custom parameters
- Call Management: Status tracking and call control

Troubleshooting

Common Issues

1. Twilio webhook not receiving calls:

- Ensure your SERVER_URL is publicly accessible
- Check ngrok is running and URL is updated
- Verify TwiML configuration

2. Deepgram connection issues:

- Validate DEEPGRAM_API_KEY
- Check WebSocket connectivity
- Review audio format compatibility

3. MCP server not loading in Claude:
- Verify mcp-config.json paths are absolute
- Check all environment variables are set
- Restart Claude Desktop after configuration changes

Debugging

Enable detailed logging by checking the console output and src/mcp/mcp-server.log file.

Development

Running in Development Mode

```bash