Whisper Speech Recognition MCP Server

Features

- Integrated with Faster Whisper for efficient speech recognition
- Batch processing acceleration for improved transcription speed
- Automatic CUDA acceleration (if available)
- Support for multiple model sizes (tiny to large-v3)
- Output formats include VTT subtitles, SRT, and JSON
- Support for batch transcription of audio files in a folder
- Model instance caching to avoid repeated loading
- Dynamic batch size adjustment based on GPU memory

Installation

Dependencies

- Python 3.10+
- faster-whisper>=0.9.0
- torch==2.6.0+cu126
- torchaudio==2.6.0+cu126
- mcp[cli]>=1.2.0

Installation Steps

1. Clone or download this repository
2. Create and activate a virtual environment (recommended)
3. Install dependencies:

pip install -r requirements.txt

PyTorch Installation Guide

Install the appropriate version of PyTorch based on your CUDA version:

- CUDA 12.6:

  pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu126

  

- CUDA 12.1:

  pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121

  

- CPU version:

  pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cpu

  

You can check your CUDA version with nvcc --version or nvidia-smi.

Usage

Starting the Server

On Windows, simply run start_server.bat.

On other platforms, run:

python whisper_server.py

Configuring Claude Desktop

1. Open the Claude Desktop configuration file:
- Windows: %APPDATA%\Claude\claude_desktop_config.json
- macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

2. Add the Whisper server configuration:

{
  "mcpServers": {
    "whisper": {
      "command": "python",
      "args": ["D:/path/to/whisper_server.py"],
      "env": {}
    }
  }
}

3. Restart Claude Desktop

Available Tools

The server provides the following tools:

1. get_model_info - Get information about available Whisper models
2. transcribe - Transcribe a single audio file
3. batch_transcribe - Batch transcribe audio files in a folder

Performance Optimization Tips

- Using CUDA acceleration significantly improves transcription speed
- Batch processing mode is more efficient for large numbers of short audio files
- Batch size is automatically adjusted based on GPU memory size
- Using VAD (Voice Activity Detection) filtering improves accuracy for long audio
- Specifying the correct language can improve transcription quality

Local Testing Methods

1. Use MCP Inspector for quick testing:

mcp dev whisper_server.py

2. Use Claude Desktop for integration testing

3. Use command line direct invocation (requires mcp[cli]):

mcp run whisper_server.py

Error Handling

The server implements the following error handling mechanisms:

- Audio file existence check
- Model loading failure handling
- Transcription process exception catching
- GPU memory management
- Batch processing parameter adaptive adjustment

Project Structure

- whisper_server.py: Main server code
- model_manager.py: Whisper model loading and caching
- audio_processor.py: Audio file validation and preprocessing
- formatters.py: Output formatting (VTT, SRT, JSON)
- transcriber.py: Core transcription logic
- start_server.bat: Windows startup script

License

MIT

Acknowledgements

This project was developed with the assistance of these amazing AI tools and models:

- GitHub Copilot - AI pair programmer
- Trae - Agentic AI coding assistant
- Cline - AI-powered terminal
- DeepSeek - Advanced AI model
- Claude-3.7-Sonnet - Anthropic's powerful AI assistant
- Gemini-2.0-Flash - Google's multimodal AI model
- VS Code - Powerful code editor
- Whisper - OpenAI's speech recognition model
- Faster Whisper - Optimized Whisper implementation

Special thanks to these incredible tools and the teams behind them.