Patronus MCP Server
Description
# Patronus MCP Server An MCP server implementation for the Patronus SDK, providing a standardized interface for running powerful LLM system optimizations, evaluations, and experiments. ## Features - Initialize Patronus with API key and project settings - Run single evaluations…
About
# Patronus MCP Server An MCP server implementation for the Patronus SDK, providing a standardized interface for running powerful LLM system optimizations, evaluations, and experiments. ## Features - Initialize Patronus with API key and project settings - Run single evaluations with configurable evaluators - Run batch…
Details
- Author
- patronus-ai
- GitHub stars
- 15
- Downloads
- 213
- Categories
- Other
Jump to
- Initialize Patronus with API key and project settings
- Run single evaluations with configurable evaluators
- Run batch evaluations with multiple evaluators
- Run experiments with datasets
- Create custom evaluator criteria in the Patronus API
- List all available evaluators and their criteria
- Custom evaluate using user-defined evaluator functions
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
Patronus MCP ServerCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Clone the repository, create a Python virtual environment, install dependencies with uv pip install -e ., then run the server with an API key via --api-key argument or the PATRONUS_API_KEY environment variable. Use the included test script for interactive evaluation or call the server’s tools through any MCP client.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"patronus mcp server": {
"patronus-mcp-server": {
"command": "python",
"args": [
"-m",
"venv",
".venv"
]
}
}
}
}McpServers
{
"patronus-mcp-server": {
"command": "python",
"args": [
"-m",
"venv",
".venv"
]
}
}Patronus MCP Server
An MCP server implementation for the Patronus SDK, providing a standardized interface for running powerful LLM system optimizations, evaluations, and experiments.
Features
- Initialize Patronus with API key and project settings
- Run single evaluations with configurable evaluators
- Run batch evaluations with multiple evaluators
- Run experiments with datasets
Installation
1. Clone the repository:
git clone https://github.com/yourusername/patronus-mcp-server.git
cd patronus-mcp-server
2. Create and activate a virtual environment:
python -m venv .venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
3. Install main and dev dependencies:
uv pip install -e .
uv pip install -e ".[dev]"
Usage
Running the Server
The server can be run with an API key provided in two ways:
1. Command line argument:
python src/patronus_mcp/server.py --api-key your_api_key_here
2. Environment variable:
export PATRONUS_API_KEY=your_api_key_here
python src/patronus_mcp/server.py
Interactive Testing
The test script (tests/test_live.py) provides an interactive way to test different evaluation endpoints. You can run it in several ways:
1. With API key in command line:
python -m tests.test_live src/patronus_mcp/server.py --api-key your_api_key_here
2. With API key in environment:
export PATRONUS_API_KEY=your_api_key_here
python -m tests.test_live src/patronus_mcp/server.py
3. Without API key (will prompt):
python -m tests.test_live src/patronus_mcp/server.py
The test script provides three test options:
1. Single evaluation test
2. Batch evaluation test
Each test will display the results in a nicely formatted JSON output.
API Usage
Initialize
from patronus_mcp.server import mcp, Request, InitRequest
request = Request(data=InitRequest(
project_name="MyProject",
api_key="your-api-key",
app="my-app"
))
response = await mcp.call_tool("initialize", {"request": request.model_dump()})
Single Evaluation
from patronus_mcp.server import Request, EvaluationRequest, RemoteEvaluatorConfig
request = Request(data=EvaluationRequest(
evaluator=RemoteEvaluatorConfig(
name="lynx",
criteria="patronus:hallucination",
explain_strategy="always"
),
task_input="What is the capital of France?",
task_output="Paris is the capital of France."
task_context=["The capital of France is Paris."],
))
response = await mcp.call_tool("evaluate", {"request": request.model_dump()})
Batch Evaluation
from patronus_mcp.server import Request, BatchEvaluationRequest, RemoteEvaluatorConfig
request = Request(data=BatchEvaluationRequest(
evaluators=[
AsyncRemoteEvaluatorConfig(
name="lynx",
criteria="patronus:hallucination",
explain_strategy="always"
),
AsyncRemoteEvaluatorConfig(
name="judge",
criteria="patronus:is-concise",
explain_strategy="always"
)
],
task_input="What is the capital of France?",
task_output="Paris is the capital of France."
task_context=["The capital of France is Paris."],
))
response = await mcp.call_tool("batch_evaluate", {"request": request.model_dump()})
Run Experiment
```python
from patronus_mcp import Request, ExperimentRequest, RemoteEvaluatorConfig, CustomEvaluatorConfig
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
