MCP Evals
About
A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.
Details
- Author
- mclenhard
- GitHub stars
- 129
- Downloads
- 397
- Categories
- Other, AI
Jump to
- LLM-based scoring on five dimensions (accuracy, completeness, relevance, clarity, reasoning)
- Built-in observability with metrics and tracing
- GitHub Action integration for PR-based evaluation
- Support for TypeScript and YAML evaluation configurations
- Metrics: tool call counts, errors, and latency distributions
- Monitoring stack with Prometheus, Grafana, and Jaeger dashboards
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
MCP EvalsCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Install via npm install mcp-evals and run evaluations using the CLI: npx mcp-evals <evals-file> <server-file>. Alternatively, add the GitHub Action to your workflow with config keys like evals_path, server_path, openai_api_key, and model. Create evaluation files in TypeScript (.ts) or YAML (.yaml/.yml) and set environment variables OPENAI_API_KEY or ANTHROPIC_API_KEY as needed.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"mcp evals": {
"mcp-evals": {
"command": "npx",
"args": [
"mcp-eval",
"path/to/your/evals.ts",
"path/to/your/server.ts"
]
}
}
}
}McpServers
{
"mcp-evals": {
"command": "npx",
"args": [
"mcp-eval",
"path/to/your/evals.ts",
"path/to/your/server.ts"
]
}
}MCP Evals
A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring, with built-in observability support. This helps ensure your MCP server's tools are working correctly, performing well, and are fully observable with integrated monitoring and metrics.
Installation
As a Node.js Package
npm install mcp-evals
As a GitHub Action
Add the following to your workflow file:
name: Run MCP Evaluations
on:
pull_request:
types: [opened, synchronize, reopened]
jobs:
evaluate:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm install
- name: Run MCP Evaluations
uses: mclenhard/mcp-evals@v1.0.9
with:
evals_path: 'src/evals/evals.ts' # Can also use .yaml files
server_path: 'src/index.ts'
openai_api_key: ${{ secrets.OPENAI_API_KEY }}
model: 'gpt-4' # Optional, defaults to gpt-4
Usage -- Evals
1. Create Your Evaluation File
You can create evaluation configurations in either TypeScript or YAML format.
Option A: TypeScript Configuration
Create a file (e.g., evals.ts) that exports your evaluation configuration:
import { EvalConfig } from 'mcp-evals';
import { openai } from "@ai-sdk/openai";
import { grade, EvalFunction} from "mcp-evals";
const weatherEval: EvalFunction = {
name: 'Weather Tool Evaluation',
description: 'Evaluates the accuracy and completeness of weather information retrieval',
run: async () => {
const result = await grade(openai("gpt-4"), "What is the weather in New York?");
return JSON.parse(result);
}
};
const config: EvalConfig = {
model: openai("gpt-4"),
evals: [weatherEval]
};
export default config;
export const evals = [
weatherEval,
// add other evals here
];
Option B: YAML Configuration
For simpler configuration, you can use YAML format (e.g., evals.yaml):
```yaml
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
