Judgmentlabs Mcp Server
About
A Model Context Protocol (MCP) server that provides seamless integration with the Judgment API for AI evaluation workflows. This server enables you to manage datasets, run evaluations, and track traces directly from your MCP-compatible environment.
Details
- Author
- suysoftware
- Downloads
- 162
- Categories
- Other
Jump to
- One-click DXT installation with no Python dependencies.
- Create, update, and retrieve datasets with flexible data handling.
- Create and delete projects; auto-creates projects when pushing datasets.
- Run evaluations and fetch detailed results.
- Create, fetch, and delete individual traces.
- Cross‑platform support (Windows, macOS, Linux).
Install via Claude Desktop’s DXT extension system (download the .dxt file from releases, then install through Settings → Extensions) or build from source. Configure your JUDGMENT_API_KEY and JUDGMENT_ORG_ID in the extension settings. After enabling and restarting Claude Desktop, use the provided tools (e.g., push_dataset, run_evaluation) through natural language prompts.
JudgmentLabs MCP Server
A Model Context Protocol (MCP) server that provides seamless integration with the Judgment API for AI evaluation workflows. This server enables you to manage datasets, run evaluations, and track traces directly from your MCP-compatible environment like Claude Desktop.
🎥 Demo Video
🚀 Features
🎯 One-Click Installation
- DXT Package: Install as a single.dxt file in Claude Desktop
- No Dependencies: All Python packages are pre-bundled
- Auto-Configuration: Easy setup through Claude Desktop's settings UI
- Cross-Platform: Works on Windows, macOS, and Linux
📊 Dataset Management
- Create & Manage Datasets: Push datasets with examples and traces to the Judgment API - Flexible Data Handling: Support for append and overwrite modes when updating datasets - Data Retrieval: Pull existing datasets from your Judgment projects - Smart Example Conversion: Automatically handles various input formats (input/question, expected_output/answer)Project Operations
- Project Creation: Create new projects in the Judgment API - Project Cleanup: Delete projects and all associated data - Auto-Creation: Projects are automatically created when pushing datasets if they don't existEvaluation & Monitoring
- Run Evaluations: Execute evaluation runs via the Judgment API - Results Retrieval: Fetch detailed evaluation results for analysis - Trace Management: Create, fetch, and delete individual traces - Real-time Monitoring: Track AI agent performance and behaviorDeveloper Experience
- Error Handling: Comprehensive error handling with helpful suggestions - Debug Logging: Built-in debugging capabilities for troubleshooting - Flexible Configuration: Environment-based configuration with sensitive data protection📋 Prerequisites
- Claude Desktop (latest version with DXT support)
- JudgmentLabs account with API access
- For manual installation: Python 3.8 or higher
🛠️ Installation
Method 1: DXT Extension (Recommended) 🎯
The easiest way to install this MCP server is through Claude Desktop's DXT extension system:
1. Download the Extension:
- Get the latest judgmentlabs-mcp-server.dxt file from the [releases page
- No Python installation required!
2. Install in Claude Desktop:
- Open Claude Desktop
- Go to Settings → Extensions
- Click "Install Extension"
- Select the downloaded .dxt file
- The extension will be automatically installed
3. Configure API Credentials:
- In Claude Desktop, go to Settings → Extensions
- Find "JudgmentLabs MCP" and click Configure
- Enter your credentials:
- JUDGMENT_API_KEY: Your API key from JudgmentLabs Dashboard
- JUDGMENT_ORG_ID: Your organization ID from the dashboard
4. Enable and Test:
- Toggle the extension to "Enabled"
- Restart Claude Desktop
- Test by asking: "Create a new project called 'test-project'"
Method 2: Build Your Own DXT Package (Advanced Users) ⚙️
For developers who want to build the DXT package from source:
Prerequisites
- Python 3.8 or higher - Node.js and npmBuild Steps
```bashSign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
