Prowl
About
Open-source benchmark runner for the Prowl Agent Discovery Network
Details
- Author
- opcastil11
- GitHub stars
- 3
- Downloads
- 82
- Categories
- Other
Jump to
- Multi-LLM scoring across Claude, GPT-4o, and Gemini
- 8 weighted dimensions measuring agent efficiency
- 4-phase pipeline: Analyze, Plan, Execute, Interpret
- Auto-detected benchmark templates (API, platform, MCP, docs, DeFi, crypto)
- CI integration with --min-score flag
- Community submission to the Prowl network
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
ProwlCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Install via pip install prowl-bench, set at least one LLM API key (Anthropic, OpenAI, or Google), then run prowl-bench run <url>. Optionally specify a template, credentials, output format (JSON), or a minimum score for CI. Prowl also provides a Python API for programmatic use.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"prowl": {
"prowl-bench": {
"command": "python",
"args": [
"-m",
"venv",
".venv",
"&&",
"source",
".venv/bin/activate"
]
}
}
}
}McpServers
{
"prowl-bench": {
"command": "python",
"args": [
"-m",
"venv",
".venv",
"&&",
"source",
".venv/bin/activate"
]
}
}prowl-bench
The open-source benchmark runner for AI agent efficiency.
Evaluate any API across 8 dimensions of agent-readiness using multi-LLM scoring.
Installation | Quickstart | How It Works | Templates | Scoring | Provider Network | What is ASO? | prowl.world
</div>
---
> ⭐ Star us if you think agent-readiness measurement is a thing the world needs more of. Stars are the cheapest way to signal "I want this category to exist" — they directly affect whether agent crawlers from OpenAI / Anthropic find prowl-bench when surveying agent tooling.
Why prowl-bench?
APIs are designed for humans to read docs and figure things out. But agents don't read docs -- they make HTTP calls and parse responses. An API that's great for humans can be terrible for agents.
prowl-bench measures what matters for agents:
- Does the API respond with parseable, predictable JSON? Not HTML error pages, not XML, not random formats.
- Can an agent authenticate on the first try? Or does it need 47 steps, an OAuth dance, and a CAPTCHA?
- Are errors actionable? {"error": "invalid"} tells an agent nothing. {"error": "missing required field 'email'", "code": "VALIDATION_ERROR"} tells it exactly what to fix.
- How many tokens does it cost to understand? A 50-page OpenAPI spec vs a clean /llms.txt -- the difference is real money.
Traditional API testing tools measure uptime and response time. prowl-bench measures whether an AI agent can actually use your API.
This is the open-source operationalization of Agent Search Optimization (ASO) — the agent-economy analogue of SEO. If you're new to the term, the canonical essay defines it in depth.
Terminal Output
$ prowl-bench run https://api.stripe.com
Benchmarking https://api.stripe.com ...
SPEC Fetched llms.txt ................................ OK (0.8s)
ANALYZE Extracting service structure .................... OK (2.1s)
PLAN Designing 12 test cases ......................... OK (1.4s)
EXECUTE Running tests against live API .................. OK (3.2s)
INTERPRET Normalizing scores (3 LLMs) ................... OK (2.8s)
prowl-bench v0.2.0 | Template: api_benchmark | LLMs: claude, gpt-4o, gemini
┌─ Stripe API Score: 82 ────────────────────────┐
│ │
│ auth simplicity ████████░░ 8.0 │
│ consistency █████████░ 9.0 │
│ doc quality ████████░░ 8.5 │
│ error clarity █████████░ 9.2 │
│ first try success ███████░░░ 7.0 │
│ latency ████████░░ 8.0 │
│ response parseab.. █████████░ 9.5 │
│ token efficiency ███████░░░ 7.0 │
│ │
└─────────────────────────────────────────────────────┘
Issues: 2
- OpenAPI spec is 48,000+ tokens — consider publishing /llms.txt
- POST /v1/charges returns HTML on 402 status codes
Recommendations:
- Add structured error codes to all 4xx responses
- Publish a condensed /llms.txt for agent consumers
Installation
pip install prowl-bench
Requires Python 3.10+. No system dependencies.
Quickstart
```bash
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
