Crawlbase MCP
About
Enables AI agents to access real-time web data with HTML, markdown, and screenshot support. SDKs: Node.js, Python, Java, PHP, .NET.
Details
- Author
- crawlbase
- GitHub stars
- 55
- Downloads
- 373
- Categories
- Developer Tools, Other, Web Scraping, AI
Jump to
- Fetches live web data for AI agents
- JavaScript rendering for modern web apps
- Proxy rotation and anti‑bot evasion
- Structured outputs: HTML, Markdown, screenshots
- Async crawling with Crawlbase Cloud Storage
- HTTP transport with per‑request token authentication
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
Crawlbase MCPCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Sign up at Crawlbase to get free Normal and JavaScript tokens. Configure the MCP server in a client like Claude Desktop, Claude Code, Cursor IDE, or Windsurf IDE by adding its JSON configuration with your tokens. Then use commands like crawl, crawl_markdown, or crawl_screenshot to fetch web data. An HTTP transport mode is also available for multi‑user or custom integrations.
crawl
Crawl a URL and return HTML content
crawl_markdown
Crawl a URL and extract clean markdown content
crawl_screenshot
Take a screenshot of a webpage
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"crawlbase mcp": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": [
"@crawlbase/mcp@latest"
],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}
}McpServers
{
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": [
"@crawlbase/mcp@latest"
],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}What is Crawlbase MCP?
Crawlbase MCP is a Model Context Protocol (MCP) server that bridges AI agents and the live web. Instead of relying on outdated training data, your LLMs can now fetch fresh, structured, real-time content — powered by Crawlbase’s proven crawling infrastructure trusted by 70,000+ developers worldwide.
It handles the complexity of scraping for you:
- JavaScript rendering for modern web apps
- Proxy rotation & anti-bot evasion
- Structured outputs (HTML, Markdown, screenshots)
How It Works
- Get Free Crawlbase Tokens → Sign up at Crawlbase ↗️, get free Normal, and JavaScript tokens.
- Set Up MCP Configuration → Configure the MCP server in your preferred client (Claude, Cursor, or Windsurf) by updating the MCP Servers settings.
- Start Crawling → Use commands like crawl, crawl_markdown, or crawl_screenshot to bring live web data into your AI agent.
Setup & Integration
Claude Desktop
1. Open Claude Desktop → Settings → Developer → Edit Config
2. Add to claude_desktop_config.json:
3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}
Claude Code
Add to your claude.json configuration:
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}
Cursor IDE
1. Open Cursor IDE → File → Preferences → Cursor Settings → Tools and Integrations → Add Custom MCP
2. Add to mcp.json:
3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}
Windsurf IDE
1. Open WindSurf IDE → File → Preferences → WindSurf Settings → General → MCP Servers → Manage MCPs → View raw config
2. Add to mcp_config.json:
3. Replace your_token_here and your_js_token_here with the tokens from your dashboard.
{
"mcpServers": {
"crawlbase": {
"type": "stdio",
"command": "npx",
"args": ["@crawlbase/mcp@latest"],
"env": {
"CRAWLBASE_TOKEN": "your_token_here",
"CRAWLBASE_JS_TOKEN": "your_js_token_here"
}
}
}
}
HTTP Transport Mode
For scenarios where you need a shared MCP server accessible over HTTP (e.g., multi-user environments, custom integrations), you can run the server in HTTP mode:
```bash
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
