AnyCrawl

by any4ai

6 stars

258 downloads

Not rated

GitHub

About

[AnyCrawl](https://anycrawl.dev) MCP Server, Powerful web scraping and crawling for Cursor, Claude, and other LLM clients via the Model Context Protocol (MCP).

Details

Author: any4ai
GitHub stars: 6
Downloads: 258
Categories: Developer Tools, AI

- Scrape single URLs with multiple output formats
- Crawl entire websites with configurable depth and limits
- Search engine integration for web search and scraping
- Multiple engines: auto, Playwright, Cheerio, Puppeteer
- Flexible output: Markdown, HTML, text, screenshots, JSON
- Async operations with status monitoring for crawl jobs

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below

Plugin Name AnyCrawl

Command (node, npx, python, etc.)

Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

From the repository

Install and run via npx with your API key: ANYCRAWL_API_KEY=YOUR-API-KEY npx -y anycrawl-mcp. For cloud usage, set the API key only; for self-hosted, additionally set ANYCRAWL_BASE_URL. The server runs in STDIO mode by default; switch to MCP (HTTP) or SSE mode by setting ANYCRAWL_MODE. Then configure your MCP client (Cursor, Claude Desktop, etc.) to connect to Anycrawl using the provided tool definitions.

`anycrawl_scrape`

Scrape a single URL and extract content in selected formats. Best for: One known page (articles, docs, product pages). Not recommended for: Multi-page coverage (use anycrawl_crawl) or open-ended discovery (use anycrawl_search). RECOMMENDED: Use 'playwright' engine for best results with dynamic content and modern websites. Usage (parameters): - url: HTTP/HTTPS URL to scrape (string, required) - engine: 'playwright' | 'cheerio' | 'puppeteer' (required, default: 'playwright') - proxy: Proxy URL (string, optional) - formats: Output formats ['markdown'|'html'|'text'|'screenshot'|'screenshot@fullPage'|'rawHtml'|'json'] (optional) - timeout: Request timeout in ms (number, optional) - retry: Enable auto-retry on failure (boolean, optional) - wait_for: Wait in ms for dynamic pages (number, optional) - include_tags: HTML tags to include (string[], optional) - exclude_tags: HTML tags to exclude (string[], optional) - json_options: { schema?, user_prompt?, schema_name?, schema_description? } (optional) - extract_source: 'html' | 'markdown' (optional) Returns: { url, status, jobId?, title?, html?, markdown?, metadata?, timestamp? } Examples: - Recommended: { "url": "https://example.com", "engine": "playwright" } - With JSON extraction: { "url": "https://news.ycombinator.com", "engine": "playwright", "formats": ["markdown"], "json_options": { "user_prompt": "Extract titles", "schema_name": "Articles" } } - With JSON schema extraction: { "url": "https://example.com/article", "engine": "playwright", "json_options": { "schema_name": "Article", "schema_description": "Extract article metadata and content", "schema": { "type": "object", "properties": { "title": { "type": "string" }, "author": { "type": "string" }, "date": { "type": "string" }, "content": { "type": "string" } }, "required": ["title", "content"] } } }

`anycrawl_crawl`

Crawl an entire website with configurable depth and limits. Best for: Multi-page coverage, site mapping, content discovery. Not recommended for: Single pages (use anycrawl_scrape) or open-ended discovery (use anycrawl_search). RECOMMENDED: Use 'playwright' engine for best results with dynamic content and modern websites. Usage (parameters): - url: Starting URL to crawl (string, required) - engine: 'playwright' | 'cheerio' | 'puppeteer' (required, default: 'playwright') - max_depth: Maximum crawl depth (number, optional, default: 10) - limit: Maximum pages to crawl (number, optional, default: 100) - strategy: Crawl strategy 'all' | 'same-domain' | 'same-hostname' | 'same-origin' (optional, default: 'same-domain') - include_paths: Path patterns to include (string[], optional) - exclude_paths: Path patterns to exclude (string[], optional) - retry: Enable auto-retry on failure (boolean, optional) - poll_seconds: Polling interval for job status (number, optional) - poll_interval_ms: Polling interval in milliseconds (number, optional) - timeout_ms: Job timeout in milliseconds (number, optional) - scrape_options: Nested scrape options for each page (object, optional) Returns: { job_id, status, message } for async jobs Examples: - Recommended: { "url": "https://example.com", "engine": "playwright", "limit": 50 } - Deep crawl: { "url": "https://docs.example.com", "engine": "playwright", "max_depth": 5, "limit": 200 } - Filtered crawl: { "url": "https://blog.example.com", "engine": "playwright", "include_paths": ["/posts/*"], "exclude_paths": ["/admin/*"] }

`anycrawl_search`

Search the web and optionally scrape results. Best for: Open-ended discovery, finding relevant content. Not recommended for: Known URLs (use anycrawl_scrape) or comprehensive site coverage (use anycrawl_crawl). RECOMMENDED: Use limit=5 for balanced performance and cost. Use 'playwright' engine for scraping results. Usage (parameters): - query: Search query string (string, required) - engine: Search engine 'google' (optional, default: 'google') - limit: Number of results to return (number, optional, default: 5) - offset: Number of results to skip (number, optional, default: 0) - pages: Number of search result pages to process (number, optional) - lang: Language code (string, optional) - country: Country code (string, optional) - safeSearch: Safe search level 0-2 (number, optional) - scrape_options: Options for scraping search results (object, optional) Returns: Array of search results with optional scraped content Examples: - Recommended: { "query": "artificial intelligence news", "limit": 5 } - With scraping: { "query": "TypeScript tutorials", "limit": 5, "scrape_options": { "formats": ["markdown"], "engine": "playwright" } } - Localized search: { "query": "machine learning", "lang": "es", "country": "ES", "limit": 5 }

`anycrawl_crawl_status`

Get the status of a crawl job. Check progress, completion status, and statistics for an ongoing or completed crawl job. Usage (parameters): - job_id: The crawl job ID (string, required) Returns: { job_id, status, start_time, expires_at, credits_used, total, completed, failed } Examples: - Check status: { "job_id": "crawl_12345" }

`anycrawl_crawl_results`

Get the results of a completed crawl job. Retrieve the scraped content and metadata from a completed crawl job. Usage (parameters): - job_id: The crawl job ID (string, required) - skip: Number of results to skip (number, optional, default: 0) Returns: { status, total, completed, creditsUsed, next?, data[] } Examples: - Get all results: { "job_id": "crawl_12345" } - Paginated results: { "job_id": "crawl_12345", "skip": 50 }

`anycrawl_cancel_crawl`

Cancel a running crawl job. Stop an ongoing crawl job and prevent further processing. Usage (parameters): - job_id: The crawl job ID to cancel (string, required) Returns: { success: boolean, message: string } Examples: - Cancel job: { "job_id": "crawl_12345" }

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "anycrawl": {
            "anycrawl-mcp": {
                "command": "npx",
                "args": [
                    "-y",
                    "anycrawl-mcp-server"
                ],
                "env": {
                    "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
                    "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
                    "LOG_LEVEL": "info"
                }
            }
        }
    }
}

McpServers

{
    "anycrawl-mcp": {
        "command": "npx",
        "args": [
            "-y",
            "anycrawl-mcp-server"
        ],
        "env": {
            "ANYCRAWL_API_KEY": "<YOUR_TOKEN>",
            "ANYCRAWL_BASE_URL": "https://api.anycrawl.dev",
            "LOG_LEVEL": "info"
        }
    }
}

AnyCrawl MCP Server

🚀 AnyCrawl MCP Server — Powerful web scraping and crawling for Cursor, Claude, and other LLM clients via the Model Context Protocol (MCP).

Features

- Web Scraping: Extract content from single URLs with multiple output formats
- Website Crawling: Crawl entire websites with configurable depth and limits
- Search Engine Integration: Search the web and optionally scrape results
- Multiple Engines: Auto mode (intelligent selection), Playwright, Cheerio, and Puppeteer
- Flexible Output: Markdown, HTML, text, screenshots, and structured JSON
- Async Operations: Non-blocking crawl jobs with status monitoring
- Error Handling: Robust error handling and logging
- Multiple Modes: STDIO (default), MCP(HTTP), SSE; cloud-ready with Nginx proxy

Installation

Running with npx

ANYCRAWL_API_KEY=YOUR-API-KEY npx -y anycrawl-mcp

Manual installation

npm install -g anycrawl-mcp-server

ANYCRAWL_API_KEY=YOUR-API-KEY anycrawl-mcp

Configuration

AnyCrawl MCP Server supports two deployment modes: Cloud Service (recommended) and Self-Hosted.

Cloud Service (Recommended)

Use the AnyCrawl cloud service at mcp.anycrawl.dev. No server setup required.

```bash

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

About

Details

Jump to

Setting up with Highlight

From the repository

anycrawl_scrape

anycrawl_crawl

anycrawl_search

anycrawl_crawl_status

anycrawl_crawl_results

anycrawl_cancel_crawl

Claude Desktop / Cursor

McpServers

AnyCrawl MCP Server

Features

Installation

Running with npx

Manual installation

Configuration

Cloud Service (Recommended)

Install

More like this

Sentry

Github

Git

GitLab

Context 7

1Panel

`anycrawl_scrape`

`anycrawl_crawl`

`anycrawl_search`

`anycrawl_crawl_status`

`anycrawl_crawl_results`

`anycrawl_cancel_crawl`