Crawl4AI MCP Server

A comprehensive Model Context Protocol (MCP) server that wraps the powerful crawl4ai library with advanced AI capabilities. Extract and analyze content fromany source: web pages, PDFs, Office documents, YouTube videos, and more. Features intelligent summarization to dramatically reduce token usage while preserving key information.

- 🔍 Google Search Integration- 7 optimized search genres with Google official operators
- 🔍 Advanced Web Crawling: JavaScript support, deep site mapping, entity extraction
- 🌐 Universal Content Extraction: Web pages, PDFs, Word docs, Excel, PowerPoint, ZIP archives
- 🤖 AI-Powered Summarization: Smart token reduction (up to 88.5%) while preserving essential information
- 💾 Disk Persistence (token-saver): Save full results to disk and return slim metadata, so agents read them from a file on demand without spending context
- 🎬 YouTube Integration: Extract video transcripts and summaries without API keys
- ⚡ Production Ready: 19 specialized tools with comprehensive error handling

- Python 3.11 or later (FastMCP requires Python 3.11+)

Install system dependencies for Playwright:

# Manual setup required due to t64 library transition sudo apt update && sudo apt install -y \ libnss3 libatk-bridge2.0-0 libxss1 libasound2t64 \ libgbm1 libgtk-3-0t64 libxshmfence-dev libxrandr2 \ libxcomposite1 libxcursor1 libxdamage1 libxi6 \ fonts-noto-color-emoji fonts-unifont python3-venv python3-pip python3 -m venv venv && source venv/bin/activate pip install playwright==1.55.0 && playwright install chromium sudo playwright install-deps

sudo bash scripts/prepare_for_uvx_playwright.sh

# After system preparation above - that's it! uvx --from git+https://github.com/walksoda/crawl-mcp crawl-mcp

# Clone the repository git clone https://github.com/walksoda/crawl-mcp cd crawl-mcp # Build and run with Docker Compose (STDIO mode) docker-compose up --build # Or build and run HTTP mode on port 8000 docker-compose --profile http up --build crawl4ai-mcp-http # Or build manually docker build -t crawl4ai-mcp . docker run -it crawl4ai-mcp

- 🔧Multi-Browser Support: Chromium, Firefox, Webkit headless browsers
- 🐧Google Chrome: Additional Chrome Stable for compatibility
- ⚡Optimized Performance: Pre-configured browser flags for Docker
- 🔒Security: Non-root user execution
- 📦Complete Dependencies: All required libraries included

UVX Installation:Add to yourclaude_desktop_config.json:

- crawl_url- Extract web page content with JavaScript support
- deep_crawl_site- Crawl multiple pages from a site with configurable depth
- crawl_url_with_fallback- Crawl with fallback strategies for anti-bot sites

- intelligent_extract- Extract specific data from web pages using LLM
- extract_entities- Extract entities (emails, phones, etc.) from web pages
- extract_structured_data- Extract structured data using CSS selectors or LLM

- extract_youtube_transcript- Extract YouTube transcripts with timestamps
- batch_extract_youtube_transcripts- Extract transcripts from multiple YouTube videos (max 3)
- get_youtube_video_info- Get YouTube video metadata and transcript availability
- extract_youtube_comments- Extract YouTube video comments with pagination

- search_google- Search Google with genre filtering
- batch_search_google- Perform multiple Google searches (max 3)
- search_and_crawl- Search Google and crawl top results
- get_search_genres- Get available search genres

- process_file- Convert PDF, Word, Excel, PowerPoint, ZIP to markdown
- get_supported_file_formats- Get supported file formats and capabilities
- enhanced_process_large_content- Process large content with chunking and BM25 filtering

- batch_crawl- Crawl multiple URLs with fallback (max 3 URLs)
- multi_url_crawl- Multi-URL crawl with pattern-based config (max 5 URL patterns)

💾 Persist Large Results to Disk (token-saver)

All information-gathering tools accept an optionaloutput_pathparameter that writes the full fetched content straight to disk and returns a slim metadata-only response. This lets an LLM fetch huge pages, long YouTube transcripts, or whole batches without blowing its context budget — read from the saved file only when needed.

- Single-file tools (e.g.crawl_url,extract_youtube_transcript) write one.md(or.jsonfor JSON-kind tools) — pass an absolute file path; the extension is auto-added if omitted. An existing regular file at that path is rejected unlessoverwrite=true.
- Batch tools (batch_crawl,multi_url_crawl,deep_crawl_site,search_and_crawl,batch_extract_youtube_transcripts) expect an absolutedirectorypath and write one.mdper URL plusindex.json. Any non-existent path is treated as a directory and created — including names containing dots such as/tmp/run.v1. If the path already exists as a regular file, the call is rejected.batch_crawl/multi_url_crawlkeep theirlistreturn shape and embed anoutput_filekey on each success item.
- Request-dict tools (search_google,batch_search_google,search_and_crawl,batch_extract_youtube_transcripts) read the persistence keys directly from their request dict.
- Common parameters:output_path(absolute;Noneor""skips persistence),include_content_in_response(defaultfalse— whentrue, content is included in the response too,still subject to anycontent_limit/content_offset/max_content_per_pageslicing),overwrite(defaultfalse).
- Writes are atomic per file (temp file +os.replace); parent directories are auto-created; the full unsliced payload is persistedbeforeany slicing or tool-internal truncation so the on-disk copy is always complete even when the response is sliced.
- Batch dict tools (deep_crawl_site,search_and_crawl,batch_extract_youtube_transcripts) skip per-item persistence for items that reportsuccess=false; these still appear inindex.jsonwithfile: nullso callers can reason about the attempt list.

{ "tool": "crawl_url", "arguments": { "url": "https://example.com/long-article", "output_path": "/tmp/crawl_out/article.md" } }

JSON structured extraction (extension auto-added):

{ "tool": "extract_structured_data", "arguments": { "url": "https://example.com/products", "extraction_type": "css", "css_selectors": {"price": ".price", "name": "h1"}, "output_path": "/tmp/crawl_out/products" } }

Enable AI agents to get structured data from unstructured web with AgentQL.

Web scraping, crawling, and change detection with AI

Official Apify MCP server for AI agents to run Actors, extract website data, and automate web scraping and crawling workflows.

1GB Free Trial, World's Leading Proxy Service Platform, Efficient Data Collection

Discover, extract, and interact with the web - one interface powering automated access across the public internet.

Automate browser interactions in the cloud (e.g. web navigation, data extraction, form filling, and more)

Easy web data access. Simplified retrieval of information from websites and online sources.

Adds powerful web scraping and search capabilities to LLM clients like Cursor and Claude.