Crawl4AI Web Scraper MCP Server

by MaitreyaM

27 stars

148 downloads

Not rated

GitHub Website

About

MCP Server leveraging crawl4ai for web scraping and LLM-based content extraction (Markdown, text snippets, smart extraction). Designed for AI agent integration.

Details

Author: MaitreyaM
GitHub stars: 27
Downloads: 148
Categories: Automation, Web Scraping

- Exposes three MCP tools: scrape_url, extract_text_by_query, and smart_extract
- scrape_url returns full webpage content as Markdown
- extract_text_by_query finds up to 5 text snippet matches with optional context size
- smart_extract uses Google Gemini LLM for structured extraction from natural language instructions
- Configurable via environment variables (API keys)
- Includes Docker configuration for containerized deployment
- Communicates over Server-Sent Events (SSE) on port 8002

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below

Plugin Name Crawl4AI Web Scraper MCP Server

Command (node, npx, python, etc.)

Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

From the repository

Run the server either with Docker (recommended) or locally. For Docker: install Docker, clone the repository, create a .env file with GOOGLE_API_KEY, build the image, and run the container mapping port 8002 with --env-file .env. For local use: ensure Python ≥3.9, clone the repo, create a virtual environment, install dependencies from requirements.txt, create a .env file with API keys, and run the main Python script. The server listens on SSE at port 8002. Configure your MCP client to connect to http://127.0.0.1:8002/sse with transport "sse".

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "crawl4ai web scraper mcp server": {
            "WEB-SCRAPING-MCP": {
                "command": "docker",
                "args": [
                    "build",
                    "-t",
                    "crawl4ai-mcp-server",
                    "."
                ]
            }
        }
    }
}

McpServers

{
    "WEB-SCRAPING-MCP": {
        "command": "docker",
        "args": [
            "build",
            "-t",
            "crawl4ai-mcp-server",
            "."
        ]
    }
}

Crawl4AI Web Scraper MCP Server

This project provides an MCP (Model Context Protocol) server that uses the crawl4ai library to perform web scraping and intelligent content extraction tasks. It allows AI agents (like Claude, or agents built with LangChain/LangGraph) to interact with web pages, retrieve content, search for specific text, and perform LLM-based extraction based on natural language instructions.

This server uses:

FastMCP: For creating the MCP server endpoint.
crawl4ai: For the core web crawling and extraction logic.
dotenv: For managing API keys via a .env file.
(Optional) Docker: For containerized deployment, bundling Python and dependencies.

Features

Exposes MCP tools for web interaction:
scrape_url: Get the full content of a webpage in Markdown format.
extract_text_by_query: Find specific text snippets on a page based on a query.
smart_extract: Use an LLM (currently Google Gemini) to extract structured information based on instructions.
Configurable via environment variables (API keys).
Includes Docker configuration (Dockerfile) for easy, self-contained deployment.
Communicates over Server-Sent Events (SSE) on port 8002 by default.

Exposed MCP Tools

scrape_url

Scrape a webpage and return its content in Markdown format.

Arguments:

url (str, required): The URL of the webpage to scrape.

Returns:

(str): The webpage content in Markdown format, or an error message.

extract_text_by_query

Extract relevant text snippets from a webpage that contain a specific search query. Returns up to the first 5 matches found.

Arguments:

url (str, required): The URL of the webpage to search within.
query (str, required): The text query to search for (case-insensitive).
context_size (int, optional): The number of characters to include before and after the matched query text in each snippet. Defaults to 300.

Returns:

(str): A formatted string containing the found text snippets or a message indicating no matches were found, or an error message.

smart_extract

Intelligently extract specific information from a webpage using the configured LLM (currently requires Google Gemini API key) based on a natural language instruction.

Arguments:

url (str, required): The URL of the webpage to analyze and extract from.
instruction (str, required): Natural language instruction specifying what information to extract (e.g., "List all the speakers mentioned on this page", "Extract the main contact email address", "Summarize the key findings").

Returns:

(str): The extracted information (often formatted as JSON or structured text based on the instruction) or a message indicating no relevant information was found, or an error message (including if the required API key is missing).

Setup and Running

You can run this server either locally or using the provided Docker configuration.

Option 1: Running with Docker (Recommended for Deployment)

This method bundles Python and all necessary libraries. You only need Docker installed on the host machine.

1. Install Docker: Download and install Docker Desktop for your OS. Start Docker Desktop.
2. Clone Repository:

    git clone https://github.com/your-username/your-repo-name.git # Replace with your repo URL

    cd your-repo-name

3. Create .env File: Create a file named .env in the project root directory and add your API keys:
``

.env

    # Required for the smart_extract tool

    GOOGLE_API_KEY=your_google_ai_api_key_here
# Optional, checked by server but not currently used by tools

    # OPENAI_API_KEY=your_openai_key_here

    # MISTRAL_API_KEY=your_mistral_key_here

    
4.  Build the Docker Image:

    
bash

    docker build -t crawl4ai-mcp-server .

    5.  Run the Container: This starts the server, making port 8002 available on your host machine. It uses --env-file to securely pass the API keys from your local .env file into the container's environment.

    
bash

    docker run -it --rm -p 8002:8002 --env-file .env crawl4ai-mcp-server

           -it: Runs interactively.

       --rm: Removes container on exit.

       -p 8002:8002: Maps host port 8002 to container port 8002.

       --env-file .env: Loads environment variables from your local .env file into the container. Crucial for API keys.

       crawl4ai-mcp-server: The name of the image you built.

6.  Server is Running: Logs will appear, indicating the server is listening on SSE (http://0.0.0.0:8002).

7.  Connecting Client: Configure your MCP client (e.g., LangChain agent) to connect to http://127.0.0.1:8002/sse with transport: "sse".
Option 2: Running Locally
This requires Python and manual installation of dependencies on your host machine.
1.  Install Python: Ensure Python >= 3.9 (check crawl4ai requirements if needed, 3.10+ recommended).

2.  Clone Repository:

    
bash

    git clone https://github.com/your-username/your-repo-name.git # Replace with your repo URL

    cd your-repo-name

    3.  Create Virtual Environment (Recommended):

    
bash

    python -m venv venv

    source venv/bin/activate # Linux/macOS

    # venv\Scripts\activate # Windows

        (Or use Conda: conda create --name crawl4ai-env python=3.11 -y && conda activate crawl4ai-env)

4.  Install Dependencies:

    
bash

    pip install -r requirements.txt

    5.  Create .env File: Create a file named .env in the project root directory and add your API keys (same content as in Docker setup step 3).

6.  Run the Server:

    
bash

    python your_server_script_name.py # e.g., python webcrawl_mcp_server.py

7.  Server is Running: It will listen on http://127.0.0.1:8002/sse.

8.  Connecting Client: Configure your MCP client to connect to http://127.0.0.1:8002/sse.
Environment Variables
The server uses the following environment variables, typically loaded from an .env file:

   GOOGLE_API_KEY: Required for the smart_extract tool to function (uses Google Gemini). Get one from Google AI Studio.

   OPENAI_API_KEY: Checked for existence but not currently used by any tool in this version.

   MISTRAL_API_KEY`: Checked for existence but not currently used by any tool in this version.
Example Agent Interaction

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

Crawl4AI Web Scraper MCP Server

About

Details

Jump to

Setting up with Highlight

Setting up with Cursor

JSON

JSON

JSON

Setting up with Claude Desktop

JSON

JSON

JSON

Setting up with Windsurf

JSON

JSON

JSON

Setting up with Cline

JSON

JSON

JSON

Setting up with Goose

JSON

JSON

JSON

From the repository

Claude Desktop / Cursor

McpServers

Crawl4AI Web Scraper MCP Server

Features

Exposed MCP Tools

`scrape_url`

`extract_text_by_query`

`smart_extract`

Setup and Running

Option 1: Running with Docker (Recommended for Deployment)

Option 2: Running Locally

Environment Variables

Example Agent Interaction

Install

About

Details

Jump to

Setting up with Highlight

From the repository

Claude Desktop / Cursor

McpServers

Crawl4AI Web Scraper MCP Server

Features

Exposed MCP Tools

scrape_url

extract_text_by_query

smart_extract

Setup and Running

Option 1: Running with Docker (Recommended for Deployment)

Option 2: Running Locally

Environment Variables

Example Agent Interaction

Install

More like this

@mendableai/firecrawl Mcp Server

Umut Naci

Playwright

Fetch

Puppeteer

Playwright

`scrape_url`

`extract_text_by_query`

`smart_extract`