Crawlab MCP Server

by crawlab-team

7 stars

279 downloads

Not rated

GitHub

Other

Description

# Crawlab MCP Server This is a Model Context Protocol (MCP) server for Crawlab, allowing AI applications to interact with Crawlab's functionality. ## Overview The MCP server provides a standardized way for AI applications to access Crawlab's features, including: - Spider…

About

Details

Author: crawlab-team
GitHub stars: 7
Downloads: 279
Categories: Other

- MCP‑based communication between AI apps and Crawlab
- Full spider lifecycle: create, read, update, delete
- Task management: run, cancel, restart, view logs
- File operations: list, read, write spider files
- Supports multiple deployment options (local, Docker, pip)
- Configuration via environment variables (.env)

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below

Plugin Name Crawlab MCP Server

Command (node, npx, python, etc.)

Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

From the repository

Install as a Python package (pip install -e .) and run the server CLI (crawlab_mcp-mcp server), or run directly with python server.py after setting up a .env file with your Crawlab API base URL and token. A Docker image is also available. Connect an AI client to the server’s URL (e.g., http://localhost:8000) to start issuing natural language commands.

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "crawlab mcp server": {
            "crawlab-mcp-server": {
                "type": "http",
                "url": "http://localhost:8000"
            }
        }
    }
}

McpServers

{
    "crawlab-mcp-server": {
        "type": "http",
        "url": "http://localhost:8000"
    }
}

Crawlab MCP Server

This is a Model Context Protocol (MCP) server for Crawlab, allowing AI applications to interact with Crawlab's functionality.

Overview

The MCP server provides a standardized way for AI applications to access Crawlab's features, including:

- Spider management (create, read, update, delete)
- Task management (run, cancel, restart)
- File management (read, write)
- Resource access (spiders, tasks)

Architecture

The MCP Server/Client architecture facilitates communication between AI applications and Crawlab:

graph TB
    User[User] --> Client[MCP Client]
    Client --> LLM[LLM Provider]
    Client <--> Server[MCP Server]
    Server <--> Crawlab[Crawlab API]
subgraph "MCP System"

        Client

        Server

    end
subgraph "Crawlab System"

        Crawlab

        DB[(Database)]

        Crawlab <--> DB

    end
class User,LLM,Crawlab,DB external;

    class Client,Server internal;
%% Flow annotations

    LLM -.-> |Tool calls| Client

    Client -.-> |Executes tool calls| Server

    Server -.-> |API requests| Crawlab

    Crawlab -.-> |API responses| Server

    Server -.-> |Tool results| Client

    Client -.-> |Human-readable response| User
classDef external fill:#f9f9f9,stroke:#333,stroke-width:1px;

    classDef internal fill:#d9edf7,stroke:#31708f,stroke-width:1px;

Communication Flow

1. User Query: The user sends a natural language query to the MCP Client
2. LLM Processing: The Client forwards the query to an LLM provider (e.g., Claude, OpenAI)
3. Tool Selection: The LLM identifies necessary tools and generates tool calls
4. Tool Execution: The Client sends tool calls to the MCP Server
5. API Interaction: The Server executes the corresponding Crawlab API requests
6. Response Generation: Results flow back through the Server to the Client to the LLM
7. User Response: The Client delivers the final human-readable response to the user

Installation and Usage

Option 1: Install as a Python package

You can install the MCP server as a Python package, which provides a convenient CLI:

```bash

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

Description

About

Details

Jump to

Setting up with Highlight

From the repository

Claude Desktop / Cursor

McpServers

Crawlab MCP Server

Overview

Architecture

Communication Flow

Installation and Usage

Option 1: Install as a Python package

Install

More like this

Sequential Thinking

Filesystem

EverArt

Google Drive

Reactive Resume

FastMCP v2 🚀