Crawlab

by crawlab-team

2 stars

Not rated

GitHub

About

Integrates with Crawlab to enable AI-driven web scraping, task automation, and data extraction workflows through spider management, file operations, and resource access tools.

Details

Author: crawlab-team
Repository: crawlab-team/crawlab-mcp
GitHub stars: 2
Categories: Productivity, Developer Tools, Workplace, File Management, AI, Communication, API
Tags: #web, #integration

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below
Plugin Name Crawlab

Command (node, npx, python, etc.) npx
Arguments
- Argument 1 -y
- Argument 2 @highlight/mcp-server
Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "/c",
                "npx",
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "cmd"
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "/c",
                "npx",
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "cmd"
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "/c",
                "npx",
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "cmd"
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "/c",
                "npx",
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "cmd"
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "/c",
                "npx",
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "cmd"
        }
    }
}

JSON

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

From the repository

1. Copy the .env.example file to .env:

   cp .env.example .env

2. Edit the .env file with your Crawlab API details:

   CRAWLAB_API_BASE_URL=http://your-crawlab-instance:8080/api

   CRAWLAB_API_TOKEN=your_api_token_here

You can install the MCP server as a Python package, which provides a convenient CLI:

pip install -e .

After installation, you can use the CLI:


1. Install dependencies:

pip install -r requirements.txt


2. Run the server:

python server.py


1. Build the Docker image:

docker build -t crawlab-mcp-server .


2. Run the container:

docker run -p 8000:8000 --env-file .env crawlab-mcp-server ```

`get_spider`

Get details of a specific spider. Parameters: spider_id (string)

`create_spider`

Create a new spider. Parameters: spider_name (string), specifications (object)

`update_spider`

Update an existing spider. Parameters: spider_id (string), updates (object)

`delete_spider`

Delete a spider. Parameters: spider_id (string)

`get_task`

Get details of a specific task. Parameters: task_id (string)

`run_spider`

Run a spider. Parameters: spider_id (string), nodes (optional array of strings)

`cancel_task`

Cancel a running task. Parameters: task_id (string)

`restart_task`

Restart a task. Parameters: task_id (string)

`get_task_logs`

Get logs for a task. Parameters: task_id (string)

`get_spider_files`

List files for a spider. Parameters: spider_id (string)

`get_spider_file`

Get content of a specific file. Parameters: spider_id (string), file_name (string)

`save_spider_file`

Save content to a file. Parameters: spider_id (string), file_name (string), content (string)

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "crawlab": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

Linux

{
    "env": [],
    "args": [
        "-y",
        "@highlight/mcp-server"
    ],
    "command": "npx"
}

Macos

{
    "env": [],
    "args": [
        "-y",
        "@highlight/mcp-server"
    ],
    "command": "npx"
}

Windows

{
    "env": [],
    "args": [
        "/c",
        "npx",
        "-y",
        "@highlight/mcp-server"
    ],
    "command": "cmd"
}

pip install git+https://github.com/crawlab-team/crawlab-mcp-server.git

After installation, you can use the CLI:

bash

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

About

Details

Jump to

Setting up with Highlight

From the repository

get_spider

create_spider

update_spider

delete_spider

get_task

run_spider

cancel_task

restart_task

get_task_logs

get_spider_files

get_spider_file

save_spider_file

Claude Desktop / Cursor

Linux

Macos

Windows

pip install git+https://github.com/crawlab-team/crawlab-mcp-server.git

Install

More like this

Time

Sentry

Github

Git

GitLab

Context 7

`get_spider`

`create_spider`

`update_spider`

`delete_spider`

`get_task`

`run_spider`

`cancel_task`

`restart_task`

`get_task_logs`

`get_spider_files`

`get_spider_file`

`save_spider_file`