mcp-server-spider: A spider MCP server
Description
# mcp-server-spider: A spider MCP server ## Overview A Model Context Protocol server for Spider crawler interaction and automation. This server provides tools to crawl and scrape web pages. Please note that mcp-server-spider is currently in early develpoment. There might be bugs…
About
# mcp-server-spider: A spider MCP server ## Overview A Model Context Protocol server for Spider crawler interaction and automation. This server provides tools to crawl and scrape web pages. Please note that mcp-server-spider is currently in early develpoment. There might be bugs and features added in the future. ###…
Details
- Author
- GeorgeLS
- GitHub stars
- 1
- Downloads
- 249
- Categories
- Other
Jump to
- Crawl a URL and return discovered links
- Scrape a URL and return structured JSON with content
- Configurable crawl depth and user agent
- Support for additional HTTP headers
- Whitelist/blacklist URLs via regular expressions
- Option to respect or bypass robots.txt
- Option to accept invalid TLS certificates
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
mcp-server-spider: A spider MCP serverCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Install and run using uvx (recommended) or install via pip (pip install mcp-server-spider) and run with python -m mcp_server_spider. The server exposes two tools: crawl and scrape, each accepting parameters such as url, headers, user_agent, depth, blacklist, whitelist, respect_robots_txt, and accept_invalid_certs.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"mcp-server-spider: a spider mcp server": {
"mcp-server-spider": {
"command": "uvx",
"args": [
"mcp-server-spider"
]
}
}
}
}McpServers
{
"mcp-server-spider": {
"command": "uvx",
"args": [
"mcp-server-spider"
]
}
}mcp-server-spider: A spider MCP server
Overview
A Model Context Protocol server for Spider crawler interaction and automation. This server provides tools to crawl and scrape web pages.Please note that mcp-server-spider is currently in early develpoment. There might be bugs and features added in the future.
Tools
1. crawl
- Crawls the given url and returns the list of URLs that were found
- Input:
- url: The url to crawl
- headers: Additional headers passed along with crawl requests
- user_agent: User agent to use for the crawl requests
- depth: The depth of link traversal
- blacklist: A list of regural expression to blacklist URLs from the crawling process
- whitelist: A list of regular expression to whitelist URLS from the crawling process
- respect_robots_txt: Whether to respect robots.txt file
- accept_invalid_certs: Whether to accept invalid certifcates or not
- Returns: List of URLs found
2. scrape
- Scrapes the given url and returns a list of JSON objects that contain the url, links and content of each page discovered
- Input: Same as crawl
- Returns: A list of JSON objects (as a string) that contain the url, links and content of each page discovered
Installation
Using uv (recommended)
When using uv no specific installation is needed. We will
use uvx to directly run mcp-server-spider.
Using PIP
Alternatively you can install mcp-server-spider via pip:
pip install mcp-server-spider
After installation, you can run it as a script using:
python -m mcp_server_spider
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
