Web Crawler MCP Server
About
A MCP server that provides a web crawling and content extraction tool for AI assistants
Details
- Author
- JonathanHsuHH
- Downloads
- 165
- Categories
- Automation
Jump to
- Extracts and cleans main text content from public web pages.
- Uses Puppeteer with stealth plugin to bypass anti-bot protections.
- Returns readable, whitespace-normalized text for LLM consumption.
- Easy integration with Claude Desktop, Cursor, and other MCP clients.
- Launches a real browser instance (headless: false) for best compatibility.
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
Web Crawler MCP ServerCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Install dependencies with npm install, build with npm run build, then run directly with node build/index.js. Alternatively, configure it as an MCP server in your client (e.g., Claude Desktop) by adding a JSON entry that points command and args to the built server.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"web crawler mcp server": {
"web-crawler-mcp-server": {
"command": "node",
"args": [
"build/index.js"
]
}
}
}
}McpServers
{
"web-crawler-mcp-server": {
"command": "node",
"args": [
"build/index.js"
]
}
}Web Crawler MCP Server
A Model Context Protocol (MCP) server that provides a web crawling and content extraction tool for AI assistants such as Claude Desktop, Cursor, and other MCP-compatible clients.
Features
- Extracts and cleans main text content from any public web page.
- Uses Puppeteer with stealth plugin to bypass anti-bot protections.
- Returns readable, whitespace-normalized text for LLM consumption.
- Easy integration with Claude Desktop and other MCP clients.
Prerequisites
- Node.js (v16 or higher)
- MCP-compatible client (e.g., Claude Desktop, Cursor)
- (Optional) Puppeteer dependencies for some Linux environments
Installation
1. Install dependencies:
npm install
2. Build the server:
npm run build
Usage
You can run the server directly:
node build/index.js
Or configure it as an MCP server in your client (e.g., Claude Desktop):
{
"mcpServers": {
"web-crawler-mcp": {
"command": "node",
"args": ["<absolute-path-to>/server/web_crawler/build/index.js"]
}
}
}
Available Tool
web-crawler
- Description: Extracts and returns the cleaned text content from a specified URL.
- Input:
- url (string, required): The URL to extract content from.
Example
{
"tool_name": "web-crawler",
"arguments": {
"url": "https://openai.com/news"
}
}
Development
- npm run build — Compile TypeScript to JavaScript.
- npm run watch — Watch and rebuild on changes.
- npm run inspector — Launch MCP Inspector for debugging.
Notes
- The server launches a real browser instance (headless: false) for best compatibility.
- Output is plain text, suitable for LLM input.
- For advanced parsing, modify the Cheerio logic in src/index.ts.
License
MIT
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
