Crawler For Llm
About
高等级的免费爬虫MCP, 完全 Markdown 化支持，支持微信公众号等一大批国内网站。A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format. This tool is essential for building knowledge bases, training
Details
- Author
- KonghaYao
- Downloads
- 336
- Categories
- Automation
Jump to
- Universal web crawler with MCP integration
- Extracts clean content and converts to Markdown
- Supports documentation, development, and Chinese platforms
- Continuously expanding list of supported websites
- Built for LLM knowledge bases and training data
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
Crawler For LlmCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Add the provided MCP server configuration to your project's settings. Use the command npx -y @langgraph-js/crawler-mcp@latest with the server name langgraph-crawler.
crawl_tool
A powerful web content extraction tool that retrieves and processes raw content from specified URLs, ideal for data collection, content analysis, and research tasks.
web_search_tool
A powerful web search tool that provides comprehensive, real-time results using search engine. Returns relevant web content with customizable parameters for result count, content type, and domain filtering. Ideal for gathering current information, news, and detailed web content analysis.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"crawler for llm": {
"langgraph-crawler": {
"command": "npx",
"args": [
"-y",
"@langgraph-js/crawler-mcp@latest"
]
}
}
}
}McpServers
{
"langgraph-crawler": {
"command": "npx",
"args": [
"-y",
"@langgraph-js/crawler-mcp@latest"
]
}
}Crawler for LLM with MCP Support
> A universal web crawler for LLM frameworks with Model Context Protocol (MCP) integration
A powerful web crawler designed specifically for LLM applications, capable of extracting clean, readable content from various web pages and converting it to Markdown format. This tool is essential for building knowledge bases, training data collection, and content aggregation for LLM applications. It includes built-in support for the Model Context Protocol (MCP) for enhanced context management.
{
"mcpServers": {
"langgraph-crawler": {
"command": "npx",
"args": ["-y", "@langgraph-js/crawler-mcp@latest"]
}
}
}
Supported Websites
> Note: This list is continuously expanding. Feel free to contribute by adding support for more websites.
Documentation & Development
- [x] NPM
- [x] Vitepress Base Website
- [x] Github
- [x] DEV Community (dev.to)
- [x] MDN
- [x] Medium
- [x] SearXNG Search Result
- [ ] freeCodeCamp.org
- [ ] Docker Hub
- [ ] StackOverflow
- [ ] Anthropic
Chinese Platforms
- [x] WeChat Public Articles (微信公众号)
- [x] Juejin (稀土掘金)
- [x] The Paper (澎湃新闻)
- [x] Jiemian (界面新闻)
- [x] Huxiu (虎嗅网)
- [x] UISDC (优设网)
- [x] CNBlogs (博客园)
- [x] SSPAI (少数派)
- [x] InfoQ
- [x] CSDN
- [ ] Zhihu (知乎)
License
Apache-2.0
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
