MCP Smart Crawler
About
MCP Smart Crawler is a Model Context Protocol (MCP) server uses Playwright to crawl web content, extract metadata, and download resources such as videos and images.
Details
- Author
- loo-y
- GitHub stars
- 4
- Downloads
- 219
- Categories
- Automation
Jump to
- Extracts metadata (title, description, images) from Xiaohongshu posts.
- Downloads videos and images from Xiaohongshu share links.
- Uses Playwright for browser automation.
- Configurable download folder via command-line argument.
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
MCP Smart CrawlerCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Configure your MCP client with the provided JSON, setting the command to npx and adding optional arguments like --download-folder to specify a local download directory. Then send a chat message containing a Xiaohongshu share link to trigger crawling and downloading.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"mcp smart crawler": {
"mcp-smart-crawler": {
"command": "npx",
"args": [
"-y",
"mcp-smart-crawler@latest",
"--download-folder",
"C:\\downloads"
]
}
}
}
}McpServers
{
"mcp-smart-crawler": {
"command": "npx",
"args": [
"-y",
"mcp-smart-crawler@latest",
"--download-folder",
"C:\\downloads"
]
}
}MCP Smart Crawler
MCP Smart Crawler is a Model Context Protocol (MCP) server uses Playwright to crawl web content, extract metadata, and download resources such as videos and images.Key Features
Extract metadata (title, description, images) from Xiaohongshu (小红书) posts. Download videos and images from Xiaohongshu share links.- Uses Playwright for browser automation.
How to Use (MCP Client Configuration)
To use this server with your MCP client, add the following configuration to your client's MCP server settings. You will need to adjust the command and args based on how you run the server script.
Example Configuration:
{
"mcpServers": {
"mcp-smart-crawler": {
"command": "npx",
"args": [
"-y",
"mcp-smart-crawler",
"--download-folder", // optional
"c:\\downloads" // optional
],
}
}
}
Example chat:
帮我查看这条小红书的内容和图片，并告诉我图片里面是什么
59 XXX发布了一篇小红书笔记，快来看吧！ 😆 OfTOBst2PsxctaX 😆 http://xhslink.com/a/xxaabbcczz，复制本条信息，打开【小红书】App查看精彩内容！
<br />
<hr />
<br />
<span id="mcp-smart-crawler-chinese"></span>
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
