Mcp_web_scrapper
About
A web scrapper to scrap mcp web servers from glama.ai
Details
- Author
- Neche-Stephen
- Downloads
- 131
- Categories
- Other
Jump to
- Scrapes MCP server links from glama.ai
- Extracts detailed server information from each link
- Uses a headless browser for data collection
- Saves output as JSON files in a dedicated folder
- Provides terminal progress during processing
Setting up with Highlight
This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:
- Download and install Highlight from highlightai.com/download
- Navigate to the plugins tab and select "Add Custom Plugin"
-
Configure the plugin with the settings below
Plugin Name
Mcp_web_scrapperCommand (node, npx, python, etc.)Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
- Enable "Start Automatically" if you want the plugin to start when Highlight launches
From the repository
Clone the repository, run npm install, then execute node scraper.js to collect server links, followed by node process-links.js to extract detailed data. Delete the servers-json folder if a fresh scrape is needed.
Claude Desktop / Cursor
Paste into your MCP client config file to install this server.
{
"mcpServers": {
"mcp_web_scrapper": {
"mcp_web_scrapper": {
"command": "node",
"args": [
"scraper.js"
]
}
}
}
}McpServers
{
"mcp_web_scrapper": {
"command": "node",
"args": [
"scraper.js"
]
}
}Getting Started with MCP Server Scraper
This project allows you to scrape and extract MCP web servers from glama.ai. Follow the steps below to set up and run the scraper efficiently.
📦 Prerequisites
- Node.js
- npm
🔧 Installation
1. Clone the repository:
git clone https://github.com/Neche-Stephen/mcp_web_scrapper
cd mcp_web_scrapper
2. Install dependencies:
npm install
---
🗃 Check for Existing Data
Before running a fresh scrape:
1. Check if the servers-json folder exists.
2. If it does and contains files, it means data has been previously scraped.
3. To run a fresh batch, delete the servers-json folder:
rm -rf servers-json
---
Step 1: Scrape MCP Server Links
Run the scraper to collect all MCP server links from the website:
node scraper.js
- The scraper will launch a headless browser and start collecting data.
- Be patient — this might take a few minutes.
- Once you see the message "Closing browser", the scraping is complete.
- You can then safely terminate the process using CTRL + C.
> Output: A file named server-links.json will be generated in a folder servers-json containing all scraped links.
---
Step 2: Process and Extract Server Data
Now, fetch and process detailed information from each server link:
node process-links.js
- The script reads from server-links.json and visits each link to scrape the necessary data.
- A progress report will be displayed in the terminal as the script runs.
- When you see the success message indicating completion, press CTRL + C to terminate the process.
> All scraped data will be stored in the servers-json directory in different files named after the mcp servers
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
