Markdown Web Crawl

by jmh108

3 stars

Not rated

GitHub

About

Python-based web crawler extracts website content into markdown files, enabling efficient content aggregation and site archiving.

Details

Author: jmh108
Repository: jmh108/md-webcrawl-mcp
GitHub stars: 3
License: MIT License
Categories: Productivity, AI, Design, File Management, Project Management, Frontend, Knowledge Base
Tags: #web

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below
Plugin Name Markdown Web Crawl

Command (node, npx, python, etc.) fastmcp
Arguments
- Argument 1 run
- Argument 2 /Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py
Environment
- OUTPUT_PATH /Users/user/Webcrawl
Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

JSON

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

From the repository

The server can be configured through environment variables:

- OUTPUT_PATH: Default output directory for saved files
- MAX_CONCURRENT_REQUESTS: Maximum parallel requests (default: 5)
- REQUEST_TIMEOUT: Request timeout in seconds (default: 30)

`extract_content`

Extract content from a specified URL and save it to a markdown file. Parameters: url (string), output_path (string)

`scan_linked_content`

Scan the linked content of a specified URL. Parameters: url (string)

`create_index`

Create an index from the scanned content map and save it to a markdown file. Parameters: content_map (input stream), output_path (string)

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "markdown web crawl": {
            "env": {
                "OUTPUT_PATH": "/Users/user/Webcrawl"
            },
            "args": [
                "run",
                "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
            ],
            "command": "fastmcp"
        }
    }
}

Linux

{
    "env": {
        "OUTPUT_PATH": "/Users/user/Webcrawl"
    },
    "args": [
        "run",
        "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
    ],
    "command": "fastmcp"
}

Macos

{
    "env": {
        "OUTPUT_PATH": "/Users/user/Webcrawl"
    },
    "args": [
        "run",
        "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
    ],
    "command": "fastmcp"
}

Windows

{
    "env": {
        "OUTPUT_PATH": "/Users/user/Webcrawl"
    },
    "args": [
        "run",
        "/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
    ],
    "command": "fastmcp"
}

MD MCP Webcrawler Project

A Python-based MCP (https://modelcontextprotocol.io/introduction) web crawler for extracting and saving website content.

Features

- Extract website content and save as markdown files - Map website structure and links - Batch processing of multiple URLs - Configurable output directory

Installation

1. Clone the repository:

git clone https://github.com/yourusername/webcrawler.git

cd webcrawler

2. Install dependencies:

pip install -r requirements.txt

3. Optional: Configure environment variables:

export OUTPUT_PATH=./output  # Set your preferred output directory

Output

Crawled content is saved in markdown format in the specified output directory.

Configuration

The server can be configured through environment variables:

- OUTPUT_PATH: Default output directory for saved files
- MAX_CONCURRENT_REQUESTS: Maximum parallel requests (default: 5)
- REQUEST_TIMEOUT: Request timeout in seconds (default: 30)

Claude Set-Up

Install with FastMCP ``

 fastmcp install server.py 
or user custom settings to run with fastmcp directly

"Crawl Server": {
"command": "fastmcp",
"args": [
"run",
"/Users/mm22/Dev_Projekte/servers-main/src/Webcrawler/server.py"
],
"env": {
"OUTPUT_PATH": "/Users/user/Webcrawl"
}
```


Development
Live Development
fastmcp dev server.py --with-editable .

Debug 
It helps to use https://modelcontextprotocol.io/docs/tools/inspector for debugging
Examples
Example 1: Extract and Save Content
mcp call extract_content --url "https://example.com" --output_path "example.md"

Example 2: Create Content Index
mcp call scan_linked_content --url "https://example.com" | \
  mcp call create_index --content_map - --output_path "index.md"

Contributing
1. Fork the repository
2. Create a feature branch (

git checkout -b feature/AmazingFeature

)
3. Commit your changes (

git commit -m 'Add some AmazingFeature'

)
4. Push to the branch (

git push origin feature/AmazingFeature

)
5. Open a Pull Request
License

Distributed under the MIT License. See LICENSE` for more information.

Requirements

- Python 3.7+
- FastMCP (uv pip install fastmcp)
- Dependencies listed in requirements.txt

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

About

Details

Jump to

Setting up with Highlight

From the repository

extract_content

scan_linked_content

create_index

Claude Desktop / Cursor

Linux

Macos

Windows

MD MCP Webcrawler Project

Features

Installation

Output

Configuration

Claude Set-Up

Development

Live Development

Debug

Examples

Example 1: Extract and Save Content

Example 2: Create Content Index

Contributing

License

Requirements

Install

More like this

Time

Context 7

1Panel

🔎 GPT Researcher

Serena

FastMCP v2 🚀

`extract_content`

`scan_linked_content`

`create_index`