Doc Scraper (Jina.ai)

by askjohngeorge

6 stars

Not rated

GitHub

About

Converts web documentation to clean markdown using Jina.ai's API, enabling easy transformation of online docs for content migration or offline use.

Details

Author: askjohngeorge
Repository: askjohngeorge/mcp-doc-scraper
GitHub stars: 6
Categories: Developer Tools, Design, Workplace, File Management, AI, Knowledge Base, Infrastructure, API
Tags: #web

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

Download and install Highlight from highlightai.com/download
Navigate to the plugins tab and select "Add Custom Plugin"
Configure the plugin with the settings below
Plugin Name Doc Scraper (Jina.ai)

Command (node, npx, python, etc.) npx
Arguments
- Argument 1 -y
- Argument 2 @highlight/mcp-server
Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.
Enable "Start Automatically" if you want the plugin to start when Highlight launches

Setting up with Cursor

Open Cursor's settings
Navigate to the MCP section
Click "Add new Global MCP server"
Copy and paste the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "/c",
                "npx",
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "cmd"
        }
    }
}

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

Setting up with Claude Desktop

Open Claude Desktop's settings
Go to the Integrations tab
Click "Add MCP Server"
Use the configuration below based on your operating system

Note: Make sure Claude Desktop is running before attempting to connect.

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "/c",
                "npx",
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "cmd"
        }
    }
}

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

Setting up with Windsurf

Open Windsurf's preferences
Go to the Integrations section
Click "Add MCP Integration"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "/c",
                "npx",
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "cmd"
        }
    }
}

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

Setting up with Cline

Open Cline's settings
Navigate to the Integrations section
Click "Add MCP Server"
Copy and paste the configuration below based on your operating system

Note: Make sure to restart Cline after adding the configuration.

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "/c",
                "npx",
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "cmd"
        }
    }
}

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

Setting up with Goose

Open Goose's preferences
Go to the Integrations tab
Select "MCP Servers"
Click "Add New Server"
Use the configuration below based on your operating system

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "/c",
                "npx",
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "cmd"
        }
    }
}

JSON

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

From the repository

The server can be run using Python:

python -m mcp_doc_scraper

To install Doc Scraper for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @askjohngeorge/mcp-doc-scraper --client claude

1. Clone the repository:

git clone https://github.com/askjohngeorge/mcp-doc-scraper.git
cd mcp-doc-scraper

2. Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows, use: venv\Scripts\activate

3. Install the dependencies:

pip install -e .

`scrape_docs`

Scrape documentation from a URL and save as markdown. Input Parameters: url (string) - The URL of the documentation to scrape; output_path (string) - The path where the markdown file should be saved.

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "doc scraper (jina.ai)": {
            "env": {},
            "args": [
                "-y",
                "@highlight/mcp-server"
            ],
            "command": "npx"
        }
    }
}

Linux

{
    "env": [],
    "args": [
        "-y",
        "@highlight/mcp-server"
    ],
    "command": "npx"
}

Macos

{
    "env": [],
    "args": [
        "-y",
        "@highlight/mcp-server"
    ],
    "command": "npx"
}

Windows

{
    "env": [],
    "args": [
        "/c",
        "npx",
        "-y",
        "@highlight/mcp-server"
    ],
    "command": "cmd"
}

Doc Scraper MCP Server

A Model Context Protocol (MCP) server that provides documentation scraping functionality. This server converts web-based documentation into markdown format using jina.ai's conversion service.

Features

Installation

Installing via Smithery

To install Doc Scraper for Claude Desktop automatically via Smithery:

npx -y @smithery/cli install @askjohngeorge/mcp-doc-scraper --client claude

1. Clone the repository:

git clone https://github.com/askjohngeorge/mcp-doc-scraper.git
cd mcp-doc-scraper

2. Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows, use: venv\Scripts\activate

3. Install the dependencies:

pip install -e .

Usage

The server can be run using Python:

python -m mcp_doc_scraper

Tool Description

The server provides a single tool:

- Name: scrape_docs
- Description: Scrape documentation from a URL and save as markdown
- Input Parameters:
- url: The URL of the documentation to scrape
- output_path: The path where the markdown file should be saved

Project Structure

doc_scraper/
├── __init__.py
├── __main__.py
└── server.py

Dependencies

- aiohttp
- mcp
- pydantic

Development

To set up the development environment:

1. Install development dependencies:

pip install -r requirements.txt

2. The server uses the Model Context Protocol. Make sure to familiarize yourself with MCP documentation.

License

MIT License

No reviews yet — be the first

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.

About

Details

Jump to

Setting up with Highlight

From the repository

scrape_docs

Claude Desktop / Cursor

Linux

Macos

Windows

Doc Scraper MCP Server

Features

Installation

Installing via Smithery

Usage

Tool Description

Project Structure

Dependencies

Development

License

Install

More like this

Sentry

Github

Git

GitLab

Context 7

Playwright

`scrape_docs`