mcp-playwright-scraper

A Model Context Protocol (MCP) server that scrapes web content and converts it to Markdown.

Overview

This MCP server provides a simple tool for scraping web content and converting it to Markdown format. It uses:

- Playwright: For headless browser automation to handle modern web pages including JavaScript-heavy sites
- BeautifulSoup: For HTML parsing and cleanup
- Pypandoc: For high-quality HTML to Markdown conversion

Tools

The server implements a single tool:

- scrape_to_markdown: Scrapes content from a URL and converts it to Markdown
- Required parameter: url (string) - The URL to scrape
- Optional parameter: verify_ssl (boolean) - Whether to verify SSL certificates (default: true)

Installation

Using uv (recommended)

When using uv no specific installation is needed. We will
use uvx to directly run mcp-playwright-scraper.

Using PIP

Alternatively you can install mcp-playwright-scraper via pip:

pip install mcp-playwright-scraper

After installation, you can run it as a script using:

python -m mcp_playwright_scraper

Prerequisites

- Python 3.11 or higher
- Playwright browser dependencies
- Pandoc (optional, will be automatically installed by pypandoc if possible)

After installation, you need to install Playwright browser dependencies:

playwright install --with-deps chromium

Configuration

Usage with Claude Desktop

Add this to your claude_desktop_config.json:

<details>
<summary>Using uvx</summary>

"mcpServers": {
  "mcp-playwright-scraper": {
    "command": "uvx",
    "args": ["mcp-playwright-scraper"]
  }
}

<details>
<summary>Using pip installation</summary>

"mcpServers": {
  "mcp-playwright-scraper": {
    "command": "python",
    "args": ["-m", "mcp_playwright_scraper"]
  }
}

Usage with Claude Code

# Basic syntax
$ claude mcp add mcp-playwright-scraper -- uvx mcp-playwright-scraper
Alternatively, with pip installation
$ claude mcp add mcp-playwright-scraper -- python -m mcp_playwright_scraper

<details>
<summary>Development/Unpublished Servers Configuration</summary>

"mcpServers": {
  "mcp-playwright-scraper": {
    "command": "uv",
    "args": [
      "--directory",
      "/path/to/mcp-playwright-scraper",
      "run",
      "mcp-playwright-scraper"
    ]
  }
}

Usage with Zed

Add to your Zed settings.json:

<details>
<summary>Using uvx</summary>

"context_servers": [
  "mcp-playwright-scraper": {
    "command": {
      "path": "uvx",
      "args": ["mcp-playwright-scraper"]
    }
  }
],

<details>
<summary>Using pip installation</summary>

"context_servers": {
  "mcp-playwright-scraper": {
    "command": "python",
    "args": ["-m", "mcp_playwright_scraper"]
  }
},

Usage with Cursor

1. Open Cursor Settings
- Navigate to Cursor Settings > Features > MCP
- Click the "+ Add New MCP Server" button
2. Configure the Server
- Name: mcp-playwright-scraper
- Type: Select stdio
- Command: Enter one of the following:

<details>
<summary>Using uvx</summary>

uvx mcp-playwright-scraper

<details>
<summary>Using pip installation</summary>

python -m mcp_playwright_scraper

Usage

Once configured in Claude Desktop, you can explicitly use the scraper with a prompt like:

Use the mcp-playwright-scraper to scrape the content from https://example.com and summarize it.

Debugging

You can use the MCP inspector to debug the server:

npx @modelcontextprotocol/inspector uvx mcp-playwright-scraper

Or if you've installed the package in a specific directory or are developing on it:

cd path/to/mcp-playwright-scraper
npx @modelcontextprotocol/inspector uv run mcp-playwright-scraper

Upon launching, the Inspector will display a URL that you can access in your browser to begin debugging.

Development

Building and Publishing

To prepare the package for distribution:

1. Sync dependencies and update lockfile:

uv sync

2. Build package distributions:

uv build

This will create source and wheel distributions in the dist/ directory.

3. Publish to PyPI:

uv publish