Webclaw
About
Web content extraction for LLM pipelines — clean markdown or structured JSON from any URL using browser-grade TLS fingerprinting, no headless browser required. CLI, REST API, and MCP server.
Details
- Author
- 0xmassi
- Categories
- Web Scraping, AI, Other
Jump to
Setup
Install Webclaw in your MCP client (Claude Desktop, Cursor, Windsurf, and others).
Repository: https://github.com/0xmassi/webclaw
Follow the installation instructions in the repository README, then restart your MCP client.
Turn websites into clean markdown, JSON, and LLM-ready context.
CLI, MCP server, REST API, and SDKs for AI agents and RAG pipelines.
Most web scraping tools give your agent one of two bad outputs:
- a blocked page, login wall, or empty app shell
- raw HTML full of nav, scripts, styling, ads, and duplicated boilerplate
webclaw turns a URL into clean content your tools can actually use.
webclaw https://example.com --format markdown
# Example Domain This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.
Use it from the terminal, wire it into Claude/Cursor through MCP, call the hosted API from your app, or self-host the OSS server.
The fastest way to connect webclaw to Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, and other MCP-compatible tools:
The installer detects supported clients and configures the MCP server for you.
brew tap 0xMassi/webclaw brew install webclaw
Download macOS, Linux, and Windows binaries fromGitHub Releases.
docker run --rm ghcr.io/0xmassi/webclaw https://example.com
cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli cargo install --git https://github.com/0xMassi/webclaw.git webclaw-mcp
If building from source fails because native build tools are missing, install the platform prerequisites:
webclaw https://stripe.com --format markdown
webclaw https://docs.anthropic.com --format llm
webclaw https://example.com/blog/post --only-main-content
webclaw https://example.com \ --include "article, main, .content" \ --exclude "nav, footer, .sidebar, .ad"
webclaw https://docs.rust-lang.org --crawl --depth 2 --max-pages 50
- HTML to Markdown for RAG
- Firecrawl-compatible API
- MCP web scraping
- Proxy-backed crawling with ColdProxy
- Cloudflare diagnostics
webclaw https://example.com/pricing --format json > pricing-old.json webclaw https://example.com/pricing --diff-with pricing-old.json
webclaw ships with an MCP server for AI agents.
Zero-install — point any MCP client at the npx launcher:
{ "mcpServers": { "webclaw": { "command": "npx", "args": ["-y", "@webclaw/mcp"] } } }
Or runnpx create-webclawto auto-detect your AI tools and write their configs for you.
Scrape these competitor pricing pages and summarize the differences.
Crawl this documentation site and prepare clean context for a RAG index.
Extract the brand colors, fonts, and logos from this company website.
Add webclaw to Claude Code, Cursor, Windsurf, and other MCP agents in one command:
Your agent gets scrape, crawl, map, extract, summarize, diff, brand, and search as native tools. Most sites extract locally with no API key. SetWEBCLAW_API_KEYto handle bot-protected and JavaScript-rendered pages.
npm install @webclaw/sdk pip install webclaw go get github.com/0xMassi/webclaw-go
import { Webclaw } from "@webclaw/sdk"; const client = new Webclaw({ apiKey: process.env.WEBCLAW_API_KEY! }); const page = await client.scrape({ url: "https://example.com", formats: ["markdown"], only_main_content: true, }); console.log(page.markdown);
from webclaw import Webclaw client = Webclaw(api_key="wc_your_key") page = client.scrape( "https://example.com", formats=["markdown"], only_main_content=True, ) print(page.markdown)
curl -X POST https://api.webclaw.io/v1/scrape \ -H "Authorization: Bearer $WEBCLAW_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "formats": ["markdown"], "only_main_content": true }'
The CLI and MCP server work locally without an account for the core extraction path.
Use the hosted API atwebclaw.iowhen you need:
- protected-site access without managing infrastructure
- JavaScript rendering
- async crawl and research jobs
- web search
- watches and production usage tracking
- SDKs for application code
export WEBCLAW_API_KEY=wc_your_key webclaw https://example.com --cloud
webclaw/ crates/ webclaw-core HTML to markdown, text, JSON, and LLM-ready output webclaw-fetch Fetching, crawling, batching, and mapping webclaw-llm Local and hosted LLM provider support webclaw-pdf PDF text extraction webclaw-mcp MCP server for AI agents webclaw-cli Command-line interface
webclaw-coreis pure extraction logic: no network I/O, small surface area, and usable independently from the fetching layer.
The most useful contributions right now are practical and small:
- add examples for real agent and RAG workflows
- improve SDK snippets
- report pages that extract poorly
- add failing fixtures for messy HTML
- improve docs for MCP clients and local setup
- test the CLI on more Linux/macOS environments
- Good first issues
- Open a bug report
- Start a discussion
URL: Command or API request: Expected output: Actual output: Format used: markdown / llm / text / json / html CLI, MCP, SDK, or API:
Third-party plugins that integrate webclaw with AI agent platforms:
Built a webclaw integration?Open a PRto add it here.
Thanks to everyone improving webclaw through issues, examples, docs, bug reports, and pull requests.
An MCP server that lets an AI agent (like Claude) read data from Discord servers through your own account. It pairs with a BetterDiscord plugin, so the Python side never handles your token — data is read straight from the already-authenticated Discord client.
MCP server + Chrome extension that connects Claude Code to Chromium browsers (Arc, Vivaldi, Brave) where the official Claude in Chrome extension doesn't work.
Web scraping skill for Claude AI. Crawl websites, extract structured data with CSS/LLM strategies, handle dynamic JavaScript content. Built on crawl4ai with complete SDK reference, example scripts, and tests.
Fetches content from deepwiki.com and converts it into LLM-readable markdown.
Extract documentation for AI agents from any site with llms.txt support. Features MCP server, REST API, batch processing, and multiple export formats.
KYB due diligence, competitive intelligence, and strategic accounts research for AI agents. KYB covers corporate registries, sanctions screening (OFAC/UN/EU/UK), FCA Register lookup, directors, UBOs, and adverse media. CI covers pricing, hiring, product, sentiment, and corporate events. New accounts get 3 free trial calls. Pay-per-call after: $0.40 per /findings, $2.00 per /run with Claude analysis. Top up from $20 at getsonar.report.
Scrape Google Maps business leads (names, phones, emails, websites, ratings) from Claude, Cursor or any MCP client via the gmapsscraper.io API.
The Import.io Web Scraper MCP server brings hosted scraping, browser automation, proxy routing, captcha handling, and structured data extraction into your AI tools. Point Claude, ChatGPT, Cursor, or any MCP client at our hosted endpoint. Import.io runs the scraping engine, you just connect.
Screenshot & Render API for AI Agents. MCP Server lets Claude, Cursor capture webpages and render HTML.
Give Claude web browsing powers - fetch webpages, extract data, discover sitemaps, summarize content
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
