Docs Fetch MCP Server

Fetch web page content with recursive exploration.

A Bun-based Model Context Protocol (MCP) server for fetching documentation pages and bounded documentation crawls.

The server exposes one MCP tool,fetch_doc_content. It fetches a root URL, extracts readable Markdown, ranks links, and can crawl linked pages within explicit depth, page, timeout, and scope limits. Results are returned as structured JSON with crawl metadata, page content, ranked links, truncation flags, and per-page errors.

- Fast static fetch path withaxios
- Optional Puppeteer rendering for client-rendered or thin pages
- Markdown extraction with headings, lists, code blocks, tables, blockquotes, and links
- Real crawl depth semantics:

- depth: 1returns only the root page
- depth: 2includes direct child links
- depth: 3includes grandchildren, up to the maximum of5

- Bun>=1.3.0
- Puppeteer's browser installation ifrenderisautooralways

This project uses Bun for dependency management, runtime execution, builds, and tests.

{ "mcpServers": { "docs-fetch": { "command": "bun", "args": ["/path/to/docs-fetch-mcp/build/index.js"] } } }

{ "mcpServers": { "docs-fetch": { "command": "bun", "args": ["/path/to/docs-fetch-mcp/src/index.ts"] } } }

Fetch one URL and optionally crawl linked pages.

{ "url": "https://example.com/docs", "depth": 2, "maxPages": 8, "pathPrefix": "/docs", "render": "auto" }

{ "rootUrl": "https://example.com/docs", "normalizedRootUrl": "https://example.com/docs", "explorationDepth": 2, "maxPages": 8, "pagesExplored": 3, "pagesFailed": 1, "timedOut": false, "durationMs": 1234, "crawl": { "sameOrigin": true, "pathPrefix": "/docs", "maxConcurrency": 3, "render": "auto", "perPageTimeoutMs": 10000 }, "content": [ { "url": "https://example.com/docs", "finalUrl": "https://example.com/docs", "depth": 0, "status": 200, "title": "Documentation", "description": "Example documentation", "canonicalUrl": "https://example.com/docs", "headings": ["Documentation"], "content": "# Documentation\n\n...", "contentLength": 2400, "truncated": false, "fetchedWith": "http", "links": [ { "url": "https://example.com/docs/api", "text": "API reference", "score": 18.5, "internal": true } ] } ], "errors": [ { "url": "https://example.com/docs/missing", "depth": 1, "error": "Request failed with status code 404", "status": 404 } ] }

- Crawling is breadth-first.
- URLs are normalized before dedupe and enqueue.
- sameOrigin: truerejects links whose origin differs from the normalized root URL.
- pathPrefixfurther restricts links to a path prefix on the root origin.
- includeLinks: falsehides links in returned page objects but does not disable link discovery for crawling.
- render: "never"uses only the HTTP fetch path.
- render: "always"uses Puppeteer for every fetched page.
- render: "auto"tries HTTP first, then uses Puppeteer when HTTP fails or extracted content is very thin.
- Browser fallback currently preserves the rendered response status and extracts the page body even for non-2xx pages.

src/index.ts CLI entrypoint src/server.ts MCP server wiring and tool registration src/config/tool-options.ts Shared tool schema/default/range metadata src/tool/fetch-doc-content-args.ts Runtime argument validation src/crawler/docs-crawler.ts Crawl orchestration and queue management src/crawler/page-fetcher.ts HTTP fetch, browser fallback, extraction coordination src/browser/browser-manager.ts Reusable Puppeteer browser/page handling src/content/content-extractor.ts Page metadata/content extraction facade src/content/main-content-selector.ts Main content selection src/content/link-extractor.ts Link normalization, dedupe, and ranking src/content/markdown-renderer.ts HTML-to-Markdown rendering src/content/text-cleanup.ts Shared text normalization helpers src/utils/url.ts URL normalization and scope utilities src/types/index.ts Shared TypeScript types

The MCP schema and runtime option normalization share the same metadata source insrc/config/tool-options.ts. Keep new options there first, then wire behavior through validation and crawler options.

bun install bun run dev bun run test bun run typecheck bun run build

- bun run dev: run the MCP server from TypeScript source.
- bun run test: run Bun tests undersrc.
- bun run typecheck: run TypeScript with--noEmit.
- bun run build: emitbuild/index.jswith a Bun shebang.
- bun run start: run the built MCP server.

Tests are colocated with the modules they cover:

- src/crawler/docs-crawler.test.ts: crawl depth, scoping, failures, and link visibility.
- src/crawler/page-fetcher.test.ts: fetch/render fallback characterization.
- src/content/content-extractor.test.ts: Markdown extraction and truncation.
- src/tool/fetch-doc-content-args.test.ts: boundary validation and schema/default sync.
- src/utils/url.test.ts: URL normalization and scope helpers.

When changing behavior, add or update characterization tests first. For refactors, keepbun run test,bun run typecheck, andbun run buildgreen.

- Userender: "never"for fast static documentation crawls and tests.
- UsepathPrefixfor documentation sites that share a domain with marketing pages, blogs, or apps.
- Puppeteer browser installation can be skipped only if callers userender: "never".
- The custom Markdown renderer is intentionally covered by characterization tests; preserve output compatibility unless making an explicit behavior change.

Local-first MCP server that captures web URLs (X, Reddit, YouTube, Wikipedia, articles) as typed data + Markdown into a self-hosted capture/store/recall substrate, with offline semantic recall. Six tools over a local khiipd daemon; run khiipd serve first.

Access the Internet Archive's Wayback Machine to retrieve archived web pages and check for available snapshots of URLs.

Real-time web data, structured for agents

An MCP server for the Urlbox Screenshot API. It enables your client to take screenshots, generate PDFs, extract HTML/markdown, and more from websites.

Zillow property data for AI agents — search listings by city or ZIP, look up any US address, and get 50+ fields per property including prices, Zestimates, price history, and sold data.

BrowserAct MCP Server is a standardized MCP service that lets MCP clients connect to the BrowserAct platform to discover and run browser automation workflows, access results/files and related storage, and trigger real-world actions via natural language.

A server for web crawling and content extraction using the Crawl4AI library.

Integrates web crawling and Retrieval-Augmented Generation (RAG) into AI agents and coding assistants.

CrawlForge MCP is a production-ready MCP server with 18 web scraping tools for AI agents. It gives Claude, Cursor, and any MCP-compatible client the ability to fetch URLs, extract structured data with CSS/XPath selectors, run deep multi-step research, bypass anti-bot detection with TLS fingerprint randomization, process documents, monitor page changes, and more. Credit-based pricing with a free tier (1,000 credits/month, no credit card required).

An MCP server for crawling WeChat articles. It supports single and batch crawling with multiple output formats, designed for AI tools like Cursor.