SteadyFetch

by carsonlabs

Not rated
GitHub

About

Reliable web fetching for AI agents with retry, circuit breaker, caching, and anti-bot bypass

Details

Author
carsonlabs
Categories
Web Scraping, Automation, AI

Setup

Install SteadyFetch in your MCP client (Claude Desktop, Cursor, Windsurf, and others).

Repository: https://github.com/carsonlabs/steadyfetch

Follow the installation instructions in the repository README, then restart your MCP client.

Reliable web fetching for AI agents.Stop losing hours to Cloudflare blocks, timeouts, and flaky scrapes.

SteadyFetch is an MCP server that gives your AI agents reliable web access with automatic retry, circuit breaker protection, caching, and anti-bot bypass — out of the box.

Every AI agent that touches the web hits the same wall:

- Cloudflare blocks your requests
- Sites return CAPTCHAs
- Pages timeout or load partially
- Rate limits kill your batch jobs
- You waste hours debugging flaky scrapes

One MCP tool call. SteadyFetch handles the rest.

Agent calls fetch_url("https://example.com") → Checks cache (instant if hit) → Checks circuit breaker (fail-fast if domain is down) → Stealth browser fetch with anti-bot bypass → On failure: retry with exponential backoff → Fallback: plain HTTP fetch → Cache the result → Return clean markdown + raw HTML
{ "mcpServers": { "steadyfetch": { "command": "steadyfetch" } } }
docker build -t steadyfetch . docker run -p 8200:8200 steadyfetch

Retry with backoff— Exponential backoff + jitter prevents retry storms. 3 browser attempts before falling back to HTTP.

Circuit breaker— Per-domain failure tracking. After 5 consecutive failures, the domain is circuit-broken for 2 minutes. Prevents wasting time on sites that are blocking you.

Caching— Disk-backed cache with configurable TTL. Repeat fetches are instant. 500MB default limit.

Anti-bot bypass— Stealth browser with magic mode, navigator patching, and human-like behavior simulation via Crawl4AI.

Graceful degradation— If the browser can't get through, falls back to plain HTTP. If HTTP fails, returns a clear error with domain health status. Never hangs, never silently fails.

Free tier gives you basic HTTP fetching and domain health checks. Pro unlocks JS rendering, anti-bot bypass, caching, and clean markdown output.

Upgrade to Pro on MCPize— $19/mo or $190/yr.

Web scraping, crawling, and change detection with AI

Hyperbrowser is the next-generation platform empowering AI agents and enabling effortless, scalable browser automation.

Leverage Notte Web AI agents & cloud browser sessions for scalable browser automation & scraping workflows

AI-powered browser automation MCP server — navigate sites, fill forms, extract data, and handle logins via Claude Code CLI

Scrape any URL with JavaScript rendering and get back clean markdown — built for AI agents, LLM pipelines, and autonomous research workflows.

An AI-powered web scraping system for high-volume automation and advanced data extraction strategies.

Ultra-fast web scraper and deep discussion crawler delivering clean Markdown for AI Agents

AI tools for web scraping, crawling, browser control, and web search via the Oxylabs AI Studio API.

AI-powered tools for web scraping, crawling, and browser automation.

An intelligent web scraping tool using AI and browser automation to extract structured data from websites.

No reviews yet — be the first

Sign in to leave a review

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.