mcp-retrieval

by role1776

Not rated
GitHub

Description

MCP server in Go with three read-only tools: web search, image search, and page scraping to Markdown. No API keys or accounts required - search runs through DuckDuckGo Lite, image search through Bing Images, and pages are extracted with a readability parser. stdio and HTTP…

About

MCP server in Go with three read-only tools: web search, image search, and page scraping to Markdown. No API keys or accounts required - search runs through DuckDuckGo Lite, image search through Bing Images, and pages are extracted with a readability parser. stdio and HTTP transports, MIT licensed

Details

Author
role1776
Categories
Search, Other, Web Scraping

Setup

Install mcp-retrieval in your MCP client (Claude Desktop, Cursor, Windsurf, and others).

Repository: https://github.com/role1776/mcp-retrieval

Follow the installation instructions in the repository README, then restart your MCP client.

An MCP server that gives an LLM three web tools: search, image search, and page scraping — no API keys required.

Tools·Quick start·Configuration·Retrieval engine·Architecture·Contributing

mcp-retrievalis aModel Context Protocolserver written in Go. It exposes web retrieval capabilities to any MCP-compatible client (Claude Desktop, IDE agents, custom LLM apps) as three read-only tools. Under the hood it uses theretrieval-golibrary to search the web and fetch pages, returning results as clean Markdown ready to hand to a model.

The library needsno API keys: web search goes through DuckDuckGo Lite, image search through Bing Images, and page fetching runs the HTML through a readability extractor before converting it to Markdown. To stay reliable against bot protection it impersonates real browsers at the TLS level and can rotate both browser fingerprints and proxies — seeRetrieval engine.

Both transports the MCP SDK supports are available and expose the identical tool set:

- stdio— the client launches the binary and talks over stdin/stdout (the default, ideal for desktop clients).
- http— a long-running streamable HTTP server (useful for remote/shared deployments).

All three are annotated asread-only. Each tool returns a structured JSON payload that matches its output schema; the SDK mirrors the same JSON into the text content block for clients that do not readstructuredContent.

Bothqueries/urlslists are capped atmax_queries(10) items per call. Queries must be ≤ 512 characters; URLs ≤ 2048 characters andhttp/httpsonly.

Every call fans out across the input list and returns one entry per query/URL, each with its ownstatussuccess,failed, ortimeout— so a partial failure still returns the items that did work.

countis the number of items actually returned, and it can belower than the requestedmax_results/max_images: duplicates within a single query's results are removed before the limit is applied, and the upstream may simply have fewer items to give. A smallercountis a normal outcome, not an error.

Deduplication isper query, not across queries. Each entry is deduplicated on its own, so a link found by two of the queries in the same call appears in both entries — dedupe the union yourself if you need it.

Request-level failures are returned as a tool result withisError: trueand a plain-text message, not as a JSON-RPC error — the model reads the message and can correct the call itself. Per-item failures never do this; they stay inside the payload asstatus: "failed"/"timeout".

A call fails outright only when the input is rejected before any work starts, or wheneveryitem in it fails:

The all-failed messages deliberately do not distinguish timeouts from other causes: a mixed batch can fail for several reasons at once, and the per-itemstatusalready carries that detail whenever at least one item survives.

- web_scrapehandles HTML only.Pages are run through a readability extractor, which needs article markup, sotext/plainresponses yield nothing and come back asstatus: "failed". Raw-file hosts are the common case:raw.githubusercontent.com,github.com/.../raw/...,cdn.jsdelivr.net. Scrape the rendered page instead of the raw file.
- web_search_imagesrelevance is not guaranteed.For some queries Bing Images serves a page that is not a result set, and it is parsed as though it were — the tool then returns unrelated images withstatus: "success". Treat image results as best-effort and verify them before showing them to a user.
- No JavaScript.Pages are fetched as-is; content rendered client-side is invisible to the extractor.

Pick whichever fits — all of them give the identical server.

docker pull ghcr.io/role1776/mcp-retrieval:latest

Prebuilt binary— grab the archive for your platform from thelatest release, unpack it, and putmcp-retrievalon yourPATH.

MCP Bundle— for clients that install.mcpbfiles, downloadmcp-retrieval_<version>_<os>_<arch>.mcpbfrom thelatest releaseand open it with your client. The bundle carries the compiled binary, so it needs neither Docker nor Go. Pick the file matching your OSandCPU architecture: a bundle holds one native binary.

go install github.com/Role1776/mcp-retrieval/app/cmd/mcp-retrieval@latest # needs Go 1.25.5+

Or build the binary in place (the Go module lives inapp/):

# defaults: stdio transport, no configuration needed ./bin/mcp-retrieval # with an explicit env file ./bin/mcp-retrieval -env /absolute/path/to/.env

Point your client at the built binary. Example Claude Desktop config:

{ "mcpServers": { "retrieval": { "command": "/absolute/path/to/mcp-retrieval", "env": { "MAX_RESULTS": "20" } } } }

Theenvblock is optional —"command"alone is enough.

Run the image on stdio. Configuration still travels through theenvblock, but Docker needs each variable named on the command line with-efor it to reach the process:

{ "mcpServers": { "retrieval": { "command": "docker", "args": [ "run", "-i", "--rm", "-e", "MAX_RESULTS", "-e", "DEFAULT_TIMEOUT_MS", "ghcr.io/role1776/mcp-retrieval:latest" ], "env": { "MAX_RESULTS": "20", "DEFAULT_TIMEOUT_MS": "5000" } } } }

-iis required — without it the container gets no stdin and the client sees the server die immediately. Clients that install from theMCP Registrybuild this invocation themselves and prompt for the variables declared inserver.json.

SetMCP_TRANSPORT=httpand the server listens onSERVER_PORTatMCP_PATH(defaulthttp://localhost:8080/mcp).

Everything is configured throughenvironment variables, and each value is validated before startup: a non-numeric or non-positive value is a startup error. Relationshipsbetweenlimits are not checked at startup — seeLimits. Variables already present in the environment win over a.envfile, so an MCP client'senvblock always takes effect. Every field has a sensible default, so the server runs with no configuration at all (stdio transport).

See.env.examplefor the full list at its default values, ready to copy to.env.

The version advertised to clients is not configurable: it is stamped into the binary at build time from the git tag.

When a proxy is configured, each outbound request gets a unique session id appended to the login, so the upstream provider rotates the exit IP per request.

Each value is checked on its own — it must be greater than zero — but theDEFAULT_,MIN_andMAX_triples arenotcross-checked against each other at startup. An inconsistent set does not stop the server; it is reconciled per request instead:

- a value the caller omits, or passes as zero or negative, falls back to the matchingDEFAULT_;
- the result is then clamped into
[MIN_, MAX_], so aDEFAULT_larger than itsMAX_simply yieldsMAX_;
- ifMIN_
exceedsMAX_*, the maximum wins.

The effective limit is therefore always within the configured maximum, and misconfiguration degrades to a working server rather than a failed start. The trade-off is that it degradessilently: a typo such asMAX_RESULTS=2instead of20produces no warning, only quietly smaller responses. Worth double-checking these values when results look truncated.

The project follows a clean, layered structure. Dependencies point inward toward the domain, and each layer talks to the next through interfaces.

app/ the Go module: sources plus its build files (Dockerfile, .dockerignore, .goreleaser.yaml) cmd/mcp-retrieval/main.go entry point: parse flags, load config, run app internal/ app/ wiring + lifecycle (build server, run, graceful shutdown) config/ config loading (.env → env vars → validate) domain/ core types (Query, Link, Document, Snippet, Image) and errors dto/web/ request/response shapes for the MCP tools transport/mcp/ MCP layer router/ registers every tool group on the MCP server web/ tool handlers utils/ schema helpers and error → tool-result mapping usecase/web/ business logic: validation, parallelism, timeouts, dedupe/limit/rerank adapter/web/ retrieval-go client wiring (search, images, scrape, proxy) pkg/ reusable building blocks (mcpserver, server, logger, validator)
MCP client → transport/mcp/web (handler) → usecase/web → adapter/web → retrieval-go → the web ↑ maps errors ↑ validates, fans out, limits results

Search and scrape both fan out across the input list concurrently and aggregate per-item results, each with its own status (success,failed,timeout). A call only fails outright wheneveryitem in it fails.

All network work is delegated toretrieval-go, configured inapp/internal/adapter/web. Worth knowing:

- Sources.Web search usesDuckDuckGo Lite; image search usesBing Images; page fetching runs the raw HTML through areadabilityextractor and converts the main article toMarkdown(tables included). No search-engine API keys are required.
- Browser impersonation.The adapter enablesWithBrowserRotation(), so each request is sent from one of ~11 real browser profiles picked at random. Every profile pairs a genuineTLS/JA3 fingerprint(via
uTLS) with a matchingUser-Agentand client-hint headers — Chrome 133/131/120 (Windows/macOS/Linux), Edge 131, Firefox 120 (Windows/macOS), Safari 18.4 (macOS), and iOS 18.4 Safari. This makes the traffic look like ordinary browsers rather than a Go HTTP client, which is what keeps the free sources reachable.
- Proxy rotation.WhenPROXY_HOSTis configured, the adapter installs a proxy factory that appends a uniquesession-<id>to the proxy username on every request. With a session-based residential/rotating proxy provider, that yields afresh exit IP per request, spreading load and avoiding rate limits. Without a proxy, requests go out directly.
- Response handling.Responses are transparently decompressed (gzip,br,zstd,deflate), and keep-alive is disabled (WithDisableKeepAlive()) so pooled connections don't pin a single fingerprint/IP across requests.

None of this needs configuration to work — the defaults above are applied automatically. Only proxy credentials are optional extras.

Everything Go lives inapp/, so either use the makefile from the repository root or pass-C appto the toolchain:

make build # compile the binary make test # run tests go -C app build ./... # compile everything go -C app test ./... # run tests go -C app vet ./... # static checks

SeeCONTRIBUTING.mdfor pull-request guidelines.

Search global news using natural language. Webz.io News Search API returns the most relevant articles and content, with filters for source, country, language, date, sentiment, and category.

A private, local research assistant that searches the web and scrapes content using DuckDuckGo.

Fetch, convert, and search AWS documentation pages, with recommendations for related content.

Search campgrounds around the world on campertunity, check availability, and provide booking links.

The Ferryhopper MCP Server exposes ferry routes, schedules and booking redirects so an AI assistant can discover connections across Europe and the Mediterranean and send users to Ferryhopper to complete bookings.

All-in-One SEO & Web Intelligence Toolkit API from FetchSERP.

MCP server that provides read-only access to HyperKitty, the web-based email archive component of Mailman 3.

At Sunrise Apps, we believe AI agents should be limitless, especially when it comes to visual data. We created ImageSorcery to bridge the critical gap in AI's ability to interact with and manipulate images directly, all while upholding the highest standards of privacy and security.

Just Domain is the domain registrar for businesses built with AI. Its remote MCP server checks availability and returns first-year and renewal pricing, plus a link to register on justdomain.ai, with DNS and WHOIS privacy in the same place. No account, no API key, read only. Endpoint: https://mcp.justdomain.ai/

Search the web using Kagi's search API

No reviews yet — be the first

Sign in to leave a review

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.