wet-mcp
About
Web search, content extraction, and media download
Details
- Author
- n24q02m
- Categories
- Search, Knowledge Base, Other
Jump to
Setup
Install wet-mcp in your MCP client (Claude Desktop, Cursor, Windsurf, and others).
Repository: https://github.com/n24q02m/wet-mcp
Follow the installation instructions in the repository README, then restart your MCP client.
Web search, content extraction, and library docs for AI agents -- 5-strategy scraping, runs without API keys.
Current release: v3.x.media(action="analyze")was removed in the v2.0.0 BREAKING release. Useimagine-mcp'sunderstandaction for vision/audio/video analysis. Seedocs/migration.mdfor the upgrade recipe.
- Features
- Status
- Quick install
- Configuration
- Documentation
- Tools
- CLI
- Comparison
- Security
- Build from Source
- Deploy to Cloudflare
- Smithery
- Trust Model
- License
- Web Search-- Embedded SearXNG metasearch (Google, Bing, DuckDuckGo, Brave) with query expansion, TTL cache (1 h general / 5 min time-sensitive), standardized citation format, and 200-token snippet cap. Optional cloud search backends (Tavily, Brave, Exa) as a fallback chain viaSEARCH_BACKENDS
- Academic Research-- Search Google Scholar, Semantic Scholar, arXiv, PubMed, CrossRef, BASE
- Library Docs-- Auto-discover and index documentation with FTS5 hybrid search, HyDE-enhanced retrieval, and version-specific docs
- Content Extract-- 5-strategy escalation chain vian24q02m-web-coreScrapingAgent(basic_http->tls_spoof-> render backends fromBROWSER_BACKENDS(native/browserless/cf-browser-rendering) -> optional key-gatedcaptcha), markitdown bridge for low-tier HTML/MD fallback, smart chunks structured output (clean text + markdown + JSON-LD + code blocks + metadata), batch processing (up to 50 URLs), deep crawling, site mapping
- Local File Conversion-- Convert PDF, DOCX, XLSX, CSV, HTML, EPUB, PPTX to Markdown
- Media-- List + download images / videos / audio files.analyzewas removed in v2.0.0 -- useimagine-mcp.understandfor vision/audio inference
- Anti-bot-- Stealth strategies bypass Cloudflare, Medium, LinkedIn, Twitter
- Zero Config-- Built-in local reference embedding + reranking through fastretrieval, no API keys needed. Optional cloud providers (Jina AI, Gemini, OpenAI, Cohere, xAI, Anthropic) selected per task via theEMBEDDING_MODELS/RERANK_MODELS/LLM_MODELSmodel chains for higher-quality vectors and LLM features
- Sync-- Cross-machine sync of indexed docs via Google Drive (OAuth Device Code, no browser redirect)
# Method 1 (default): plugin install via Claude Code /plugin marketplace add n24q02m/claude-plugins /plugin install wet-mcp@n24q02m-plugins # Method 2 (CLI): direct uvx invocation claude mcp add wet -- uvx wet-mcp # Method 3 (recommended for HTTP / multi-device / OAuth) docker run -d --name wet-mcp-http -p 8084:8080 \ -v wet-data:/data -e MCP_TRANSPORT=http \ -e PUBLIC_URL=https://wet.example.com \ n24q02m/wet-mcp:latest # Method 4 (remote): point a client at an HTTP deployment claude mcp add --transport http wet https://<your-host>/mcp
The HTTP endpoint speaks Streamable HTTP and is OAuth-gated -- your client is prompted to authenticate in the browser on first connect (no API key to paste). Stand one up via Method 3 or theDeploy to Cloudflaresection.
Full setup matrices live at the canonical docs sitemcp.n24q02m.com/servers/wet-mcp/setup/and the paste-to-agent snippets atclaude-plugins/plugins/wet-mcp/setup-with-agent.md(per Spec F single source of truth).
wet runs zero-config out of the box: web search uses an embedded local SearXNG, and embedding/reranking fall back to the bundled local ONNX models through fastretrieval when no cloud keys are set. For higher-quality results, point each task at a cloud model chain. All settings are plain environment variables (no app prefix) -- in the HTTP self-host mode they are entered through the browser setup form instead.
Model chains(CSVprovider/model,provider/model; order = fallback). Leave a chain empty to use the local ONNX models (embedding/rerank) or to disable LLM features (LLM):
Provider keys-- the provider is inferred from each model's prefix; supply the matching key (litellm<PROVIDER>_API_KEYconvention):
Any other litellm provider works via env passthrough -- seelitellm provider docsfor its key name.
FASTRETRIEVAL_CACHE_PATHcontrols the local model cache. The oldQWEN3_EMBED_CACHE_PATHname is still honored when the new name is absent.
Search backends--SEARCH_BACKENDS(CSV, runtime fallback chain) oversearxng(default, local) plus optional cloud providerstavily/brave/exa. Point at an external SearXNG withSEARXNG_URL. Cloud providers needTAVILY_API_KEY/BRAVE_API_KEY/EXA_API_KEY.
Browser render backends--BROWSER_BACKENDS(CSV, escalation chain) picks the headless render leg ofextract:native(in-process chromium, the zero-config default),browserless(self-host render service -- setBROWSERLESS_URL+BROWSERLESS_TOKEN), andcf-browser-rendering(Cloudflare Browser Rendering -- setCF_ACCOUNT_ID+CF_BROWSER_RENDERING_TOKEN). Empty chain falls back tonative. SetCAPSOLVER_API_KEYto append an optional, key-gated CAPTCHA tier as the last escalation step.
Robots policy-- setRESPECT_ROBOTS_TXT=trueto enforcerobots.txtacross both theextractstrategy chain and the Crawl4AI-backedcrawl,sitemap, andlist_mediaactions. The default isfalseto preserve existing deployment behaviour; configure this process-level policy explicitly when the operator requires robots enforcement.
Disable local fallbacks-- opt out of the heavy in-process local fallbacks per capability (e.g. on a slim container that renders/searches/embeds via cloud backends only):DISABLE_LOCAL_BROWSER,DISABLE_LOCAL_SEARCH,DISABLE_LOCAL_EMBED,DISABLE_LOCAL_RERANK.
Docs sync--SYNC_ENABLED(defaulttrue),GOOGLE_DRIVE_CLIENT_ID(required for sync),SYNC_FOLDER(defaultwet-mcp),SYNC_INTERVAL(default300s). Sync uses Google Drive over the OAuth Device Code flow (no browser redirect).
HTTP self-host--MCP_TRANSPORT=http,PUBLIC_URL=<your-domain>. The setup form is gated byMCP_RELAY_PASSWORD; multi-user deployments also requireCREDENTIAL_SECRET(per-user vault key) andMCP_DCR_SERVER_SECRET.
{ "mcpServers": { "wet": { "command": "uvx", "args": ["wet-mcp"], "env": { "EMBEDDING_MODELS": "jina_ai/jina-embeddings-v5-text-small", "RERANK_MODELS": "jina_ai/jina-reranker-v3", "LLM_MODELS": "gemini/gemini-3-flash-preview", "JINA_AI_API_KEY": "jina_xxx", "GEMINI_API_KEY": "AIza_xxx" } } } }
Stable architecture with two transports:stdio(default, local) andHTTP(self-host, OAuth-gated). No daemon-bridge layer and no auto-spawn from stdio. Themedia.analyzeaction was removed in the v2.0.0 BREAKING release -- seedocs/migration.mdfor the upgrade recipe. Current release line: v3.x.
Full docs atmcp.n24q02m.com/servers/wet-mcp/setup/:
- Setup-- install methods for Claude Code, Codex, Gemini CLI, Cursor, Windsurf, mcp.json
- Modes overview-- stdio / local-relay / remote-relay / remote-oauth
- Multi-user setup-- per-JWT-sub credential model
In-repo references (Spec F single source of truth: setup docs live inclaude-plugins/plugins/wet-mcp/):
- docs/ARCHITECTURE.md-- web-core ScrapingAgent integration, strategy chain, storage layout, LLM provider dispatch
- docs/BENCHMARKS.md-- v1.x baseline coverage / latency placeholders + tier-1 fixture metrics
Install with AI agent-- paste this to your AI coding agent:
Install MCP serverwet-mcpfollowing the steps athttps://raw.githubusercontent.com/n24q02m/claude-plugins/main/plugins/wet-mcp/setup-with-agent.md
6 MCP tools (3 domain +config+help+config__open_relay). The legacysetuptool merged intoconfigaction dispatch.
Media boundary: For vision / audio understanding (image captioning, OCR, audio transcription, video summarization), useimagine-mcp.media.analyzewas removed in wet v2.0.0 -- useimagine-mcp.understandinstead.
Thewet-mcpconsole script starts the server and also exposes a few one-shot operator subcommands. A bare invocation (or any leading-dash flag) starts the server; a leading positional argument is dispatched as a subcommand.
wet-mcp # start the server over stdio (default transport) wet-mcp --http # start the server over Streamable HTTP (self-host mode) wet-mcp auth google # authorize the Google credential provider for Drive sync wet-mcp logout # clear the local Google Drive sync token wet-mcp warmup # pre-download local models + run auto-setup (SearXNG, browser) to avoid first-run delays wet-mcp docs reindex <library> # drop the cached docs index for <library>; the next docs search re-indexes it
auth googleaccepts an optional bring-your-own OAuth client via--client-idand--client-secret(single-user / local machine only; the token is written to the local store). Each subcommand prints a JSON result and exits.
- SSRF prevention-- URL validation on crawl targets
- Graceful fallbacks-- Cloud → Local embedding, multi-tier crawling
- Error sanitization-- No credentials in error messages
- File conversion sandboxing-- OptionalCONVERT_ALLOWED_DIRSrestriction
git clone https://github.com/n24q02m/wet-mcp.git cd wet-mcp uv sync uv run wet-mcp
Run your own single-user wet instance serverless on Cloudflare (Containers + D1 + Vectorize + KV).
Prerequisites:a Cloudflare account on theWorkers Paid plan— required for Containers, D1, and Vectorize (the Cloudflare free tier does not include them) — and thewranglerCLI.
- git clone https://github.com/n24q02m/wet-mcp && cd wet-mcp
- wrangler login
- Provision resources and apply the D1 schema:
wrangler d1 create wet-docs wrangler d1 execute wet-docs --file migrations/0001_init_wet.sql --remote wrangler d1 execute wet-docs --file migrations/0002_project_context.sql --remote wrangler d1 execute wet-docs --file migrations/0003_version_index_state.sql --remote wrangler vectorize create wet-docs-vectors --dimensions 768 --metric cosine wrangler kv namespace create wet-kv
docker pull ghcr.io/n24q02m/wet-mcp:beta docker tag ghcr.io/n24q02m/wet-mcp:beta wet-mcp:beta wrangler containers push wet-mcp:beta # prints registry.cloudflare.com/<ACCOUNT_ID>/wet-mcp:beta
wrangler secret put CREDENTIAL_SECRET wrangler secret put JINA_AI_API_KEY wrangler secret put GOOGLE_VERTEX_EXPRESS_API_KEY wrangler secret put XAI_API_KEY wrangler secret put MCP_RELAY_PASSWORD wrangler secret put MCP_DCR_SERVER_SECRET wrangler secret put SEARXNG_URL wrangler secret put BROWSERLESS_URL # render backend (BROWSER_BACKENDS default = browserless,cf-browser-rendering) wrangler secret put BROWSERLESS_TOKEN wrangler secret put CF_BROWSER_RENDERING_TOKEN
Storage maps to Cloudflare viaMCP_STORAGE_BACKEND=cf-kv(credentials/tokens, encrypted),DOCS_DB_BACKEND=cf-d1(docs + BM25 full-text), and Vectorize (embeddings). Web search uses a SearXNG instance (SEARCH_BACKEND=searxng,SEARXNG_URL) or Tavily (SEARCH_BACKEND=tavily); embed/rerank are forced cloud viaEMBEDDING_MODELS/RERANK_MODELS.
wet-mcp ships asmithery.yamlso it can be installed and run throughSmithery. The manifest declares a stdio start command (uvx --python 3.13 wet-mcp) with an empty config schema -- no config is required to start, and providers and credentials are configured at runtime via the server's own config flow (seeConfiguration).
This plugin implementsTC-Local(machine-bound, single trust principal). Seemcp-core trust modelfor full classification.
Search global news using natural language. Webz.io News Search API returns the most relevant articles and content, with filters for source, country, language, date, sentiment, and category.
Fetch, convert, and search AWS documentation pages, with recommendations for related content.
Search campgrounds around the world on campertunity, check availability, and provide booking links.
The Ferryhopper MCP Server exposes ferry routes, schedules and booking redirects so an AI assistant can discover connections across Europe and the Mediterranean and send users to Ferryhopper to complete bookings.
All-in-One SEO & Web Intelligence Toolkit API from FetchSERP.
MCP server that provides read-only access to HyperKitty, the web-based email archive component of Mailman 3.
At Sunrise Apps, we believe AI agents should be limitless, especially when it comes to visual data. We created ImageSorcery to bridge the critical gap in AI's ability to interact with and manipulate images directly, all while upholding the highest standards of privacy and security.
Just Domain is the domain registrar for businesses built with AI. Its remote MCP server checks availability and returns first-year and renewal pricing, plus a link to register on justdomain.ai, with DNS and WHOIS privacy in the same place. No account, no API key, read only. Endpoint: https://mcp.justdomain.ai/
Research tools, including a Sqlite-backed document stash
Semantic search over 9 free-license stock photo sources. Hosted remote server with OAuth — no API key to paste.
SerpApi MCP Server for Google and other search engine results
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.




