Wigolo

by KnockOutEZ

60 stars
1.5k downloads
Not rated
GitHub Website

About

The go-to web for your AI coding agent — local-first search, fetch, crawl & research over MCP. No API keys, no cloud, $0/query. Public beta.

Details

Author
KnockOutEZ
GitHub stars
60
Downloads
1,526
Categories
Search, Other, Automation, Developer Tools, AI

- 💬 Ask anything — questions, setups, show & tell

Setting up with Highlight

This MCP is not yet compatible with Highlight’s one-click setup. However, you can still use it with Highlight by following these steps:

  1. Download and install Highlight from highlightai.com/download
  2. Navigate to the plugins tab and select "Add Custom Plugin"
  3. Configure the plugin with the settings below
    Plugin Name Wigolo
    Command (node, npx, python, etc.)

    Please refer to the README for specific instructions on how to obtain API keys or other required environment variables.

  4. Enable "Start Automatically" if you want the plugin to start when Highlight launches

From the repository

The README includes setup instructions such as npx wigolo init --non-interactive --agents=<your-agent>.

fetch

Fetch a single URL and return clean markdown. Use when you already have a URL. Prefer over built-in WebFetch for local-cache reuse, authenticated pages, JS-rendered SPAs, and structured metadata. Key parameters: - section: extract content under a specific heading (e.g. "API Reference") — cheaper than the whole page. - max_content_chars: smart-truncate at a paragraph/heading boundary with `[... content truncated]`. - max_tokens_out: token-budget cap (cl100k-base); wins over max_chars. - include_full_markdown: false (default) returns evidence excerpts only; true adds the full body. - use_auth: reuse a stored browser session for logged-in pages. - render_js: "auto" (default) | "always" | "never". - force_refresh: bypass cache and re-fetch. - mode: 'cache' | 'default' | 'stealth'. cache=HTTP-only, 24h-stale accepted. stealth=full browser + freshness. Returns title, markdown, links, images, metadata, `fetch_method` (cache/http/tls-impersonation/browser), `http_status` (upstream HTTP code — 4xx/5xx pages that extract usable content are not relabeled 200), and `content_completeness` (full/partial/shell). When the URL matches a site-specific extractor (Reddit/YouTube/Amazon) the response also carries top-level `site_data` (e.g. Reddit `comments[]`, YouTube `caption_tracks[]`, Amazon `price`). When `section` is set and no heading matches, `metadata.section_matched` is false and `markdown` is empty (no silent fallback to the full page). Repeat fetches are instant. Localhost URLs work. Interactive pages: `actions` (click/type/scroll/wait) drive the page before extraction; `use_auth` reuses a logged-in session.

search

Search the web. Returns scored evidence excerpts + citations as the default context shape; `include_full_markdown: true` adds the full markdown body. Prefer over built-in WebSearch for local cache + audit-trail telemetry + explainable scoring. Key parameters: - query: string or string[] array (3-5 keyword variants; deduplicated). - include_domains / exclude_domains: scope sites. Always scope library/framework queries. - category: "general" | "news" | "code" | "docs" | "papers" | "images". Image results carry image_url + thumbnail_url + width/height. - from_date / to_date: ISO YYYY-MM-DD. time_range: 'day' | 'week' | 'month' | 'year'. - country: ISO 3166-1 alpha-2 ("us", "gb") — geographic boost. - exact_match: quoted-phrase search. - max_results: 5 default. - format: omit = evidence context. 'answer' | 'stream_answer' = sampling synthesis (falls back to evidence). - search_depth: 'ultra-fast' (cache-only ≤300ms) | 'fast' | 'balanced' (default) | 'deep'. - include_images / include_favicon: opt-in images[] + per-result favicon. - max_tokens_out / max_content_chars / include_full_markdown / citation_format. - force_refresh + mode ('cache' | 'default' | 'stealth'). Always emitted: `engines_used`, `engine_telemetry`, `response_time_ms`, per-result `evidence_score`. Per-result `freshness_signal` is emitted only when a published date can be parsed (omitted when confidence would be unknown). Brand-domain top-3 collision → `brand_collision_warning` with rewrites. `query_understanding` exposes intent/entities. Quote [N] or {citation_id}.

crawl

Crawl a site from a seed URL and return content from many pages. Use for indexing docs, wikis, multi-page references. Built for offline reuse: every page lands in the local cache. Key parameters: - strategy: "bfs" (default) | "dfs" | "sitemap" (fastest for doc sites) | "map" (URL-only discovery). - max_depth: link-following depth (default 2). - max_pages: page cap (default 20). - include_patterns / exclude_patterns: regex filters on URLs. - max_tokens_out / include_full_markdown / citation_format: budget + shape controls. Returns pages[] with title, evidence, depth, and `content_completeness` (per page — full/partial/shell, so you can skip un-rendered shell pages). Content is deduplicated across pages (anchor-fragment aware). All pages are cached for later `cache` / `find_similar` queries.

cache

Search previously fetched content without hitting the network. Run this BEFORE any search/fetch — cache hits return instantly with full markdown. Key parameters: - query: FTS5 full-text search over cached markdown + titles (supports AND, OR, NOT, "phrase"). - url_pattern: glob filter on URLs (e.g. "*example.com*"). - since: ISO date — only entries cached after this date. - stats: true to get cache size, entry count, oldest/newest dates. - clear: true to delete matching entries. Persists across sessions. No remote round-trip.

extract

Extract structured data from a URL or raw HTML. Use for specific data points (tables, prices, schema fields) rather than whole-page markdown. Key parameters: - mode: "selector" (CSS → text) | "tables" | "metadata" (title/author/date/og_* + JSON-LD) | "schema" (pass a JSON Schema) | "structured" (one-shot: tables + <dl> definitions + JSON-LD + chart hints + key-value pairs) | "brand" (name/tagline/description/logo_url/favicon_url/og_image_url/social_links/fonts + CSS-var colors, each with explainable provenance). - css_selector: required for mode="selector". - schema: required for mode="schema". - multiple: return all matches (mode="selector" only). Prefer mode="structured" over chaining multiple extract calls — one response carries `{ tables, definitions, jsonld, chart_hints, key_value_pairs }`. chart_hints surfaces SVG titles, aria-labels, figcaptions for charts whose data is JS-rendered. Metadata parity with `fetch` (same og_/canonical_url shape). `mode: "brand"` walks JSON-LD Organization/Brand/WebSite → OG/Twitter Card meta → `<link rel=icon>` → CSS custom properties → heuristic header/footer DOM; `provenance` records the winning source. Provenance enums: logo ∈ {json-ld, og:logo, link[rel=icon], heuristic, unknown}; colors ∈ {css-vars, palette-extraction, unknown}; fonts ∈ {css-vars, css-rule, inline-style, google-fonts-link, unknown}. Honesty: `name` and `logo_url` are unset when no explicit source emits them — favicons never promote to `logo_url`. `mode: "schema"` is evidence-only: LLM-sourced fields not present in source text are returned as `null` with a warning.

find_similar

Find content related to a URL or concept. Best after a successful crawl/fetch — the local cache makes recommendations cheap. Concept-only queries on a cold cache often return 0-2 weak matches; warm the cache first via `crawl` / `fetch` for materially better results. Key parameters: - url: known-good page; its content + embeddings drive similarity. - concept: free-text alternative to url. Thin cache → expect `cold_start` to fire. - max_results: default 5. - include_cached: true (default) to search cache first; false = web only. - threshold: minimum fused score (0-1, default 0.5). - include_ranking_debug: opt-in per-result `ranking_debug` { fts5_rank, embedding_rank, web_rank, rrf_score } so you can audit which signal won. - max_tokens_out / include_full_markdown / citation_format: budget + shape controls. Pass either url or concept. Three signals fused via RRF: keyword (FTS5), embeddings, optional live web. Each result carries `match_signals` with `embedding_rank`, `fts5_rank`, `fused_score`. When local signals are weak (cache empty, no hits, or concept mode returns only 1-2 cache matches), the response carries `cold_start` — pass it verbatim to the user (tune `WIGOLO_FIND_SIMILAR_COLD_START_THRESHOLD` to adjust). Returns results[], method ("hybrid" | "embedding" | "fts5" | "search"), cache_hits, search_hits, embedding_available, total_time_ms.

research

Multi-step research on a complex question. Decomposes into sub-queries, searches in parallel, fetches sources, synthesizes a cited report. Beats chaining `search` + `fetch` manually for multi-source synthesis. LLM-optional: with a synthesis LLM configured, the returned `report` is a written answer. Without one it returns a structured, cited `brief` (key_findings/highlights/sections) — YOU write the final answer from it; do not hand the user the raw structure as a weak result. For the best research quality a free Gemini API key (or any provider) is strongly recommended. Key parameters: - question: the research question. - depth: 'quick' (~15s, 2 sub-queries) | 'standard' (~40s, 4 sub-queries, default) | 'comprehensive' (~80s, 7 sub-queries). - max_sources: override per-depth source count. - include_domains / exclude_domains: scope. - schema: optional JSON Schema — structures the report. - stream: progress notifications per phase. - max_tokens_out / include_full_markdown / citation_format: budget + shape controls. Returns report (markdown with [N]), citations[], sources[], sub_queries[], depth, total_time_ms, sampling_supported, and `brief` with `topics`, `highlights`, `key_findings`, `sections` (overview.cross_references, comparison, gaps — gaps lists any named sub-entity research could not corroborate).

agent

Natural-language data gathering across sources. Plans queries + URLs from a prompt, executes in parallel, optionally extracts structured fields, synthesizes. Full step transparency. LLM-optional: with a synthesis LLM configured it writes the summary; without one it returns gathered evidence + a step log (plus schema-shaped fields when a schema is given) — YOU write the summary from the returned evidence, never present the raw step log as a poor result. For best agent results configure a free LLM key (e.g. Gemini). Key parameters: - prompt: NL description of what to gather (e.g. "pricing for the top 5 CRM tools"). - urls: optional seed URLs. - schema: optional JSON Schema — extracts matching fields from each page and merges. - max_pages: default 10. - max_time_ms: default 60000. - stream: progress notifications per step. - max_tokens_out / include_full_markdown / citation_format: budget + shape controls. Pipeline: plan → search+fetch in parallel within budget → optional schema extraction → synthesize. `steps[]` exposes every action with timing. Uses MCP sampling when supported; falls back to keyword extraction otherwise. Returns result, sources[], pages_fetched, steps[], total_time_ms, sampling_supported.

diff

Compute a diff between two markdown bodies or two URL fetches. Key parameters: - old: { url?, markdown?, content_hash? } — left-hand side. URL form reads from cache; cache miss returns a structured `cache_miss` error (no network re-fetch). - new: { url?, markdown? } — right-hand side. Same cache rules as `old`. - output: 'unified' (default, git-style patch) | 'hunks' (structured array) | 'summary' (line counts only). - granularity: 'line' (default) | 'word' | 'section'. Section walks H1/H2/H3 boundaries and tags each hunk with `section_title`. Returns `{ changed, summary, unified_diff?|hunks?, truncated? }`. `summary` always present (added/removed/modified lines + total_changed_chars). Above the 5000-line cap the engine emits `truncated: true` plus an approximate summary — never silently degrades.

watch

Schedule lazy re-checks of a URL and surface diffs on change. Persistent across sessions — jobs survive MCP server restarts. LAZY EXECUTION — NOT CRON. There is no background daemon. Checks fire ONLY when: 1. You explicitly call `watch({ action: 'check', job_id })`, OR 2. Any OTHER wigolo tool runs and the job's interval has elapsed (overdue jobs are triggered fire-and-forget in the background). A job on an idle MCP server will not fire until the next tool call. Do not expect cron-like accuracy. Key parameters: - action: 'create' | 'list' | 'check' | 'pause' | 'resume' | 'delete'. - url (create-only): must be a public http/https URL. Loopback, RFC 1918 ranges, link-local, and non-http(s) schemes are rejected at registration. - interval_seconds (create-only): minimum 60 — sub-minute polling is refused to respect target-site rate limits. - selector (create-only, optional): CSS selector for future selector-scoped diffs. Persisted now for forward compatibility; the diff is currently full-page. - notification (create-only, optional): 'inline' (default — change reports come back on the next `action: 'check'`) or a public webhook URL. Same SSRF guard as `url`. Webhook delivery is best-effort POST — no retry / no queue / no backoff. - job_id (check/pause/resume/delete). `list` returns each job's `staleness_seconds` so you can see how overdue each check is: negative = not yet due, positive = overdue by N seconds. Pair with `action: 'check'` to force one immediately. Idempotent `create`: identical url + interval + selector returns the existing `job_id` — does not duplicate the row.

Claude Desktop / Cursor

Paste into your MCP client config file to install this server.

{
    "mcpServers": {
        "wigolo": {
            "wigolo": {
                "command": "npx",
                "args": [
                    "-y",
                    "wigolo"
                ]
            }
        }
    }
}

McpServers

{
    "wigolo": {
        "command": "npx",
        "args": [
            "-y",
            "wigolo"
        ]
    }
}

<div align="center">

wigolo — the go-to web for your agent

Local-first web intelligence over MCP — no keys, no cloud, no metered bill.

<sub>works with&nbsp;&nbsp;Claude Code · Cursor · Codex · Gemini CLI · VS Code · Windsurf · Zed · Antigravity</sub>

npm
node
MCP
license
status

Quickstart · Tools · Why wigolo · Benchmark · Architecture · Configuration · Feedback · FAQ · Contribute

</div>

---

wigolo runs on your machine as an MCP server and gives an AI coding agent one durable surface for everything web-related — search, fetch, crawl, extract, cache, find-similar, research, and autonomous gather loops. The core tools need no API keys, nothing it touches leaves ~/.wigolo/, and there's no bill that grows with how much your agent thinks.

<div align="center">

wigolo demo — Claude Code answering a live web question through wigolo, no API keys

</div>

Quickstart

Requires Node ≥ 20 and ~1.5 GB of free disk. macOS, Linux, and Windows.

One command installs the local engine (search, browser, on-device models), auto-wires it into your agent, and sets up the MCP connection:

npx wigolo init --non-interactive --agents=<your-agent>

- <your-agent> — one or more of claude-code · cursor · codex · gemini-cli · vscode · windsurf · zed · antigravity (comma-separated). wigolo writes the MCP config and instructions for you — nothing else to set up.
- Any other MCP-capable agent? Omit --agents — the engine still installs headlessly, and you point your agent at wigolo's MCP server (npx wigolo mcp) yourself.

That's the whole setup — search, fetch, crawl, extract, cache, and find-similar work with no API key. Check it's healthy:

npx wigolo doctor

Not for you? npx wigolo config --uninstall --yes removes everything, cleanly.

Manual MCP setup (any other agent)

The --agents flag has a built-in installer for each agent listed above — but it can't cover every agent in the world. For anything else — your own custom or in-house agent, or any MCP-capable client we don't wire automatically yet — set wigolo up by hand: it's just another MCP server. Install the engine once, then register it:

npx wigolo init --non-interactive        # engine only: models, browser, cache — no agent wiring

Most clients use an mcpServers block in a JSON config file:

{
  "mcpServers": {
    "wigolo": {
      "command": "npx",
      "args": ["-y", "wigolo"]
    }
  }
}

wigolo with no subcommand starts the MCP stdio server (that is the default). If you installed it globally, use "command": "wigolo", "args": [] instead.

The file location — and the exact key — vary by client:

| Agent | Config file | Servers key |
|-------|-------------|-------------|
| Cursor | ~/.cursor/mcp.json | mcpServers |
| Windsurf | ~/.codeium/windsurf/mcp_config.json | mcpServers |
| Gemini CLI | ~/.gemini/settings.json | mcpServers |
| Antigravity | ~/.antigravity/mcp.json | mcpServers |
| VS Code | user mcp.json (Command Palette → MCP: Open User Configuration) | servers |
| Zed | ~/.config/zed/settings.json | context_servers |
| Claude Code | (no file) run claude mcp add wigolo --scope user -- npx -y wigolo (--scope user = global; drop it for project-only) | — |
| Codex | ~/.codex/config.toml (TOML, not JSON) | [mcp_servers.wigolo] |
| Any other | wherever it registers MCP servers | its MCP-servers key |

Codex uses TOML instead of JSON:

[mcp_servers.wigolo]
command = "npx"
args = ["-y", "wigolo"]

To enable answer synthesis (below) for a hand-wired agent, add the provider and key to the server's env:

{
  "mcpServers": {
    "wigolo": {
      "command": "npx",
      "args": ["-y", "wigolo"],
      "env": { "WIGOLO_LLM_PROVIDER": "gemini", "GEMINI_API_KEY": "<your-free-key>" }
    }
  }
}

Let an AI set it up for you

Setup is simple enough to hand off to an AI. Ask your coding agent (Claude Code, Cursor, …) — or any chat assistant (ChatGPT, Claude, Gemini) — to do it, and it can follow the steps above. Paste a prompt like:

> Set up the wigolo MCP server for my agent. wigolo is a local-first MCP server installed with npx wigolo init --non-interactive (engine only — no API keys). Then register it in my agent's MCP config as an mcpServers entry { "command": "npx", "args": ["-y", "wigolo"] }. Note the per-client differences: VS Code uses the servers key with "type": "stdio"; Zed uses context_servers; Codex uses TOML [mcp_servers.wigolo]; Claude Code uses the CLI claude mcp add wigolo --scope user -- npx -y wigolo. My agent is <name> and its MCP config is at <path, or "wherever it registers MCP servers">.

That prompt is self-contained, so even an assistant with no web access can act on it. If the assistant can browse, point it at this README (the Manual MCP setup section above has every client's exact config path) or the project's machine-readable llms.txt — both carry the full procedure, including the optional LLM-synthesis env below.

Optional — enable answer synthesis

research, agent, and search format=answer use an LLM to write the final answer. Turn them on by setting a provider and its key (in your shell, or in your agent's MCP env block). WIGOLO_LLM_PROVIDER names the LLM — set it alongside the key:

export WIGOLO_LLM_PROVIDER=gemini
export GEMINI_API_KEY=<your-key>      # free from https://aistudio.google.com/apikey — the free tier is plenty

Any provider works — use anthropic + ANTHROPIC_API_KEY, openai + OPENAI_API_KEY, or groq + GROQ_API_KEY. To stay fully local and keyless, set WIGOLO_LLM_PROVIDER=ollama (or a local server URL) instead. Gemini is suggested because its free tier is more than enough for wigolo.

Run with Docker

A prebuilt image runs the MCP server without installing Node yourself. It bundles the browser engine and on-device models, and the default command is the stdio MCP server.

docker run -i --rm -v wigolo-data:/data ghcr.io/knockoutez/wigolo

The -i flag keeps stdin open for the MCP protocol, and the volume persists the local cache and models across runs (first run downloads the models). Wire it into Claude Code:

claude mcp add wigolo --scope user -- docker run -i --rm -v wigolo-data:/data ghcr.io/knockoutez/wigolo

Any MCP client works the same way: set command to docker and args to the run flags above. The image is also on Docker Hub as towhid69420/wigolo.

The tools

| Tool | What it does |
|------|--------------|
| 🔎 search | Multi-engine web search (18 direct adapters) with rank fusion, ML cross-encoder reranking, and an explainable per-result score. Pass a query array for parallel breadth. |
| 📄 fetch | Load one URL through a tiered router (HTTP → TLS-impersonation → headless browser) that auto-escalates on anti-bot challenges or SPA shells. Clean markdown + metadata + links. |
| 🕸️ crawl | Multi-page crawl — BFS, DFS, sitemap, or map-only. Per-domain rate limits, robots.txt respect, boilerplate dedup. |
| 🧩 extract | Structured data from a page: tables, metadata, JSON-LD, brand identity, named schemas (Article / Recipe / Product / …), or any custom JSON Schema. |
| 💾 cache | Query everything already seen — keyword (BM25) or hybrid (BM25 + on-device vectors). Plus stats, clear, and change detection. |
| 🧲 find_similar | Pages similar to a URL or a concept, via 3-way fusion of keyword + semantic + live web. |
| 🧠 research | Decompose a question → fan out sub-queries → fetch sources → synthesize a cited report (or a structured brief the host LLM writes from). |
| 🤖 agent | Autonomous gather loop: plan → search → fetch → extract → synthesize, with a step log, time budget, and optional output schema. |
| 🔁 diff + ⏱️ watch | See exactly what changed on a page since last visit; re-check on a schedule and deliver changes to a webhook. |

Why it's different

wigolo isn't the free stand-in you settle for until the budget clears — it's built to hold the same line as the paid services in this lane, and it brings receipts. What actually separates it:

- Built for agents, not humans. One MCP call fans out many queries across many engines in parallel — something a serial host tool-loop can't replicate — with transparent per-result scoring and budget-aware output.
- Honest output. Stale cache, failed fetches, degraded backends, and truncation are surfaced in the result, never disguised as empty-but-successful data.
- $0 per query, free to re-query. Default search talks to public engines through direct adapters; the reranker and embeddings run on-device. Every response is cached, so asking again is instant and costs nothing.
- Private by default. Cache, embeddings, models, and config live under ~/.wigolo/. Nothing reaches a third party unless you explicitly opt into an LLM for synthesis.

wigolo is a focused web layer for one agent on one machine — not a hosted SaaS, a vector database other apps query, or a browser-automation framework. Within that lane it goes toe-to-toe with the paid services on result quality — and the meter, the key, and the data-egress simply aren't there.

Here's what one real result looks like, dissected — including the failed engine and the weak result, because those are part of the answer too:

<div align="center">

<picture>
<source media="(prefers-color-scheme: dark)" srcset="assets/promo/anatomy-dark.svg">
Anatomy of a wigolo result: explainable score decomposition, live engine telemetry, surfaced degradation, self-flagged junk — one real query, captured live
</picture>

</div>

Benchmark

> All four tools converged on the same core answer — and only one of them handed back verbatim, byte-pinned evidence while doing it.

One cold query, run live inside a single Claude Fable 5 session and fanned out to four web tools on equal footing — built-in WebSearch, wigolo, Tavily, and Exa — then reported by the agent itself under one rule: judge on the evidence alone, no favoritism. The query: when to choose logical vs streaming replication in Postgres.

The headline is in the report itself: all four tools converged on the same core answer. Same top source as the paid tools, same conclusions — parity demonstrated, not asserted. On top of that, wigolo was the only tool of the four to return verbatim quoted excerpts pinned to byte-offset source spans with citation IDs, an explainable per-result score decomposition (cross-encoder, lexical, engine consensus), and live per-engine telemetry — and when two of its results were weak, its own scorer flagged them as junk on-screen. The cloud tools earn their line too: Exa rendered the official docs' comparison matrix in full. Both edges, stated straight, by the same model that drove all four tools.

One honest query, not a leaderboard — run your own and you'll see the same shape: the keyless local tool standing shoulder to shoulder with the paid services, handing your agent evidence the others don't, at $0 with nothing leaving your machine. Here's the full run:

<div align="center">

wigolo vs built-in WebSearch, Tavily, and Exa on one real query, driven by Claude Fable 5

</div>

Same fight, different physics

The paid tools are genuinely good — that's what makes the parity interesting. The differences that remain aren't quality, they're physics:

| | wigolo | Firecrawl | Exa | Tavily |
|---|:---:|:---:|:---:|:---:|
| Multi-engine web search | ✅ | ✅ | ✅ | ✅ |
| Fetch & structured extraction | ✅ | ✅ | ✅ | ✅ |
| Whole-site crawl & map | ✅ | ✅ | — | ✅ |
| Verbatim excerpts pinned to byte-offset source spans | ✅ | — | — | — |
| Explainable per-result score decomposition | ✅ | — | — | — |
| Persistent local memory — re-query instantly, offline | ✅ | — | — | — |
| Query data stays on your machine | ✅ | — | — | — |
| API key / account | none | required | required | required |
| Cost per query | $0 | metered | metered | metered |

<sub>Feature standing as of July 2026 — check each vendor's docs for current state.</sub>

That last row is the one that compounds — agents don't ask once, they ask in bursts:

<div align="center">

<picture>
<source media="(prefers-color-scheme: dark)" srcset="assets/promo/meter-dark.svg">
The meter: a metered cloud API's cost climbs with every query while wigolo stays flat at zero dollars — illustrative pricing
</picture>

</div>

Architecture

A single Node process speaking MCP (JSON-RPC over stdio). Everything heavy is local and lazy-loaded, so a zero-key install pays nothing for the parts it isn't using.

flowchart TD
    A["🤖 AI coding agent<br/>any MCP client"]
    A -->|MCP over stdio| B["<b>wigolo</b><br/>10 tools · dynamic instructions<br/>in-process browser pool + cache + models"]

B --> C{"Tool layer"}
C --> T1["search · fetch · crawl · extract"]
C --> T2["cache · find_similar · research · agent"]

T1 --> F["⚙️ Fetch router<br/>HTTP → TLS-impersonation → headless browser<br/><i>per-domain learning</i>"]
T1 --> S["⚙️ Search<br/>18 engines → RRF fusion → cross-encoder rerank<br/><i>explainable evidence score</i>"]
T2 --> DB[("🗄️ SQLite<br/>url cache · FTS5 keyword · sqlite-vec")]
T2 --> ML["🧠 On-device ML<br/>BGE-small embeddings (384d)<br/>MiniLM cross-encoder reranker"]

F -.->|optional| LLM["☁️ Cloud LLM<br/>synthesis only · opt-in"]
S -.->|optional| SX["🔀 Aggregator backend<br/>opt-in legacy / hybrid"]

F --> WEB["🌍 Public web"]
S --> WEB

style B fill:#7c3aed,stroke:#5b21b6,color:#fff
style WEB fill:#0ea5e9,stroke:#0369a1,color:#fff
style DB fill:#1e293b,stroke:#334155,color:#fff
style LLM stroke-dasharray: 5 5
style SX stroke-dasharray: 5 5

- Code beats model. Deterministic work — canonicalization, rank fusion, dedup, schema matching, hashing — never touches an LLM. The model is reserved for judgment, opt-in, and capped per request. LLM-filled fields are checked against the source and nulled if absent, so hallucinations don't reach your output.
- Routing on observable signals. The fetch ladder escalates to a real browser on what it sees — SPA markers, challenge bodies, thin content — not domain guesses. It learns per-domain and unlearns when a site stops needing it.
- Transparent, honest results. Every result carries a score breakdown and a query-understanding block; degraded state is always surfaced, never hidden.

<div align="center">

<picture>
<source media="(prefers-color-scheme: dark)" srcset="assets/promo/ladder-dark.svg">
wigolo's fetch escalation ladder: plain HTTP, then TLS-impersonation on challenge signals, then a headless browser on SPA shells — learned and unlearned per domain
</picture>

<picture>
<source media="(prefers-color-scheme: dark)" srcset="assets/promo/fusion-dark.svg">
18 search engines fused with reciprocal rank fusion, reranked by an on-device cross-encoder — one scored answer, real output scores
</picture>

</div>

Configuration

A clean install works out of the box. A few settings meaningfully raise output quality — set them as environment variables or in your agent's MCP env block.

# 1. Synthesis — the biggest lever. research / agent / search-answer need an LLM

to write the final text. Set the provider AND its key (a key alone is ignored).

export WIGOLO_LLM_PROVIDER=gemini # names the LLM; free tier is plenty (or anthropic/openai/groq) export GEMINI_API_KEY=<your-key> # that provider's key (ANTHROPIC_API_KEY / OPENAI_API_KEY / …)

...or fully local & keyless: export WIGOLO_LLM_PROVIDER=ollama (or a local http URL)

2. Wider retrieval funnel

export WIGOLO_SEARCH=hybrid # core engines + aggregator fallback export WIGOLO_GITHUB_TOKEN=... # GitHub code search 10 → 30 req/min + org-private

3. Land more fetches, stay warm

export WIGOLO_TLS_TIER=auto # per-domain TLS-impersonation past Cloudflare/DataDome export WIGOLO_EAGER_WARMUP=1 # pay the ~1s model load up front, not on first search

For repeated interactive use, run wigolo serve so the browser pool, embeddings, and reranker stay resident across calls. To stay 100% on-device, a local LLM endpoint + WIGOLO_TLS_TIER=auto is the honest minimal set.

Per-call habits that pay off: query arrays (["a","b","c"]) for parallel breadth · search_depth: "deep" for queries that matter · include_domains as a hard filter for docs lookups.

<details>
<summary><b>CLI commands</b></summary>

| Command | What it does |
|---------|--------------|
| wigolo / wigolo mcp | Start the MCP stdio server (the default command). |
| wigolo init | Set up wigolo: install components, wire into detected agents. --non-interactive --agents=<csv> --provider=<name> --search=<backend> for CI. |
| wigolo setup mcp | Re-write just the MCP server entries, without the full wizard. |
| wigolo doctor | Cold-start health check — no network fetches. |
| wigolo verify | End-to-end smoke test (fetch, crawl, extract, search, rerank, embed). |
| wigolo serve | HTTP daemon — keeps subsystems warm across multiple clients. |
| wigolo shell | Interactive REPL (--json for piping). |
| wigolo config | Settings TUI; or headless --set K=V, --export, --import, --cleanup, --uninstall --yes. |
| wigolo status | Plain-text status summary. |
| wigolo health | Ping a running daemon's /health. |
| wigolo backfill | Embed cached pages that have no vector yet (--batch-size, --dry-run). |
| wigolo plugin add\|list\|remove | Manage custom extractor / search-engine plugins. |
| wigolo uninstall | Remove wigolo from agent configs (keeps your cache). |

</details>

<details>
<summary><b>Environment variables — search &amp; engines</b></summary>

| Var | Default | Effect |
|-----|---------|--------|
| WIGOLO_SEARCH | core | core (direct engines) / searxng (legacy) / hybrid (core + fallback). |
| BRAVE_API_KEY | — | When set, Brave joins the engine pool (env-only, never persisted). |
| WIGOLO_GITHUB_TOKEN | — | Lifts GitHub code search 10 → 30 req/min; enables org-private search (env-only). |
| SEARXNG_URL | — | External aggregator URL; when set, skips local bootstrap. |
| SEARXNG_MODE | native | native (Python venv) or docker. |
| SEARXNG_PORT | 8888 | Port for the native aggregator. |
| SEARXNG_QUERY_TIMEOUT_MS | 8000 | Per-query timeout to the aggregator. |
| WIGOLO_MULTI_QUERY_CONCURRENCY | 5 | Max parallel (query × engine) tasks. |
| WIGOLO_MULTI_QUERY_MAX | 10 | Max unique queries after normalization. |
| WIGOLO_QUERY_EXPAND_VARIANTS | 5 | Heuristic query-expansion variants. |
| SEARCH_NARROW_RENDER_MAX_CANDIDATES | 3 | Max candidates for which a domain-scoped (include_domains) search renders result pages in the browser engine during enrichment — recovers real content from JS-heavy documentation sites. Bounded to a few URLs; broad searches never escalate. 0 disables. |

</details>

<details>
<summary><b>Environment variables — fetch, network &amp; TLS</b></summary>

| Var | Default | Effect |
|-----|---------|--------|
| USER_AGENT | rotating Chrome UAs | Override the User-Agent header. |
| FETCH_TIMEOUT_MS | 10000 | HTTP request timeout. |
| FETCH_MAX_RETRIES | 2 | Retry budget for 429 / 502 / 503 / network errors. |
| MAX_REDIRECTS | 5 | Manual-mode redirect cap. |
| PLAYWRIGHT_LOAD_TIMEOUT_MS | 15000 | Browser page.load wait. |
| PLAYWRIGHT_NAV_TIMEOUT_MS | 30000 | Browser navigation timeout. |
| SEARCH_FETCH_TIMEOUT_MS | 15000 | Per-result hydration fetch in search. |
| SEARCH_TOTAL_TIMEOUT_MS | 30000 | Aggregate search budget. |
| USE_PROXY / PROXY_URL | false / — | Route fetch through a proxy. |
| WIGOLO_TLS_TIER | off | off / auto (per-domain learned) / on (always try TLS first). |
| WIGOLO_TLS_BROWSER | chrome_142 | TLS fingerprint profile (<browser>_<version>). |
| WIGOLO_TLS_SUCCESS_THRESHOLD | 3 | Successes before a domain flips to TLS-first. |

</details>

<details>
<summary><b>Environment variables — browser pool &amp; auth</b></summary>

| Var | Default | Effect |
|-----|---------|--------|
| MAX_BROWSERS | 3 | Max concurrent contexts per browser type. |
| BROWSER_IDLE_TIMEOUT | 60000 | Idle context eviction (ms). |
| BROWSER_FALLBACK_THRESHOLD | 3 | HTTP failures on a domain before forcing the browser. |
| WIGOLO_BROWSER_TYPES | auto (all 3) | CSV of browsers to use (chromium, firefox, webkit). |
| WIGOLO_CDP_URL | — | Chrome DevTools endpoint for a remote / logged-in browser. |
| WIGOLO_AUTH_STATE_PATH | — | Playwright storageState.json (cookies / localStorage). |
| WIGOLO_CHROME_PROFILE_PATH | — | Full Chrome User Data dir (copied to temp per use). |

</details>

<details>
<summary><b>Environment variables — cache &amp; crawl</b></summary>

| Var | Default | Effect |
|-----|---------|--------|
| CACHE_TTL_SEARCH | 86400 | Search result cache TTL (s). |
| CACHE_TTL_CONTENT | 604800 | Page content cache TTL (7 days). |
| WIGOLO_FAST_STALE_MAX_HOURS | 24 | In cache mode, accept entries up to this age. |
| WIGOLO_FAST_TIMEOUT_MS | 800 | Tight timeout for cache-mode fallback fetches. |
| CRAWL_CONCURRENCY | 2 | Per-public-domain concurrent fetches. |
| CRAWL_DELAY_MS | 500 | Per-public-domain inter-request delay. |
| CRAWL_PRIVATE_CONCURRENCY | 10 | Per-private-domain concurrency (localhost / RFC1918). |
| CRAWL_PRIVATE_DELAY_MS | 0 | Per-private-domain delay. |
| RESPECT_ROBOTS_TXT | true | When false, robots.txt is not fetched. |
| VALIDATE_LINKS | true | When false, broken-link probe is skipped. |
| WIGOLO_CRAWL_INDEX | — | 1 → crawled pages enqueued for embedding. |
| WIGOLO_WAIT_FOR_INDEX | — | 1 → embedding queue runs synchronously per page. |

</details>

<details>
<summary><b>Environment variables — reranker, embedding &amp; relevance</b></summary>

| Var | Default | Effect |
|-----|---------|--------|
| WIGOLO_RERANKER | onnx | onnx (cross-encoder) / none (consensus + authority + recency boosts only). |
| WIGOLO_RERANKER_MODEL | Xenova/ms-marco-MiniLM-L-6-v2 | Cross-encoder model ID. |
| WIGOLO_RERANKER_IDLE_TIMEOUT_MS | 300000 | Hold the model warm 5 min after last use. |
| WIGOLO_EMBEDDING_MODEL | BAAI/bge-small-en-v1.5 | Embedding model (384-dim). |
| WIGOLO_EMBEDDING_IDLE_TIMEOUT | 1800000 | Idle unload (30 min). |
| WIGOLO_EMBEDDING_MAX_TEXT_LENGTH | 8000 | Truncation before embedding. |
| WIGOLO_RELEVANCE_THRESHOLD | 0 | Min relevance for the agent's post-fetch filter. |
| WIGOLO_FIND_SIMILAR_COLD_START_THRESHOLD | 0.02 | Fused score below which find_similar emits cold_start. |

</details>

<details>
<summary><b>Environment variables — LLM integration (all optional)</b></summary>

| Var | Default | Effect |
|-----|---------|--------|
| WIGOLO_LLM_PROVIDER | — | anthropic / openai / gemini / groq / custom URL (Ollama, vLLM, LM Studio). |
| WIGOLO_LLM_MODEL | — | Universal model override. |
| WIGOLO_LLM_MODEL_{ANTHROPIC\|OPENAI\|GEMINI\|GROQ} | — | Per-provider model override (highest precedence). |
| WIGOLO_LLM_MAX_CALLS_PER_REQUEST | 1 | Hard ceiling on LLM calls per tool invocation. |
| WIGOLO_LLM_CACHE_TTL_DAYS | 7 | LLM response cache TTL. |
| WIGOLO_LOCAL_LLM | off | Opt-in keyless local language model tier: off (default) / auto (auto-detect a local model server) / an explicit http(s):// endpoint. Off keeps the keyless path unchanged. |
| WIGOLO_LOCAL_LLM_MODEL | — | Preferred model name for the local tier; unset auto-picks an installed model. |
| WIGOLO_LOCAL_LLM_BASE_URL | http://localhost:11434 | Endpoint probed when WIGOLO_LOCAL_LLM=auto (falls back to WIGOLO_LLM_BASE_URL, then the default local server). |
| ANTHROPIC_API_KEY / OPENAI_API_KEY | — | Read on every call; never persisted. |
| GEMINI_API_KEY / GOOGLE_API_KEY | — | Gemini provider key (either name; read on every call, never persisted). |
| GROQ_API_KEY | — | Same. |
| WIGOLO_LLM_API_KEY | — | Generic key for whichever provider WIGOLO_LLM_PROVIDER names. The provider-specific var wins; ignored during auto-detect. |

Keys can also live in the OS keychain or an AES-encrypted file (wigolo init / wigolo config) — never in config.json.

</details>

<details>
<summary><b>Environment variables — daemon, warmup, paths, logging &amp; misc</b></summary>

| Var | Default | Effect |
|-----|---------|--------|
| WIGOLO_DATA_DIR | ~/.wigolo | Root for cache, models, keys, plugins, aggregator venv. |
| WIGOLO_CONFIG_PATH | ${DATA_DIR}/config.json | Persisted config path. |
| WIGOLO_DAEMON_PORT | 3333 | Listen port for wigolo serve. |
| WIGOLO_DAEMON_HOST | 127.0.0.1 | Bind address. |
| WIGOLO_EAGER_WARMUP | — | 1 → pre-warm embed + rerank on startup (fire-and-forget). |
| WIGOLO_BOOTSTRAP_MAX_ATTEMPTS | 3 | Aggregator bootstrap retry limit. |
| WIGOLO_HEALTH_PROBE_INTERVAL_MS | 30000 | Background backend-health probe period. |
| WIGOLO_PLUGINS_DIR | ${DATA_DIR}/plugins | Plugin discovery root. |
| LOG_LEVEL | info | debug / info / warn / error. |
| LOG_FORMAT | json | json or human-friendly text. |
| WIGOLO_TELEMETRY | — | 1 → local NDJSON event log (off by default, no PII). |
| WIGOLO_TELEMETRY_ENDPOINT | — | Also POST events fire-and-forget to this URL. |
| WIGOLO_TUI_REDUCED_MOTION | — | 1 → disable TUI spinners / animations. |

</details>

<details>
<summary><b>Common per-call options (tool arguments)</b></summary>

| Option | Tools | Notes |
|--------|-------|-------|
| mode | fetch, search, crawl, extract, find_similar | cache (fast, stale-OK) / default (smart routing) / stealth (full browser, no cache). |
| search_depth | search | ultra-fast (cache only) / fast / balanced (default) / deep (evidence + rerank highlights). |
| query | search | string or string[] — arrays fan out in parallel. |
| include_domains / exclude_domains | search, find_similar, research | Hard whitelist / blacklist (host-suffix match). |
| format | search | answer / stream_answer — triggers LLM synthesis with citations. |
| citation_format | search, crawl, research, agent | numbered / json / anthropic_tags. |
| time_range / from_date / to_date | search | Recency bounds. |
| render_js | fetch | auto / always / never. |
| use_auth | fetch, crawl | Route through configured auth (CDP > Chrome profile > storage state). |
| actions | fetch | Sequential browser actions (click, type, wait, wait_for, scroll, screenshot). |
| section | fetch | Extract a markdown subtree at a heading. |
| strategy | crawl | bfs / dfs / sitemap / auto / map. |
| mode (extract) | extract | selector / tables / metadata / schema / structured / brand. |
| named_schema | extract | Article / Recipe / Product / CodeSnippet / Paper / EventListing. |
| depth | research | quick / standard / comprehensive. |
| max_pages / max_time_ms | agent | Per-invocation page cap (default 3) and wall-clock budget. |
| max_tokens_out | most | Aggregate output-token budget (default 4000). |
| include_full_markdown | fetch, crawl, research, agent | false → evidence excerpts instead of full bodies. |

</details>

Beta & feedback

wigolo is in public beta. Everything documented here works and is held to a 6,000-test suite — beta is about the polish bar, not stability. It stays beta until enough people have used it, kicked it, and starred it that calling it v1 means something.

That makes your feedback the whole game right now. Every report is read, usually the same day:

- 🐛 Report a bug — broke, misbehaved, surprised you
- 💡 Request a feature — something it should do
- 💬 Ask anything — questions, setups, show & tell

And if wigolo earns a place in your setup, the ways to keep it alive: a ⭐ star (it's how open source gets found), a ☕ coffee (there's no paid tier and never will be), or just an email — it goes straight to the one developer who wrote the code.

FAQ

<details>
<summary><b>Free? What's the catch?</b></summary>

No catch by design. The expensive parts — ranking, embeddings, the browser engine — run on your hardware, so there's no per-query cost to recover and no reason for a meter. Sustained by donations; the AGPL license legally prevents a bait-and-switch into a closed hosted product.

</details>

<details>
<summary><b>Is the quality really on par with the paid services?</b></summary>

Run one query and judge — the benchmark section above is a live 4-way run, not a chart. Everyday agent queries land at parity; the paid tools still win some deep-extraction edge cases, and crawling is where wigolo is strongest. Every result shows its scoring, so you don't have to take anyone's word for it.

</details>

<details>
<summary><b>Won't public search engines block or rot?</b></summary>

It's engineered for exactly that: 18 engines fused with rank fusion (any one failing barely moves results), a tiered fetch ladder with per-domain learning, and an optional aggregat

No reviews yet — be the first

Sign in to leave a review

Use Google, GitHub, or an email account so ratings stay tied to real people.

Email sign in

No reviews posted yet.