search-scrape

CortexScout (cortex-scout) — Search and Web Extraction Engine for AI Agents

CortexScout is the Deep Research & Web Extraction module within the Cortex-Works ecosystem.

Designed for agent workloads that require token-efficient web retrieval, reliable anti-bot handling, and optional Human-in-the-Loop (HITL) fallback.

CortexScout provides a single, self-hostable Rust binary that exposes search, extraction, andstateful browser automationcapabilities over MCP (stdio) and an optional HTTP server. Output formats are structured and optimized for downstream LLM use.

It is built to handle the practical failure modes of web retrieval (rate limits, bot challenges, JavaScript-heavy pages) through progressive fallbacks: native retrieval → Chromium CDP rendering →Stateful E2E Testing→ HITL workflows.

Legacy names remain callable as compatibility aliases (web_search_json,web_fetch_batch,web_crawl,fetch_then_extract,human_auth_session). Agents should prefer the unified primary tools above.

While CortexScout runs as a standalone tool today, it is designed to integrate with CortexDB and CortexStudio for multi-agent scaling, shared retrieval artifacts, and centralized governance.

🎭 The "Playwright Killer" (Stateful Browser Automation)

CortexScout includes a built-in, stateful CDP automation engine designed specifically for AI Agents, completely replacing heavy frameworks like Playwright or Cypress for E2E testing workflows.

The main tradeoff versus raw Playwright MCP is packaging, not capability shape: Cortex Scout keeps the browser surface inside one stateful omni-tool so agents spend fewer turns and fewer tokens coordinating multi-step flows.

This repository includes captured evidence artifacts that validate extraction and HITL flows against representative protected targets.

Seeproof/README.mdfor methodology and raw outputs.

./cortex-scout --port 5000 curl http://localhost:5000/health

git clone https://github.com/cortex-works/cortex-scout.git cd cortex-scout cargo build --release --manifest-path mcp-server/Cargo.toml --bin cortex-scout-mcp

cargo build --release --manifest-path mcp-server/Cargo.toml --all-features --bin cortex-scout-mcp

// mcp.json (global): top-level key is "servers" // settings.json (workspace): use "mcp.servers" instead { "servers": { "cortex-scout": { "type": "stdio", "command": "env", "args": [ "RUST_LOG=warn", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS=90", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SCRAPE_URL=90", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SEARCH_STRUCTURED=120", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_VISUAL_SCOUT=45", "CORTEX_SCOUT_BROWSER_LAUNCH_TIMEOUT_SECS=12", "CORTEX_SCOUT_BROWSER_TAB_PROBE_TIMEOUT_SECS=4", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS=20", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_INITIAL_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_RETRY_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_FORCED_CDP_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_NATIVE_CDP_FALLBACK=25", "SEARCH_ENGINES=google,bing,duckduckgo,brave", "LANCEDB_URI=/YOUR_PATH/cortex-scout/lancedb", "HTTP_TIMEOUT_SECS=30", "MAX_CONTENT_CHARS=10000", "/YOUR_PATH/cortex-scout/mcp-server/target/release/cortex-scout-mcp" ] } } }

Default behavior is direct/no-proxy. AddIP_LIST_PATHandPROXY_SOURCE_PATHonly if you want proxy tools available. If you wantproxy_controlavailable without routing normal traffic through proxies, pointIP_LIST_PATHat an emptyip.txtfile and let agents populate it on demand.

Important:Always useRUST_LOG=warn, notinfo. Atinfolevel, the server emits hundreds of log lines per request to stderr, which can confuse MCP clients that monitor stderr.

Windows:Windows has noenvcommand. Use thecommand+envobject format instead — seedocs/IDE_SETUP.md.

{ "servers": { "cortex-scout": { "type": "stdio", "command": "env", "args": [ "RUST_LOG=warn", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS=90", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SCRAPE_URL=90", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_SEARCH_STRUCTURED=120", "CORTEX_SCOUT_TOOL_TIMEOUT_SECS_VISUAL_SCOUT=45", "CORTEX_SCOUT_BROWSER_LAUNCH_TIMEOUT_SECS=12", "CORTEX_SCOUT_BROWSER_TAB_PROBE_TIMEOUT_SECS=4", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS=20", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_INITIAL_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_CDP_RETRY_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_FORCED_CDP_ATTEMPT=25", "CORTEX_SCOUT_SCRAPE_STAGE_TIMEOUT_SECS_NATIVE_CDP_FALLBACK=25", "SEARCH_ENGINES=google,bing,duckduckgo,brave", "LANCEDB_URI=/YOUR_PATH/cortex-scout/lancedb", "HTTP_TIMEOUT_SECS=30", "MAX_CONTENT_CHARS=10000", "OPENAI_BASE_URL=https://openrouter.ai/api/v1", "OPENAI_API_KEY=sk-or-v1-...", "DEEP_RESEARCH_LLM_MODEL=moonshotai/kimi-k2.5", "DEEP_RESEARCH_ENABLED=1", "DEEP_RESEARCH_SYNTHESIS=1", "DEEP_RESEARCH_SYNTHESIS_MAX_TOKENS=4096", "/YOUR_PATH/cortex-scout/mcp-server/target/release/cortex-scout-mcp" ] } } }

Createcortex-scout.jsonin the same directory as the binary (or repository root). All fields are optional; environment variables act as fallback.

{ "deep_research": { "enabled": true, "llm_base_url": "http://localhost:1234/v1", "llm_api_key": "", "llm_model": "lfm2-2.6b", "synthesis_enabled": true, "synthesis_max_sources": 3, "synthesis_max_chars_per_source": 800, "synthesis_max_tokens": 1024 } }

- Callmemory_searchbefore any new research run — skip live fetching if similarity ≥ 0.60 andskip_live_fetchistrue.
- For topic discovery useweb_searchfor URL-only discovery, orweb_search(include_content=true)to search and scrape top results in one round-trip.
- For known URLs useweb_fetch(mode="single")withoutput_format="clean_json", and setquery+strict_relevance=trueto keep only relevant sections.
- On 403/429: callproxy_controlwithaction:"grab"to refresh the proxy list, then retry withuse_proxy:true.
- For auth-gated pages: runvisual_scoutwhenauth_risk_score >= 0.4, then usehitl_web_fetch(auth_mode="challenge")for CAPTCHA walls orhitl_web_fetch(auth_mode="auth")for login walls.
- For deep research:deep_researchhandles multi-hop search + scrape + LLM synthesis automatically. Tunedepth(1–3) andmax_sourcesper run cost budget.
- For UI automation and E2E testing: usescout_browser_automatewith step arrays for tabs, locator assertions, screenshots/PDF, route mocks, file uploads, and browser-state setup. If blocked by first-time login/CAPTCHA, callscout_agent_profile_auth, then resume automation.

Why doesdeep_researchwith Ollama orqwen3.5sometimes fail or fall back to heuristic mode?

Some reasoning-capable local models return OpenAI-compatible/v1/chat/completionsresponses withmessage.reasoningpopulated butmessage.contentempty. Cortex Scout now retries local Ollama endpoints through native/api/chatwiththink:falsewhen that pattern is detected.

Recommended config for local 4B-class Ollama models:

- llm_api_key: ""incortex-scout.jsonis valid and means "no auth required"
- Keepsynthesis_max_sourcesat1-2
- Keepsynthesis_max_chars_per_sourcearound600-1000
- Keepsynthesis_max_tokensaround512-768

If you still see slow or unstable synthesis, reducesynthesis_max_sourcesbefore increasing token limits.

Why do I see Chromium profile lock errors?

Each headless request uses a unique temporary profile, so normal scraping and deep_research are safe from profile lock races. Only HITL flows (likehitl_web_fetch) using a real browser profile can hit a lock if you run them concurrently or have Brave/Chrome open on the same profile. To avoid: run HITL calls one at a time, and close all browser windows before reusing a profile.
- Use a recent build (2026-03-05 or newer)
- Avoid persistent profile paths unless you need a logged-in session
- Run HITL/profile flows sequentially
- Close all browser windows before reusing a profile
- Let Cortex Scout use its own temp profiles for concurrent research

My MCP client connects but tools fail or time out immediately. What should I check first?

Search global news using natural language. Webz.io News Search API returns the most relevant articles and content, with filters for source, country, language, date, sentiment, and category.

A Google search server using Playwright for parallel keyword searches.

Scrapes Google search results using a headless browser. Requires Chrome to be installed.

Adds powerful web scraping and search capabilities to LLM clients like Cursor and Claude.

Scrape, crawl, and extract data from any website using the Firecrawl API.

Give Claude, Cursor, ChatGPT, Kilo, and other MCP clients access to scraping, web search, screenshots, and network tools.

A collection of servers for file system operations, Google search, web automation, and executing terminal commands.

Hosted, Stateless & Multitenant Firecrawl MCP server enables AI assistants to crawl, scrape, and extract structured web data through Firecrawl.

Search, extract, crawl, map, and research the web — from any AI agent or terminal.

A server for web scraping, Google searches, and website URL lookups using the Olostep API.