Crawl4AI

- JS-aware crawling: full headless-browser rendering withwait_until=networkidledefaults
- Schema-based extraction: derive a CSS selector schema once via LLM, apply it forever with no further LLM cost
- LLM extraction: per-request structured extraction when a schema is not worth deriving
- Content filtering: BM25 relevance filter and quality-based pruning, plain markdown or markdown-fit output
- Concurrent batch crawling: multi-URL processing with per-job concurrency caps
- Session management: persistent sessions for authenticated, multi-step flows
- CLI and SDK: both thecrwlcommand-line tool and thecrawl4aiPython SDK

Clone the repo into the skills directory your agent host loads from:

# Claude Code git clone https://github.com/brettdavies/crawl4ai-skill.git ~/.claude/skills/crawl4ai

For other agent hosts (Codex, Cursor, OpenCode, Cline, custom agents), clone into whichever directory your host scans for SKILL.md-format bundles. Refer to your host's documentation for the skills directory location. The bundle root containsSKILL.md, so the skill registers automatically once the directory is on the host's skills search path.

The skill calls into the Crawl4AI Python library, which must be installed in the runtime your agent uses:

pip install crawl4ai crawl4ai-setup crawl4ai-doctor

crawl4ai-doctorvalidates the install and confirms a headless browser is available.

crwl https://example.com -c "wait_until=networkidle,page_timeout=60000" -o markdown crwl https://example.com -o json -v --bypass-cache

crwl https://docs.example.com -o markdown > docs.md

# Derive the schema once (uses LLM) ./scripts/generate_schema.py https://shop.example.com "products with name, price, image" shop_schema.json # Apply the saved schema (no LLM cost per request) ./scripts/extract_with_schema.py https://shop.example.com shop_schema.json products.json

News aggregation with relevance filtering

for url in news1.com news2.com news3.com; do crwl "https://$url" -f templates/filter_bm25.yml -o markdown-fit done

Contributions welcome. Open a pull request.

An MCP server that lets an AI agent (like Claude) read data from Discord servers through your own account. It pairs with a BetterDiscord plugin, so the Python side never handles your token — data is read straight from the already-authenticated Discord client.

MCP server + Chrome extension that connects Claude Code to Chromium browsers (Arc, Vivaldi, Brave) where the official Claude in Chrome extension doesn't work.

KYB due diligence, competitive intelligence, and strategic accounts research for AI agents. KYB covers corporate registries, sanctions screening (OFAC/UN/EU/UK), FCA Register lookup, directors, UBOs, and adverse media. CI covers pricing, hiring, product, sentiment, and corporate events. New accounts get 3 free trial calls. Pay-per-call after: $0.40 per /findings, $2.00 per /run with Claude analysis. Top up from $20 at getsonar.report.

Scrape Google Maps business leads (names, phones, emails, websites, ratings) from Claude, Cursor or any MCP client via the gmapsscraper.io API.

The Import.io Web Scraper MCP server brings hosted scraping, browser automation, proxy routing, captcha handling, and structured data extraction into your AI tools. Point Claude, ChatGPT, Cursor, or any MCP client at our hosted endpoint. Import.io runs the scraping engine, you just connect.

Screenshot & Render API for AI Agents. MCP Server lets Claude, Cursor capture webpages and render HTML.

Give Claude web browsing powers - fetch webpages, extract data, discover sitemaps, summarize content

Fetches content from deepwiki.com and converts it into LLM-readable markdown.

Extract documentation for AI agents from any site with llms.txt support. Features MCP server, REST API, batch processing, and multiple export formats.