CrawlAPI

Official JavaScript/TypeScript SDK forCrawlAPI— web scraping API built for AI agents.

const CrawlAPI = require('crawlapi-js'); const client = new CrawlAPI({ apiKey: 'YOUR_RAPIDAPI_KEY' }); // Scrape a URL → clean markdown const result = await client.scrape('https://example.com'); console.log(result.data.markdown); // Scrape multiple URLs in parallel const batch = await client.batch([ 'https://example.com', 'https://example.org' ]); // Search + scrape top results in one call const search = await client.search('LangChain web scraping 2025', { num: 5 });

Available onRapidAPI. Free tier: 50 calls/day, no credit card.

const result = await client.scrape('https://example.com', { formats: ['markdown'], // 'markdown' | 'html' | 'text' | 'structured' waitFor: 1000, // ms to wait after page load (JS-heavy pages) timeout: 30000 // max ms (default 30s, max 60s) }); console.log(result.data.markdown); console.log(result.data.metadata.title);

Scrape up to 10 URLs in parallel. Failed URLs return an error field rather than failing the whole request.

const result = await client.batch([ 'https://example.com', 'https://example.org', 'https://example.net' ]); result.data.forEach(item => { if (item.success) console.log(item.data.markdown); else console.error(item.url, item.error); });

Search the web and automatically scrape the top N results.

const result = await client.search('best pizza in New York', { num: 5, // 1-10 results (default 5) formats: ['markdown'] }); result.data.forEach(item => { console.log(item.title, item.url); if (item.success) console.log(item.data.markdown); });

const { Tool } = require('langchain/tools'); const CrawlAPI = require('crawlapi-js'); const client = new CrawlAPI({ apiKey: process.env.RAPIDAPI_KEY }); const webTool = new Tool({ name: 'WebScraper', description: 'Scrape any URL and return clean markdown content. Input should be a URL.', func: async (url) => { const result = await client.scrape(url); return result.data.markdown; } });

const CrawlAPI = require('crawlapi-js'); const client = new CrawlAPI({ apiKey: process.env.RAPIDAPI_KEY }); // Tool definition const tools = [{ type: 'function', function: { name: 'scrape_url', description: 'Scrape a webpage and return its content as clean markdown', parameters: { type: 'object', properties: { url: { type: 'string', description: 'The URL to scrape' } }, required: ['url'] } } }]; // Handle tool call async function handleToolCall(url) { const result = await client.scrape(url); return result.data.markdown; }

All endpoints accept standard JSON POST requests:

POST https://crawlapi.net/v1/scrape X-RapidAPI-Key: YOUR_KEY Content-Type: application/json { "url": "https://example.com", "formats": ["markdown"] }

Full OpenAPI spec:crawlapi.net/openapi.json

import { CrawlAPI, ScrapeResponse } from 'crawlapi-js'; const client = new CrawlAPI({ apiKey: process.env.RAPIDAPI_KEY! }); const result: ScrapeResponse = await client.scrape('https://example.com');

CrawlAPI ships with a built-inMCP server— plug it directly into any MCP-compatible AI client.

{ "mcpServers": { "crawlapi": { "command": "node", "args": ["/path/to/crawlapi-js/mcp-server.js"], "env": { "CRAWLAPI_KEY": "your_rapidapi_key" } } } }

3. Restart Claude Desktop.You'll see three new tools:

- scrape_url— scrape any page to markdown
- batch_scrape— scrape up to 10 URLs in parallel
- search_and_scrape— search + scrape top results in one shot

Same config, different location. Check your editor's MCP docs. Themcp-server.jsfile works with any stdio-transport MCP client.

"Usescrape_urlto fetch web pages as clean markdown. Usesearch_and_scrapeto research topics from the live web. Both support JavaScript-rendered pages."

Get your API key atRapidAPI— free tier included.

- 🌐crawlapi.net
- 📦RapidAPI listing
- 📄OpenAPI spec
- 🤖llms.txt
- 🔌MCP server

AI tools for web scraping, crawling, browser control, and web search via the Oxylabs AI Studio API.