Skrapr

🎉 Cloud version available["Your AI Agent"] l1@--> | scrape_with_schema | skrapr["Skrapr"] subgraph server [Skrapr MCP Server] skrapr l2@--> | use tool | playwright["Playwright MCP"] end playwright l3@--> |navigates| website["Website"] skrapr l4@--> | think with | openai["Azure OpenAI"] skrapr l5@-.-> | json data | agent classDef customer stroke:#00f, stroke-width: 2 classDef external stroke:#63af3e,stroke-width: 2,stroke-dasharray: 4 classDef group fill:#ffffa455, stroke:#ff7c7c, stroke-width: 2 agent:::customer agent:::someclass website:::external openai:::external server:::group classDef animate stroke-dasharray: 20,stroke-dashoffset: 900,animation: dash 25s linear infinite; class l1 animate l2@{ animate: true } l3@{ animate: true } l4@{ animate: true } l5@{ animate: true }

- scrape_with_schema:

- Title: Extract data from a website based on a JSON schema
- Description: Navigate and scrape the given website to extract data matching the structure of the provided JSON Schema.
- Parameters:

- url: The URL of the website to extract data from.
- jsonSchema: The JSON Schema describing the data structure to extract.
- instruction: Optional instructions to guide how the scraping should be performed.

Once configured, you can use Skrapr's capabilities directly in any MCP client (Claude Desktop,n8n,librechat, ...).

Extract product information from all categories at https://example.com/products. I want name, description and price for each product.

{ "type":"object", "properties":{ "products":{ "type":"array", "items":{ "type":"object", "properties":{ "name":{ "type":"string" }, "price":{ "type":"string" }, "description":{ "type":"string" } } } } } }

{ "products": [ { "name": "Product 1", "price": "10", "description": "Description of product 1" }, { "name": "Product 2", "price": "20", "description": "Description of product 2" } ] }

For a typical MCP Host integration (Claude Desktop), here's how to set it up:

{ "mcpServers": { "skrapr": { "command": "dotnet", "args": [ "run", "--project", "path/to/Skrapr/Skrapr.csproj" "-e", "AzureOpenAi__ApiKey=your-api-key", "-e", "AzureOpenAi__Endpoint=https://your-resource.openai.azure.com/", "-e", "AzureOpenAi__DeploymentName=your-deployment-name", "-e", "PlaywrightMcp__IsLocal=true", ] } } }

docker run \ -p 80:80 \ -e AzureOpenAi__ApiKey=your-api-key \ -e AzureOpenAi__Endpoint=https://your-resource.openai.azure.com/ \ -e AzureOpenAi__DeploymentName=your-deployment-name \ -e PlaywrightMcp__IsLocal=true \ skrapr

The MCP server will be available athttp://localhost:5000.

{ "mcpServers": { "skrapr": { "url": "http://localhost:5000/sse" } } }

I am hosting a Skrapr instance on Cloud with everything configured.

If you are interested to try it out,reach me hereto get access to the beta.

{ "mcpServers": { "skrapr": { "url": "https://[ask-for-url]/sse" } } }

Playwright can be executed locally or remotely.

Setting environment variablePlaywrightMcp__IsLocal=truewill make Playwright run locally.

You need to have Node.js installed to make it work.

It usesnpx @playwright/mcpto run Playwright.

Setting environment variablePlaywrightMcp__Endpointwill use Playwright on a remote host.

Skrapr uses an AI agent internally aiming to :

- Understand website structure and content
- Autonomously navigate across the website
- Make strategic navigation decisions to locate the requested information
- Produce a structured JSON result based on a provided JSON schema.

- Microsoft's Semantic Kernelto build the agent
- Playwright MCPto expose tools for websites navigation
- Azure OpenAIfor AI capabilities and data extraction according to user-defined JSON schemas

Thanks to MCP (Model Context Protocol), you can connect Skrapr to any agent whatever the language.

Interact with WebScraping.AI for web data extraction and scraping.