Agenty
About
Web scraping, crawling, and change detection with AI
Details
- Author
- Unknown
- Categories
- Web Scraping, Automation, AI
Jump to
- Convert pages to Markdown— Ask to convert a URL or raw HTML into clean, readable Markdown with themarkdowntool for LLM-friendly content.
- Capture screenshots— Usescreenshotto capture full-page or clipped PNG/JPEG images with custom viewports and device emulation.
- Scrape rendered HTML— Usescrapeto get the full rendered HTML of JavaScript-heavy pages after dynamic content loads.
- Extract structured data— Useextractto pull specific fields like prices, headlines, or reviews via CSS selectors or XPath.
- Generate PDFs— Usepdfto export a page as a PDF with format, margins, landscape, and header/footer options.
- Discover site structure— Usesitemapto read a site's sitemap.xml and return all discovered URLs, with exclusion patterns.
Agenty AI exposes a Model Context Protocol (MCP) server that gives AI assistants like Claude, Cursor, and any MCP-compatible client direct access to browser automation tools. You can capture screenshots, generate PDFs, scrape web pages, extract structured data, convert pages to Markdown and more using LLM models
In this article, I will covers how to connect to the Agenty’s MCP server and how to use each available tool.
- An Agenty AI account and API key
- An MCP-compatible client such as Claude Desktop, Cursor, Windsurf, or any tool that supports the Model Context Protocol
- Node.js 18 or later if you are using the CLI or a local configuration file
Add the following configuration to your MCP client. For Claude Desktop, this goes inclaude_desktop_config.json. For Cursor, it goes in your.cursor/mcp.jsonfile.
{ "mcpServers": { "agenty": { "url": "https://api.agenty.ai/mcp", "headers": { "Authorization": "Bearer YOUR_API_KEY" } } } }
ReplaceYOUR_API_KEYwith the key from your Agenty AI > Settings > API Key. Once added, restart your client and the Agenty tools will appear in the tool list.For clients that support OAuth or token-based login, you can authenticate directly through the Agenty AI sign-in flow without manually pasting a key.
The Agenty MCP server provides eleven tools covering the most common browser automation needs. Each tool accepts a URL or raw HTML as input along with optional parameters for controlling the browser behavior.
This is particularly useful for feeding web content into LLMs, building RAG pipelines, or saving readable versions of articles.
“ConvertWeb scraping - Wikipediato Markdown”
“Get the Markdown content of this blog post:https://example.com/article”
This is useful when you want a visual snapshot of a page, need to document a UI, or want to verify how a site looks at a specific screen size.
“Take a full-page screenshot ofhttps://example.comand return it as a PNG”
“Capture a mobile screenshot ofhttps://agenty.aiusing an iPhone 14 viewport”
Use this when you need the raw HTML of a page exactly as a user would see it after all scripts have run.
“Scrape the HTML ofhttps://news.ycombinator.comand return the full rendered content”
“Scrape this page and wait 3 seconds for dynamic content to load:https://example.com/dashboard”
“Extract every link fromhttps://example.com/blogand give me a list of article URLs”
“Generate a PDF ofhttps://agenty.aiin A4 format with print backgrounds enabled”
“Create a landscape PDF ofhttps://example.com/report”
“Extract the product names and prices fromhttps://example.com/products”
“Get all the headlines fromhttps://news.ycombinator.com”
Use this to quickly understand the full structure of a website or to build a crawl list for a large site.
“Find all URLs in the sitemap forhttps://agenty.ai”
“Get the sitemap forhttps://example.comand exclude any URLs containing /blog/”
This is helpful for debugging broken links, verifying canonical URLs, and auditing affiliate or shortened links.
“Trace all redirects forhttps://bit.ly/some-link”
“Show me the redirect chain forhttps://example.com/old-page”
“Find all email addresses onhttps://example.com/contact”
“Scrape emails from these three pages:https://site1.com,https://site2.com,https://site3.com”
“Get the rendered HTML content ofhttps://example.com/app”
The snapshot tool captures the accessibility tree of a page — a structured list of all interactive and readable elements including buttons, links, inputs, and headings. This is designed for AI agents that need to navigate or reason about a page without processing raw HTML.
“Take an accessibility snapshot ofhttps://example.comso I can understand the page structure”
Here are a few multi-step examples that combine tools:
Audit a website’s content:
“Get the sitemap forhttps://example.com, then convert the top 5 pages to Markdown and summarize each one”
Generate a visual report:
“Take a screenshot ofhttps://agenty.aion mobile and desktop viewports and show me both”
Build a contact list:
“Find all links onhttps://example.com/team, then scrape email addresses from each team member page”
Verify link health:
“Extract all links fromhttps://example.com/blogand trace the redirect chain for any that don’t return 200”
Archive a page:
“Converthttps://example.com/articleto Markdown and generate a PDF version, both with absolute links”
All navigation-based tools (screenshot, pdf, scrape, content, markdown, links, extract, redirects) share a common set of browser configuration options:
When using the screenshot tool via MCP, the result is returned as a hosted URL by default. When calling the API directly,responseTypedefaults tobuffer. SetresponseType: 'url'to receive a hosted link instead.
When using the PDF tool via MCP, the result is returned as a hosted URL by default. When calling the API directly,responseTypedefaults tobuffer. SetresponseType: 'url'to receive a hosted link instead.
Each API key has a monthly request limit based on your plan. Long-running pages with heavy JavaScript or large PDFs consume more resources. UseblockAdsandblockTrackersto speed up requests and reduce bandwidth usage on scraping workloads.
The Agenty AI request and response schema is intentionally kept compatible with the most widely used browser automation APIs includingBrowserless,Firecrawl, Cloudflare Browser Rendering, and similar services. If you are already using one of these providers, you can switch to Agenty AI by updating your base URL and API key without rewriting your existing requests.
Hyperbrowser is the next-generation platform empowering AI agents and enabling effortless, scalable browser automation.
Leverage Notte Web AI agents & cloud browser sessions for scalable browser automation & scraping workflows
AI-powered browser automation MCP server — navigate sites, fill forms, extract data, and handle logins via Claude Code CLI
Scrape any URL with JavaScript rendering and get back clean markdown — built for AI agents, LLM pipelines, and autonomous research workflows.
An AI-powered web scraping system for high-volume automation and advanced data extraction strategies.
Ultra-fast web scraper and deep discussion crawler delivering clean Markdown for AI Agents
AI tools for web scraping, crawling, browser control, and web search via the Oxylabs AI Studio API.
AI-powered tools for web scraping, crawling, and browser automation.
An intelligent web scraping tool using AI and browser automation to extract structured data from websites.
Reliable web fetching for AI agents with retry, circuit breaker, caching, and anti-bot bypass
Sign in to leave a review
Use Google, GitHub, or an email account so ratings stay tied to real people.
No reviews posted yet.
