CRW Web Scraper
Open-source web scraper for AI agents with scrape, crawl, and map tools
At a glance
Not tested yet- GitHub stars
- 1,276
- Last commit
- 2026-11-19
- Licence
- AGPL-3.0
- Transport
- stdio
- Auth
- none
docker run -i --rm ghcr.io/fastcrw/crw:latest
What it does
CRW Web Scraper is an MCP server for fastCRW, an engine that turns URLs into clean markdown or structured JSON. It gives AI agents web operations: scrape, crawl, map, search and extract. It is for people who want their coding agents to read and search the web. It runs locally as a small Rust binary or through a managed API.

Key features
- Scrape one URL to markdown, HTML, links, screenshots, or schema JSON.
- Crawl a bounded site and collect its pages.
- Map discovers URLs without scraping every page.
- Search the web and optionally scrape selected results.
- Extract structured fields from one or many URLs.
- Runs locally and free with no account needed, or through the managed Cloud API.
- Crawl and map follow
robots.txtby default.
Quick start
Run the one-command install:
curl -fsSL https://fastcrw.com/install | sh
To use the Cloud, paste your key into the same command:
curl -fsSL https://fastcrw.com/install | CRW_API_KEY=crw_live_... sh
Scrape a page:
crw https://example.com
Search the web after crw setup:
crw search "rust async runtime"
The installer picks up Claude Code, Cursor, Codex, Gemini CLI, OpenCode and Windsurf when they are already set up. Add CRW_NO_AGENTS=1 to skip that step, or run crw setup to choose interactively. You can also install the MCP server with:
npx -y crw-mcp@latest install
Add it to Claude Code from a terminal:
claude mcp add crw -- docker run -i --rm ghcr.io/fastcrw/crw:latest
Or add this to .mcp.json in your project root (Claude Code) or .cursor/mcp.json (Cursor):
{
"mcpServers": {
"crw": {
"command": "docker",
"args": [
"run",
"-i",
"--rm",
"ghcr.io/fastcrw/crw:latest"
]
}
}
}Environment variables
CRW_API_KEY: the Cloud API key. The installer connects it and keeps it in~/.config/crw/config.tomlrather than copying it into each tool.CRW_NO_AGENTS: skips the step that registers the MCP server with your AI tools.
Example prompts
- Scrape https://example.com and give me the page as markdown.
- Search the web for rust async runtime and scrape the top results.
- Map this site and list its URLs without scraping every page.
- Extract the product name and price from these URLs as structured fields.
What to watch out for
- The Cloud API needs an account and an API key. It offers 1000 free credits with no credit card.
- Local mode runs free with no account, but you choose renderers, search, auth, proxies and capacity yourself.
- Capabilities and response shapes can differ between the managed API and local or self-hosted deployments.
- The engine and MCP server are AGPL-3.0; the Python and TypeScript SDKs are MIT.
- Respect website policies. Crawl and map follow
robots.txtby default.
Not tested by tools4agents yet. Read the source repository before you install it.
Similar tools
Firecrawl MCP Server
Crawls, scrapes, and converts dynamic web pages into clean, LLM-ready Markdown or structured JSON data.
Fetch
Retrieves a URL and converts the page to markdown so an agent can read it without burning tokens on raw HTML.
Context7 MCP Server
Injects up-to-date documentation and code examples from official sources directly into AI prompts.
Apify MCP Server
Connects AI agents to the Apify platform to run web scrapers, crawlers, and data extraction actors across 1,000+ websites.