BACK TO MCP SERVERS
searchany

CRW Web Scraper

Open-source web scraper for AI agents with scrape, crawl, and map tools

At a glance

Not tested yet
GitHub stars
1,276
Last commit
2026-11-19
Licence
AGPL-3.0
Transport
stdio
Auth
none
Install
docker run -i --rm ghcr.io/fastcrw/crw:latest
Stats checked 2026-11-19

What it does

CRW Web Scraper is an MCP server for fastCRW, an engine that turns URLs into clean markdown or structured JSON. It gives AI agents web operations: scrape, crawl, map, search and extract. It is for people who want their coding agents to read and search the web. It runs locally as a small Rust binary or through a managed API.

fastCRW
Image: fastcrw/crw (AGPL-3.0)

Key features

  • Scrape one URL to markdown, HTML, links, screenshots, or schema JSON.
  • Crawl a bounded site and collect its pages.
  • Map discovers URLs without scraping every page.
  • Search the web and optionally scrape selected results.
  • Extract structured fields from one or many URLs.
  • Runs locally and free with no account needed, or through the managed Cloud API.
  • Crawl and map follow robots.txt by default.

Quick start

Run the one-command install:

curl -fsSL https://fastcrw.com/install | sh

To use the Cloud, paste your key into the same command:

curl -fsSL https://fastcrw.com/install | CRW_API_KEY=crw_live_... sh

Scrape a page:

crw https://example.com

Search the web after crw setup:

crw search "rust async runtime"

The installer picks up Claude Code, Cursor, Codex, Gemini CLI, OpenCode and Windsurf when they are already set up. Add CRW_NO_AGENTS=1 to skip that step, or run crw setup to choose interactively. You can also install the MCP server with:

npx -y crw-mcp@latest install

Add it to Claude Code from a terminal:

claude mcp add crw -- docker run -i --rm ghcr.io/fastcrw/crw:latest

Or add this to .mcp.json in your project root (Claude Code) or .cursor/mcp.json (Cursor):

{
  "mcpServers": {
    "crw": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/fastcrw/crw:latest"
      ]
    }
  }
}

Environment variables

  • CRW_API_KEY: the Cloud API key. The installer connects it and keeps it in ~/.config/crw/config.toml rather than copying it into each tool.
  • CRW_NO_AGENTS: skips the step that registers the MCP server with your AI tools.

Example prompts

  • Scrape https://example.com and give me the page as markdown.
  • Search the web for rust async runtime and scrape the top results.
  • Map this site and list its URLs without scraping every page.
  • Extract the product name and price from these URLs as structured fields.

What to watch out for

  • The Cloud API needs an account and an API key. It offers 1000 free credits with no credit card.
  • Local mode runs free with no account, but you choose renderers, search, auth, proxies and capacity yourself.
  • Capabilities and response shapes can differ between the managed API and local or self-hosted deployments.
  • The engine and MCP server are AGPL-3.0; the Python and TypeScript SDKs are MIT.
  • Respect website policies. Crawl and map follow robots.txt by default.

Not tested by tools4agents yet. Read the source repository before you install it.

GitHub SourceGitHub Stars
Last verified 2026-11-19

Similar tools