Robots.txt & AI Crawler Checker

dev.workers.quietworklabs.robots-check/robots-checkVersion 1.0.0

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Status

Responds248 msNov 14, 2026, 14:29 UTCAnswered our handshake without sign-in.

Registry data may be out of date

We could not refresh it, so this page shows the last copy.

Last checked: Nov 14, 2026, 14:29 UTC
Latency: 248 ms
Remotes responding: 1 of 1
Sign-in: No sign-in
Hosting: Hosted outside the EU (detected)
Registry data as of: Nov 14, 2026, 22:14 UTC

Data from the official MCP Registry, checked by run.zero. How we check

Connect

Streamable HTTP

https://robots-check.quietworklabs.workers.dev/mcpResponds248 msNov 14, 2026, 14:29 UTC

No sign-in

Claude Code

claude mcp add --transport http robots-check https://robots-check.quietworklabs.workers.dev/mcp

Claude Desktop and claude.ai

1
2
3

Settings > Connectors > Add custom connector
Name: robots-check
Remote MCP server URL: https://robots-check.quietworklabs.workers.dev/mcp

Cursor

{
  "mcpServers": {
    "robots-check": {
      "url": "https://robots-check.quietworklabs.workers.dev/mcp"
    }
  }
}

VS Code

code --add-mcp '{"name":"robots-check","type":"http","url":"https://robots-check.quietworklabs.workers.dev/mcp"}'

Built your own server? run.zero hosts MCP servers on European infrastructure.

Host your own MCP server on run.zero

Tools

3 tools

Tools of Robots.txt & AI Crawler Checker
	Description		Read-only	Idempotent	Open world
`ai_crawler_access`AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.	AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, Meta-ExternalAgent, CCBot, Bytespider, Amazonbot and more) grouped by training/search/user purpose, plus page meta robots, X-Robots-Tag (noindex/noai) and llms.txt presence. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.	Not declared	Yes	Not declared	Yes
`robots_check`Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else , longest match, Allow wins ties, and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.	Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else , longest match, Allow wins ties, and $ wildcards, 4xx=allow all, 5xx=disallow all). Returns allowed, the deciding rule and line, crawl-delay and sitemaps. Deterministic, no AI. Price: $0.001 USDC per call via x402 (Base); the first 5 calls per day per client are free.	Not declared	Yes	Not declared	Yes
`sitemap_check`Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.	Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, relative/off-host/duplicate/http URLs, gzip support. Returns sample URLs or child sitemaps. Deterministic, no AI. Price: $0.002 USDC per call via x402 (Base); the first 5 calls per day per client are free.	Not declared	Yes	Not declared	Yes

Details

Registry status: Active
Published: Oct 18, 2026
Updated: Oct 18, 2026
Website: https://robots-check.quietworklabs.workers.dev/⁠ (opens in a new tab)
Repository: Not given
Categories (suggested): Search and webSuggested categories are assigned automatically from the registry text and may be wrong. Verified operators can correct them.
GDPR: Not declared

Are you the operator? Claim this listing