RobotsGate

dev.workers.mike-tusa.robotsgate/robotsgateVersion 0.3.1

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

Status

Responds198 msNov 14, 2026, 14:28 UTCAnswered our handshake without sign-in.

Registry data may be out of date

We could not refresh it, so this page shows the last copy.

Last checked: Nov 14, 2026, 14:28 UTC
Latency: 198 ms
Remotes responding: 1 of 1
Sign-in: No sign-in
Hosting: Hosted outside the EU (detected)
Registry data as of: Nov 14, 2026, 22:14 UTC

Data from the official MCP Registry, checked by run.zero. How we check

Connect

Streamable HTTP

https://robotsgate.mike-tusa.workers.dev/mcpResponds198 msNov 14, 2026, 14:28 UTC

No sign-in

Claude Code

claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp

Claude Desktop and claude.ai

1
2
3

Settings > Connectors > Add custom connector
Name: robotsgate
Remote MCP server URL: https://robotsgate.mike-tusa.workers.dev/mcp

Cursor

{
  "mcpServers": {
    "robotsgate": {
      "url": "https://robotsgate.mike-tusa.workers.dev/mcp"
    }
  }
}

VS Code

code --add-mcp '{"name":"robotsgate","type":"http","url":"https://robotsgate.mike-tusa.workers.dev/mcp"}'

Built your own server? run.zero hosts MCP servers on European infrastructure.

Host your own MCP server on run.zero

Tools

4 tools

Tools of RobotsGate
	Description		Read-only	Idempotent	Open world
`check_site`Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.	Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.	No	Yes	Yes	Yes
`generate_robots`Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.	Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.	No	Yes	Yes	No
`list_crawlers`Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.	Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.	No	Yes	Yes	No
`validate_robots`Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.	Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.	No	Yes	Yes	No

Details

Registry status: Active
Published: Oct 16, 2026
Updated: Oct 16, 2026
Website: https://robotsgate.mike-tusa.workers.dev/⁠ (opens in a new tab)
Repository: Not given
Categories (suggested): Search and webSuggested categories are assigned automatically from the registry text and may be wrong. Verified operators can correct them.
GDPR: Not declared

Are you the operator? Claim this listing