← Back to directory

Site checks: Read a site's robots.txt and say whether a crawler may fetch a URL ...

https://rengi.dev/v1/net-tools/robots

Overview

Site checks: Read a site's robots.txt and say whether a crawler may fetch a URL and by which rule, which AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) are let in or shut out, and what the sitemap holds. Full schema: https://rengi.dev/v1/net-tools/schema

Protocol x402 (POST)
Price $0.01
Payment Asset USD Coin
Payment Network eip155:8453
Category uncategorized
Provider Rengi Forge
Source bazaar
Indexed 2026-11-21 17:10:38

Health

Status unknown
Latency (p50) —
Uptime (30d) —
Reliability Score —
Last Checked Never
Last Healthy Never
Consecutive Failures 0

x402 Payment Validation

Payment Requirements —
Asset Verified —
Facilitator —

Input Schema

{ "body": { "url": "https://rengi.dev/forge", "userAgent": "GPTBot" }, "bodyType": "json", "method": "POST", "type": "http" }

Output Schema

{ "example": { "aiCrawlers": { "ClaudeBot": true, "GPTBot": false }, "allowed": false, "crawlDelay": null, "exists": true, "findings": [], "matchedRule": "Disallow: /", "robotsUrl": "https://example.org/robots.txt", "sitemap": { "lastModified": "2026-11-01", "nested": 0, "pages": 42, "sample": [ "https://example.org/" ], "url": "https://example.org/sitemap.xml" }, "sitemaps": [ "https://example.org/sitemap.xml" ] }, "type": "json" }