Site checks: Read a site's robots.txt and say whether a crawler may fetch a URL ...
https://rengi.dev/v1/net-tools/robotsOverview
Site checks: Read a site's robots.txt and say whether a crawler may fetch a URL and by which rule, which AI crawlers (GPTBot, ClaudeBot, PerplexityBot and others) are let in or shut out, and what the sitemap holds. Full schema: https://rengi.dev/v1/net-tools/schema
Protocol
x402 (POST)
Price
$0.01
Payment Asset
USD Coin
Payment Network
eip155:8453
Category
uncategorized
Provider
Rengi Forge
Source
bazaar
Indexed
2026-11-21 17:10:38
Health
Status
unknown
Latency (p50)
—
Uptime (30d)
—
Reliability Score
—
Last Checked
Never
Last Healthy
Never
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
—
Asset Verified
—
Facilitator
—
Input Schema
{
"body": {
"url": "https://rengi.dev/forge",
"userAgent": "GPTBot"
},
"bodyType": "json",
"method": "POST",
"type": "http"
}
Output Schema
{
"example": {
"aiCrawlers": {
"ClaudeBot": true,
"GPTBot": false
},
"allowed": false,
"crawlDelay": null,
"exists": true,
"findings": [],
"matchedRule": "Disallow: /",
"robotsUrl": "https://example.org/robots.txt",
"sitemap": {
"lastModified": "2026-11-01",
"nested": 0,
"pages": 42,
"sample": [
"https://example.org/"
],
"url": "https://example.org/sitemap.xml"
},
"sitemaps": [
"https://example.org/sitemap.xml"
]
},
"type": "json"
}