Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, ...
https://api.strale.io/x402/robots-txt-parseOverview
Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, crawl delays, and whether all crawlers are blocked.
Protocol
x402
Price
$0.02
Payment Asset
USD Coin
Payment Network
Base
Category
uncategorized
Provider
strale
Source
bazaar
Indexed
2026-04-15 14:10:02
Health
Status
healthy
Latency (p50)
157ms
Uptime (30d)
100.0%
Reliability Score
100/100
Last Checked
2026-10-08 23:29:58
Last Healthy
2026-10-08 23:29:58
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
—
Facilitator
—
Input Schema
{
"discoverable": true,
"method": "GET",
"queryParams": {
"url": {
"required": true,
"type": "string"
}
},
"type": "http"
}
Output Schema
{
"blocks_all_crawlers": {
"type": "boolean"
},
"exists": {
"type": "boolean"
},
"rules": {
"type": "array"
},
"sitemaps": {
"type": "array"
},
"user_agents": {
"type": "array"
}
}
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-10-08 23:29:58 | healthy | 402 | 183ms | |
| 2026-10-08 05:50:44 | healthy | 402 | 113ms | |
| 2026-10-07 17:20:15 | healthy | 402 | 184ms | |
| 2026-10-07 02:54:31 | healthy | 402 | 140ms | |
| 2026-10-06 20:11:42 | healthy | 402 | 138ms | |
| 2026-10-06 09:09:37 | healthy | 402 | 157ms |