Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, ...
https://api.strale.io/x402/robots-txt-parseOverview
Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, crawl delays, and whether all crawlers are blocked.
Protocol
x402
Price
$0.02
Payment Asset
USD Coin
Payment Network
Base
Category
uncategorized
Provider
strale
Source
bazaar
Indexed
2026-04-15 14:10:02
Health
Status
healthy
Latency (p50)
134ms
Uptime (30d)
100.0%
Reliability Score
100/100
Last Checked
2026-08-22 14:18:33
Last Healthy
2026-08-22 14:18:33
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
—
Facilitator
—
Input Schema
{
"discoverable": true,
"method": "GET",
"queryParams": {
"url": {
"required": true,
"type": "string"
}
},
"type": "http"
}
Output Schema
{
"blocks_all_crawlers": {
"type": "boolean"
},
"exists": {
"type": "boolean"
},
"rules": {
"type": "array"
},
"sitemaps": {
"type": "array"
},
"user_agents": {
"type": "array"
}
}
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-08-22 14:18:33 | healthy | 402 | 158ms | |
| 2026-08-22 03:29:51 | healthy | 402 | 125ms | |
| 2026-08-22 00:33:36 | healthy | 402 | 377ms | |
| 2026-08-21 14:59:55 | healthy | 402 | 111ms | |
| 2026-08-21 12:53:56 | healthy | 402 | 536ms | |
| 2026-08-21 07:45:20 | healthy | 402 | 642ms | |
| 2026-08-20 20:55:39 | healthy | 402 | 134ms | |
| 2026-08-20 14:36:30 | healthy | 402 | 120ms | |
| 2026-08-20 09:38:46 | healthy | 402 | 109ms | |
| 2026-08-20 05:47:37 | healthy | 402 | 116ms | |
| 2026-08-19 20:19:37 | healthy | 402 | 217ms | |
| 2026-08-19 16:16:43 | healthy | 402 | 109ms |