Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, ...
https://api.strale.io/x402/robots-txt-parseOverview
Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, crawl delays, and whether all crawlers are blocked.
Protocol
x402
Price
$0.02
Payment Asset
USD Coin
Payment Network
Base
Category
uncategorized
Provider
strale
Source
bazaar
Indexed
2026-04-15 14:10:02
Health
Status
healthy
Latency (p50)
214ms
Uptime (30d)
100.0%
Reliability Score
95/100
Last Checked
2026-06-12 08:42:13
Last Healthy
2026-06-12 08:42:13
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
—
Facilitator
—
Input Schema
{
"discoverable": true,
"method": "GET",
"queryParams": {
"url": {
"required": true,
"type": "string"
}
},
"type": "http"
}
Output Schema
{
"blocks_all_crawlers": {
"type": "boolean"
},
"exists": {
"type": "boolean"
},
"rules": {
"type": "array"
},
"sitemaps": {
"type": "array"
},
"user_agents": {
"type": "array"
}
}
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-06-12 08:42:13 | healthy | 402 | 238ms | |
| 2026-06-12 04:35:21 | healthy | 402 | 395ms | |
| 2026-06-12 02:11:22 | healthy | 402 | 208ms | |
| 2026-06-11 20:56:22 | healthy | 402 | 221ms | |
| 2026-06-11 16:12:30 | healthy | 402 | 652ms | |
| 2026-06-11 07:30:02 | healthy | 402 | 200ms | |
| 2026-06-11 02:06:04 | healthy | 402 | 214ms | |
| 2026-06-10 22:22:08 | healthy | 402 | 207ms | |
| 2026-06-10 17:33:05 | healthy | 402 | 212ms | |
| 2026-06-10 14:25:42 | healthy | 402 | 217ms | |
| 2026-06-10 08:16:56 | healthy | 402 | 199ms | |
| 2026-06-10 04:19:15 | healthy | 402 | 209ms | |
| 2026-06-09 23:45:02 | healthy | 402 | 216ms | |
| 2026-06-09 18:27:55 | healthy | 402 | 242ms | |
| 2026-06-09 12:49:43 | healthy | 402 | 211ms |