Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, ...
https://api.strale.io/x402/robots-txt-parseOverview
Fetch and parse a website's robots.txt. Returns rules per user-agent, sitemaps, crawl delays, and whether all crawlers are blocked. SQS: 96/100.
Protocol
x402
Price
$0.01
Payment Asset
USD Coin
Payment Network
Base
Category
uncategorized
Provider
strale
Source
bazaar
Indexed
2026-04-15 14:10:02
Health
Status
healthy
Latency (p50)
198ms
Uptime (30d)
100.0%
Reliability Score
95/100
Last Checked
2026-04-16 20:11:05
Last Healthy
2026-04-16 20:11:05
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
Known USDC
Facilitator
—
Input Schema
{
"discoverable": true,
"method": "GET",
"queryParams": {
"url": {
"required": true,
"type": "string"
}
},
"type": "http"
}
Output Schema
{
"blocks_all_crawlers": {
"type": "boolean"
},
"exists": {
"type": "boolean"
},
"rules": {
"type": "array"
},
"sitemaps": {
"type": "array"
},
"user_agents": {
"type": "array"
}
}
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-04-16 20:11:05 | healthy | 402 | 230ms | |
| 2026-04-16 19:18:46 | healthy | 402 | 194ms | |
| 2026-04-16 18:35:59 | healthy | 402 | 209ms | |
| 2026-04-16 16:59:00 | healthy | 402 | 206ms | |
| 2026-04-16 16:36:47 | healthy | 402 | 220ms | |
| 2026-04-16 15:23:55 | healthy | 402 | 194ms | |
| 2026-04-16 14:53:35 | healthy | 402 | 197ms | |
| 2026-04-16 13:32:37 | healthy | 402 | 190ms | |
| 2026-04-16 12:30:18 | healthy | 402 | 196ms | |
| 2026-04-16 11:02:48 | healthy | 402 | 778ms | |
| 2026-04-16 10:21:07 | healthy | 402 | 190ms | |
| 2026-04-16 09:00:59 | healthy | 402 | 188ms | |
| 2026-04-16 08:41:03 | healthy | 402 | 210ms | |
| 2026-04-16 07:19:33 | healthy | 402 | 223ms | |
| 2026-04-16 06:33:42 | healthy | 402 | 198ms | |
| 2026-04-16 05:23:52 | healthy | 402 | 188ms | |
| 2026-04-16 04:07:08 | healthy | 402 | 186ms | |
| 2026-04-16 02:58:31 | healthy | 402 | 223ms | |
| 2026-04-16 02:22:07 | healthy | 402 | 185ms | |
| 2026-04-16 01:17:12 | healthy | 402 | 210ms |