Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per...
https://api.x402node.dev/web/robotsOverview
Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com.
Protocol
x402
Price
$0.0050
Payment Asset
USD Coin
Payment Network
eip155:8453
Category
uncategorized
Provider
x402node
Source
bazaar
Indexed
2026-05-28 19:24:38
Health
Status
healthy
Latency (p50)
458ms
Uptime (30d)
100.0%
Reliability Score
95/100
Last Checked
2026-10-28 06:32:25
Last Healthy
2026-10-28 06:32:25
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
Known USDC
Facilitator
—
Input Schema
{
"method": "GET",
"queryParams": {
"domain": "string"
},
"type": "http"
}
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-10-28 06:32:25 | healthy | 402 | 4322ms | |
| 2026-10-28 05:17:52 | healthy | 402 | 1625ms | |
| 2026-10-28 04:20:23 | healthy | 402 | 208ms | |
| 2026-10-28 03:26:35 | healthy | 402 | 172ms | |
| 2026-10-28 02:21:59 | healthy | 402 | 658ms | |
| 2026-10-28 01:19:46 | healthy | 402 | 1099ms | |
| 2026-10-28 00:17:37 | healthy | 402 | 1495ms | |
| 2026-10-27 23:20:55 | healthy | 402 | 145ms | |
| 2026-10-27 22:19:34 | healthy | 402 | 500ms | |
| 2026-10-27 21:18:41 | healthy | 402 | 519ms | |
| 2026-10-27 20:59:11 | healthy | 402 | 3093ms | |
| 2026-10-27 20:00:01 | healthy | 402 | 218ms | |
| 2026-10-27 18:56:48 | healthy | 402 | 145ms | |
| 2026-10-27 18:46:07 | healthy | 402 | 145ms | |
| 2026-10-27 18:34:33 | healthy | 402 | 458ms | |
| 2026-10-26 15:01:45 | healthy | 402 | 427ms | |
| 2026-10-26 06:46:44 | healthy | 402 | 445ms | |
| 2026-10-25 13:07:54 | healthy | 402 | 430ms |