Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per...
https://api.x402node.dev/web/robotsOverview
Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com.
Protocol
x402
Price
—
Payment Asset
USD Coin
Payment Network
eip155:8453
Category
uncategorized
Provider
x402node
Source
bazaar
Indexed
2026-05-28 19:24:38
Health
Status
healthy
Latency (p50)
466ms
Uptime (30d)
100.0%
Reliability Score
95/100
Last Checked
2026-09-09 13:54:38
Last Healthy
2026-09-09 13:54:38
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
Known USDC
Facilitator
—
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-09-09 13:54:38 | healthy | 402 | 425ms | |
| 2026-09-09 12:36:56 | healthy | 402 | 431ms | |
| 2026-09-09 03:14:19 | healthy | 402 | 430ms | |
| 2026-09-08 16:19:15 | healthy | 402 | 1468ms | |
| 2026-09-08 02:17:24 | healthy | 402 | 1303ms | |
| 2026-09-07 21:39:49 | healthy | 402 | 466ms | |
| 2026-09-07 11:43:05 | healthy | 402 | 1420ms | |
| 2026-09-07 04:54:00 | healthy | 402 | 1168ms | |
| 2026-09-06 15:40:07 | healthy | 402 | 430ms |