Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per...
https://api.x402node.dev/web/robotsOverview
Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com.
Protocol
x402
Price
—
Payment Asset
USD Coin
Payment Network
eip155:8453
Category
uncategorized
Provider
x402node
Source
bazaar
Indexed
2026-05-28 19:24:38
Health
Status
healthy
Latency (p50)
430ms
Uptime (30d)
100.0%
Reliability Score
95/100
Last Checked
2026-07-23 10:59:21
Last Healthy
2026-07-23 10:59:21
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
Known USDC
Facilitator
—
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-07-23 10:59:21 | healthy | 402 | 141ms | |
| 2026-07-23 05:09:58 | healthy | 402 | 415ms | |
| 2026-07-23 01:14:54 | healthy | 402 | 413ms | |
| 2026-07-22 23:33:42 | healthy | 402 | 417ms | |
| 2026-07-22 20:16:41 | healthy | 402 | 438ms | |
| 2026-07-22 18:31:13 | healthy | 402 | 411ms | |
| 2026-07-21 23:25:25 | healthy | 402 | 432ms | |
| 2026-07-21 19:05:44 | healthy | 402 | 432ms | |
| 2026-07-21 11:33:16 | healthy | 402 | 432ms | |
| 2026-07-21 09:27:07 | healthy | 402 | 430ms |