Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per...
https://api.x402node.dev/web/robotsOverview
Fetch and parse robots.txt for a domain. Returns parsed allow/disallow rules per user-agent, sitemap URLs, crawl-delay directives, and raw content. Built for AI agents planning ethical web crawling and SEO audit. Use ?domain=example.com.
Protocol
x402
Price
$0.0050
Payment Asset
USD Coin
Payment Network
Base
Category
uncategorized
Provider
x402node
Source
bazaar
Indexed
2026-05-28 19:24:38
Health
Status
healthy
Latency (p50)
440ms
Uptime (30d)
100.0%
Reliability Score
92/100
Last Checked
2026-06-02 05:34:45
Last Healthy
2026-06-02 05:34:45
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
—
Facilitator
—
Input Schema
{
"discoverable": true,
"method": "GET",
"type": "http"
}
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-06-02 05:34:45 | healthy | 402 | 421ms | |
| 2026-06-02 01:34:33 | healthy | 402 | 432ms | |
| 2026-06-01 20:49:01 | healthy | 402 | 447ms | |
| 2026-06-01 17:34:06 | healthy | 402 | 438ms | |
| 2026-06-01 12:42:34 | healthy | 402 | 448ms | |
| 2026-06-01 08:23:22 | healthy | 402 | 422ms | |
| 2026-06-01 06:42:58 | healthy | 402 | 436ms | |
| 2026-06-01 00:50:59 | healthy | 402 | 446ms | |
| 2026-05-31 21:56:31 | healthy | 402 | 419ms | |
| 2026-05-31 17:50:32 | healthy | 402 | 440ms | |
| 2026-05-31 15:09:30 | healthy | 402 | 436ms | |
| 2026-05-31 08:37:23 | healthy | 402 | 441ms | |
| 2026-05-31 06:32:02 | healthy | 402 | 445ms | |
| 2026-05-31 01:19:56 | healthy | 402 | 1001ms | |
| 2026-05-30 20:59:48 | healthy | 402 | 438ms | |
| 2026-05-30 18:46:33 | healthy | 402 | 466ms | |
| 2026-05-30 12:14:47 | healthy | 402 | 470ms | |
| 2026-05-30 11:07:53 | healthy | 402 | 857ms |