Overview
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
Protocol
x402
Price
—
Payment Asset
USD Coin
Payment Network
eip155:8453
Category
data/web-intelligence
Provider
netintel
Source
bazaar
Indexed
2026-06-21 06:10:43
Health
Status
healthy
Latency (p50)
169ms
Uptime (30d)
100.0%
Reliability Score
100/100
Last Checked
2026-08-07 23:31:23
Last Healthy
2026-08-07 23:31:23
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
Known USDC
Facilitator
—
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-08-07 23:31:23 | healthy | 402 | 166ms | |
| 2026-08-07 21:19:41 | healthy | 402 | 190ms | |
| 2026-08-07 14:30:45 | healthy | 402 | 168ms | |
| 2026-08-07 10:33:03 | healthy | 402 | 165ms | |
| 2026-08-07 05:23:43 | healthy | 402 | 168ms | |
| 2026-08-06 22:45:51 | healthy | 402 | 180ms | |
| 2026-08-06 16:20:54 | healthy | 402 | 170ms | |
| 2026-08-06 13:52:22 | healthy | 402 | 167ms | |
| 2026-08-06 09:33:19 | healthy | 402 | 169ms | |
| 2026-08-06 01:13:13 | healthy | 402 | 184ms | |
| 2026-08-05 23:54:10 | healthy | 402 | 167ms | |
| 2026-08-05 17:38:18 | healthy | 402 | 165ms | |
| 2026-08-05 10:52:09 | healthy | 402 | 156ms | |
| 2026-08-05 07:07:39 | healthy | 402 | 185ms | |
| 2026-08-05 00:51:04 | healthy | 402 | 189ms |