Overview
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
Protocol
x402
Price
—
Payment Asset
USD Coin
Payment Network
eip155:8453
Category
data/web-intelligence
Provider
netintel
Source
bazaar
Indexed
2026-06-21 06:10:43
Health
Status
healthy
Latency (p50)
167ms
Uptime (30d)
100.0%
Reliability Score
92/100
Last Checked
2026-06-21 23:04:07
Last Healthy
2026-06-21 23:04:07
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
Known USDC
Facilitator
—
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-06-21 23:04:07 | healthy | 402 | 167ms | |
| 2026-06-21 10:04:17 | healthy | 402 | 164ms |