Overview
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
Protocol
x402
Price
—
Payment Asset
USD Coin
Payment Network
eip155:8453
Category
data/web-intelligence
Provider
NetIntel
Source
bazaar
Indexed
2026-06-21 06:10:43
Health
Status
healthy
Latency (p50)
184ms
Uptime (30d)
100.0%
Reliability Score
100/100
Last Checked
2026-09-22 15:23:52
Last Healthy
2026-09-22 15:23:52
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
Known USDC
Facilitator
—
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-09-22 15:23:52 | healthy | 402 | 188ms | |
| 2026-09-22 12:41:46 | healthy | 402 | 178ms | |
| 2026-09-21 22:46:51 | healthy | 402 | 184ms | |
| 2026-09-21 15:55:51 | healthy | 402 | 181ms | |
| 2026-09-21 08:05:15 | healthy | 402 | 166ms | |
| 2026-09-21 00:48:11 | healthy | 402 | 192ms | |
| 2026-09-20 09:37:35 | healthy | 402 | 224ms | |
| 2026-09-20 01:15:56 | healthy | 402 | 166ms |