Overview
robots.txt API: fetch and parse a site robots.txt with user-agent rules and sitemap links. Param: url. · Keywords: crawl rules, seo, scraping, data · Built with @piprail/sdk
Protocol
x402
Price
$0.0030
Payment Asset
USDC
Payment Network
base
Category
data
Provider
x402render
Source
self-registered
Indexed
2026-06-28 00:44:22
Health
Status
healthy
Latency (p50)
164ms
Uptime (30d)
100.0%
Reliability Score
97/100
Last Checked
2026-07-01 20:54:42
Last Healthy
2026-07-01 20:54:42
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
—
Facilitator
—
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-07-01 20:54:42 | healthy | 402 | 146ms | |
| 2026-07-01 17:56:31 | healthy | 402 | 172ms | |
| 2026-07-01 13:54:44 | healthy | 402 | 164ms |