Cited Brief: may a crawler fetch this URL?
https://cited-brief.cited-brief.workers.dev/robots-checkOverview
Reads a site's robots.txt and says whether a crawler may fetch a URL (RFC 9309), with the matching rule, crawl-delay and sitemaps. No AI; an unreadable robots.txt is not charged.
Protocol
x402 (POST)
Price
$0.01
Payment Asset
USDC
Payment Network
Base
Category
tools
Provider
Cited Brief
Source
self-registered
Indexed
2026-10-13 07:12:27
Health
Status
healthy
Latency (p50)
725ms
Uptime (30d)
100.0%
Reliability Score
87/100
Last Checked
2026-10-18 14:22:52
Last Healthy
2026-10-18 14:22:52
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
Known USDC
Facilitator
—
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-10-18 14:22:52 | healthy | 402 | 1226ms | |
| 2026-10-18 02:11:15 | healthy | 402 | 725ms | |
| 2026-10-17 06:56:25 | healthy | 402 | 617ms | |
| 2026-10-16 19:46:01 | healthy | 402 | 101ms | |
| 2026-10-16 09:13:55 | healthy | 402 | 659ms | |
| 2026-10-16 01:02:56 | healthy | 402 | 742ms |