Agent Extraction API: robots-check
https://freetier-agent-api.pages.dev/api/robots-checkOverview
Check whether a URL may be crawled per the site's robots.txt for a given user-agent. $0.001 per call in USDC on Base (x402). 1 free call per caller; failed calls are never charged.
Protocol
x402
Price
$0.0030
Payment Asset
USD Coin
Payment Network
eip155:8453
Category
tools
Provider
FreeTier Kits
Source
self-registered
Indexed
2026-10-17 20:17:06
Health
Status
unknown
Latency (p50)
614ms
Uptime (30d)
97.0%
Reliability Score
83.5/100
Last Checked
2026-10-31 07:00:32
Last Healthy
2026-10-31 05:59:52
Consecutive Failures
1
x402 Payment Validation
Payment Requirements
—
Asset Verified
—
Facilitator
—
Input Schema
{
"method": "GET",
"queryParams": {
"ua": "MyAgent",
"url": "https://example.com/some/page"
},
"type": "http"
}
Output Schema
{
"example": {
"allowed": true,
"crawl_delay": null,
"matched_rule": null,
"robots_status": 404,
"sitemaps": [],
"sources": [
{
"name": "robots.txt (live)",
"url": "https://example.com/robots.txt"
}
],
"url": "https://example.com/some/page",
"user_agent": "MyAgent"
},
"type": "json"
}