Everything a crawler or SEO agent needs before visiting a site: robots.txt rules...
https://aipaygen.com/v1/site_crawlOverview
Everything a crawler or SEO agent needs before visiting a site: robots.txt rules for your user agent (is this URL allowed? crawl delay?), the site's sitemaps with page URLs and last-modified dates (following one sitemap-index level), and the internal/external links on the page.
Protocol
x402
Price
$0.0020
Payment Asset
USD Coin
Payment Network
eip155:8453
Category
uncategorized
Provider
aipaygen
Source
bazaar
Indexed
2026-11-15 04:08:33
Health
Status
unknown
Latency (p50)
—
Uptime (30d)
—
Reliability Score
—
Last Checked
Never
Last Healthy
Never
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
—
Asset Verified
—
Facilitator
—
Input Schema
{
"method": "GET",
"queryParams": {
"max_urls": 20,
"url": "https://www.cloudflare.com/learning/",
"user_agent": "GPTBot"
},
"type": "http"
}