← Back to directory

Everything a crawler or SEO agent needs before visiting a site: robots.txt rules...

https://aipaygen.com/v1/site_crawl

Overview

Everything a crawler or SEO agent needs before visiting a site: robots.txt rules for your user agent (is this URL allowed? crawl delay?), the site's sitemaps with page URLs and last-modified dates (following one sitemap-index level), and the internal/external links on the page.

Protocol x402
Price $0.0020
Payment Asset USD Coin
Payment Network eip155:8453
Category uncategorized
Provider aipaygen
Source bazaar
Indexed 2026-11-15 04:08:33

Health

Status unknown
Latency (p50) —
Uptime (30d) —
Reliability Score —
Last Checked Never
Last Healthy Never
Consecutive Failures 0

x402 Payment Validation

Payment Requirements —
Asset Verified —
Facilitator —

Input Schema

{ "method": "GET", "queryParams": { "max_urls": 20, "url": "https://www.cloudflare.com/learning/", "user_agent": "GPTBot" }, "type": "http" }