Vextorium · robots.txt & AI crawler rules
https://api.vextorium.com/v1/robots-txtOverview
Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared sitemaps. Check before your agent scrapes.
Protocol
MPP (POST)
Price
—
Payment Asset
—
Payment Network
—
Category
uncategorized
Provider
Vextorium
Source
self-registered
Indexed
2026-11-22 22:19:35
Health
Status
healthy
Latency (p50)
—
Uptime (30d)
—
Reliability Score
—
Last Checked
Never
Last Healthy
Never
Consecutive Failures
0