← Back to directory

Vextorium · robots.txt & AI crawler rules

https://api.vextorium.com/v1/robots-txt

Overview

Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared sitemaps. Check before your agent scrapes.

Protocol MPP (POST)
Price —
Payment Asset —
Payment Network —
Category uncategorized
Provider Vextorium
Source self-registered
Indexed 2026-11-22 22:19:35

Health

Status healthy
Latency (p50) —
Uptime (30d) —
Reliability Score —
Last Checked Never
Last Healthy Never
Consecutive Failures 0

Also available via

x402
healthy
$0.0030
0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913 on eip155:8453
View details →