Robots.txt Analyzer
https://netintel-production-440c.up.railway.app/robots-txt/analyzeOverview
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
Protocol
x402
Price
$0.01
Payment Asset
USD Coin
Payment Network
Base
Category
data/web-intelligence
Provider
netintel-production-440c
Source
bazaar
Indexed
2026-03-15 03:02:32
Health
Status
healthy
Latency (p50)
173ms
Uptime (30d)
100.0%
Reliability Score
100/100
Last Checked
2026-08-11 08:42:09
Last Healthy
2026-08-11 08:42:09
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
Known USDC
Facilitator
—
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-08-11 08:42:09 | healthy | 402 | 169ms | |
| 2026-08-11 04:29:40 | healthy | 402 | 176ms | |
| 2026-08-10 23:45:38 | healthy | 402 | 172ms | |
| 2026-08-10 16:08:29 | healthy | 402 | 203ms | |
| 2026-08-10 13:47:03 | healthy | 402 | 171ms | |
| 2026-08-10 08:12:05 | healthy | 402 | 180ms | |
| 2026-08-10 01:19:02 | healthy | 402 | 173ms | |
| 2026-08-09 19:38:47 | healthy | 402 | 193ms | |
| 2026-08-09 16:47:19 | healthy | 402 | 171ms | |
| 2026-08-09 12:23:17 | healthy | 402 | 194ms | |
| 2026-08-09 05:55:20 | healthy | 402 | 203ms | |
| 2026-08-09 01:59:02 | healthy | 402 | 168ms | |
| 2026-08-08 18:13:23 | healthy | 402 | 167ms | |
| 2026-08-08 14:41:31 | healthy | 402 | 169ms | |
| 2026-08-08 10:51:29 | healthy | 402 | 195ms |