Overview
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
Protocol
x402
Price
$0.01
Payment Asset
USDC
Payment Network
base
Category
data/web-intelligence
Provider
NetIntel
Source
bazaar
Indexed
2026-06-21 06:10:43
Health
Status
healthy
Latency (p50)
356ms
Uptime (30d)
100.0%
Reliability Score
95/100
Last Checked
2026-10-31 07:04:02
Last Healthy
2026-10-31 07:04:02
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
Known USDC
Facilitator
—
Input Schema
{
"method": "GET",
"queryParams": {
"domain": "example.com"
},
"type": "http"
}
Output Schema
{
"example": {
"domain": "example.com",
"findings": [],
"found": true,
"grade": "B",
"path_check": {
"allowed": false,
"matched_rule": "Disallow: /private/",
"path": "/private/page",
"rule_type": "disallow",
"user_agent": "*"
},
"raw_content_preview": "User-agent: *\nDisallow: /private/\n...",
"robots_url": "https://example.com/robots.txt",
"rules": [
{
"allow": [
"/public/"
],
"crawl_delay": null,
"disallow": [
"/private/",
"/admin/"
],
"user_agent": "*"
}
],
"score": 85,
"sitemaps": [
"https://example.com/sitemap.xml"
],
"status_code": 200
},
"type": "json"
}