Purchase: Public Robots.txt Crawler Policy — Rules, Delay & Sitemaps
https://api.the402.ai/v1/services/svc_049286bebcf3424b/purchaseOverview
Purchase: Public Robots.txt Crawler Policy — Rules, Delay & Sitemaps
Protocol
x402
Price
$0.05
Payment Asset
USD Coin
Payment Network
Base
Category
uncategorized
Provider
the402
Source
bazaar
Indexed
2026-07-18 22:14:54
Health
Status
healthy
Latency (p50)
281ms
Uptime (30d)
100.0%
Reliability Score
92/100
Last Checked
2026-07-22 00:51:51
Last Healthy
2026-07-22 00:51:51
Consecutive Failures
0
x402 Payment Validation
Payment Requirements
Valid
Asset Verified
—
Facilitator
—
Input Schema
{
"body": {
"additionalProperties": false,
"properties": {
"url": {
"description": "Public HTTPS website or robots.txt URL.",
"format": "uri",
"maxLength": 2048,
"type": "string"
},
"userAgent": {
"default": "*",
"description": "Crawler user-agent whose effective rule group should be selected.",
"maxLength": 100,
"type": "string"
}
},
"required": [
"url"
],
"type": "object"
},
"bodyType": "json",
"discoverable": true,
"method": "POST",
"type": "http"
}
Output Schema
{
"properties": {
"job_id": {
"type": "string"
},
"status": {
"type": "string"
},
"thread_id": {
"type": "string"
}
},
"type": "object"
}
Recent Health Checks
| Time | Status | HTTP | Latency | Error |
|---|---|---|---|---|
| 2026-07-22 00:51:51 | healthy | 402 | 265ms | |
| 2026-07-21 17:39:08 | healthy | 402 | 623ms | |
| 2026-07-21 12:37:31 | healthy | 402 | 281ms | |
| 2026-07-21 08:01:54 | healthy | 402 | 264ms |