← Back to directory

Check if a URL is allowed to be fetched according to the website's robots.txt fi...

https://minifetch.com/api/v1/x402/preflight/url-check

Overview

Check if a URL is allowed to be fetched according to the website's robots.txt file. Use this super-cheap endpoint before using our extraction endpoints to avoid spending extra on un-fetchable URLs & get more granular info about why a URL may or may not return data

Protocol x402
Price $0.0010
Payment Asset USD Coin
Payment Network Base
Category uncategorized
Provider minifetch
Source bazaar
Indexed 2026-03-02 01:01:28

Health

Status healthy
Latency (p50) 455ms
Uptime (30d) 100.0%
Reliability Score 95/100
Last Checked 2026-03-21 19:42:08
Last Healthy 2026-03-21 19:42:08
Consecutive Failures 0

x402 Payment Validation

Payment Requirements Valid
Asset Verified Known USDC
Facilitator —

Input Schema

{ "method": "GET", "queryParams": { "url": "https://github.com" }, "type": "http" }

Output Schema

{ "example": { "results": [ { "data": { "allowed": true, "crawlDelay": 1, "message": "Access allowed by robots.txt", "url": "https://github.com" } } ], "success": true }, "type": "json" }

Recent Health Checks

Time Status HTTP Latency Error
2026-03-21 19:42:08 healthy 402 452ms
2026-03-21 18:31:05 healthy 402 488ms
2026-03-21 18:01:46 healthy 402 449ms
2026-03-21 16:49:08 healthy 402 445ms
2026-03-21 15:59:08 healthy 402 451ms
2026-03-21 15:05:53 healthy 402 460ms
2026-03-21 14:11:43 healthy 402 572ms
2026-03-21 13:06:25 healthy 402 467ms
2026-03-21 12:29:47 healthy 402 450ms
2026-03-21 11:17:47 healthy 402 453ms
2026-03-21 10:26:24 healthy 402 469ms
2026-03-21 09:30:30 healthy 402 450ms
2026-03-21 08:19:02 healthy 402 469ms
2026-03-21 07:01:56 healthy 402 455ms
2026-03-21 06:27:29 healthy 402 452ms
2026-03-21 05:06:57 healthy 402 455ms
2026-03-21 04:19:07 healthy 402 449ms
2026-03-21 03:25:31 healthy 402 456ms
2026-03-21 02:11:36 healthy 402 533ms
2026-03-21 01:18:56 healthy 402 454ms