Fetch and parse robots.txt for each User-agent block.
Accepts either a full robots.txt URL or a site host, then reports the discovered user-agents and Allow/Disallow rules. Drop this helper wherever you want to fetch and parse robots.txt for each User-agent block with a single HTTP call. Send the documented inputs and the API streams back polished results you can hand directly to your users.
/checks/robots-txt-validatorWhere this helper shines
Fetch and parse robots.txt for each User-agent block. keeps checks helpers on autopilot. Use it whenever you need to fetch and parse robots.txt for each User-agent block without pausing a sprint or asking design/dev to jump in.
- Trigger this helper whenever you need to fetch and parse robots.txt for each User-agent block without waiting on a full deploy.
- Drop it into automations (Zapier, Make, workers) to fetch and parse robots.txt for each User-agent block as part of your workflow.
- Share the GET URL so support and success can fetch and parse robots.txt for each User-agent block safely using their Dataczar API key.
Copy & ship snippets
GET
Instant link
GET https://api.dataczar.app/checks/robots-txt-validator?url=https%3A%2F%2Fexample.com%2Fsample.png&site=sample-site&dryrun=true&ak=demoOpen directly in any browser, webhook, or low-code workflow.
curl
curl GET
curl "https://api.dataczar.app/checks/robots-txt-validator?url=https%3A%2F%2Fexample.com%2Fsample.png&site=sample-site&dryrun=true&ak=demo"Swap ak=demo with your Dataczar API key.
curl
curl POST
curl -X POST "https://api.dataczar.app/checks/robots-txt-validator?ak=demo" \
-H 'Content-Type: application/json' \
-d '{
"url": "https://example.com/sample.png",
"site": "sample-site",
"dryrun": "true"
}'Send JSON payloads from back-end jobs, queues, or workers.
Parameters
Query Parameters
| Name | Type | Description |
|---|---|---|
| url | string | Explicit robots.txt URL. |
| site | string | Base domain (https://example.com). |
| dryrun | boolean |
All helper calls accept your Dataczar key via the ak query parameter. Need one? Create or copy a key on Dataczar Connect and drop it into each request.
Responses
Status 200
Parsed robots groups.
{
"ok": true,
"data": {
"url": "https://example.com/robots.txt",
"group_count": 2,
"groups": [
{
"user_agent": "*",
"rules": [
{
"type": "disallow",
"value": "/private"
}
]
}
]
}
}Examples
Grab your Dataczar API key
Every helper runs with the same simple API key. Sign in (or create a free Dataczar account) at Dataczar Connect to issue keys in seconds, then drop the key into any `ak=` parameter and start shipping.
Tip: reuse the same key across these helpers, or rotate keys per project from the Connect dashboard.