Bot Info

Last updated: 2026-05-27

AgentFit is an on-demand HTTP audit tool. This page describes how we crawl, what User-Agents we send, and how site operators can block or report us.

What we fetch

When a user submits your site's documentation URL for audit, the service makes a curated set of HTTP requests:

The homepage of the submitted host
/robots.txt
/sitemap.xml (if present)
/llms.txt and /llms-full.txt (the llmstxt.org spec)
Standard OpenAPI / Swagger paths (/openapi.json, /openapi.yaml, /swagger.json, /api/openapi.json, etc.)
A small sample of documentation pages from the sitemap (typically 3-5, capped at ~10)
A handful of common error-page slugs (/errors, /error-codes, /fundamentals/errors) as a last-resort fallback

Total request budget per audit: ≤~80 URLs, typically 20-40. A full audit completes in 20-60 seconds.

User-Agent strings

Different criteria use different User-Agents to probe how a site behaves under different client signals:

Purpose	User-Agent value
Default (most requests)	`curl/8.7.1`
Search-bot probe (checks robots.txt + cloaking)	`Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)`
Browser probe (checks JS-rendered availability)	`Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36`

We do not currently send a service-branded User-Agent (agentfit/...). Branded UAs trigger WAF blocks on many sites and would degrade audit quality without warning.

Rate limits and politeness

≤ 3 concurrent requests per host
~500 ms jitter between requests to the same host
10 MB body cap per response (truncated after that)
15 s default timeout per request
One single retry on 429 / 503 with Retry-After honoured (capped at 10 s)

robots.txt

AgentFit fetches your robots.txt as part of every audit (it is one of the scored criteria). We do not currently parse it to gate our crawl, but our request volume is intentionally low and bounded so a permissive policy is rarely a concern.

If your robots.txt disallows our User-Agent, the audit will still score the file (as a fetch). To block us cleanly, see the next section.

Blocking AgentFit

To prevent further audits of your site:

WAF / CDN rule — block any of the User-Agent strings above. Cloudflare / Fastly / Akamai rules apply cleanly. We respect the block; failed requests result in an "unknown" or "absent" audit criterion, never a retry storm.
IP-based block — emails to i@gumeniuk.com will get you the current outbound IP range.
Takedown of existing reports — see Privacy / Removing your audit.

Headless rendering (opt-in)

The service has an optional headless Chrome fallback that re-renders a sample page through a real Chrome process when the static HTML looks like a JS-only shell. This is OPT-IN per deployment (HEADLESS_ENABLE=true) and runs at most ONE Chrome render per audit. The default production deployment runs without it.

Reporting abuse or misbehaviour

If our crawler is causing problems for your site — too many requests, unexpected behaviour, or incorrect audit output — please contact i@gumeniuk.com. Response within 72 hours.

audit a site · browse · privacy · terms · agentfit.dev