AI crawler tracking
AI crawlers do not run JavaScript. That is not a limitation of our tracker, it is what a crawler is: it requests your HTML and reads it. No browser-based analytics tool can see them, ours included.
So measuring them needs one extra piece: a small server-side beacon that runs where your pages are served and reports each crawler request to Zenovay.
Available on every plan, including Free. Crawl events do not count against your monthly event quota.
Install it with one command
Run this in your project and answer the prompts:
npx @zenovay/cli@latest init
The CLI detects your framework, writes the beacon into your request path, and leaves everything else alone. It will:
- create the file if it does not exist,
- extend an existing middleware at a known-safe insertion point,
- or print the snippet and change nothing if it cannot do either safely.
That last case is deliberate. A malformed middleware takes your whole site down, not just your analytics, so the CLI never guesses at a file it does not fully recognise.
Re-running init is safe. If the beacon is already installed it says so and does nothing; if an older version is installed it upgrades that block in place.
What the beacon does
// zenovay:ai-crawl-beacon v2
const ZENOVAY_ENDPOINT = 'https://api.zenovay.com/c/YOUR_TRACKING_CODE';
const ZENOVAY_BROWSER = /(?:Chrome|CriOS|Firefox|FxiOS|Edg|EdgiOS|OPR|Version)\/\d/;
const ZENOVAY_BOT = /bot|crawler|spider|agent|fetch|scrap|\+https?:\/\//i;
It runs after your page responds, so it adds nothing to your response time, and it is wrapped so a tracking failure can never break a page.
The filter is worth understanding, because it explains what you will and will not see. It skips what is plainly a human browser and forwards everything else, rather than looking for a list of known crawler names. Crawler names are an open set that grows every week; browsers are a closed set that changes slowly. A beacon that matched names would go blind to every crawler launched after it was installed.
Identification happens on our side, against a registry of 159 crawlers. That means a newly added crawler starts appearing in your dashboard without you touching your site.
The beacon reports the request path and the crawler's user agent. It never sends your page content. The crawler's IP is used during the request to verify the crawler is genuine, and what the analytics tables keep is a hash of it, not the address. See Privacy-first for where addresses do appear in plaintext.
Supported runtimes
| Where your pages are served | What the CLI writes |
|---|---|
| Cloudflare Pages | functions/_middleware.ts |
| Next.js | middleware.ts |
| Astro | src/middleware.ts |
| SvelteKit | src/hooks.server.ts |
| Nuxt | server/middleware/zenovay.ts |
| Express | a snippet to paste above your routes |
| Hono | a snippet to paste above your routes |
A purely static site with no server in front of it cannot report crawlers, because there is nothing running when the crawler asks for the page. Putting it behind Cloudflare Pages (or any of the above) is enough.
What you will see
Crawls are grouped by why the crawler came, which is the part that actually changes what you would do about it:
- AI answers — someone asked an assistant a question and it fetched your page to answer, live. ChatGPT, Claude, Perplexity, Gemini.
- Indexing — search and answer engines building an index. Google, Bing, Apple, DuckDuckGo.
- Training — collecting content for model training corpora. OpenAI, Anthropic, Meta, Amazon.
- Agents — something browsing on a person's behalf. ChatGPT Agent, Claude Computer Use.
- Scrapers — commercial crawlers selling crawl feeds on.
- Other — a crawler that identified itself but matches nothing in our registry. New crawlers and small operators land here; it only appears when there is one.
Verified vs unverified
Anything can claim to be ChatGPT in a user agent. Where an operator publishes address ranges or supports reverse DNS, we check the request against them, and a request that claims an identity it cannot support is flagged rather than counted as genuine.
Operators that publish nothing to check against are recorded as unverified. That is an honest "we cannot tell", not a suspicion.
Turning it off
Per site, in Domains → your site → Settings → Advanced. Existing data is kept; new crawls stop being recorded. To remove it entirely, delete the beacon block from the file the CLI wrote — it is self-contained and marked with a zenovay:ai-crawl-beacon comment.
Related
- AI Influence — the dashboard view, including which crawlers your
robots.txtallows - AI crawler directory — every crawler we identify, what it is for, and who runs it
On this page