Verified Bots
Which crawlers are real, and how to prove it.
Three ways to know it is really them.
A user agent is a claim, not proof — anything can send one. The tier is how much of that claim a bot's operator has actually made checkable.
Tier 1
Fully verifiable
The operator publishes a machine-readable IP feed, or supports Web Bot Auth. You can confirm a request came from them without trusting anything it told you.
Tier 2
Verifiable
No live feed, but a documented reverse-DNS pattern, ASN or static range. Checkable, with a round trip and a little more care.
Tier 3
Listed only
The identity is known and the user agent is documented, but nobody published a way to check it. Recorded honestly as unverifiable rather than quietly implied to be safe.
Of the 223 bots catalogued, 120 carry a recipe you can actually run — 59 at tier one and 61 at tier two. The directory says which, including when the answer is that there is not one.
How it works
Five steps, in the order they happen.
- 01
One bot, one file
Every entry is a YAML file validated against a published JSON schema. Adding a bot or correcting a range is a pull request, not a private database edit.
- 02
Sources refresh daily
Operators publish their own ranges, and the pipeline pulls them every day rather than trusting a list someone wrote down once.
- 03
A bad feed cannot poison it
CIDR sanity checks, a diff alarm when a range list swings unexpectedly, and automatic fallback to the last known-good feed when a source looks wrong.
- 04
Every bot gets a tier
Tier 1 is fully verifiable — a machine-readable IP feed or Web Bot Auth. Tier 2 is verifiable by reverse DNS, ASN or static ranges. Tier 3 is listed only: the identity is known, but nobody published a way to check it.
- 05
Then it is just files
It builds to all.json, per-category and per-tier sets, individual bot profiles, user-agent patterns, and plain .ips lists a firewall can read directly.
The facts
Everything below is verifiable. Nothing is rounded up.
- Bots catalogued
- 223 across 10 categories
- With a verification recipe
- 120 of 223
- IP ranges tracked
- 9,232
- Largest categories
- Search engines 53, SEO tools 38, Monitoring 37
- AI crawlers and assistants
- 32
- Refresh
- Daily, with poisoned-feed safeguards
- Formats
- JSON, .ips lists, RSS
- Licence
- MIT for code, CC-BY-4.0 for data
- Affiliation
- Independent — not affiliated with Cloudflare
- Figures as of
- 9 August 2026