After a marketplace's Cloudflare custom rule blocked bot scores below 30, its search crawlers and in-house uptime checker were blocked too; why, and what fixes each?
answer
- automation is not malice
- verified status is a separate flag
- a private checker is never verified
- scoped Skip ahead of the score rule
basics
~10 sCrawlers and checkers are automated, so they score below 30. Adding not cf.bot_management.verified_bot lets Cloudflare's verified crawlers pass; the in-house checker is not verified, so it needs a narrowly scoped Skip rule placed first.
solid answer
~40 sThe bot score measures automation, not intent, so a search crawler and an uptime script both score low and a bare `cf.bot_management.score lt 30` rule blocks them. Search crawlers and many monitoring services are **verified bots**: Cloudflare has confirmed who operates them through Web Bot Auth signatures, published IP lists or reverse DNS, and `cf.bot_management.verified_bot` is true for them, so the rule needs `and not cf.bot_management.verified_bot`. The in-house checker appears in no directory, so that flag is false for it; give it a *Skip* rule ordered before the score rule and matched on its source IP list and health-check path only, never on its `User-Agent`. Then scope the blocking rule to checkout, add `not cf.bot_management.static_resource`, prefer Managed Challenge for scores 2-29, and confirm in Security Events that crawler hits stopped.
code
plaintext · 8 linesRule 1 - action Skip (all remaining custom rules)
(ip.src in $uptime_checkers and http.request.uri.path eq "/healthz")
Rule 2 - action Block
(cf.bot_management.score eq 1 and not cf.bot_management.verified_bot)
Rule 3 - action Managed Challenge
(starts_with(http.request.uri.path, "/checkout") and cf.bot_management.score lt 30 and not cf.bot_management.verified_bot and not cf.bot_management.static_resource)go deeper
Recall that a low bot score means automated, not malicious, and that verified bots are flagged separately by cf.bot_management.verified_bot.
Explain how a bot becomes verified, Web Bot Auth, IP lists or reverse DNS, why an in-house script never is, and how cf.client.bot covers zones without the add-on.
Diagnose from Security Events, fix both populations with different tools, keep the checker's exception to its path, and prove the fix in Bot Analytics and crawl statistics.
Treat exceptions as owned assets: who may add an IP to the skip list, how often it is reviewed, and how a new partner's automation gets in without widening the hole.
## Why the rule caught good automation A **bot score** says how likely a request is to be automated. It does not say whether the automation is welcome. Search engine crawlers fetch pages with no human behind them, and a script that requests `/healthz` every minute from a data centre looks exactly like what it is. Both score low, so a rule such as `(cf.bot_management.score lt 30)` with *Block* stops them along with the scalpers it was written for. The symptoms follow quickly: listing pages drop out of search results, and the on-call engineer is paged for an outage that is really the edge blocking the monitor. ## Verified bots and what makes one verified A **verified bot** is one Cloudflare has confirmed is transparent about who it is and does not abuse that access. Two bars apply: - **Honest self-identification**, deterministically: a cryptographic **Web Bot Auth** signature, a published IP list with a stable `User-Agent`, or reverse DNS that resolves to the operator's domain. - **Non-abusive behaviour**: it obeys `robots.txt` and crawl directives and keeps reasonable request rates. A bot that breaches the policy is removed from the allowlist. Verified bots and agents are listed in **BotBase** and shared through Cloudflare Radar's bots and agents directory. Since 1 July 2026, BotBase classifies each one by behaviour (Search, Agent, Training, Monitoring & Operations and others) and labels it Direct or Intermediary. Rules keep working on the older taxonomy: `cf.verified_bot_category` still matches legacy strings such as `"Search Engine Crawler"` and `"Monitoring & Analytics"`. In rules, verification is a boolean separate from the score: | Field | Type | What it tells a rule | | --- | --- | --- | | `cf.bot_management.verified_bot` | Boolean | The request comes from a known good bot; needs the Enterprise Bot Management add-on | | `cf.client.bot` | Boolean | The same information, without the add-on | | `cf.verified_bot_category` | String | The verified bot's type and purpose, in the legacy categories | ## Fixing the crawlers Exclude verified bots from every score-based rule, either inline or with a *Skip* rule evaluated first: - inline: `(cf.bot_management.score lt 30 and not cf.bot_management.verified_bot)`; - or a first rule `(cf.bot_management.verified_bot)` with *Skip* for all remaining custom rules, Cloudflare's own baseline. Do not exempt crawlers by `User-Agent`. Anyone can send a crawler's user agent string; verification exists precisely because that string is only a claim. ## Fixing the in-house checker The team's checker runs from its own servers and is in no directory, so `cf.bot_management.verified_bot` is false for it and no zone setting changes that. BotBase does accept submissions, but approval requires Web Bot Auth or IP validation and a disclosed purpose that matches the observed traffic; for a monitor that only ever checks the team's own site, an exception the team controls is simpler: 1. Put the checker's egress addresses in an IP list. 2. Add a custom rule, ordered before the score rules, matching that list **and** the health-check path, with *Skip* for the remaining custom rules. 3. Keep the exception that narrow: the same IPs requesting `/checkout` are still scored. ## Rolling the fix out and proving it 1. Open **Security Events**, filter on the blocking rule, and look at the user agents, ASNs and verified-bot status of what it caught. 2. Deploy the corrected expressions, first with a non-blocking action if the rule is broad, then enforce. 3. Narrow the score rule to the paths automated buyers target, add `not cf.bot_management.static_resource`, and use *Managed Challenge* for scores 2-29 so a real buyer who lands there can still pass. 4. Watch Bot Analytics and the crawler's own crawl statistics recover over the following days. On zones without the add-on, the same idea applies: Super Bot Fight Mode's **Verified bots** setting on Allow keeps crawlers in, and a custom rule can use `cf.client.bot`. ## Mistakes to avoid while fixing it - **Allowlisting by user agent.** It turns a public string into a bypass for every scalper who copies it. - **A Skip with no path condition.** It exempts the checker's addresses everywhere, including checkout, so a compromised monitoring host becomes a free pass. - **Blocking verified bots to stop scraping.** Verified crawlers obey crawl directives; price scrapers that pretend to be crawlers are not verified and stay in scope of the score rules.
- How do you check whether a specific crawler is a Cloudflare verified bot?Look it up in Cloudflare Radar's bots and agents directory, which shares BotBase. In Security Events or Security Analytics, filter its requests by verified-bot status: a genuine crawler shows as verified, while a scraper borrowing the crawler's user agent does not. `cf.verified_bot_category` then tells you which category it belongs to.
- Why not exempt the in-house checker by its User-Agent string?A user agent is a claim any client can make. A rule that skips bot scoring for a fixed string hands every scalper that string as a bypass. Match the checker on what it cannot easily fake, its source IP list, and only on the path it monitors, so even a request from those addresses to checkout is still scored.
- Good bots get high bot scores, so a score threshold never touches crawlers.
- Allowing crawlers by their User-Agent string is as safe as verified_bot.
- The in-house monitor will become a verified bot on its own over time.
- Skip every security feature for the monitor's IPs on every path.
- Verified bots must be blocked at checkout to stop price scraping.
Assumes Cloudflare as of 2026-10-10: since 1 July 2026 BotBase classifies verified bots by behaviour, while cf.verified_bot_category still matches the legacy category strings.