skip to content

After a marketplace's Cloudflare custom rule blocked bot scores below 30, its search crawlers and in-house uptime checker were blocked too; why, and what fixes each?

level: seniorimportance: should knowfreq 34%

answer

  1. automation is not malice
  2. verified status is a separate flag
  3. a private checker is never verified
  4. scoped Skip ahead of the score rule

basics

~10 s

Crawlers and checkers are automated, so they score below 30. Adding not cf.bot_management.verified_bot lets Cloudflare's verified crawlers pass; the in-house checker is not verified, so it needs a narrowly scoped Skip rule placed first.

solid answer

~40 s

The bot score measures automation, not intent, so a search crawler and an uptime script both score low and a bare `cf.bot_management.score lt 30` rule blocks them. Search crawlers and many monitoring services are **verified bots**: Cloudflare has confirmed who operates them through Web Bot Auth signatures, published IP lists or reverse DNS, and `cf.bot_management.verified_bot` is true for them, so the rule needs `and not cf.bot_management.verified_bot`. The in-house checker appears in no directory, so that flag is false for it; give it a *Skip* rule ordered before the score rule and matched on its source IP list and health-check path only, never on its `User-Agent`. Then scope the blocking rule to checkout, add `not cf.bot_management.static_resource`, prefer Managed Challenge for scores 2-29, and confirm in Security Events that crawler hits stopped.

code

plaintext · 8 lines
plaintext
Rule 1 - action Skip (all remaining custom rules)
(ip.src in $uptime_checkers and http.request.uri.path eq "/healthz")

Rule 2 - action Block
(cf.bot_management.score eq 1 and not cf.bot_management.verified_bot)

Rule 3 - action Managed Challenge
(starts_with(http.request.uri.path, "/checkout") and cf.bot_management.score lt 30 and not cf.bot_management.verified_bot and not cf.bot_management.static_resource)

go deeper

for a junior

Recall that a low bot score means automated, not malicious, and that verified bots are flagged separately by cf.bot_management.verified_bot.

for a middle

Explain how a bot becomes verified, Web Bot Auth, IP lists or reverse DNS, why an in-house script never is, and how cf.client.bot covers zones without the add-on.

for a senior

Diagnose from Security Events, fix both populations with different tools, keep the checker's exception to its path, and prove the fix in Bot Analytics and crawl statistics.

for a principal

Treat exceptions as owned assets: who may add an IP to the skip list, how often it is reviewed, and how a new partner's automation gets in without widening the hole.

## Why the rule caught good automation A **bot score** says how likely a request is to be automated. It does not say whether the automation is welcome. Search engine crawlers fetch pages with no human behind them, and a script that requests `/healthz` every minute from a data centre looks exactly like what it is. Both score low, so a rule such as `(cf.bot_management.score lt 30)` with *Block* stops them along with the scalpers it was written for. The symptoms follow quickly: listing pages drop out of search results, and the on-call engineer is paged for an outage that is really the edge blocking the monitor. ## Verified bots and what makes one verified A **verified bot** is one Cloudflare has confirmed is transparent about who it is and does not abuse that access. Two bars apply: - **Honest self-identification**, deterministically: a cryptographic **Web Bot Auth** signature, a published IP list with a stable `User-Agent`, or reverse DNS that resolves to the operator's domain. - **Non-abusive behaviour**: it obeys `robots.txt` and crawl directives and keeps reasonable request rates. A bot that breaches the policy is removed from the allowlist. Verified bots and agents are listed in **BotBase** and shared through Cloudflare Radar's bots and agents directory. Since 1 July 2026, BotBase classifies each one by behaviour (Search, Agent, Training, Monitoring & Operations and others) and labels it Direct or Intermediary. Rules keep working on the older taxonomy: `cf.verified_bot_category` still matches legacy strings such as `"Search Engine Crawler"` and `"Monitoring & Analytics"`. In rules, verification is a boolean separate from the score: | Field | Type | What it tells a rule | | --- | --- | --- | | `cf.bot_management.verified_bot` | Boolean | The request comes from a known good bot; needs the Enterprise Bot Management add-on | | `cf.client.bot` | Boolean | The same information, without the add-on | | `cf.verified_bot_category` | String | The verified bot's type and purpose, in the legacy categories | ## Fixing the crawlers Exclude verified bots from every score-based rule, either inline or with a *Skip* rule evaluated first: - inline: `(cf.bot_management.score lt 30 and not cf.bot_management.verified_bot)`; - or a first rule `(cf.bot_management.verified_bot)` with *Skip* for all remaining custom rules, Cloudflare's own baseline. Do not exempt crawlers by `User-Agent`. Anyone can send a crawler's user agent string; verification exists precisely because that string is only a claim. ## Fixing the in-house checker The team's checker runs from its own servers and is in no directory, so `cf.bot_management.verified_bot` is false for it and no zone setting changes that. BotBase does accept submissions, but approval requires Web Bot Auth or IP validation and a disclosed purpose that matches the observed traffic; for a monitor that only ever checks the team's own site, an exception the team controls is simpler: 1. Put the checker's egress addresses in an IP list. 2. Add a custom rule, ordered before the score rules, matching that list **and** the health-check path, with *Skip* for the remaining custom rules. 3. Keep the exception that narrow: the same IPs requesting `/checkout` are still scored. ## Rolling the fix out and proving it 1. Open **Security Events**, filter on the blocking rule, and look at the user agents, ASNs and verified-bot status of what it caught. 2. Deploy the corrected expressions, first with a non-blocking action if the rule is broad, then enforce. 3. Narrow the score rule to the paths automated buyers target, add `not cf.bot_management.static_resource`, and use *Managed Challenge* for scores 2-29 so a real buyer who lands there can still pass. 4. Watch Bot Analytics and the crawler's own crawl statistics recover over the following days. On zones without the add-on, the same idea applies: Super Bot Fight Mode's **Verified bots** setting on Allow keeps crawlers in, and a custom rule can use `cf.client.bot`. ## Mistakes to avoid while fixing it - **Allowlisting by user agent.** It turns a public string into a bypass for every scalper who copies it. - **A Skip with no path condition.** It exempts the checker's addresses everywhere, including checkout, so a compromised monitoring host becomes a free pass. - **Blocking verified bots to stop scraping.** Verified crawlers obey crawl directives; price scrapers that pretend to be crawlers are not verified and stay in scope of the score rules.

  • How do you check whether a specific crawler is a Cloudflare verified bot?
    Look it up in Cloudflare Radar's bots and agents directory, which shares BotBase. In Security Events or Security Analytics, filter its requests by verified-bot status: a genuine crawler shows as verified, while a scraper borrowing the crawler's user agent does not. `cf.verified_bot_category` then tells you which category it belongs to.
  • Why not exempt the in-house checker by its User-Agent string?
    A user agent is a claim any client can make. A rule that skips bot scoring for a fixed string hands every scalper that string as a bypass. Match the checker on what it cannot easily fake, its source IP list, and only on the path it monitors, so even a request from those addresses to checkout is still scored.

  • Good bots get high bot scores, so a score threshold never touches crawlers.
  • Allowing crawlers by their User-Agent string is as safe as verified_bot.
  • The in-house monitor will become a verified bot on its own over time.
  • Skip every security feature for the monitor's IPs on every path.
  • Verified bots must be blocked at checkout to stop price scraping.

Assumes Cloudflare as of 2026-10-10: since 1 July 2026 BotBase classifies verified bots by behaviour, while cf.verified_bot_category still matches the legacy category strings.