skip to content

Pattern & Entropy Scanning

Searching code, configuration and version-control history for credential-shaped strings by known format, by randomness, and by whether they still work. Asked because a noisy scanner gets switched off.

on this pageshow

questions

5

A credential scan over a repository's checked-out default branch reports no findings — what did that scan not look at?

level: juniorimportance: must knowfreq 62%

answer

  1. one snapshot is not the corpus
  2. branches and tags, not just the default
  3. history keeps what the tree dropped
  4. deleted branches still hold file versions
  5. forks and clones sit outside your scan

basics

~20 s

One snapshot of one branch was searched. Every other branch and tag, every earlier version of every file on all of them, and every copy living outside that repository — forks, clones, mirrors, exported archives — were never read.

solid answer

~50 s

A checked-out tree is the current state of a single branch, and a credential usually leaves the current state long before anyone runs a scan: someone noticed, edited the file and carried on. The material worth searching is the history — every version of every file, on every branch and tag, including branches that were merged and deleted, whose commits many hosting systems keep for a while. Outside the repository the scanner cannot reach at all: a fork is a separate repository, and every clone on a laptop or build agent holds a full copy of whatever history it fetched. So a clean result means clean in the surface that scan was pointed at, which is a much smaller claim than it sounds. Configuring the scan over all branches and full history, and enumerating known forks and mirrors, is what makes the statement worth quoting.

code

pseudocode · 15 lines
pseudocode
# the surface a scan can be pointed at, widest first
for each branchOrTag in allBranches(repo) + allTags(repo):
    for each commit in history(branchOrTag):
        for each file in filesChangedIn(commit):
            scan(contentsOf(file, commit))

# often forgotten: recorded commits no longer on any branch tip
for each commit in commitsNoLongerOnAnyBranch(repo):
    for each file in filesChangedIn(commit):
        scan(contentsOf(file, commit))

# outside every loop above, and outside this scanner's reach:
#   forks owned by other people
#   clones on laptops and build agents
#   mirrors and exported archives

go deeper

for a junior

Know that the checked-out files are one snapshot of one branch, and that a scan pointed at them misses every earlier version, every other branch, and every copy held somewhere else.

for a middle

Explain why the current state is the least likely place a committed credential survives, and which parts of the history a first scan configuration usually leaves out.

for a senior

Show that you scope the verdict: say which surfaces were searched, which are out of reach, and how you enumerate forks and mirrors before quoting a result to anyone.

for a principal

Decide what the organisation will claim from a scan and to whom. The standard that matters is whether a clean result is recorded with its surface attached, so nobody later reads it as a guarantee it never was.

## What a checked-out tree actually is The checked-out tree is one snapshot: the content of one branch at one moment. A scan pointed at it answers a narrow question — *does any file, as it stands right now on this branch, contain something credential-shaped?* That is a useful question, but it is not the question people think they asked when they say the repository is clean. Credential material has a characteristic life in a repository. It is added, it works, and at some point someone notices and edits the file. The current state is therefore the one place a committed credential is **least** likely to be, and a scan restricted to it is structurally biased towards finding nothing. ## The history is the corpus A version-control repository keeps every version of every file that was ever recorded, not just the latest. A scan configured over history reads all of them. Several parts of that history are easy to leave out of a first configuration: - **Other branches.** Long-lived release branches and maintenance branches each carry their own file versions. - **Tags.** A tag pins a point in history that may predate the cleanup. - **Review branches that were never merged.** The change was abandoned; the recorded content is still there. - **Branches that were merged and deleted.** Their commits are often no longer reachable from any branch tip, but the objects usually remain on the hosting system until it prunes them, and hosting systems differ in when that happens. - **Earlier versions of a file that still exists.** Nothing about the file's current contents tells you what it held two years ago. ## Copies the scan cannot reach at all The second half of the answer is not about configuration; it is about reach. Some copies are outside the scanner's control entirely: - **Forks.** A fork is a separate repository with its own owner and its own scan settings, and it holds the history it was forked from. You may not be able to enumerate every fork of a repository at all. - **Clones.** Every developer laptop and build agent that ever fetched the repository holds a complete copy of the history it fetched, and nothing you do centrally touches it. - **Mirrors and exported archives.** A backup export, an internal mirror or a release archive carries whatever it captured. - **Caches inside the hosting system.** Some hosts will still serve a commit by its identifier after it stops being reachable from any branch; behaviour differs between systems, so the safe assumption is that it is reachable. | Surface | Read by a default-branch scan | Read by an all-branches history scan | Reachable by you at all | |---|---|---|---| | Current files on the default branch | yes | yes | yes | | Older versions of those files | no | yes | yes | | Other branches and tags | no | yes | yes | | Commits on deleted branches | no | sometimes, until pruned | partly | | Forks | no | no | rarely | | Clones on laptops and agents | no | no | no | ## What a clean result may honestly claim The verdict is scoped to the surface. "No findings" from a default-branch scan means *nothing matched, in one snapshot of one branch, with the engines that were enabled*. It is not a statement about the repository, and it is certainly not a statement about every copy of the repository. When an external review asks what the scan actually covered, the answer is a list of surfaces, not a yes or no. ## Widening the search, in order 1. Move from the checked-out tree to **full history on the default branch**. 2. Extend to **every branch and every tag**, including ones nobody has touched in a year. 3. Add whatever the hosting system still holds from **deleted branches**, where the tooling can see it. 4. Enumerate **forks and known mirrors**, and get them scanned by whoever owns them. 5. Accept that **clones are out of reach**, and let that shape what you conclude from any find rather than what you scan. What you then do about a credential that history turns out to contain — and why removing it from history is not the end of the work — is a separate subject with its own answer. This question is only about whether the scan could see it in the first place.

  • Why is the current state of a branch the least likely place to find a committed credential?
    Because the obvious reaction to noticing one is to edit the file. That edit changes the current state and leaves the recorded earlier version untouched, so the value survives exactly where a tree-only scan does not look.
  • A fork was made before the cleanup. What can you actually do about it?
    Nothing to its contents unilaterally — it is someone else's repository. You can enumerate the forks you can see, ask their owners to scan and act, and treat the value as reachable regardless. The scan result for your repository says nothing about theirs.
  • Does scanning every branch and tag make the result complete?
    No. It makes it complete for that repository as the hosting system holds it. Clones already taken, mirrors and exported archives are unaffected by anything you run centrally, so the honest claim stays scoped to what was searched.

saying these in an interview costs you the question

  • Says a clean working tree means the repository is clean
  • Thinks history only matters while the branch still exists
  • Assumes a fork inherits the parent repository's scan coverage
  • Believes a value removed from the current files is unreadable
  • Counts a default-branch scan as full coverage of the repository
open as a page

A scanner runs both a known-format catalogue and an entropy heuristic — why does a human-chosen database password clear both?

level: middleimportance: must knowfreq 58%

basics

~20 s

Neither engine has anything to work with: no issuer stamped a recognisable prefix, length or checksum on a password a person typed, and at under twenty characters it sits below the minimum length an entropy score needs before it means anything.

open as a page

A credential scanner's entropy heuristic flags a 40-character random build identifier on every run — why does that value score like a secret?

level: juniorimportance: should knowfreq 50%

basics

~20 s

An entropy heuristic measures randomness and length, not meaning: a 40-character hexadecimal build identifier has the same character spread as a freshly minted key, so it clears the threshold. Nothing in the string says what produced it.

open as a page

The first credential scan across every branch and fork of your search service returns 3,000 findings and the team mutes it — what do you change?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Split it into two jobs: a one-off backlog, ranked by whether each value is still accepted and what it reaches; and a narrow, high-precision check on new material that nobody may mute. Suppress single findings, never engines.

open as a page

A teammate proposes confirming a flagged credential by calling the service with it — whose records does that attempt land in?

level: seniorimportance: nice to knowfreq 27%

basics

~20 s

The service being called records it — a successful authentication attributed to the credential's identity, from your address. If that service belongs to a partner, your test appears in their records as their credential used from somewhere new.

open as a page