Temporal and Group Overlap

## The independence assumption the split relies on A held-out validation set estimates generalisation only if the held-out rows are, in the relevant sense, new. Random row-level splitting assumes rows are exchangeable draws. Behavioural data almost never is: one user generates hundreds of sessions, one patient many visits, one device many readings, one merchant many transactions. Those rows share a hidden cause - the entity itself. ## What the model actually learns With 400 sessions from one user scattered across folds, roughly four fifths of them sit in training whenever that user's remaining rows are being scored. The model does not need to identify the user explicitly to exploit this. Ordinary features act as a fingerprint: a rare city plus a specific device type plus a characteristic session length can single a person out almost uniquely. Even a linear model will tilt its coefficients toward whatever combination that heavy user shows, because those rows are numerous and mutually consistent. A high-capacity model does it far more aggressively, effectively storing "rows that look like this behave like that". So the model learns *this user*, and is then rewarded for it at validation time. Nothing generalises. ## Three distinct damages **1. Optimistic point estimate.** The validation score measures within-entity prediction while you intend to report cross-entity prediction. The two can differ enormously - it is entirely normal for a model that looks strong on seen users to be near-useless on new ones, because the useful signal was identity, not behaviour. **2. A metric dominated by a few entities.** Those 400 rows out of, say, 5,000 are 8% of the metric from one person. Aggregate performance now partly reports how well the model fits a handful of heavy users, and improvements that only help them look like general improvements. Reporting the metric averaged per entity as well as per row exposes this. **3. Understated uncertainty.** Fold scores agree suspiciously well, because folds share entities and are therefore correlated. The fold-to-fold spread, which people routinely use as an error bar, becomes far too tight, and a difference between two candidate models looks significant when it is not. ## When entity overlap is actually fine This is the part weaker candidates miss. The split should mirror deployment. If the model exists to score the *next* session of customers you already have - a churn score refreshed monthly on an existing base, a recommender for logged-in users - then seeing a user in training is exactly what happens in production, and forbidding it produces a needlessly pessimistic estimate. What still must hold in that case is the time direction: train on their earlier rows, validate on their later ones. "Same user, later time" is a legitimate evaluation; "same user, random rows" is not, because it lets the model see a user's future while predicting their past. If instead the model will meet strangers - onboarding risk scoring, cold-start ranking, a diagnostic deployed to a new clinic - the held-out entities must be unseen, and a row-level split answers the wrong question entirely. ## Detecting it The direct check is trivial and worth automating: intersect the entity identifiers of each pair of folds; the intersection should be empty when it is meant to be. Alongside it, count rows per entity and look at the tail - a Zipf-like distribution where the top 1% of entities hold a large share of rows is a warning that the estimate is entity-sensitive. The hard case is when there is no identifier column at all. Near-duplicates still create the same coupling: the same marketplace listing reposted four times with a tweaked title and price yields four rows whose targets are essentially one observation. If they straddle folds the model is scored on a row it has already read. Here you have to reconstruct the grouping - hash or fuzzy-match the stable fields, cluster rows that are near-identical on their features, or exploit a surrogate key such as an image fingerprint or a normalised text body - and then treat each cluster as the unit that must stay intact. ## The remedy, stated carefully The rule is: the unit of splitting must be the unit of independence. Whatever the entity is - user, patient, listing, merchant - all of its rows go to one side. Deduplicating instead is usually wrong: it discards real repeated observations that carry information, and it does not help when the duplicates are merely similar rather than identical. Nor does a different random seed help; the problem is structural, not a bad draw. One consequence to expect and accept: the honest score is lower, sometimes dramatically. That number is the one worth optimising, because it is the only one production can reproduce.