When you call the reference customers a vendor gives you, why is that alone not enough to trust reference checks, and what should you actually do to get more honest signal?

## What a reference check is A reference check is a **structured conversation with a vendor's existing customer** intended to validate claims made during the sales process against lived experience. The vendor typically supplies a list of two to four reference customers as part of the RFP response; the evaluator schedules calls, usually 30-45 minutes, and asks about: - implementation experience - ongoing product performance - support responsiveness - whether the customer would choose the product again Done well, this is one of the highest-signal steps in the entire evaluation, because it's the only point in the process where you hear from someone with no financial stake in your decision. ## The structural problem: selection bias The structural problem is **selection bias**: the vendor chooses who's on the reference list, and no vendor hands out a customer having a bad experience. Worse, many vendors maintain a small rotating pool of reference customers, often ones who received a discount or extra support specifically in exchange for being available for reference calls, so the same few voices get reused across many prospective buyers' evaluations, and their experience may not be representative of a typical customer at all. A reference check that stops at calling the names on the list and asking if they're happy is, in effect, asking the vendor's biggest fans if they're fans, and getting yes is not informative. ## What honest signal costs Getting past vendor-curated references costs effort: - Sourcing independent references means networking through professional contacts, industry communities, conference contacts, or asking your own team if anyone has prior experience with the vendor, none of which is guaranteed to produce a willing contact, and it takes real calendar time to chase down. - There's also a diminishing-returns point: for a low-stakes, low-cost tool, spending days hunting independent references is disproportionate. The effort should **scale with contract size and criticality** — a multi-year, mission-critical platform deserves this rigor, a low-cost SaaS tool for one team usually doesn't. ## Where it goes wrong 1. The most common failure is **asking only satisfaction questions** like 'are you happy with the product,' which invites a socially easy, non-committal positive answer even from someone with real complaints, especially since the reference knows the vendor will likely hear about the call. A better failure-resistant technique is asking specific, falsifiable, behavioral questions: what was the support ticket response time on the last major incident, what feature was promised that hasn't shipped yet, would you re-sign today knowing everything you know now — questions that are harder to answer vaguely and that surface concrete data points rather than vibes. 2. A second failure is **only talking to the champion** who bought and championed the tool internally, who has an ego stake in it succeeding, rather than the engineers or analysts who actually use it daily and have no stake in defending the purchase decision. 3. A third failure is **stale references**: a name given as a reference from years ago may predate the vendor's product roadmap changes, team turnover, an acquisition, or a pricing model change, so the experience described no longer reflects what a new customer would get. ## A worked example A team evaluating a monitoring and observability vendor calls the three vendor-provided references, all of whom report smooth implementations and responsive support, unsurprising, since the vendor selected them. The evaluation lead separately posts in a relevant industry community asking if anyone uses the vendor's platform at scale, good or bad experiences welcome, and gets two independent responses: - one confirms the positive experience - another describes a period where the vendor's ingestion pipeline silently dropped data during a traffic spike and support took days to acknowledge the root cause That second, unsolicited data point, which never would have surfaced from the vendor's own reference list, directly informs a decision to add a specific SLA clause around data-loss incidents to the contract, something the vendor-curated references gave no reason to even ask about.