skip to content

A staged rollout leaves two app versions live against one backend. What must your test coverage prove about that overlap?

level: seniorimportance: must knowfreq 58%

answer

  1. Users update on their own schedule
  2. Two client builds, one deployed service
  3. Run the previous release's suite too
  4. New fields optional for the older build
  5. Start a task, update, finish it

basics

~20 s

Coverage must prove the previous app version still works against the backend the new one requires. Run both versions' suites against the same deployed service in one pass, and cover a user who updates partway through a flow.

solid answer

~50 s

Treat the overlap as a cross-product of two client builds against one service, not as two independent releases. Three claims need cases. First, the **previous app version keeps working unchanged** against the deployment the new release targets. Second, any request or response field the new version introduces is **optional for the older build**, so an unknown field is ignored rather than fatal. Third, records the new build writes stay readable by the older one. Mechanically, keep the previous release's suite runnable and point it at the same deployed service the new suite targets, in the same pipeline run. A green suite for the new version says nothing about the installs still on the old build. Add one case where a task starts on the old build and finishes after an update. If the previous version's suite goes red, the service change is what has to stop.

code

pseudocode · 13 lines
pseudocode
deploy(service, build = new_release_backend)

# both live client builds, one deployment, one pipeline run
for client in [previous_release_client, new_release_client]:
    result = run_suite(client.suite, against = service)
    require(result == PASS, client.version + ' must stay green')

# one task that spans the version change
install(previous_release_client)
start_multi_step_task(step = 1)
install(new_release_client)          # user updates mid-task
resume_multi_step_task()
assert_task_completes()

go deeper

for a junior

Be ready to say why a release does not reach every install at once, and that older app builds keep calling the same backend for days or weeks afterwards.

for a middle

Explain the mechanics: additive, optional fields on the wire; a defined service behaviour when a field is absent; and why the service change ships before the client that depends on it.

for a senior

Show how you prove it. An interviewer expects the previous release's suite pinned and pointed at the new deployment in the same run, a case where a task spans an update, and a rule that a red old-version suite stops the release.

for a principal

Own the exit condition. Decide what share of active installs justifies keeping compatibility code and its cases, publish that threshold, and retire the tolerance and its tests together rather than accumulating branches nobody dares delete.

## Why two app versions are live at once An installed application updates on the user's schedule, not the team's. A new version reaches devices over hours or days: some installs take it immediately, most arrive later, and a tail never arrives at all because automatic updates are off, storage is full, or the device no longer accepts the build. A staged release makes this deliberate — the new version is held to a fraction of installs so that problems surface small — but even an unstaged release produces the same condition. **From the first install of a new app version until the last of the previous one is gone, two client builds are talking to one deployed service.** That window is the normal condition of a shipped product, and it is the condition most suites never exercise. A suite is written alongside the new client, runs the new client, and goes green — which says nothing about the installs that are still on the old build and are hitting the same service you just changed. ## The claims that need their own cases | Combination | The claim under test | How it breaks | |---|---|---| | Previous app version against the new deployment | The old build keeps working unchanged | The service added a required request field, tightened validation, or stopped returning a field the old build reads | | New app version against the new deployment | The new build works | The one combination ordinary release testing already covers | | Previous app version reading records the new build wrote | The old build understands what the new one produces | The new build writes a shape or a category value the old build has no branch for | | One task that spans the version change | A flow started on the old build finishes on the new one | A half-written server-side record is picked up by different client code | Only the second row is normally automated. The first and third are where a rollout gets halted. ## Designing the cases 1. **Keep the previous release's suite runnable and point it at the new deployment.** Not a copy of a few cases — the pinned suite from the last release, executed in the same pipeline run as the new one, against the same deployed service. It is the only mechanical proof that the old build still works. 2. **Assert that wire changes are additive.** A new request field is optional and the service has a defined behaviour when it is absent. A new response field is ignorable. Removing or renaming anything is a change the old build cannot survive, so it is a separate, later release. 3. **Write a forward-compatibility case for the old build.** Have the new build produce a record carrying a category value the old build predates, then assert the old build renders something sane rather than failing. That unknown values must degrade gracefully is a design decision, and this case is what pins it down. 4. **Write one case that spans the update.** Start a multi-step task on the previous build, install the new build, resume, and assert the task completes. This is the only case where a single logical operation is split across two client versions. 5. **Make the old-version suite a release gate.** If it goes red, the service change stops. Rolling the client back does not help the installs that already took it. ## What this coverage is not - It is not coverage of which devices or platform versions the build runs on; that is a different selection question with its own cases. - It is not a check that locally saved data survives an upgrade or a rollback; that is its own compatibility concern and needs its own cases. - It is not the rollout machinery itself — how the fraction is chosen, how it is halted. Case design assumes the overlap exists and proves the product works inside it. ## Order of operations, and the exit The safe sequence is always the same, and it is three releases rather than one: - **First**, ship the service change that tolerates both shapes, and prove the previous client's suite is still green against it. - **Then** ship the client version that depends on the new shape. - **Finally**, once the old version's share of active installs is negligible, remove the tolerance and the compatibility cases together. That last step is the one teams skip, and skipping it is how a service accumulates branches nobody can date or delete. Decide the threshold in advance — a share of active installs, measured rather than guessed — and treat compatibility code and the cases protecting it as one unit with a planned end. The overlap is temporary by design, and the code that serves it should be too.

  • The new version needs a request field the old one never sends. How do you ship it without breaking the old installs?
    Make the field optional on the wire and give the service a defined behaviour when it is absent — a default, or a path that skips the feature. Ship the service change first and prove the previous version's suite is still green against it, then release the client that populates the field. Removing the tolerance is a separate, later change once the old version's share is negligible.
  • How long do you keep testing the previous app version?
    Until its share of active installs falls below a threshold you decided in advance and can measure — not until the next release simply feels done. Keep the pinned suite in the pipeline for that whole period. When you retire it, retire the service-side tolerances it was protecting in the same change, so compatibility code does not outlive its reason.
  • Why is a case that updates the app partway through a task worth the trouble?
    Because it is the only case where one logical operation is split across two client builds. Whatever the first half left half-written — a draft, a queued upload, a multi-step form's server-side record — is now read by different code. Users hit this constantly, and it is invisible to a suite that installs one version and never changes it.

It is the same problem as a bridge open to traffic while it is being widened: the new lanes have to work, and so do the old ones, with the same cars on both at once.

saying these in an interview costs you the question

  • Assumes everyone is on the newest version after release day
  • Runs only the new version's suite before shipping
  • Adds a required request field and ships the client simultaneously
  • Treats old-version failures as the user's fault for not updating
  • Claims a forced update makes the overlap disappear