skip to content

How would you roll out a new version of a payments webhook function with a weighted alias shift and automatic rollback?

level: middleimportance: must knowfreq 65%

answer

  1. small share first, then the rest
  2. two versions behind one name
  3. alarms scoped to the newcomer
  4. bake time and a minimum count
  5. rollback restores the old weights

basics

~20 s

Publish the new version, weight the alias to send it a small share of invocations, and watch its own alarms through a bake period. An alarm makes the deploy tool repoint the alias back; a quiet bake promotes the new version.

solid answer

~50 s

The deployment template declares the function, its permissions, the `live` alias and the shift policy, so the rollout is reviewed like code. The pipeline publishes the new version and runs a pre-traffic check, such as a signed test webhook of each event type invoked directly against it. The deployment tool then sets the alias weights so a small share of invocations reaches the new version, and watches alarms scoped to that version: its error rate, its latency and a business signal such as signature failures. If an alarm fires during the bake, the tool restores full weight to the old version; if the bake passes with enough invocations, it promotes the new version. Two function-specific traps: weights usually apply per invocation, so a sender's retry can reach either version, and only triggers wired to the alias are shifted at all.

code

yaml · 12 lines
yaml
function: payments-webhook
alias: live
release:
  shift: canary
  newVersionWeight: 10        # percent of invocations during the bake
  bakePeriod: 10m
  minInvocations: 500
  preTrafficCheck: send-signed-sample-of-each-event-type
  rollbackOn:
    - alarm: webhook-errors-new-version
    - alarm: webhook-p99-latency-new-version
    - alarm: webhook-signature-failures-new-version

go deeper

for a junior

Recall that a new version first gets a small share of traffic through the alias, and that an alarm can send all traffic back to the old version.

for a middle

Explain the sequence: publish, pre-traffic check, weighted shift, bake against per-version alarms, then promote or restore the old weights.

for a senior

Show the traps you have met: diluted combined alarms, quiet overnight canaries, triggers bypassing the alias, and data changes the old version cannot read after rollback.

for a principal

Discuss which shift shape and alarm set a team should make the default for every function, and when an all-at-once release is an acceptable exception.

## What a weighted alias shift is A **weighted alias shift** sends a chosen share of the invocations that reach an alias to a new published version, while the rest keep running the old one. Instead of moving the alias all at once, the deployment tool points the alias at two versions with weights, watches the new version, and then either promotes it to full traffic or puts the weights back. On a payments webhook released several times a day, this is what stops one bad release from rejecting or mishandling every payment notification at once. ## The release, step by step 1. **Package and declare.** A deployment template declares the function, its permissions, its triggers, the `live` alias and the shift policy, so the rollout itself is reviewed and versioned like code. 2. **Publish** the new version. It exists, but no traffic reaches it yet. 3. **Run a pre-traffic check.** Invoke the new version directly with a signed test webhook of every event type the function handles, and fail the release before any real sender reaches it. 4. **Shift a small share** of invocations through the alias weights, for example 10% in this scenario. 5. **Bake.** Watch alarms scoped to the new version for a fixed period and until it has handled a minimum number of invocations. 6. **Promote or roll back.** If no alarm fired, the tool gives the new version full weight; if one did, it restores full weight to the old version and marks the deployment failed. ## Choosing the shift shape | Shape | What the alias does | Fits when | |---|---|---| | All at once | Moves fully to the new version | Low-risk changes, or a function with almost no traffic | | Canary | A small share for one bake period, then the rest | Most releases of a busy function | | Linear | Raises the share in equal steps with a bake between each | Changes whose failure may only show under load | Whatever the shape, the gate between steps is the same: the new version must stay inside its alarms for the bake period, and it must have handled enough invocations for those alarms to mean something. More steps give more chances to stop, at the cost of a longer release. For a function shipped several times a day, a single canary step with strong, version-scoped alarms is often a better trade than a slow linear ramp watched by weak ones. ## The rollback alarm is the real work A shift is only as good as the signal that stops it. - **Scope alarms to the new version.** At a small share, a combined error rate barely moves even when the new version is badly broken. Use the platform's per-version metrics, or derive them from structured logs that record the executed version. - **Watch latency as well as errors.** A version that answers slowly can make the sender time out and resend while the function itself reports success. - **Add a business signal**, such as signature-verification failures, rejected payments or events arriving at a failure destination. - **Require enough traffic.** Overnight, a small share may be a handful of invocations, so make promotion wait for a minimum count or drive synthetic traffic. ## What is specific to functions - **Routing is usually per invocation.** On most platforms the weight is applied to each invocation, not to each client, so one sender's retry of an event can reach the other version. Both versions must accept the same payload and deduplicate on the event id. - **Only traffic through the alias is shifted.** A trigger wired to a fixed version, or to an editable working copy, ignores the weights entirely. - **The new version starts cold.** Its share lands on fresh sandboxes, so its first latency readings run high; set the latency alarm with that in mind. - **Both versions share downstream state.** A schema or message-format change must be readable by both versions, introduced in expand-then-contract steps, or a rollback will break the old version. ## What rollback does not undo Moving the alias back stops new invocations reaching the bad version. It does not reverse payments the bad version already processed, events it already published or rows it already rewrote. That is why the shift starts small, why the pre-traffic check exists, and why the alarms must be sensitive enough to fire while the share is still small.

  • Why might a webhook sender's retry of the same event reach a different version than its first attempt?
    On most platforms the alias weight is applied to each invocation independently, not to each sender, so a retry is just another draw. Both versions must therefore accept the same payload shape and deduplicate on the event id; otherwise a retry handled by the other version can double-process a payment.
  • Why can a canary pass overnight and still be broken?
    A small share of quiet traffic may be a handful of invocations, too few for an error-rate alarm to mean anything, and rare event types may never arrive. Gate promotion on a minimum invocation count as well as time, and cover every event type with a pre-traffic check or synthetic traffic.
  • What must a database or payload change look like when both versions run side by side?
    It must be readable by both. Add the new column or field first in a change the old version tolerates, let the new version write it, and remove the old shape only in a later release. Otherwise rolling the alias back leaves the old version facing data it cannot read.

  • A 10% shift means 10% of customers see the new version.
  • The rollback alarm can safely watch the function's combined error rate.
  • Once the alias weights are set, every trigger follows them automatically.
  • Rolling back is safe even after the new version changed the data shape.
  • A canary that showed no errors overnight proves the version is healthy.