How would you build a weighted decision matrix to choose between competing architecture options, and what are its main failure modes?
answer
- Criteria from scenarios, not adjectives
- Weights before scores, 100-point budget
- Hard constraints = knock-out filter, not a row
- Coarse 1/3/9 scale, avoid false precision
- Sensitivity check: does the winner flip?
basics
~20 sList the options as columns and the criteria (quality attributes) as rows, give each criterion a weight agreed with stakeholders, score every option per criterion, then sum weight × score. Its value is the discussion it forces, not the number it prints.
solid answer
~50 sSteps: (1) derive criteria from ranked quality-attribute scenarios, not from a generic list; (2) get stakeholders to set weights *before* seeing scores, ideally by forced ranking or spending a fixed 100 points; (3) mark any absolute constraint as a knock-out filter rather than a weighted row, so a data-residency violation cannot be out-voted by convenience; (4) score each option against evidence — benchmarks, spikes, prior incidents — on a coarse scale like 1/3/9; (5) compute weighted totals; (6) run a sensitivity check by perturbing weights to see whether the winner flips. Failure modes: reverse-engineering weights to justify a pre-made choice; false precision from invented decimals; criteria that double-count the same concern; ignoring cost of reversal and team skills; and treating a 5% gap as decisive. Publish the matrix inside an ADR so reviewers can attack the inputs.
code
text · 7 linesweighted_total(option) = sum over criteria of weight[c] * score[option][c]
# sanity checks before trusting the total
assert weights_were_set_before_scores
assert hard_constraints_applied_as_knockout_filters # not as weighted rows
assert no_two_criteria_measure_the_same_concern
# sensitivity: does the winner survive +/-20% on every weight?go deeper
Describe the table: options as columns, criteria as rows, weights, scores, weighted sum — and note the number is an aid, not the answer.
Add process discipline: criteria from measurable scenarios, weights set with stakeholders before scoring, coarse scale, evidence-based scores, sensitivity check.
Emphasise knock-out constraints, independent criteria, second-order and long-run costs, independent scoring to avoid anchoring, and publishing it in an ADR.
Discuss when the matrix is worth its cost at all (irreversible, contested, expensive decisions), how to run the elicitation without turning it into theatre, and how to make the reasoning auditable across the organisation.
## What the tool is A **weighted decision matrix** (also called a Pugh matrix, decision grid or trade study) is a table used to compare architecture options against several criteria at once when no option dominates on every criterion. ``` criterion weight OptionA OptionB OptionC p99 latency < 200ms 30 9 3 9 team can operate it 25 9 3 1 cost at 10x traffic 20 3 9 3 time to first release 15 9 3 1 data residency (EU) 10 9 9 9 ----------------------------------------------------- weighted total 810 450 510 ``` Scores here use a 1/3/9 scale (poor / adequate / strong). Total = sum of (weight × score). ## Building it properly 1. **Derive criteria from scenarios, not adjectives.** "Fast" is unusable. "p99 checkout read < 200 ms at 2 000 rps" is scoreable. Include criteria people forget: operability with *this* team's skills, cost at projected scale, hiring and market support, migration cost, and **reversibility** (what does it cost to undo?). 2. **Keep criteria independent.** "Scalability", "performance" and "throughput" as three rows triple-count one concern and silently distort the result. 3. **Set weights before scores, and with stakeholders.** A good technique is a fixed budget: hand out 100 points to distribute, which forces real prioritisation instead of "all rows are 10". Doing weights after scores invites motivated reasoning. 4. **Separate constraints from criteria.** Hard requirements (regulatory data residency, an existing contract, a mandated security control) are **knock-out filters** applied first: an option that fails is removed, never merely penalised. Otherwise a high score elsewhere can out-vote a legal requirement. 5. **Score against evidence.** Prefer measurements from a spike, a load test, a vendor benchmark you re-ran, or a documented past incident. Where you only have opinion, mark it as such and consider timeboxing a spike for the rows that dominate the weight. 6. **Use a coarse scale.** 1/3/9 or 1–5. Fine-grained scales imply precision you do not have and make ties look meaningful. 7. **Do a sensitivity analysis.** Nudge each weight by ±20% and see if the winner changes. If it flips easily, the matrix is telling you the options are close — then decide on reversibility, team preference or cost of delay, and say so. ## Failure modes (what interviewers probe) - **Reverse-engineering.** Someone already picked the technology; the weights get tuned until it wins. The tell is weights invented after scoring, or a criterion that exists only because one option is good at it. - **False precision.** Totals like 7.42 vs 7.38 presented as a decision. Aggregation hides that the two options differ on qualitatively different axes. - **Missing criteria.** Operational burden, on-call load, licence terms, exit cost, and organisational fit are the usual omissions — and they are often what actually kills a choice. - **Ignoring second-order and long-run costs.** A cheap option today that doubles infra cost at 10× traffic, or that needs a skill the team lacks. - **Group-think in scoring.** One senior voice anchors everyone. Mitigation: score independently first, then compare and discuss only where scores diverge widely (a Delphi-style pass). - **Treating the number as the decision.** The matrix's real product is a *documented, attackable argument*. When totals are within noise, say "they tie; we choose B because it is a two-way door". ## When not to use it For a reversible, low-cost decision, the matrix costs more than being wrong — just choose and move. Reserve it for expensive, hard-to-reverse choices with genuine stakeholder disagreement, or where you must show your reasoning to people who were not in the room.
- Two options score within 3% of each other. What do you do?Treat it as a tie — the model is not that precise. Break it on criteria the matrix handles badly: reversibility (prefer the two-way door), team familiarity, cost of delay, and whether one option keeps more future options open. State explicitly that the scores tied and name the real tiebreaker.
- How do you stop weights being gamed to fit a pre-chosen answer?Fix and publish the weights before any scoring, elicit them with a forced 100-point allocation from stakeholders rather than from the proposer, score independently before discussing, and run a sensitivity analysis so any weight-tuning that flips the result becomes visible.
- Where does a hard regulatory requirement belong in the matrix?Nowhere in the weighted rows — it is an eligibility filter applied first. Options that violate it are eliminated. Encoding it as a weighted criterion lets convenience scores outvote a legal obligation.
Choosing a house. You do not average bedrooms, commute and price into one number and buy the highest score — but writing the columns down stops you buying on kerb appeal, and 'must be within the school catchment' is a filter you apply before scoring anything, not a point you can outweigh with a nice kitchen.
saying these in an interview costs you the question
- Assigning weights after seeing the scores
- Reporting two-decimal totals as if they were measurements
- Listing scalability, performance and throughput as separate rows (double counting)
- Putting a legal or security constraint in as a weighted row instead of a knock-out filter
- Omitting operability, exit cost and team skills from the criteria
- Presenting the total as the decision rather than as an argument to be attacked