Chapter 12 — Capstone: System Design & Architecture Quality Strategy and Evidence Portfolio

Metadata

Opening Quote

MSQE principle: The evidence will not tell you what to do. It will tell you what you would have to believe in order to do each thing. Say which of those beliefs you hold, and why, and you have made an architecture contribution.

Opening Story

The following is an illustrative scenario using the synthetic Atlas Commerce platform. Every figure, record, and observation in this chapter is synthetic and is provided for teaching. None of it describes a real company, product, or system.

Atlas Commerce has a decision to make about checkout and fulfilment, and it has run out of ways to defer it.

The immediate trigger is a request from the fulfilment side of the business. They want to ship changes to how partial shipments are grouped, and they want to do it during campaign periods rather than between them. Today that is impossible: fulfilment ships inside the same deployment artefact as checkout, and no one will release checkout during a campaign.

Behind that request sit a year of accumulated conditions. Checkout and fulfilment share the orders table with four other consumers, at least one of which nobody has fully enumerated. The payment provider degrades under load and has twice produced orders whose payment outcome Atlas could not establish. A promotion three weeks ago produced a seven-hour fulfilment backlog, three hundred and forty orders in PAYMENT_UNKNOWN, twelve duplicate fulfilments, and a cached order-status response served across customers. And fourteen months ago a migration that was planned for six weeks left a dual-write running that nobody owns.

The engineering director has asked for a recommendation. Not a design — a recommendation, with the reasoning attached, that another engineer can inspect and disagree with.

You have been given an evidence packet. It is incomplete. Parts of it conflict. Some of the most decision-relevant material is a measurement of something adjacent to what anyone actually wants to know. Two of the three options on the table are supported by different subsets of it.

There is no correct answer waiting to be found. There is a defensible recommendation to be constructed, and an indefensible one, and the difference between them is entirely in whether the reasoning survives inspection.

Why This Chapter Matters

Every chapter in Part XI has ended with a decision that someone else owned. This one ends with a recommendation that you own — its scope, its evidence, its limitations, and its residual risk.

That is the shape of the work at the level this part is aimed at. Nobody hands a Quality Engineer an architecture decision. What they hand over is a situation, a partial evidence base, several people with incompatible confident views, and a deadline. The contribution is not to be right. It is to produce something that makes the decision inspectable: what is known, what is assumed, what the evidence cannot reach, what each option would require you to believe, and what would change your mind.

Three things distinguish this capstone from an exercise.

The evidence conflicts. Not because it is faulty, but because real evidence about a real system does. Two competent measurements can support opposite conclusions when they measure different things under different conditions, and reconciling that is the work.

No option is the intended answer. Option C is not the advanced choice, Option A is not the timid one, and Option B is not the balanced compromise. Each is defensible on some readings of the packet and rejectable on others.

The brief is not solved for you. This chapter demonstrates how to fill in FACT, EVIDENCE, ASSUMPTION, and LIMITATION, because those are craft skills with right and wrong answers. It deliberately does not fill in DECISION, CONSEQUENCE, RESIDUAL RISK, or REVISION TRIGGER. Those are yours.

Learning Objectives

By the end of this capstone, you should be able to:

• define a bounded architecture-quality claim and a decision scope from an ambiguous business request;
• read an evidence packet critically, classifying items as fact, interpretation, limitation, or gap;
• identify where evidence conflicts, and determine whether the conflict is real or an artefact of differing boundaries;
• state what each architectural option requires you to believe, and which of those beliefs is unverified;
• integrate performance, security, reliability, testability, operability, and evolution consequences without combining them into a score;
• design a migration sequence with intermediate architectures, evidence gates, and honest reversibility;
• name the specialist evidence still required and the part of the discipline that owns producing it;
• write an Architecture Decision Brief using the required fields, with a decision scope you can defend; and
• assemble a portfolio that another engineer can inspect, challenge, and revise.

What This Capstone Is and Is Not

You will not draw a target architecture. You may sketch boundaries if it helps you think, but a diagram is an architecture description (Chapter 1) and no diagram is part of the required submission. Nothing in this capstone requires code, cloud access, a modelling tool, or a real system.

The Decision

The request as received is: "Let fulfilment ship independently of checkout, without breaking anything."

That sentence is not yet decidable (Chapter 1). Before anything else, it needs a scope, a bounded claim, and a set of things it is explicitly not about.

The engineering director has provided one constraint on scope: whatever is recommended must be safe to begin before the next campaign, which is eleven weeks away. Work may extend past it; irreversible steps taken before it require justification.

Your first task in the workflow below is to convert the request into one or more bounded claims. The packet is arranged to support that work, not to pre-empt it.

The Evidence Packet

Every item carries a stable identifier. Use the identifier when citing evidence in your brief, so that a reviewer can check what you relied on. Where an item repeats a measurement from an earlier chapter, that is stated, and the value is identical — reused evidence is the same evidence, not a new observation.

Items are grouped for readability. The grouping is not a priority order.

Context, responsibilities, and structure

State, consistency, and caching

Communication, workload, and performance

Dependency behaviour and failure history

Capability constraints

Change, migration, and cost

Option assumptions, gaps, risks, and triggers

Numerical Evidence

Each calculation below is inspectable and independently checkable. None produces a score, and none decides an option.

N-1 — Synchronous checkout path composition (from ARCH-SYNC-01, ARCH-PERF-01, ARCH-PAY-01)

N-2 — Promotional fulfilment backlog (from ARCH-QUEUE-01, ARCH-ASYNC-01)

N-3 — Change coupling, re-read (from ARCH-CHG-01)

N-4 — Canary detection floor for a staged rollout (from ARCH-LOAD-01, ARCH-MIG-01)

N-5 — Storage consequence of a state split (from ARCH-COST-01)

Where the Evidence Conflicts

This is the section to read twice. The conflicts below are not defects in the packet; they are what a real evidence base looks like when several competent measurements are placed side by side.

Conflict 7 deserves particular attention because it is the kind most often missed. Two measurements appear to disagree, and they do not — they have different boundaries. Reconciling them produces a hypothesis nobody had: that the release coupling may be organisational rather than architectural, in which case an architectural change addresses a symptom of something else. Checking that is cheap and it appears in none of the three options.

The Three Options

Each option below is defensible on some reading of the packet. None is the intended answer, and the order is not a ranking.

Option A — Improve the existing modular architecture

Enforce the fulfilment module boundary with a build-time dependency check; remove direct orders access from checkout; record the orderStatus enumeration as open or closed and define required unknown-value behaviour; introduce a domain port for the payment adapter; add a retry budget with backoff and jitter; correct deduplication keying to the business operation; generate the correlation identifier before the payment call; make PAYMENT_UNKNOWN enumerable. One artefact, one store, one deployment.

Option B — Extract a bounded responsibility and introduce a bounded asynchronous workflow

Checkout validates, creates the order in an explicit pending state, and returns; a worker resolves payment asynchronously with reconciliation. Fulfilment becomes a separately deployed unit consuming an order-ready event. Shared store retained.

Option C — Adopt a larger decomposition and migration

Fulfilment becomes a separately deployed unit with its own store; consumers migrate one at a time; the shared orders dependency is dissolved over a multi-stage migration with dual-write, comparison, authority transfer, per-consumer migration, and decommissioning.

A note on combination. The options are not mutually exclusive, and a defensible recommendation may sequence them, take part of one, or reject all three in favour of something the packet supports better. If you propose a combination, the same discipline applies: what does each component require you to believe, and what does it cost.

There is no single pivot

Chapter 11 worked a narrower decision — how to respond to three promotion findings — and reached a conditional position resting on one unverified fact: whether the provider's reconciliation query is contractually available and functions under degradation. That reasoning is sound for the decision it addressed, and it does not transfer to this one.

This decision is wider, and the packet is deliberately arranged so that no single fact determines the recommendation. Six substantially independent tensions are in play, each capable of changing the answer on its own:

Resolving the reconciliation question alone leaves five of these six untouched. A recommendation that treats it as decisive — including one imported from Chapter 11 — will be strong on one axis and silent on the rest, and stage 11 of the workflow exists to catch exactly that.

Two consequences for your work. Test whether Chapter 11's conditional still holds under this packet, rather than assuming it does; the wider evidence may support, qualify, or overturn it. And state which axes your recommendation is exposed to and which it is not — a recommendation that is robust on four axes and fragile on two is a better contribution than one that claims to have settled everything.

The Investigation Workflow

Twelve stages. They are revisitable, not sequential: evidence found at stage 7 routinely invalidates work at stage 1, and a workflow that does not loop is a checklist pretending to be an investigation. Record when you return to an earlier stage and what caused it — that record is part of the portfolio.

The Architecture Decision Brief

Your brief must use these fields, with these names. Four optional fields may be added where they carry information the required fields do not.

Optional, used only where needed: EVIDENCE GAP, UNCERTAINTY, MIGRATION CONDITION, ESCALATION.

Worked examples of four fields

The four fields below are demonstrated because they are craft skills with recognisably better and worse answers. The remaining fields — and in particular DECISION, CONSEQUENCE, RESIDUAL RISK, and REVISION TRIGGER — are deliberately not filled in. Those are the judgement the capstone exists to exercise, and a worked version would remove it.

FACT — only what the evidence directly supports, with its scope attached.

EVIDENCE — source, scope, and relevance, including relevance you are declining to claim.

ASSUMPTION — an unverified condition stated as something checkable, with its status.

LIMITATION — what no amount of evidence within reach will establish, distinguished from a gap that could be closed.

The distinction in the final row is the one most often lost. A limitation carried as a gap generates work that cannot succeed; a gap carried as a limitation quietly accepts a risk that was cheap to remove.

The Portfolio

Your submission is a System Design & Architecture Quality Strategy and Evidence Portfolio. It should be something you could show a colleague as an example of your reasoning, without either overstating it or apologising for it.

Completion criteria

A portfolio is complete when a reviewer who disagrees with your recommendation can nonetheless say: I can see exactly what you relied on, what you assumed, what you could not establish, and what would change your mind.

Common ways this capstone goes wrong

Choosing an option and then reading the packet for support. The tell is that ambiguous evidence — N-3, conflict 7 — appears on only one side of your analysis.

Importing Chapter 11's conditional wholesale. Chapter 11 reached a defensible position on a narrower decision by treating the provider reconciliation contract as pivotal. Carrying that conclusion here without testing it against the wider packet leaves five of the six axes unexamined — most consequentially the operability floor in ARCH-OBS-01 and ARCH-OPS-01, which can make an option unworkable even when its reconciliation question resolves favourably.

Treating any one fact as decisive. A submission whose entire case rests on a single unverified condition has usually found the axis it understands best rather than the one that matters most.

Producing a score. Weighting the options across quality attributes and totalling them. This looks rigorous and it converts your judgement into arithmetic while concealing whose judgement it was.

Recommending Option C because it is the most sophisticated. ARCH-TEAM-01 and ARCH-FAIL-01 (iv) exist in the packet specifically so that this reading has to be argued for rather than assumed.

Recommending Option A because it is the safest. It does not address the request. That may still be the right recommendation — but it requires you to tell the fulfilment side of the business that their problem is not being solved, and to say why that is right.

Leaving RESIDUAL RISK empty or writing "none." After a real decision it is never none. ARCH-RISK-01 alone supplies four items you cannot remove.

Treating a limitation as a gap. Filing payment failure modes 8 and 9 as future work implies they will be resolved.

Engineering Perspective

Three habits distinguish a portfolio that holds up under challenge.

Cite what you declined to rely on. Stating that ARCH-PERF-02 was offered and does not bear on the claims is stronger than omitting it, because a reviewer who knows the benchmark exists will otherwise assume you missed it. The same applies to evidence that cuts against you.

Write the strongest version of the case you are rejecting. If your account of the option you did not choose is weaker than its advocates would give, you have not evaluated it — you have dismissed it. Stage 11 exists for this.

Name the one fact. Every recommendation rests more heavily on one belief than on the others. Identifying it, saying it is unverified, and saying how it could be checked converts a position into a proposal. It is also the single most useful sentence you can give a decision owner, because it tells them where to spend their attention.

Industry Perspective

Architecture-evaluation practice at the Software Engineering Institute treats the output of an evaluation as risks, sensitivity points, and trade-off points elicited against stated quality scenarios — not as a verdict on the architecture.¹ This capstone adopts that output shape deliberately. An evaluation concluding "Option B is best" is unfalsifiable and unusable; one concluding "Option B turns on a provider contract nobody has read, and here is what changes if it does not exist" gives a decision owner somewhere to act.

ISO/IEC/IEEE 42010:2022 supplies the frame for what you are producing: architecture decisions and their rationale are part of an architecture description, expressed for stakeholders with concerns.² Your portfolio is a description. It communicates reasoning; it does not, by existing, establish that the reasoning is right. That is why its value lies in being challengeable, and why the completion criteria are about inspectability rather than about conclusions.

ISO/IEC 25010:2023 supplies the quality vocabulary, and — as Chapter 11 noted — pointedly does not supply a means of combining characteristics into a single figure.³ That absence is the standard declining to make a business judgement. Your portfolio should decline in the same way.

Common Misconceptions and Pitfalls

"The capstone has a correct answer."

It has defensible and indefensible reasoning. Two engineers can recommend different options from this packet and both produce excellent portfolios, provided each is explicit about what they believe and why.

"More evidence would settle it."

Some of it, yes — ARCH-GAP-01 lists six closeable gaps. But ARCH-RISK-01 lists four things that no evidence will settle, and the decision must be made under them.

"A recommendation needs certainty."

It needs a stated confidence, an identified load-bearing assumption, and a revision trigger. Certainty is not available and claiming it is the failure mode this whole part has been arguing against.

"The three options are the option set."

They are three defensible starting points. Conflict 7 suggests a fourth line of enquiry that none of them addresses.

"Recommending the cheapest option is conservative."

It is a position with consequences, like any other. Option A leaves the fulfilment request unmet, and that is a business cost that belongs in CONSEQUENCE.

QA → QE Transition

The transition in this capstone is from assessing one architecture diagram or test result to contributing a transparent recommendation that another engineer can inspect, challenge, and revise.

A QA Engineer given this situation would test what exists, verify what changes, and report defects accurately against each. That work is necessary, and Parts III through X are where it is learned properly.

A Quality Engineer does something the packet cannot do for itself. They convert an undecidable request into bounded claims. They read a benchmark and say what it is evidence of. They notice that the most-cited number in the packet does not settle the question it was gathered to settle. They find the two measurements that appear to disagree and do not. They state, for each option, the belief it requires and whether anyone has checked it. They keep performance, security, and reliability apart when everyone wants a single answer. They say plainly which risks will still be there afterwards, and whose they are. And they hand the whole thing to the person who owns the decision, in a form that lets that person disagree with them intelligently.

That is the capability Part XI has been building, and it does not require owning the architecture. It requires being the person in the room who can say exactly what is known, exactly what is not, and exactly what would change their mind.

Summary

This capstone provides a synthetic, incomplete, internally conflicting evidence packet about the Atlas Commerce checkout and fulfilment boundary, together with three initially defensible options and a twelve-stage revisitable investigation workflow. The work is to convert an undecidable request into bounded claims, read each evidence item for what it actually measures, engage with the conflicts rather than selecting around them, state what each option requires you to believe, keep the quality implications separate rather than scoring them, design a migration with honest reversibility, and produce an Architecture Decision Brief and portfolio that another engineer can inspect and challenge. No option is the intended answer, the strongest evidence in the packet is genuinely ambiguous, and several risks cannot be removed by any amount of further work. The recommendation you produce is yours; the decision belongs to an accountable owner elsewhere.

Key Takeaways

• Convert the request into bounded claims before evaluating any option.
• Read each evidence item for what it measures, not for what it is offered as supporting.
• Ambiguous evidence stays ambiguous; presenting it as support is the most common failure.
• Two measurements that appear to conflict may have different boundaries — reconciling them can produce a hypothesis nobody held.
• State, per option, what it requires you to believe and whether anyone has checked it.
• Keep performance, security, and reliability implications separate; produce no composite score.
• Distinguish gaps, which get plans, from limitations, which get owned residual-risk entries.
• A wide decision rarely turns on one fact; name your exposure per axis and say which axes you are robust against.
• Argue the strongest version of the case you are rejecting.
• Contribute a recommendation with its reasoning attached; the decision belongs to an accountable owner.

Review Questions

1. The request is "let fulfilment ship independently of checkout, without breaking anything." Identify four things that must be added before it is decidable.
2. N-3 was gathered to determine whether separation would relieve change coupling. Explain precisely why it does not settle that question, and what would.
3. ARCH-PERF-02 is a competently run benchmark. State what it is evidence of, and why rerunning it better would not help.
4. Conflict 5 says availability arithmetic that varies by option is measuring the wrong failure class. Explain.
5. Conflict 7 involves two measurements that appear to disagree and do not. What is the reconciliation, and what new line of enquiry does it open?
6. Distinguish, with an example from the packet, an evidence gap from a limitation, and state the different response each requires.
7. Why does N-4's detection-floor calculation matter more given ARCH-FAIL-01 (i) and ARCH-OBS-01?
8. What would have to be true for Option A to be the right recommendation despite not addressing the stated request?

Interview Questions

1. Describe a time you recommended against the option you personally preferred. What changed your mind?
2. How do you present a recommendation when the evidence genuinely does not settle the question?
3. A stakeholder asks you to score three architecture options out of ten. How do you respond?
4. How do you distinguish a risk you can reduce from one you can only own?
5. What does a good architecture recommendation contain that a bad one omits?

Practical Exercise

Complete the capstone.

Working from the evidence packet, the numerical evidence, and the conflict analysis in this chapter, produce a System Design & Architecture Quality Strategy and Evidence Portfolio containing an Architecture Decision Brief.

Required.

1. Scope and claims. Convert the request into at least two bounded, falsifiable quality claims. State explicitly what your recommendation is not about.
2. Evidence register. For every packet item you rely on, cite its identifier, state what you take from it, and state what you decline to take from it. Include at least two items you decline to rely on, with reasons.
3. Classification. Mark every relied-upon item fact, interpretation, limitation, or evidence gap.
4. Conflict analysis. Engage with at least four conflicts, including at least one you cannot resolve and at least one you determine is an artefact of differing boundaries rather than a real disagreement.
5. Option analysis. For each of A, B, and C — and any combination you construct — state what it requires you to believe, which of those beliefs is verified, its strongest argument for, and its strongest argument against. Your account of the option you reject most firmly must be one its advocates would recognise.
6. Quality scenarios. Produce at least four scenarios using the Chapter 6 scaffold, of which at least one has a degraded environment. Mark each response measure obtainable or not, given ARCH-OBS-01.
7. Integrated implications. State performance/scale, security/trust, and reliability/recovery implications separately for your recommendation. No combined rating.
8. Migration and reversibility. For your recommendation, define at least three stages with intermediate architectures, evidence gates, reversibility with an expiry per kind, rollback symmetry, and a decommissioning trigger with a named owner for any temporary debt.
9. Numerical work. Produce at least one new bounded calculation of your own, with context, population, assumptions, units, calculation, interpretation, and limitation. It must inform a decision you actually make.
10. Specialist handoffs. Name at least three questions for Parts VIII or X, phrased as questions those disciplines could answer.
11. Decision brief. All sixteen required fields. RESIDUAL RISK must contain at least three items, at least one drawn from ARCH-RISK-01 that no work will remove. REVISION TRIGGER entries must be observable events.
12. Axis exposure. Work all six axes from There is no single pivot. For each, state whether your recommendation is exposed or robust, and why. For every axis you are exposed to, name the unverified fact, say how it could be checked and by whom, and state what you would recommend instead if it proved false. A submission that names only one such fact has not engaged with the packet. Then state explicitly whether Chapter 11's conditional position survives this wider evidence base — and if it does not, what changed it.
13. Ownership statement. State that you are contributing a recommendation and name the role that owns the decision.

Constraints.

• No composite score, weighted total, or numeric option ranking anywhere.
• No target architecture diagram is required; if you draw one, label it as an architecture description and state what it does not establish.
• Do not present any figure in this chapter as describing a real system.
• Use only the packet, your own derived calculations, and clearly labelled assumptions.

Self-challenge. Before submitting, complete stage 11: write, in no more than 300 words, the strongest available argument that your recommendation is wrong. If you cannot construct one, you have not understood the packet.

Further Reading

• ISO/IEC/IEEE 42010:2022 — Software, systems and enterprise — Architecture description
• ISO/IEC 25010:2023 — Systems and software Quality Requirements and Evaluation (SQuaRE) — Product quality model
• R. Kazman, M. Klein, and P. Clements — ATAM: Method for Architecture Evaluation — SEI technical report; a facilitated method, not an MSQE requirement.

References

Chapter Checklist

Before considering this capstone complete, confirm that you can:

• Convert an ambiguous request into bounded, falsifiable architecture claims with an explicit scope.
• Cite evidence by identifier and state what you decline to take from it.
• Classify each item as fact, interpretation, limitation, or evidence gap, and respond to each correctly.
• Engage with a conflict you cannot resolve, without resolving it rhetorically.
• Distinguish a real conflict from an artefact of differing evidence boundaries.
• State what each option requires you to believe, and which belief is unverified.
• Keep performance, security, and reliability implications separate.
• Design intermediate architectures with evidence gates and reversibility that has an expiry.
• Identify, per axis, where a recommendation is exposed and where it is robust, rather than naming one decisive fact.
• Argue the strongest case against your own conclusion.
• Present a recommendation without claiming to approve an architecture.

Chapter Navigation

Previous: Chapter 11 — Integrated Architecture Decisions: Scale, Security, Reliability, and Residual Risk · Next: Part XI overview