Chapter 10 — Evolution, Migration, Reversibility, and Architecture Debt

Metadata

Opening Quote

MSQE principle: A migration plan describes two architectures people want and several they will actually run. The ones nobody planned are where the system lives longest and fails worst.

Opening Story

The following is an illustrative scenario using the synthetic Atlas Commerce platform.

Atlas decides to separate fulfilment from checkout. The plan is sensible: introduce a fulfilment service, dual-write fulfilment jobs to both the old table and the new service for a period, migrate consumers one at a time, then decommission the old path. Three stages, six weeks, with a rollback at each stage.

Stage one goes well. Stage two goes well. Stage three is deferred because the promotion is approaching and nobody wants to touch fulfilment during it.

Fourteen months later, Atlas is still dual-writing.

The old table is still populated because the finance reconciliation job reads it, and nobody has ever fully enumerated what the finance job needs. The new service is authoritative for everything else. The dual-write is performed by a shim that was written in an afternoon and has no owner; the engineer who wrote it has moved teams. The shim has developed a quiet bug: when the new service returns a duplicate-job error, the shim swallows it, so the two stores have drifted by an unknown number of rows. Nobody knows the number because nobody built a comparison — the plan assumed the dual-write period would last six weeks and that a comparison would not be worth building.

Rolling back is now discussed occasionally and never seriously. Technically the old path still works. Practically, fourteen months of orders exist only in the new service's representation; support has retrained on the new console; the partner integration was pointed at the new service in stage two; and the person who understood the old path's edge cases has left.

The migration is not failing. It succeeded at everything it planned. What it did not plan was the architecture Atlas has actually been running for fourteen months — one with two stores, an unowned shim, unknown drift, and a rollback that exists only on paper.

Why This Chapter Matters

Every previous chapter ended by identifying a change worth considering. This chapter is about the fact that a change is not an event.

Between the current architecture and the intended one lies a sequence of intermediate architectures. Each is a real system that must serve real customers, and each has its own boundaries, failure modes, consistency properties, and evidence needs. Teams plan endpoints carefully and intermediates casually, which is backwards: the endpoints are designed and reviewed, while the intermediate is the one that will be running when something goes wrong — and, as the opening story shows, is frequently the one that runs longest.

The second theme is that reversibility is not one property. "We can roll back" is usually a statement about deployment artefacts, and deployment is the easiest of five things to reverse. Data, operations, organisation, and commitments to third parties all reverse at different rates, and they decay from the moment a migration starts.

The third is temporary architecture debt: the shims, dual-writes, translation layers, and duplicated state that exist only to make a migration possible. They are legitimate and often unavoidable. They become permanent when nobody records what would trigger their removal or who owns it.

The chapter does not prescribe a transformation programme, a cloud-migration playbook, an organisation redesign, or the strangler pattern. It develops migration as an evidence-led sequence and supplies the evolution conditions used by Chapters 11 and 12.

Learning Objectives

By the end of this chapter, you should be able to:

• treat each intermediate architecture as a system requiring its own boundaries, failure analysis, and evidence;
• distinguish five kinds of reversibility and explain how each decays over a migration;
• assess rollback symmetry, and identify when roll-forward is the only real recovery;
• recognise temporary architecture debt and specify its decommissioning trigger, evidence, and owner;
• design shadow or parallel-running evidence and state what it can and cannot establish;
• treat ownership transfer as a migration step with its own risk;
• assess a blast-radius calculation, including why reducing peak exposure can increase total exposure; and
• produce an Evolution, Migration, and Reversibility Strategy comparing staged options.

The Intermediate Architecture Is a Real Architecture

A migration produces a sequence of states, and each state deserves the analysis Chapters 2 through 8 applied to the endpoints. The most useful discipline is to write down, for every intermediate:

The opening story fails on the last three. No comparison was built because the intermediate was expected to be brief; no maximum tolerable duration was recorded; and the silent failure mode — a swallowed duplicate error — was exactly the kind a comparison would have caught.

Intermediate architectures are often worse than both endpoints. Dual-writing is worse than one store and worse than two clean stores. That is normal and acceptable for a bounded period. It becomes a problem when the period is not bounded, and the boundary is a decision someone must own.

Reversibility Is Five Things

"Can we roll back?" is answered too quickly because it is heard as a question about deployment.

Atlas's fourteen-month intermediate is deployment-reversible and irreversible on the other four. Saying "we can roll back" is true of the artefact and false of the system, and the honest statement is: "we can redeploy the old code; we cannot restore the state, the operational knowledge, or the partner integration."

Two practical consequences.

Reversibility has an expiry, and it should be stated. "This stage is data-reversible for seven days, after which backfill would be required" is a fact worth writing in the plan, because it converts an assumed safety net into a scheduled decision.

Some things should be made deliberately hard to reverse, late. Pointing the partner at the new service (stage two in the story) is a near-irreversible step. Sequencing near-irreversible steps last, after cheap evidence has accumulated, is the core of an evidence-led migration.

Rollback, Roll-Forward, and Asymmetry

Chapter 8 introduced rollback asymmetry for contract changes; it generalises.

Rollback is symmetric when reverting the change returns the system to its prior state with nothing left behind. This is true for stateless changes behind a stable interface and rarely true otherwise.

Rollback is asymmetric when reverting the code does not revert the consequences — state written in a new format, events published to consumers who already acted, external systems already notified. Here, reverting the deployment leaves a system the old code cannot correctly interpret.

Roll-forward — fixing forward rather than reverting — is the honest plan for asymmetric changes. It is not a lesser option; for most state-affecting migrations it is the only real one. What it requires is preparation: the ability to deploy a fix quickly, the ability to identify affected records, and a correction path for data written incorrectly.

The failure to avoid is a plan that names rollback as the recovery for a change where rollback is asymmetric. That is not a plan; it is an assumption that will be tested during an incident. The test is simple and should be applied to every stage: if we revert this deployment, what remains that the old code cannot handle? If the answer is anything at all, the recovery is roll-forward and the plan should say so.

Temporary Architecture Debt

Temporary architecture debt is structure that exists only to enable a migration: dual-writes, shims, translation layers, compatibility endpoints, duplicated state, feature flags routing between old and new paths.

It is legitimate. Incremental migration is generally safer than a cutover, and incremental migration requires these mechanisms. The problem is not incurring the debt; it is that the mechanisms are written under time pressure, are nobody's product, and outlive their purpose by default.

Every piece of temporary architecture debt needs four things recorded when it is created:

The opening story's shim had a purpose and none of the other three. That is the general failure: purpose is obvious, trigger is vague, evidence is unbuilt, and the owner is whoever wrote it.

Two further disciplines help. Give temporary debt a visible expiry, so that passing it is an event rather than a non-event — a dated entry in the plan, reviewed. And build the comparison before the debt, because a dual-write without a comparison is an unobserved intermediate, and unobserved intermediates drift silently.

Shadow and Parallel-Running Evidence

Shadow running sends real traffic to the new path alongside the old, discards the new path's output, and compares. Parallel running uses both outputs, typically with the old path authoritative.

Both are strong evidence for a narrow question, and both are commonly over-read.

The last row is where shadow running usually goes wrong. A divergence rate of 0.3% sounds tolerable and is meaningless until someone reads a sample: 0.3% of divergences that are timestamp formatting is fine, and 0.3% that are order totals is not. A shadow run without a divergence-triage step produces a number rather than evidence.

Shadow running also has real costs — doubled load on shared dependencies, and, if the new path has side effects that are not fully suppressed, duplicated external actions. Chapters 3 and 4 apply directly: a shadow path that calls the payment provider is not a shadow.

Numerical Reasoning: Blast Radius and Exposure

The following is a bounded, synthetic worked example in which correct arithmetic supports a conclusion opposite to the one it appears to support.

The general lesson: a blast-radius model that measures only the peak will always favour staging, and one that measures only the integral will always favour cutover. Choosing the metric chooses the answer, which is why the metric must be justified against the system's actual failure profile.

Worked Reasoning: Three Migration Approaches

Atlas must separate fulfilment from checkout while preserving order status, PAYMENT_UNKNOWN handling, support workflows, contract compatibility, and customer-visible state.

Option A — Modular refactor in place

Enforce the fulfilment module boundary; no deployment, process, or data change.

Option B — Bounded extraction, shared store retained

Fulfilment becomes a separately deployed unit; the store stays shared.

Option C — Larger decomposition with a separate fulfilment store

Fulfilment becomes a separate unit and owns its own store.

The decision record

This record makes three different kinds of decision, and the difference matters. One option is authorized because it is cheap, reversible, and independently valuable. One is rejected for this cycle on grounds that are explicitly circumstantial rather than architectural — recording why it was rejected, and what would revive it, is what keeps a rejection from hardening into permanent received wisdom. One is held open pending a specific piece of evidence with a named owner. A record in which every option is deferred is usually a record that has avoided the work; a record in which one option simply wins is usually one that has skipped the constraints.

The Evolution, Migration, and Reversibility Strategy

This chapter's professional artefact is the Evolution, Migration, and Reversibility Strategy — an MSQE teaching artefact. It is organised by stage, not by endpoint, and each stage records:

Three rules keep it honest.

Every stage must be independently defensible. If a stage delivers nothing on its own and only makes sense as part of the whole, the plan is a cutover disguised as a sequence, and its stage gates will be skipped under pressure.

Every temporary-debt item has a named owner and an observable trigger. "When migration completes" is not a trigger. "When the finance job reads exclusively from the new store, verified over one monthly cycle" is.

Reversibility is stated with an expiry. "Data-reversible for seven days" is a plan. "We can roll back" is a hope.

Engineering Perspective

Build the comparison before the dual-write. This inverts the usual order and is the single highest-value change to a migration plan. A dual-write without a comparison is an unobserved intermediate; the opening story's unknown drift is the predictable result. The comparison also has value after the migration, as a regression check.

Sequence irreversible steps last. Pointing an external consumer at the new path, deleting the old store, and disbanding the team that knew the old system are all near-irreversible. Each should come after cheap evidence has accumulated, and each should be an explicit decision rather than a consequence of a stage completing.

Ask what the intermediate does under the failure you already know about. Atlas knows about payment degradation. A migration plan should state what a dual-store intermediate does when PAYMENT_UNKNOWN reconciliation must run across two stores, because that condition will recur and the intermediate will be running when it does.

Treat ownership transfer as a stage. It has a risk profile, an evidence gate — can the receiving team operate it, demonstrated how? — and it is frequently the least reversible step in the plan. Designing the organisation is Part XII's; noticing that the transfer is a migration step with its own risk belongs here.

Industry Perspective

The strangler-fig approach — incrementally replacing a system by routing functionality to new implementations until the old one can be retired — is widely used and widely cited.¹ Its appeal is real: it avoids a single high-risk cutover and delivers value incrementally.

It is not universally correct, and this chapter declines to prescribe it. Its costs are precisely the ones this chapter is about: it requires a long-lived intermediate architecture, a routing mechanism that is temporary architecture debt, and a decommissioning phase that teams routinely fail to complete — the fourteen-month dual-write is a strangler migration that stopped at stage two. Where a system is small, where the cutover is genuinely reversible, or where the team lacks the capacity to operate two paths simultaneously, a bounded cutover can be the lower-risk option. The approach is a tool with a cost profile, not a default.

Evolutionary-architecture practitioner literature makes the complementary case that architecture should be designed for incremental change rather than for a target state.² Part XI takes the compatible position without the prescription: the intermediate states are the design problem, and a plan that has designed only its endpoint has designed the easy half.

Common Misconceptions and Pitfalls

"We can roll back."

Usually true of the deployment artefact and false of the data, operations, organisation, and external commitments. Ask what remains that the old code cannot handle.

"It's temporary."

Temporary architecture debt becomes permanent by default. Without a named owner and an observable decommissioning trigger, "temporary" is a description of intent, not of duration.

"The shadow run showed 0.3% divergence."

Meaningless without triage. Read a sample and classify: formatting divergences and total divergences are the same number and different findings.

"Staged migration is safer."

It reduces peak exposure and increases total exposure, sometimes by an order of magnitude. Which is safer depends on whether your failure modes are fast and loud or slow and silent — and Atlas's have been slow and silent.

"We'll enumerate the consumers during the migration."

The enumeration is the gating condition for the consumer-migration stage. Starting a migration whose compatibility condition cannot yet be satisfied means discovering the missing consumer at the decommissioning step, which is the most expensive possible moment.

"The strangler pattern is the right way to do this."

It is one approach with a specific cost profile: a long intermediate, routing debt, and a decommissioning phase that is frequently abandoned. It is often right, and it is not a default.

QA → QE Transition

The transition in this chapter is from validating a final cutover to evaluating a sequence of intermediate states, their evidence gates, their compatibility conditions, their failure paths, and their reversibility.

A QA Engineer given the fulfilment migration would plan regression coverage for the end state and a smoke check for each release — necessary, and not sufficient. A Quality Engineer asks what architecture runs between stage three and stage seven; which store is authoritative on each day of it; what invariant stops being enforceable and what replaces it; what silent divergence could accumulate and what would detect it; how long the intermediate is tolerable as distinct from planned; what remains after a rollback that the old code cannot read; which step is the one that cannot be undone and why it is not last; and who owns the shim after its author changes team.

Then they say the thing that prevents the opening story: build the comparison before the dual-write, and write down the trigger that removes it.

Summary

A migration is a sequence of intermediate architectures, each of which is a real system with its own boundaries, failure modes, authoritative-store decisions, and evidence needs — and the intermediate frequently runs longest and is analysed least. Reversibility is five distinguishable properties: deployment, data, operational, organisational, and commercial. Only the first fails to decay, and it is the least important. Rollback is symmetric only when nothing persists that the old code cannot interpret; for most state-affecting changes, roll-forward is the honest recovery and requires its own preparation. Temporary architecture debt is legitimate and becomes permanent by default unless it carries a named owner and an observable decommissioning trigger. Shadow running establishes output agreement on the traffic seen and establishes nothing about failure behaviour or the significance of divergences without triage. A blast-radius model measuring peak exposure always favours staging and one measuring total exposure always favours cutover, so the choice of metric must be justified against the system's actual failure profile.

Key Takeaways

• Design the intermediate architectures, not only the endpoint; they run longest and are analysed least.
• Record which store is authoritative for each fact, per stage, before the stage starts.
• Reversibility is five things; four of them decay from the moment a migration begins.
• Ask what remains after a rollback that the old code cannot handle; if anything does, plan roll-forward.
• Every temporary-debt item needs a purpose, an observable trigger, safety evidence, and a named owner.
• Build the store comparison before the dual-write, not after it goes wrong.
• A divergence rate without triage is a number, not evidence.
• Staging reduces peak exposure and increases total exposure; the metric you choose chooses the answer.
• Sequence near-irreversible steps last, and treat ownership transfer as a stage with its own risk.
• Every stage should be independently defensible, or the plan is a cutover in disguise.

Review Questions

1. Atlas dual-wrote for fourteen months. Name the three things missing from the temporary-debt record that would have prevented it.
2. Which of the five kinds of reversibility did Atlas retain after fourteen months, and which had it lost?
3. In the blast-radius example, both figures are correct and point opposite ways. Which should Atlas weight more heavily, and why?
4. What does a shadow run establish, and what does it specifically fail to establish about failure behaviour?
5. Why is Option C's compatibility condition currently unsatisfiable, and what would satisfy it?
6. Give an example of a migration step that is deployment-reversible and commercially irreversible.

Interview Questions

1. A team says a migration can be rolled back at any stage. How do you test that claim?
2. How would you decide between a staged migration and a bounded cutover?
3. What would you require before agreeing to a dual-write period?
4. How do you prevent a temporary compatibility shim from becoming permanent?

Practical Exercise

Produce an Evolution, Migration, and Reversibility Strategy for the following synthetic Atlas Commerce migration.

Atlas is replacing its pricing engine. Today, prices are computed inside the catalogue module from tables in the shared store. The new engine is a separate service with its own rules store and an API. Checkout, catalogue listing, the promotions module, the partner feed, and the nightly margin report all obtain prices today by reading the shared tables directly. The new engine changes how rounding is applied for multi-item discounts, so some prices will legitimately differ. The promotion begins in six weeks.

Your strategy must:

• define at least three stages, each with a stage goal that is independently defensible;
• for each stage, describe the intermediate architecture, including which component is authoritative for price on each day of that stage;
• state a planned duration and a maximum tolerable duration for each stage, and say what happens when the second is reached;
• record every item of temporary architecture debt with purpose, observable decommissioning trigger, safety evidence, and a named owner role;
• assess all five kinds of reversibility per stage, with an expiry for each that decays;
• determine rollback symmetry per stage and give a roll-forward plan wherever it is asymmetric;
• design shadow or parallel-running evidence, state what it cannot establish, and specify a divergence-triage step — noting that some divergences are intended because rounding changed;
• identify the near-irreversible step and justify its position in the sequence;
• state a customer-impact signal per stage and how it would be detected; and
• give each stage a reason to defer.

Then answer, in no more than 150 words: the rounding change means the old and new engines will legitimately disagree. Explain what this does to shadow-run comparison as an evidence mechanism, and what you would do instead. Do not implement the migration. Use synthetic data only.

Further Reading

• M. Fowler — StranglerFigApplication — practitioner guidance, not a standard; one approach with a specific cost profile.
• N. Ford, R. Parsons, and P. Kua — Building Evolutionary Architectures: Support Constant Change. O'Reilly Media, 2017 — practitioner literature.
• ISO/IEC/IEEE 42010:2022 — Architecture description

References

Chapter Checklist

Before moving on, confirm that you can:

• Describe an intermediate architecture with its authoritative store, new failure modes, and lost invariants.
• State a maximum tolerable duration distinct from a planned one.
• Distinguish five kinds of reversibility and give each an expiry.
• Determine rollback symmetry and produce a roll-forward plan where it is absent.
• Record temporary architecture debt with an observable trigger and a named owner.
• Specify a divergence-triage step for shadow-run evidence.
• Explain why a blast-radius metric choice determines the migration conclusion.

Chapter Navigation

Previous: Chapter 9 — Architecture Evidence, Fitness Functions, and Decision Records · Next: Chapter 11 — Integrated Architecture Decisions: Scale, Security, Reliability, and Residual Risk (planned; not yet drafted)