Chapter 3 — Communication, Time, and Failure Across Boundaries

Metadata

Opening Quote

MSQE principle: A request that does not return has not told you that it failed. It has told you nothing. Designing for "nothing" is the difference between a system that can be recovered and a system that can only be apologised for.

Opening Story

The following is an illustrative scenario using the synthetic Atlas Commerce platform.

Atlas checkout calls its third-party payment provider synchronously. The client timeout is five seconds. The retry policy, added a year ago to improve success rates during brief network blips, allows two retries.

During a promotion, the provider's median response time rises from 180 ms to just over four seconds. Most calls now exceed the timeout. The retry policy does what it was told to do: each checkout attempt becomes up to three payment attempts.

Here is what the provider sees. It receives every attempt. It processes many of them successfully — the provider is slow, not broken. It authorises the card, records the transaction, and returns a response that arrives after Atlas has already given up and moved on.

Here is what Atlas sees: a timeout. Its code treats the timeout as a failure, marks the order PAYMENT_FAILED, and shows the customer an apologetic message suggesting they try again. Some customers do. Some customers now have three authorisations on one card for one basket.

The next morning, support has a queue of customers asking whether they have been charged. Atlas cannot answer. It holds no record of a successful payment, because it never received one. It holds no record of an attempted payment that might have succeeded, because a timeout was modelled as a failure rather than as an unknown outcome. The only party that knows what happened is the provider, and the only way to ask is a reconciliation query nobody built.

Every individual decision here was defensible. Five seconds is a reasonable timeout. Retrying transient failures is reasonable. Reporting a failure to the customer rather than hanging is reasonable. The architecture failure is in the seam between them: a timeout is not a failure, and a system that cannot represent "unknown" will lie.

Why This Chapter Matters

Chapter 2 mapped where boundaries fall. This chapter is about what happens when something crosses one.

The moment a call crosses a process boundary, three properties appear that did not exist for an in-process function call. It takes measurable time. It can fail independently of its caller. And it can fail in a way that gives the caller no information — which is a genuinely different outcome from success or failure, and the one most systems are least equipped to represent.

Communication choice is therefore an architectural decision about time, availability, feedback, failure, and evidence — not a technology preference. Choosing synchronous request/response is choosing to make the caller's availability depend on the callee's. Choosing asynchronous messaging is choosing to accept ordering, duplication, and delayed-visibility concerns in exchange for that independence. Neither is free, and the trade is decided by which failure you can afford and what you can truthfully tell a customer.

This chapter stays within its scope. It does not configure brokers, teach a streaming platform, repeat Part IV's protocol curriculum, or implement Part VIII's telemetry. It supplies the interaction assumptions that Chapter 4 (state and consistency), Chapter 5 (styles), and Chapter 11 (integrated decisions) build on.

Learning Objectives

By the end of this chapter, you should be able to:

• distinguish request/response, command, event, queue, callback, polling, and stream interactions by what each commits the caller to;
• explain temporal coupling and its consequences for availability and latency;
• define idempotency precisely and distinguish delivery guarantees from business effects;
• reason about timeout, retry, ordering, duplicate delivery, backpressure, and unknown outcome as design conditions rather than incidents;
• perform and critique bounded numerical reasoning about latency, fan-out, retry amplification, and backlog, stating assumptions, units, and limitations; and
• produce an Interaction, Time, and Failure Analysis for a synthetic Atlas Commerce decision.

Interaction Forms and What They Commit You To

The useful classification is not by technology but by what the caller commits to.

Two distinctions inside this table matter more than the rest.

Command versus event. A command names an intended effect and usually has one intended handler: charge this payment. An event states a fact about the past and has zero or more consumers: payment was authorised. Confusing them produces a specific architecture defect — an "event" that actually requires a particular consumer to act, which is a command with the accountability removed. If the publisher cares whether someone handled it, it is not an event.

Queue versus stream. A queue distributes work items, typically consumed once, and ordering is usually incidental. A stream is an ordered log that consumers read at their own position and can replay. Choosing a stream because it is fashionable, then treating it as a queue, produces surprise when a consumer's position resets and reprocesses a week of history.

Temporal Coupling, Latency, and Availability

Temporal coupling exists when two components must be available at the same moment for work to complete. Synchronous request/response creates it by definition; asynchronous forms exist largely to remove it.

Its consequences are direct. The caller's availability is bounded by the callee's, because it cannot complete without it. The caller's latency includes the callee's, plus network time. And the caller's failure modes now include the callee's, plus a new one that belongs to neither: the timeout.

This is where the availability arithmetic in the Industry Perspective section belongs — and where it needs handling with care, which is why the numerical section below treats it explicitly rather than as a throwaway multiplication.

The important architectural question is not "how do we avoid temporal coupling?" It is: does the user outcome genuinely require the answer before we can respond truthfully? For Atlas checkout, the honest answer is contested. "Your payment succeeded" requires the answer. "We have accepted your order and will confirm payment shortly" does not. The design choice follows from which statement the business is willing to make — which is a product decision as much as a technical one.

Timeout, Retry, and the Unknown Outcome

A timeout is a decision by the caller to stop waiting. It is not information about the callee. After a timeout the caller knows exactly one thing: it did not receive a response within the bound it chose. The request may have been lost in transit, may be queued at the callee, may have been fully processed with the response lost, or may still be executing.

Systems that model only success and failure cannot represent this, and so they guess. Atlas guessed "failed," and the guess was wrong often enough to charge customers for orders that do not exist.

The architectural requirement is a third terminal category: unknown, with a defined path out of it. That path is normally one of:

• Reconciliation — query the callee for the authoritative outcome, using a caller-generated correlation identifier.
• Idempotent retry — repeat the operation in a way that cannot cause a second effect.
• Compensation — perform an offsetting action once the true outcome is known.
• Human resolution — surface the ambiguity to an operator with enough evidence to resolve it.

Each is a design commitment with a cost. What is not acceptable is having none of them and reporting a guess as a fact.

Retry interacts with this badly when it is added without idempotency. A retry after a timeout is, by definition, a retry of an operation whose outcome is unknown. If the operation is not idempotent, the retry may duplicate an effect that already occurred. Atlas's retry policy was safe under its original assumption — brief network blips where the request never arrived — and unsafe under the condition that actually occurred, where the request arrived and was processed slowly. The assumption was never recorded, so it was never revisited.

Retries also need bounds beyond a count: a budget (total time across all attempts), backoff (increasing delay), and jitter (randomised delay to prevent synchronised retry waves). Without backoff and jitter, a dependency recovering from overload is immediately re-overloaded by every client retrying in unison.

Idempotency, Delivery Guarantees, and Duplicate Effects

Idempotency means that performing an operation more than once has no additional intended effect beyond the first successful application. Reading a resource is naturally idempotent. Setting a value to X is idempotent. Incrementing a counter is not. Charging a card is not — unless the operation carries a key that lets the receiver recognise a repeat and return the original outcome instead of performing a new one.

That last mechanism is the practical one. An idempotency key is a caller-generated identifier for a business operation, not a message. The receiver stores the key with the outcome; a second request bearing the same key returns the stored outcome rather than acting again. The critical design details are the key's scope (one key per checkout attempt, not per HTTP request), its retention window, and what the receiver does when the same key arrives with a different payload.

Two conflations must be avoided.

At-least-once delivery is not the same as duplicate business effect. Delivery guarantees are properties of a transport: at-most-once may lose messages; at-least-once may deliver a message more than once. Whether a duplicate delivery produces a duplicate effect is a property of the handler, not the transport. An at-least-once transport with an idempotent handler produces one effect. An at-most-once transport with an application-level retry loop can still produce several. Blaming the broker for duplicate charges usually misattributes the defect.

"Exactly-once" requires careful bounding. Exactly-once delivery across an unreliable network between independently failing endpoints is not achievable in general; a sender that receives no acknowledgement cannot distinguish a lost message from a lost acknowledgement, and must choose between risking loss and risking duplication. What is achievable is effectively-once processing: at-least-once delivery combined with idempotent handling or deduplication. That guarantee is real but bounded — it holds only within the deduplication window, only for the scope of the key, and only while the deduplication store is available and consistent. A platform advertising "exactly-once" is describing effectively-once processing under stated conditions. Part XI requires those conditions to be stated whenever the property is claimed.

Ordering, Duplication, and Backpressure

Ordering is not guaranteed by most asynchronous transports except within a defined scope — commonly a partition or a single-consumer queue. A design that requires global ordering across an entire event type is usually requiring something the transport does not provide. The architectural response is either to scope ordering to a key (all events for one order share a partition), or to make handlers order-insensitive by carrying a version or sequence number and ignoring stale updates.

Out-of-order arrival produces a distinctive class of defect: a SHIPPED event processed before the PAID event it logically follows leaves an order in a state its own state machine considers impossible. Handlers that assume arrival order will either crash or, worse, silently write an incorrect state.

Backpressure is the mechanism by which a system under load signals upstream that it cannot accept more work. Without it, a fast producer and a slow consumer produce one of three outcomes: unbounded queue growth until memory or storage is exhausted; silent message loss; or a cascade in which the producer's own resources are consumed by work that will never complete. Bounded queues with explicit rejection, consumer-driven pull, and rate limiting are all forms of backpressure. The architectural point is that an unbounded queue does not remove the capacity problem; it relocates it and delays the symptom, usually to a moment when the backlog is large enough to be its own incident.

Numerical Reasoning About Interactions

The following are bounded, synthetic worked examples. Each states context, population, assumptions, units, calculation, interpretation, limitation, and decision relevance. They exist to structure judgement, not to produce precision the evidence cannot support.

Example 1 — Call-chain latency allocation

Example 2 — Fan-out failure probability, and why the arithmetic can mislead

Example 3 — Retry amplification

Example 4 — Backlog and drain time

Worked Reasoning: Synchronous Versus Asynchronous Checkout

Atlas must decide how checkout obtains a payment outcome. The two options are the ones in the opening story and its alternative.

Option A — Synchronous confirmation. Checkout calls payment and waits. The customer sees a confirmed or declined outcome before the page returns.

Option B — Accept and resolve asynchronously. Checkout validates the cart, creates the order in an explicit PAYMENT_PENDING state, enqueues the payment work, and returns immediately. A worker performs the payment call and transitions the order. The customer sees "order received; payment confirming."

Tracing the failure path

The required trace, for each option, is: timeout → retry → duplicate or unknown outcome → evidence gap → recovery requirement → customer-state consequence.

Applying the reasoning model

Notice that the analysis produced two decisions of different shapes, and that only one of them was the Quality Engineer's to make. Rejecting the retry policy is squarely an evidence-led engineering conclusion: the policy is unsafe against the failure mode that actually occurred, and no further information would rescue it. Choosing between synchronous and asynchronous checkout is not, because it turns on what the business is prepared to say to a customer — and framing that question accurately is a more useful contribution than answering it.

The idempotency key and the explicit unknown state sit underneath both, required by either design.

The Interaction, Time, and Failure Analysis

The Interaction, Time, and Failure Analysis is this chapter's professional artefact. It is an MSQE teaching artefact. For each interaction that crosses a boundary, it records:

Two fields carry disproportionate weight. Customer-visible state forces the analysis to connect a technical failure mode to something a person experiences, which is what makes it a quality artefact rather than a design note. Unknown-outcome path is the field most commonly left blank, and a blank is a finding.

Engineering Perspective

Interaction failures are hard to evidence because the interesting conditions are rare and difficult to reproduce. Three practical positions help.

Design the observation before the interaction. If a timeout produces an unknown outcome, something must record that an attempt was made, with a correlation identifier, before the call is issued. This is an architecture requirement, not a logging preference: if it is not written before the call, it cannot exist after a crash. Telemetry implementation belongs to Part VIII; the requirement belongs here.

Prefer bounded fault simulation over waiting for production. A controlled experiment in which a dependency is made slow — rather than made to fail — is disproportionately valuable, because slow is the condition most systems handle worst and test least. Chapter 7 develops what an architecture must provide for such simulation to be possible at all.

Treat the retry policy as a first-class artefact. Retry counts are frequently set once, in a hurry, against an assumption about the failure mode that is never recorded. Every retry policy should carry the failure mode it assumes, because that assumption is what makes it safe or dangerous.

Industry Perspective

RFC 9110 defines HTTP semantics, including the property that certain request methods are idempotent — a repeated request has the same intended effect as a single one — and that this is a property of the method's semantics rather than a guarantee about any particular implementation.¹ The distinction is exactly the one this chapter draws: a specification can define what an operation is supposed to mean, but whether a specific handler behaves idempotently is a property of that handler.

The specification is used here only for that protocol claim. RFC 9110 is a protocol specification, not an architecture standard, and it does not settle an architecture decision. Part IV owns HTTP semantics as a curriculum.

Common Misconceptions and Pitfalls

"A timeout means the operation failed."

It means no response arrived within the chosen bound. The operation may have completed. This single misconception generates the largest class of correctness defects in distributed systems, including the Atlas duplicate charges.

"At-least-once delivery causes duplicate charges."

Duplicate delivery causes duplicate effects only when the handler is not idempotent. The defect is in the handler. Conversely, at-most-once delivery does not prevent duplicate effects if the application retries.

"Exactly-once is available; we just enable it."

Effectively-once processing is available, bounded by a deduplication window, key scope, and store availability. Those bounds must be stated. A design that assumes unbounded exactly-once semantics will fail at the edge of a window nobody wrote down.

"Retries make the system more reliable."

Retries improve success against transient, non-duplicating failures. Against a slow dependency they amplify load (Example 3) and, without idempotency, duplicate effects. A retry policy is safe only relative to a stated failure mode.

"Asynchronous means faster."

It means the caller does not wait. Total time to a completed business outcome is frequently longer. What asynchrony buys is availability independence and load absorption, not speed.

"The queue will absorb it."

An unbounded queue relocates a capacity problem and delays its symptom (Example 4). Absorption is real and useful; it is not unlimited, and the backlog outlasts the event.

QA → QE Transition

The transition in this chapter is from testing a happy-path API response to assessing how timing, loss, duplication, delayed failure, and user feedback affect a system claim.

A QA Engineer verifies that the payment adapter returns the right thing when the provider returns the right thing. A Quality Engineer asks what the system does when the provider returns nothing; whether "nothing" is representable in the state model; whether a retry after "nothing" can charge twice; what the customer is told while the answer is unknown; whether anyone can later determine the truth; and who owns the path from unknown to resolved. The first set of questions is necessary. The second set is where the Atlas incident lived.

Summary

Crossing a boundary introduces time, independent failure, and — most consequentially — outcomes that carry no information. Communication choice is an architectural decision about temporal coupling, availability, feedback, and evidence, not a technology preference. A timeout is not a failure, and a system that cannot represent "unknown" will report a guess as a fact. Idempotency is a property of the handler, not the transport; delivery guarantees and business effects are different things, and exactly-once claims require explicit bounds. Bounded numerical reasoning about latency, fan-out, retry amplification, and backlog is useful for structuring judgement, provided each example states its assumptions and its limitations — including the case where correct arithmetic supports a weak inference because failures are correlated.

Key Takeaways

• Classify interactions by what the caller commits to, not by the technology used.
• Temporal coupling makes the callee's availability and latency the caller's own.
• A timeout yields an unknown outcome; every design needs a defined path out of unknown — reconciliation, idempotent retry, compensation, or human resolution.
• Idempotency means repetition adds no further intended effect; an idempotency key is scoped to a business operation, with a stated retention window.
• At-least-once delivery does not by itself cause duplicate effects; a non-idempotent handler does.
• Effectively-once processing is achievable and bounded; unbounded exactly-once is not.
• Ordering is usually guaranteed only within a partition or single consumer; handlers should tolerate reordering.
• Unbounded queues relocate capacity problems rather than solving them, and backlogs outlast the events that create them.
• Correct arithmetic can support a weak inference: check what fan-out dependencies share before assuming independence.

Review Questions

1. Why is a timeout not evidence that the operation failed, and what must the architecture provide instead?
2. Distinguish at-least-once delivery from duplicate business effect. Where does the defect live?
3. Under what conditions is "effectively-once" a defensible claim, and what must be stated alongside it?
4. In Example 2, the arithmetic is correct. Explain why the inference may still be wrong, and what you would ask to find out.
5. In Example 3, retry amplification is 2.8×. What would change if exponential backoff with jitter and a 20-second total budget were added?
6. Why does asynchronous checkout not make checkout faster, and what does it actually buy?

Interview Questions

1. A service adds retries to improve reliability. What would you want to know before agreeing it is safe?
2. How would you design evidence that a payment attempt occurred, when the response never arrived?
3. What is the difference between a command and an event, and what goes wrong when they are confused?
4. A team says their queue means they cannot be overwhelmed. How would you test that claim?

Practical Exercise

Produce an Interaction, Time, and Failure Analysis for the following synthetic Atlas Commerce interaction.

Atlas is adding a loyalty-points award at checkout. When an order is paid, checkout calls the loyalty service to award points. The loyalty service is internally operated, occasionally restarts during deployment, and takes 40–90 ms normally. Product wants the points balance shown on the confirmation page. The team proposes a synchronous call with three retries.

Complete every field of the analysis. Your submission must:

• list the timeout-with-unknown-outcome failure mode separately from callee failure and callee slowness;
• state whether the award operation is idempotent, by what mechanism, and with what key scope and retention;
• state what the customer is told at each stage and identify at least one point where the message could be untrue;
• fill in the unknown-outcome path with a named owner, or record its absence as a finding;
• state what backpressure exists, or that none does; and
• identify at least one piece of evidence that does not currently exist and would be needed.

Then perform one bounded numerical example of your choice — retry amplification during a loyalty-service restart, or the latency contribution of the synchronous call to the checkout path. State context, population, assumptions, units, calculation, interpretation, limitation, and decision relevance. Finally, in no more than 120 words, state whether the synchronous proposal should proceed, what you would need to know first, and who owns that decision. Use synthetic data only; no production system or live third-party integration.

Further Reading

• RFC 9110 — HTTP Semantics (STD 97) — protocol specification; see especially the treatment of idempotent and safe methods.
• RFC 9111 — HTTP Caching — protocol specification; relevant where a staleness window is expressed through HTTP caching semantics.
• Timeouts, retries, and backoff with jitter — vendor-published practitioner guidance, not a standard; useful for the mechanics of retry budgets, backoff, and jitter.

References

Chapter Checklist

Before moving on, confirm that you can:

• Classify an interaction by what the caller commits to and state the temporal coupling it creates.
• Explain why a timeout produces an unknown outcome and name four paths out of it.
• Define idempotency precisely and specify an idempotency key's scope and retention.
• Distinguish delivery guarantees from business effects, and bound an exactly-once claim.
• Perform a bounded numerical example with assumptions, units, interpretation, and limitation.
• Identify when correct arithmetic supports a weak inference because of correlated failure.
• Complete an Interaction, Time, and Failure Analysis including the unknown-outcome path and its owner.

Chapter Navigation

Previous: Chapter 2 — Boundaries, Responsibilities, Coupling, and Dependencies · Next: Chapter 4 — State Ownership, Consistency, and Transactional Boundaries