Microservices from Zero: Architectures, Trade-offs, and Migration

115 min read

Published

A senior-engineer reference from architecture foundations through Domain-Driven Design (DDD) service boundaries, transactional consistency patterns, migration playbooks, design anti-patterns, and a full worked design for money-moving commerce.

Microservices are an architectural style for structuring a system as independently deployable services that own their data and collaborate over the network. They are a rational response to specific scaling, team, and release constraints—not a default for every product. This guide teaches the vocabulary, decision rules, and failure modes you need before you split a complex transactional application (orders, payments, stock, ledger).

Running example — transactional commerce:
Catalog · Cart · Inventory · Orders · Payments · Ledger/Accounting · Fulfillment · Notifications · Identity
Later sections design this estate end-to-end: aggregates, invariants, saga, outbox, APIs, and failure compensations.

How to read this guide (start here)

Every code block has a colored label. Do only what the label says.

This article is conceptual and design-heavy. Prefer Read blocks. Follow parts in order unless a part explicitly says you can skip it.

Suggested path: architecture qualities → styles & tables → decision criteria → Domain-Driven Design (DDD) boundaries → design patterns catalog → communication & data → platform → migrate vs not → strategies & playbook → design mistakes → commerce design → first-cut learning path → checklist.

Unpacking rule used throughout: for each important idea you will see definition → why it matters → concrete example → failure if ignored → decision rule.

Glossary of abbreviations (read once)

Abbreviations are also expanded on first use in the body. This list is the lookup table:

1. Software architecture

What this part is for: define architecture precisely and name the qualities you will trade off later.
What you should do: read; map each quality to your current system before any style debate.

Software architecture is the set of structural decisions that are expensive to reverse: the major components (or services), their responsibilities, their interfaces, where state is stored, how failures propagate, and how the system is built, deployed, and operated over time. Architecture is not the choice of framework, language, or cloud vendor—those are implementation details that should be replaceable within the architectural constraints.

Why it matters: every team already has an architecture, even if nobody drew it. Accidental architecture concentrates coupling in the wrong places: shared mutable tables, undocumented call chains, and “temporary” modules that become permanent choke points.

Example: In commerce, deciding that Payments may never write to the Orders table, and that Inventory reservations are authoritative only inside Inventory, is an architectural decision. Choosing Postgres vs MySQL for Inventory is not.
Failure if ignored: You discover the real architecture during an incident—when a “small” schema change locks checkout, or when two teams cannot release without a weekend coordination meeting.

Decision rule

Treat a choice as architectural when reversing it requires coordinated data migration, multi-team interface change, or a sustained rise in operational risk. Document those choices; leave framework preferences out of the architecture record.

Architecture ≈ boundaries + contracts + data ownership + deployment topology + change/failure cost

1.1 Qualities you actually trade

Architecture is a continuous trade among competing qualities. Name them explicitly so debates stop being about fashion.

CAP (Consistency, Availability, Partition tolerance — as a decision lens, not a slogan): Brewer’s CAP conjecture (and the PACELC elaboration: if Partitioned then Availability vs Consistency; Else Latency vs Consistency) says that under a network partition, a replicated data system cannot simultaneously guarantee linearizable consistency and full availability. Even without partitions, you trade latency vs consistency (PACELC’s “else”). For checkout money paths, prefer consistency of the ledger and payment capture; for product search rankings, prefer availability and accept stale reads. Decision rule: classify each read/write path as C-preferring or A-preferring under failure, and design the UI/support process accordingly.

READ — Architecture quality sketch. Do not paste into a terminal.
  Deployability  <----->  Coordination cost
  Evolvability   <----->  Contract stability / coupling
  Availability   <----->  Blast radius of failure
  Consistency    <----->  Latency / partition tolerance
  Operability    <----->  Number of independently failing parts

2. Architecture styles in depth

What this part is for: define each style with historical and technical precision so comparisons later are fair.
What you should do: read diagrams; keep the commerce example in mind.

An architecture style is a named family of structural constraints (how you package deployable units, share data, and communicate). Real systems mix styles; naming the dominant style still helps you choose deliberately and spot anti-patterns.

2.1 Monolith

Definition: A monolith is one primary deployable application that contains most business capabilities in a single process (or a tightly versioned set of processes shipped as one unit), typically against one primary transactional database.

Why it matters: ACID (Atomicity, Consistency, Isolation, Durability) transactions across catalog, cart, order, and payment tables are straightforward; local method calls are cheap and typed; debugging is one stack trace.

READ — Monolith. Do not paste into a terminal.
        [ Clients ]
             |
        [ Web / API ]
             |
   +---------+---------+
   |   Commerce Monolith|
   | catalog cart order |
   | pay ship notify    |
   +---------+---------+
             |
      [ One primary database (DB) ]
Example: Place order, decrement stock, and record payment intent in one database transaction with foreign keys enforcing integrity.
Failure if ignored (at scale): Hot paths force you to scale the whole app; a memory leak in reporting takes down checkout; release trains become organizational rituals.

Decision rule

Prefer a monolith (ideally modular) when team size is small-to-medium, domain seams are still being discovered, and strong transactional integrity across capabilities is more valuable than independent deployability.

2.2 Modular monolith

Definition: A modular monolith is still one deployable unit, but the codebase is partitioned into modules with enforced dependency rules: modules expose public APIs; other modules must not reach into private persistence or domain types. In practice this is Ports and Adapters inside one process: modules depend on ports, not on each other’s ORM tables.

Why it matters: You keep single-process transactions and cheap calls while practicing the same boundary discipline microservices require. Extraction later becomes swapping an in-process adapter for a remote one—not archaeology.

READ — Modular monolith. Do not paste into a terminal.
   +---------------- Commerce deployable ----------------+
   |  [Catalog] --> [Cart] --> [Orders] --> [Payments]   |
   |       ^            |           |            |       |
   |       |            v           v            v       |
   |   module DBs logically separated (schemas/tables)   |
   +-----------------------------------------------------+
                         one process / one release train
Example: Orders module depends on a PaymentsPort interface implemented in-process; Orders never imports Payments’ ORM (Object-Relational Mapper) entities.
Failure if ignored: “Modules” that share tables and circular imports become a monolith with marketing labels—extraction will still tear the schema apart.

Decision rule

If you anticipate microservices but cannot yet staff independent on-call, start here. Enforce module boundaries with package rules, separate schemas, and architecture tests before you introduce a network.

2.3 SOA versus microservices (historical differences)

Service-oriented architecture (SOA) emerged in the 2000s as an enterprise approach: coarse-grained services, often shared via an enterprise service bus (ESB), with heavyweight standards (SOAP — Simple Object Access Protocol; WS-* web-service standards), centralized governance, and shared canonical data models. Services were frequently large, multi-capability applications; integration logic often lived in the bus.

Microservices (popularized in the 2010s) emphasize: smaller services aligned to business capabilities, decentralized data management (database-per-service), smart endpoints / dumb pipes (prefer thin messaging or HTTP over a logic-heavy ESB), independent deployability, and product-aligned teams. Governance is distributed; contracts are versioned APIs and events, not a single enterprise schema.

DimensionClassic SOAMicroservices
Integration hubOften ESB with orchestration/transformPrefer broker or direct RPC; logic in services
DataShared enterprise models commonDatabase-per-service; published language
DeployOften coordinated enterprise releasesIndependent service deploys as a goal
Team modelProject / integration-centricProduct / capability-centric
Tech diversityConstrained by platform standardsPolyglot allowed (with ops cost)

Decision rule

If your “services” share one database and can only ship together through a central integration team, you have SOA-flavored coupling (or a distributed monolith)—not microservices—regardless of container count.

2.4 Microservices definition

Definition: A microservice architecture structures an application as a suite of services that (1) are independently deployable, (2) are organized around business capabilities, (3) own their persistent state, and (4) communicate via network protocols with explicit contracts (APIs and/or events).

“Micro” is relative to the capability boundary—not a line-count target. A Payments service that owns capture, refund, and provider adapters may be large in code and still be one correct microservice.

READ — Microservices estate (simplified). Do not paste into a terminal.
  Clients --> [ API Gateway / BFF ]
                 |        |        |
            [Orders]  [Payments] [Inventory]
               | DB      | DB       | DB   (each = Database)
                 \        |        /
                  \       v       /
                   [ Message broker ]
                          |
                   [Notifications] ...
Example: Inventory can deploy a reservation-algorithm fix without redeploying Catalog; Orders consumes InventoryReserved events without sharing Inventory’s tables.
Failure if ignored: Calling every Docker container a microservice while retaining shared database (DB) writes and lockstep releases produces network latency without the benefits.

Decision rule

A component counts as a microservice only if it can be versioned, deployed, scaled, and failed independently while preserving its business invariants through its own data store and contracts.

2.5 Distributed monolith and nanoservice anti-patterns

Distributed monolith: many network-separated processes that must be released, tested, and reasoned about as one unit—because of shared databases, synchronous call webs, shared libraries that require synchronized versions, or ambient temporal coupling.

Nanoservices: services split below a useful business capability (e.g., separate deployables for CreateUser, UpdateUser, DeleteUser). Chatty RPC, distributed transactions, and operational noise dominate.

Example (distributed monolith): Orders, Payments, and Inventory each have a container, but all three write the same orders schema and share a “common” JAR that embeds domain entities—so a field rename requires a fleet release.
Failure if ignored: You pay microservices’ operational tax while retaining monolith coordination costs—the worst of both styles.

Decision rule

If two deployables cannot change schemas or shared libraries on independent cadences, merge them (modular monolith) or sever the shared dependency before calling the design “microservices.”

2.6 Serverless as runtime, not architecture

Serverless (functions-as-a-service, managed containers that scale to zero) is a hosting and scaling model. You can implement a monolith, modular modules, or microservices on serverless runtimes. Serverless does not choose your bounded contexts, data ownership, or consistency model.

Decision rule

Select architecture for boundaries and invariants first; select serverless vs long-running processes for traffic shape, cold-start tolerance, and ops preference second.

3. Comparison tables

What this part is for: put styles side-by-side on deployability, data, failure, teams, and complexity.
What you should do: skim now; return when making a style decision.
DimensionMonolithModular monolithSOA — Service-Oriented Architecture (classic)Microservices
Deployable unitOne primary appOne app, internal modulesFew large services + busMany independently deployable services
Process boundaryUsually oneOneMultipleMultiple by design
Data ownershipShared database typicalLogical separation possibleOften shared/canonicalDatabase-per-service
Cross-capability transaction (TX)Local ACID easyLocal ACID easyOften distributed / ESBSagas; avoid Two-Phase Commit (2PC)
Inter-component callIn-processIn-process via portsSOAP/ESB/RPCHTTP/gRPC/events
Failure isolationLowLow (process)MediumHigh if designed
Team autonomyLow–mediumMediumMedium (integration hub)High when ownership clear
Ops complexityLowLow–mediumHighHighest
Consistency defaultStrong inside databaseStrong inside databaseMixedEventual across services
Scaling grainWhole appWhole appPer large servicePer service
SymptomLikely style mismatchCorrective direction
Every release needs a war roomMonolith too coupled or distributed monolithModularize; extract highest-churn boundary
Partial outage takes all UI downSync call chains / no bulkheadsTimeouts, fallbacks, async for non-critical
“Services” share write databaseNot microservicesSplit schemas or merge deployables
Hundreds of tiny CRUDish servicesNanoservicesMerge by capability / aggregate
Cannot explain who owns refundsBlurred bounded contextDDD mapping; single write model for money
READ — Complexity vs autonomy curve (qualitative). Do not paste into a terminal.
  Autonomy / independent scale
        ^
        |                          * microservices (if mature)
        |                    *
        |              * SOA
        |        * modular monolith
        |  * monolith
        +-------------------------------------------> ops & consistency cost

4. When each style is the rational choice

What this part is for: replace slogans with decision criteria.
What you should do: score your context against the criteria; write the choice down with evidence.

4.1 Choose a monolith when

4.2 Choose a modular monolith when

4.3 Choose microservices when

Rational microservices ⇔ (deploy OR scale OR org isolation pain) > (distributed consistency + ops + cognitive load cost)

Decision rule

Do not adopt microservices to “modernize.” Adopt them when at least one of deploy, scale, or org-isolation pains is severe and you can staff platform and saga design for the money paths you will split.

5. Microservices pillars and Domain-Driven Design (DDD) essentials for boundaries

What this part is for: define the pillars and the DDD tools that make service cuts safe.
What you should do: practice naming bounded contexts on the commerce domain before cutting networks.

5.1 Pillars

  1. Business capability alignment — Services reflect what the business does (fulfill an order, capture a payment), not technical layers (UserRepositoryService).
  2. Independent deployability — A service ships on its own cadence with backward-compatible contracts.
  3. Data ownership — Only the owning service writes its store; others use APIs/events/read models.
  4. Automation and observability — CI/CD (Continuous Integration / Continuous Delivery), health, metrics, traces, and correlation are prerequisites, not optional polish.

5.2 Domain-Driven Design essentials

Ubiquitous language — The shared vocabulary of domain experts and developers inside one model. Definition: terms like “reservation,” “capture,” “settlement,” and “refund” mean one thing in that context. Why it matters: ambiguous language produces wrong service cuts (Order vs Payment “status” wars). Example: Inventory’s “reserved” is not Payments’ “authorized.” Failure if ignored: APIs that reuse the same word for different invariants. Decision rule: write a glossary per bounded context; forbid overloaded terms across contexts without translation.

Bounded context — A boundary within which a particular domain model is valid and consistent. Definition: Catalog’s Product (marketing attributes, SEO) differs from Inventory’s StockItem (SKU, on-hand, reserved). Why it matters: forcing one enterprise Product entity across teams creates coupling. Example: Catalog publishes ProductPublished; Inventory consumes SKU identity only. Failure: one shared Product table edited by five teams. Decision rule: split services primarily along bounded contexts, not along database tables of an old ER diagram.

Aggregate — A cluster of domain objects treated as a consistency boundary with a root that enforces invariants. Definition: all changes that must be atomic together go through the aggregate root and are persisted in one transaction in that service’s database. Why it matters: aggregates define the maximum size of strong consistency you should expect without a saga. Example: InventoryItem aggregate enforces on_hand - reserved >= 0. Failure: updating stock from Orders and Payments concurrently without a single enforcer. Decision rule: if an invariant spans two aggregates in different services, design a saga or rethink the boundary—do not reach across databases.

Domain events vs integration events — A domain event records something meaningful that happened inside a model (often used inside a service). An integration event is a published contract for other bounded contexts (stable name, versioned payload, no leakage of private entity graphs). Decision rule: never publish ORM entities; publish intent-focused integration events with correlation and causation IDs.

READ — Context map sketch. Do not paste into a terminal.
  [Identity] --customerId--> [Cart] --checkout--> [Orders]
       |                                          |     \
       |                                          v      v
       |                                    [Payments] [Inventory]
       |                                          |
       +------------------------------------> [Ledger]
  Catalog --SKU--> Inventory / Pricing
  Orders --> Fulfillment --> Notifications

Decision rule for a service cut

Cut where (a) ubiquitous language shifts, (b) a different team can own the lifecycle, (c) an aggregate’s invariants fit in one database transaction, and (d) cross-context collaboration can be API or event without dual-writes to the same tables.

5.3 Context mapping relationships

Bounded contexts collaborate through explicit relationships:

Example: Fulfillment is a customer of Orders’ published language (OrderConfirmed). Fulfillment’s ACL maps order lines into pick-list items; it never imports Orders’ ORM package.
Failure if ignored: “Just add the field to the shared event” without versioning turns every consumer into an accidental partnership.

6. Benefits and costs (operational tax)

What this part is for: quantify costs conceptually so benefits are not free in your plan.
What you should do: list benefits you need; estimate tax you will pay.

6.1 Benefits (when earned)

6.2 Costs — the operational tax

Think in categories you must budget for continuously:

Rough tax signal: if platform + saga/read-model work > 30–40% of feature capacity for a year, either invest in platform deliberately or delay further splits.
Example: Extracting Notifications may cost one engineer-week of plumbing and save weekly coordinated releases. Extracting Payments without ledger design may cost months of reconciliation debt.
Failure if ignored: Leadership counts “number of services” as progress while feature velocity drops under incident load and unclear money states.

Decision rule

Approve each extraction with an explicit tax budget: who owns on-call, which SLOs, which saga, which observability dashboards, and which rollback story.

7. Design patterns catalog (what they are before we use them)

What this part is for: teach the named patterns this article relies on—so later sections are applications, not name-dropping.
What you should do: read each pattern card once; return here when a later section cites the pattern.

A design pattern is a reusable solution to a recurring design problem, with known trade-offs—not a product and not a framework. In distributed systems the same names (Saga, Outbox, Circuit Breaker) appear everywhere; teams fail when they adopt the name without the responsibilities, failure modes, and decision rules. Each card below follows: problem → definition → how it works → example → failure if ignored → decision rule.

How to use this catalog

Skip deep into §8+ only after you can explain Saga vs 2PC, Outbox vs “publish after commit,” and Circuit Breaker vs “retry forever” in your own words. Commerce design (§16) assumes these cards.

7.1 Ports and Adapters (Hexagonal Architecture)

Problem: Domain logic becomes entangled with HTTP controllers, ORM entities, message SDKs, and vendor APIs—so you cannot test invariants without standing up infrastructure, and extraction means rewriting the core.

Definition: Keep the domain (use-cases / aggregates) at the center. A port is an interface the domain needs or exposes (e.g., PaymentsPort.authorize(...), PlaceOrder application service). An adapter is a concrete implementation at the edge (HTTP controller, Kafka consumer, Postgres repository, Stripe client). Dependencies point inward: adapters depend on ports; the domain does not depend on frameworks.

How it works: Orders orchestrates checkout against InventoryPort and PaymentsPort. In a modular monolith those ports are in-process adapters; later the same ports become HTTP/gRPC or messaging adapters without rewriting aggregates.

Example: CheckoutService calls inventory.reserve(cmd) through an interface. Unit tests use a fake inventory; production uses a local module adapter, then a remote client—same domain code.
Failure if ignored: Controllers and ORM entities become the “model.” Microservices extraction copies framework glue instead of a boundary.

Decision rule

Every cross-capability dependency goes through a port. If you cannot swap the adapter without touching aggregates, the boundary is fake.

7.2 Anti-Corruption Layer (ACL)

Problem: An upstream system (legacy monolith, PSP, partner API) speaks a different language and model. Consuming it raw infects your ubiquitous language with foreign fields and invariants.

Definition: An ACL is a translation layer at the boundary of your context: inbound foreign payloads/models become your domain types; outbound calls map your intents into the foreign API. It is a form of adapter focused on model isolation, not just transport.

How it works: Payments receives Stripe webhooks → ACL maps to PaymentIntentCaptured / reason codes → domain updates. Your ledger never stores Stripe’s raw enum soup as truth.

Example: Fulfillment’s ACL maps OrderConfirmed.lines[] into PickListItem. Catalog marketing names never appear in warehouse picking logic.
Failure if ignored: Shared DTOs across services recreate a canonical enterprise model; every upstream change forces downstream rewrites.

Decision rule

If the foreign model’s vocabulary would pollute yours, introduce an ACL. Prefer translating at the edge over “just reuse their JSON.”

7.3 Database-per-service

Problem: Multiple services reading and writing the same tables couple release schedules, create distributed race conditions, and destroy independent deployability—the schema becomes the real monolith.

Definition: Each service is the only writer of its persistent data (dedicated database, schema, or credentials that forbid foreign writers). Other services get data through APIs, integration events, or projected read models—not shared joins.

How it works: Inventory alone updates stock and reservations. Orders stores order lines and saga state. Reporting builds a denormalized store from events.

Example: Finance cannot JOIN payments p JOIN orders o across service databases. A projector builds OrderFacts from PaymentCaptured + OrderConfirmed.
Failure if ignored: “Microservices” with one shared OLTP database are a distributed monolith: network latency plus schema coupling.

Decision rule

One writer per datum. If two services must update the same row in one user action, either merge them or design a saga with clear ownership—not a shared table.

7.4 Saga

Problem: A business transaction spans multiple services (reserve stock → authorize payment → capture → ledger). You cannot wrap them in one ACID database transaction without distributed Two-Phase Commit (2PC), which couples availability and holds locks across slow external calls.

Definition: A saga is a sequence of local transactions, each inside one service’s database, coordinated so the overall business outcome is either completed or semantically compensated. It provides process-level consistency, not single-commit atomicity. Named after long-lived transactions literature (Garcia-Molina & Salem); popularized for microservices by Richardson and others.

How it works:

  1. Each step commits locally (e.g., Inventory reserves stock).
  2. Progress is recorded (saga state machine and/or events).
  3. The next step runs (command or reaction to an event).
  4. On failure, previously successful steps run compensating transactions (release reservation, void auth)—undoing business effect, not necessarily bitwise DB rollback.
  5. Timeouts and reconciliation close gaps when messages are lost or PSP state diverges.
READ — Saga vs 2PC. Do not paste into a terminal.
  2PC (usually avoided across services):
  Coordinator --prepare--> A,B,C  then --commit--> A,B,C
  (locks + availability coupled; PSP latency inside the TX = disaster)

  SAGA (preferred):
  Local TX Inventory.reserve     COMMIT
  Local TX Payments.authorize    COMMIT
  Local TX Payments.capture      COMMIT
  Local TX Ledger.post           COMMIT
  on fail after reserve: Local TX Inventory.release  COMMIT
  UI may show intermediate states (Pending / AwaitingPayment)
Example: Checkout saga states Pending → Reserved → Authorized → Captured → Confirmed. Payment decline after reserve → compensate with ReleaseStock → terminal Failed with reason code for support.
Failure if ignored: Teams “fake” a saga with a long synchronous call chain and hope retries fix money—producing double charges, stranded reservations, and unexplainable support tickets.

Decision rule

Use a saga when invariants span multiple aggregates/services. Prefer keeping strongly related invariants in one aggregate (and one local ACID TX) when you can. Never treat saga as optional for money/stock paths you have already split.

7.5 Compensating transaction

Problem: Step N of a saga failed after steps 1…N−1 already committed. You cannot roll back those commits with a global undo log across services.

Definition: A compensating transaction is a deliberate business action that reverses or neutralizes a prior local commit’s effect (refund, void authorization, release reservation, reverse ledger entry). Compensation is designed up front as part of the saga—not improvised in support chat.

How it works: Each forward step that is not naturally idempotent-terminal declares its compensation, ordering rules (compensate in reverse order), and what “already compensated” means under retries.

Example: Forward: ReserveStock. Compensate: ReleaseStock. Forward: CapturePayment. Compensate: RefundPayment (may be async and take days at the PSP—saga must allow RefundPending).
Failure if ignored: Partial success with no compensation leaves stranded stock holds and captured funds without confirmed orders—manual cleanup that does not scale.

Decision rule

Do not add a forward saga step unless you can name its compensation (or prove it is safe to leave if later steps fail). Document reason codes for operator intervention when compensation itself can fail.

7.6 Saga choreography vs saga orchestration

Problem: You need a saga, but must choose where the workflow “brain” lives.

Choreography — definition: Each service reacts to events from others; there is no central coordinator. The workflow is the emergent graph of subscriptions. Best for fan-out after a terminal business fact (OrderConfirmed → notify, index, analytics).

Orchestration — definition: A coordinator (often the Orders/Checkout service, sometimes a workflow engine) owns an explicit state machine, sends commands to participants, and interprets replies. Best when steps are conditional, ordered, and must compensate (checkout, refund).

Example: Orchestrate checkout (reserves/auths/captures). Choreograph post-confirmation side effects (email, search signal). Mixing them accidentally—half sync, half undocumented events—is an anti-pattern.

Decision rule

Orchestrate multi-step commands with compensations. Choreograph one-way reactions to terminal facts. Write the choice on the context map.

7.7 Transactional Outbox

Problem: A service must update its database and publish a message. Doing them as two separate steps is a dual-write: either can succeed alone, leaving silent divergence (DB says confirmed; no event; downstream never moves).

Definition: In the same local database transaction as the business write, insert a row into an outbox table describing the message to publish. A separate publisher process (polling or log-tail / CDC) reads unpublished outbox rows, publishes to the broker, and marks them published. Atomicity of “state change + intent to publish” is guaranteed by local ACID.

How it works:

  1. BEGIN → update business tables → insert outbox row → COMMIT.
  2. Publisher: read outbox → publish → mark published (idempotent publish).
  3. Consumers process with inbox/idempotency (at-least-once delivery still applies).
Example: Orders sets status Confirmed and inserts OrderConfirmed into outbox in one TX. Crash after COMMIT but before publish is safe: publisher retries. Crash before COMMIT publishes nothing and leaves no confirmed order.
Failure if ignored: Dual-write races: “we’ll republish from a cron that scans tables” without clear markers produces duplicates, misses, and irreproducible money states.

Decision rule

Any integration event that must not diverge from a local commit uses outbox (or proven CDC from the write store). Ban “update then publish” in money/stock paths.

7.8 Inbox (idempotent consumer)

Problem: Brokers deliver at-least-once: the same message can arrive twice. Handlers that are not idempotent double-charge, double-reserve, or double-email.

Definition: An inbox (or processed-message store) records message IDs (or idempotency keys) that have already been applied. In one local transaction the consumer: checks inbox → applies side effects → inserts inbox row → commits → acknowledges. Duplicates short-circuit.

Related: Idempotency on command APIs uses a client-supplied key with the same idea for HTTP retries, not only messaging.

Example: Payments CapturePayment with idempotencyKey=ord_123_cap. Second delivery sees the key, returns the original result, does not capture again.
Failure if ignored: “Exactly-once” assumed from broker settings alone—until the first redelivery doubles a capture.

Decision rule

Every money, stock, and ledger handler is idempotent. Idempotency keys are part of the public command contract.

7.9 CQRS and read-model projection

Problem: Write models optimized for invariants (normalized OLTP) are poor for dashboards, search, and cross-context reports. After database-per-service, SQL joins across services disappear.

Definition: CQRS (Command Query Responsibility Segregation) separates the command side (writes, invariants, aggregates) from the query side (read models shaped for specific screens/reports). In microservices, read models are often projections: projectors subscribe to integration events and upsert denormalized rows in a read store.

How it works: Orders/Payments/Inventory remain systems of record. A projector builds OrderFacts for support UI. Lag is expected; SLOs define how stale is acceptable.

Example: Support “customer 360” reads a projection updated by OrderConfirmed, PaymentCaptured, ShipmentDispatched—not live joins into three OLTP databases.
Failure if ignored: Read-only SQL users across service DBs “just for reports,” then a “quick UPDATE”—shared-database anti-pattern returns.

Decision rule

Do not open cross-service write (or casual write-capable) access for reporting. Prefer projections; if freshness cannot tolerate lag, expose an owning-service query API.

7.10 API Gateway and Backend for Frontend (BFF)

Problem: Browsers and mobile apps should not discover dozens of internal services, terminate TLS unevenly, or orchestrate checkout business rules at the edge.

API Gateway — definition: A shared edge entry for cross-cutting concerns: TLS, authentication, rate limiting, routing, request logging. It is infrastructure, not the home of domain invariants.

BFF — definition: A Backend for Frontend is an API tailored to one client experience (web BFF vs mobile BFF). It composes downstream calls and shapes DTOs for that UI. It may aggregate reads; it must not become the system of record for money or stock.

Example: Mobile BFF calls Orders for checkout status and a read-model for order history. Capture still happens inside Payments via the saga—not inside the BFF.
Failure if ignored: A “god gateway” encodes checkout rules → second ESB. Or mobile chats with twelve services → fragile clients and duplicated auth.

Decision rule

Gateway/BFF compose and authorize. Aggregates and sagas own money/stock invariants in domain services.

7.11 API Composition

Problem: A screen needs data owned by multiple services; clients should not fan out to all of them.

Definition: A composer (often BFF or gateway) calls several services (sync) and merges the response. Simple and fine for low fan-out reads; dangerous when used to implement write workflows (hidden sync saga).

Example: Order detail page: BFF gets order from Orders + shipment from Fulfillment. Refund button still hits Orders/Payments commands—not a composed write.

Decision rule

Compose reads carefully (timeouts, partial failure UX). Never implement multi-service writes solely via API composition—use an explicit saga.

7.12 Publish/Subscribe messaging

Problem: The producer should not know or wait for every consumer. Side effects must not block the core transaction path.

Definition: Producers publish events to a broker (topic/exchange); zero or more subscribers consume independently. Enables temporal decoupling and fan-out. Requires durable storage, consumer acknowledgements, poison-message policy (DLQ), and idempotent handlers.

Example: OrderConfirmed published once; Notifications, Analytics, and Search each subscribe. Checkout latency does not include SMTP.

Decision rule

Use pub/sub for facts others may react to. Use addressed commands when exactly one owner must mutate an aggregate.

7.13 Dead-Letter Queue (DLQ)

Problem: A poison message (bad payload, bug) fails forever; infinite retries block the partition/queue and amplify load.

Definition: After a retry budget, the broker/consumer moves the message to a Dead-Letter Queue for isolation, alerting, and manual or automated replay after a fix.

Example: Malformed PaymentCaptured fails schema validation three times → DLQ → on-call fixes consumer → replay.

Decision rule

Every critical consumer has a DLQ (or equivalent), an alert, and a documented replay runbook—not “delete and ignore.”

7.14 Timeout, Retry, Bulkhead, Circuit Breaker

These are stability patterns (popularized in release-it! / microservices resilience literature). They prevent one slow or failing dependency from taking down the caller.

Timeout — definition: Every outbound call has a deadline. Without timeouts, threads/connections wait forever and exhaust the process.

Retry — definition: Re-attempt a failed call. Safe only for idempotent operations; use exponential backoff and a retry budget. Blind retries on non-idempotent captures cause double charges and retry storms.

Bulkhead — definition: Isolate resources (thread pools, connection pools, even processes) per dependency so exhaustion of pool A cannot starve pool B. Named after ship compartments that limit flooding.

Circuit Breaker — definition: After error/latency thresholds, the breaker “opens” and fails fast (optionally serving a fallback) without calling the dependency; periodic probes (“half-open”) test recovery. Protects both sides from pile-on traffic during outages.

READ — Resilience stacking. Do not paste into a terminal.
  Caller
    |-- bulkhead pool for Payments
    |-- timeout 300ms
    |-- retry only if idempotent (max 2, exponential backoff)
    |-- circuit breaker: open after N failures → fail fast / fallback
    +-- never retry non-idempotent capture without idempotency key
Example: Catalog browse keeps its own pool; Payments outage opens the breaker on checkout authorize calls; product pages stay up.
Failure if ignored: One hung dependency saturates all request threads → entire node unhealthy → cascading failure across the estate.

Decision rule

Timeouts are mandatory. Retries require idempotency. Bulkheads isolate critical dependencies. Circuit breakers protect during sustained failure. Measure all four.

7.15 Correlation ID and Causation ID

Problem: A single user action fans out across services and messages. Without shared identifiers, logs and traces cannot reconstruct what happened.

Definition: A correlation ID identifies the whole conversation (one checkout attempt). A causation ID identifies the immediate parent message/request that caused this one. Propagate both on every RPC and broker message; include them in structured logs and traces.

Example: Support pastes correlationId=chk_9f2; you see Orders saga transitions, Inventory reserve, Payments capture, and the email attempt in one timeline.

Decision rule

No service ships to production money paths without correlation propagation. Treat missing IDs as a defect, not a nice-to-have.

7.16 Strangler Fig

Problem: Replacing a monolith in a big-bang rewrite is high risk; you need incremental replacement with production traffic.

Definition: Named after strangler fig vines that grow around a tree: a new implementation gradually intercepts requests for a capability at the edge (routing, facade, or branch-by-abstraction). The old code path shrinks until it can be deleted. Migration is incomplete until the old path is gone.

How it works: Route /notify/* to Notifications service; everything else to monolith. Expand routes over time; delete monolith mailer when metrics prove parity.

Example: Shadow traffic compares email sends for two weeks; then 10% → 50% → 100% cutover; monolith module deleted in the same milestone as cutover—not “later.”
Failure if ignored: Permanent dual implementations (“strangler” forever) double cost and diverge behavior.

Decision rule

Every strangler plan needs a delete criterion and an owner. Dual-run without an end date is not a strategy.

7.17 Branch by Abstraction

Problem: You want to change how a capability is implemented (local module → remote service) without a long-lived source-control branch war.

Definition: Introduce an abstraction (port/interface) in the monolith; implement the old behavior behind it; add the new remote adapter; flip traffic via config/feature flag; remove the old implementation when stable. Complements strangler for in-process seams.

Example: PaymentsPort first calls in-process module; flag switches to HTTP Payments service; in-process adapter deleted after reconciliation is clean.

Decision rule

Abstractions must not leak monolith ORM types across the future network boundary—or you have only postponed the mess.

7.18 Eventual consistency (as a designed property)

Problem: After you split writes across services, readers cannot always see a single globally up-to-date snapshot immediately after a write.

Definition: The system is eventually consistent when, given no new updates, all replicas/readers converge on the same values. During convergence, different components may disagree briefly. This is not a bug if product, UI, and support are designed for workflow states; it is a bug if you pretend strong global consistency still exists.

Example: UI shows “Payment processing…” while saga is AwaitingCapture. Support tools read saga state and reason codes—not ad-hoc guesses from three databases.

Decision rule

For each read path, state whether it is strongly consistent (same aggregate) or eventually consistent (cross-service). Design UX and ops for the latter explicitly.

Patterns are tools with failure modes. Naming Saga without compensations, Outbox, and idempotency is cargo-cult architecture.

8. Communication between services

What this part is for: choose sync vs async correctly; design sagas for money and stock.
What you should do: map each commerce edge to REST/gRPC/event and note idempotency keys.

8.1 REST vs gRPC vs messaging

Synchronous request/response — Caller waits for callee. REST (Representational State Transfer) over HTTP (Hypertext Transfer Protocol) with JSON (JavaScript Object Notation) is ubiquitous, cache-friendly for reads, easy to debug at the edge. gRPC (a Remote Procedure Call framework using HTTP/2 and Protocol Buffers) suits internal low-latency, strongly typed APIs and streaming; worse for ad-hoc browser clients without a gateway.

Asynchronous messaging — Producer publishes to a broker (queue/topic); consumers process later. Enables temporal decoupling and fan-out. Requires durable messaging, idempotent consumers, and poison-message strategy.

Edge typePreferWhy
User-facing query needing immediate answerSync (BFF/gateway → service)Latency budget visible to user
Command that must complete before UX continuesSync to orchestrator or owning serviceClear success/fail to client
Side effects (email, analytics, search index)Async eventsMust not block checkout
Multi-service business workflowSaga (async choreography or orchestrated commands)Avoid holding sync chains across many hops
Internal high-QPS (Queries Per Second) service-to-service callsgRPC or tuned HTTPEfficiency; still need timeouts

Sync chain anti-pattern: Client → A → B → C → D all synchronously. Definition: a request’s success depends on the entire chain being up within stacked timeouts. Why it matters: tail latency multiplies; one slow dependency saturates thread pools upstream. Example: Checkout calls Orders calls Payments calls Fraud calls Ledger inline. Failure: cascading timeouts and retries amplify load (retry storms). Decision rule: keep sync depth short (ideally one hop from gateway to an orchestrator or owning service); push the rest to async saga steps with explicit compensations.

READ — Sync chain vs saga. Do not paste into a terminal.
  BAD (sync chain):
  Client -> Orders -> Inventory -> Payments -> Ledger -> Notify

  BETTER:
  Client -> Orders (accept command, return 202/orderId)
             |
             +-- outbox: OrderCheckoutStarted
                    |-> Inventory (reserve)
                    |-> Payments (authorize/capture)
                    |-> Ledger (journal)
                    |-> Notify (email)

8.2 API gateway vs BFF

Pattern cards: §7.10 and §7.11.

API gateway (applied): Edge entry for TLS termination, authn, rate limits, routing—not a second ESB of checkout rules.

BFF (applied): Purpose-built API for one client experience; composes downstream calls and shapes DTOs. Keep money invariants in domain services; BFF/gateway only compose and authorize.

8.3 Applying sagas, compensations, idempotency, delivery, outbox

Full definitions live in the design patterns catalog (Saga, compensation, Outbox, Inbox, eventual consistency). This section applies them to service communication.

Eventual consistency in the UX: After a write, not all readers immediately see the new state; the system converges. For commerce, “order placed” may briefly show “payment pending.” Design UI and support for that—see §7.18.

Saga (applied): Coordinate checkout across Inventory and Payments as local transactions plus compensations—not distributed 2PC. Prefer an explicit Orders state machine for money/stock (§7.4, §7.6).

Example (checkout orchestration): Orders moves state Pending → Reserved → Authorized → Confirmed. On payment failure after reserve: command Inventory to release; mark order Failed; emit integration events for Notifications.

Idempotency (applied): Same command/message twice → same business effect once. Keys are part of the API contract. See §7.8.

At-least-once delivery: duplicates are normal; “effectively once” = idempotent handlers + outbox—not broker magic alone.

Transactional outbox / inbox (applied): Same local TX as the business write inserts an outbox row; publisher relays to the broker. Consumers dedupe via inbox. See §7.7 and §7.8. Ban dual-write (“update then publish”) on money/stock paths.

READ — Outbox sequence. Do not paste into a terminal.
  BEGIN TRANSACTION (TX)
    update orders set status='Confirmed' where id=...
    insert into outbox(id, type, payload, created_at) values (...)
  COMMIT
  -- separate publisher --
  read outbox -> publish to broker -> mark published
  consumer: BEGIN TRANSACTION; inbox dedupe; apply; COMMIT; ack
Failure if ignored: Dual-write races create silent divergence—Orders thinks paid; Payments never saw the command; support cannot reconstruct truth without manual ledger archaeology.

Decision rule

For any workflow that moves money or stock across services: use a saga with explicit states, compensating actions, idempotent commands, and outbox/inbox. Never rely on a long sync chain or “we’ll fix duplicates in support.”

8.4 Choreography versus orchestration in practice

Definitions: §7.6. Both styles implement sagas; they differ in where workflow state lives. This section is the commerce application.

Choreography example (order confirmation fan-out): Orders emits OrderConfirmed. Fulfillment creates a shipment; Notifications sends mail; Analytics records conversion. No central step list—each consumer’s subscription is the workflow. This fits reactive side effects after a terminal business decision.

Orchestration example (checkout): Orders (or CheckoutOrchestrator inside that context) owns states Pending → AwaitingReserve → AwaitingAuth → AwaitingCapture → Confirmed | Failed. It issues commands and interprets replies/events. Money and stock require this clarity because compensations are ordered and conditional.

READ — Choreography vs orchestration. Do not paste into a terminal.
  CHOREOGRAPHY (implicit graph):
  OrderConfirmed --> Fulfillment
                 --> Notifications
                 --> Search "purchased" signal

  ORCHESTRATION (explicit state machine):
  [Orders saga]
     |-- command Reserve --> Inventory -- reply -->|
     |-- command Authorize -> Payments -- reply -->|
     |-- on fail: command Release / Void ---------->|
     +-- emit OrderConfirmed only from Confirmed state

Why it matters: choreographed checkout often hides “what happens if auth fails after reserve?” in tribal knowledge. Orchestrated notification fan-out often recreates an ESB inside Orders.

Failure if ignored: hybrid accidental workflows—half the steps in events, half in undocumented sync calls—defeat both operability and autonomy.

Decision rule

Orchestrate multi-step commands that require compensations (checkout, refund). Choreograph one-way reactions to terminal facts (notify, index, analytics). Document the choice on the context map.

8.5 Message semantics: commands, events, queries

Decision rule: Do not pretend a command is an event (“StockReservationRequested” consumed by five services that all try to reserve). One aggregate owns the mutation.

9. Data ownership and multi-step workflows

What this part is for: enforce database-per-service and replace 2PC with sagas and read models.
What you should do: list each table’s writing service; ban cross-service writes.

Database-per-service — Each service owns its persistent store (physical database, schema, or tightly controlled database with only that service’s credentials). Others obtain data via API, events, or projected read models. Full card: §7.3.

Shared database anti-pattern — Multiple services reading/writing the same tables. This recreates a monolith schema with distributed race conditions and coupled releases.

READ — Ownership. Do not paste into a terminal.
  Orders database <--only-- Orders service
  Payments database <--only-- Payments service
  Inventory database <--only-- Inventory service
  Reporting database <-- writers: projectors from events (CQRS read side)

CQRS — Separate write models from query-optimized read models/projections. Full card: §7.9; reporting application in §9.1 below.

Why distributed 2PC (Two-Phase Commit) is usually avoided — Two-phase commit across services (prepare/commit with a transaction coordinator) couples availability: any participant or the coordinator blocking stalls the transaction; locks are held longer; failure modes are operationally harsh; cloud databases and polyglot stores often lack cooperative 2PC. Prefer local ACID + saga compensations + reconciliation—see the Saga vs 2PC diagram in §7.4.

Example: Instead of 2PC across Orders and Inventory, Inventory reserves stock locally; Orders confirms; if payment fails, Inventory releases. A nightly reconciler flags reservations older than TTL (Time To Live) without payment.

Decision rule

One writer per datum. Cross-service truth is assembled from events and APIs. If you believe you need 2PC, first ask whether the aggregates should live in one service.

9.1 CQRS and reporting without shared joins

Pattern definition: §7.9 CQRS and read-model projection. Here is the reporting application.

Problem: Finance wants “all captured payments with order lines and ship dates.” In a monolith that was a SQL join. After a split, those tables no longer share a database.

Approach: Keep command models normalized per service. Build read models (projections) updated by integration events into a reporting store or warehouse. The projection is eventually consistent by design; finance jobs run after lag SLO, not inside the checkout request.

Example: A OrderFacts projector listens to OrderConfirmed, PaymentCaptured, ShipmentDispatched and upserts a denormalized row keyed by orderId. Support UI reads OrderFacts; Orders service remains the write authority for cancellations.
Failure if ignored: Teams open read-only Structured Query Language (SQL) users across all service databases “just for reports,” then inevitably write a “quick fix” UPDATE—reintroducing the shared-database anti-pattern.

Decision rule

Reporting credentials never get write access to service OLTP (Online Transaction Processing) stores. If a report needs fresher data than projection lag allows, expose an owning-service query API—not foreign table access.

9.2 Multi-step workflows without 2PC — worked sketch

Suppose checkout must (1) reserve stock, (2) authorize payment, (3) capture, (4) post ledger. A distributed 2PC would hold locks across Inventory and Payments while the PSP round-trip runs—an availability and latency disaster.

Instead:

  1. Each step is a local transaction with its own commit.
  2. Progress is stored in saga state (Orders).
  3. Timeouts move the saga into compensation or “needs operator” with reason codes.
  4. Reconciliation closes gaps (e.g., capture succeeded at PSP, local Payments row missing → repair from webhook/PSP query).
Semantic atomicity (saga) ≠ single-commit atomicity (2PC). Design for the former; reserve the latter for one aggregate’s database.

10. Cross-cutting platform concerns

What this part is for: list platform capabilities that must exist before a wide migration.
What you should do: gap-check your org against each item.
No correlation IDs + no timeouts = distributed monolith of failure

Decision rule

Do not extract a money-moving service until timeouts, idempotency, structured logs with correlation, and metrics for saga states are in place for that path.

11. Why migrate versus fashion

What this part is for: separate legitimate drivers from cargo-cult motives.
What you should do: write your drivers before proposing splits.

Legitimate drivers

Non-reasons

Decision rule

Migration RFC must cite metrics (lead time, failed deploy rate, scaling cost, incident blast radius) and a named seam—not a desire to be “cloud native.”

12. When not to migrate

What this part is for: stop unsafe migrations early.
What you should do: if several items apply, improve the monolith first.
Failure if ignored: A year later you have more repositories, the same tangled domain, and a slower path to correct money states.

13. Migration strategies and failure modes

What this part is for: pick an incremental pattern and know how it fails.
What you should do: choose one primary strategy for the first extraction.

13.1 Strangler fig

Full pattern: §7.16. Applied: New implementation gradually intercepts traffic for a capability; the old monolith shrinks until the strangler owns the path. Failure modes: proxy complexity; dual writes without outbox; never deleting the old path (“strangler” becomes permanent duplication).

13.2 Extract service

Definition: Move a modular capability and its data to a new deployable; the monolith (or peers) call it via API/events. Requires database-per-service ownership for the extracted data and usually a strangler or branch-by-abstraction cutover. Failure modes: incomplete data ownership (monolith still writes extracted tables); chatty sync extraction of a non-boundary.

13.3 Branch by abstraction

Full pattern: §7.17. Applied: Introduce an interface in-process; flip implementations from local to remote behind the abstraction; remove the old implementation when stable. Failure modes: abstraction leaks monolith types; feature flags never cleaned up.

13.4 Incremental cutover

Definition: Shadow traffic, percentage rollout, or customer cohort migration with clear rollback. Failure modes: no parity checks; cutting 100% before reconciliation of money/stock divergences.

READ — Strangler. Do not paste into a terminal.
  Clients -> Edge routing
               |-- /notify/*  -> Notifications service (new)
               |-- /*         -> Monolith (old)
  Over time more paths move; monolith modules deleted, not abandoned.

Decision rule

Every strategy needs a delete/finish criterion. If the old path cannot be removed, the migration is incomplete and cost will compound.

14. Migration playbook (phases with entry/exit criteria)

What this part is for: a gated plan teams can execute.
What you should do: treat phases as gates; do not skip measurement.
PhaseEntry criteriaWorkExit criteria
0. BaselineProd existsSLOs, correlation IDs in monolith, architecture tests for modulesDashboards for checkout/payment; module map agreed
1. ModularizePhase 0 exitSeparate schemas/packages; ports; kill cross-table writesArchitecture tests green; no illegal imports
2. Platform thin sliceNeed extractionCI for second deployable; broker; log/metric/trace standardsCan deploy a “hello” service with auth and traces
3. Extract non-criticalPlatform readye.g. Notifications via eventsMonolith code deleted; error budget held
4. Extract transactionalSaga skills provenPayments/Inventory with outbox, idempotency, ledgerReconciliation clean; compensations tested
5. HardenMultiple services liveBulkheads, contract tests, chaos of dependenciesDocumented runbooks; on-call per service
Example exit test for Payments extract: replay duplicate capture messages in staging; prove single ledger credit; kill Payments mid-saga and show Orders ends in a terminal compensated state with support-visible reason codes.

15. Common microservices design mistakes and anti-patterns

What this part is for: recognize and avoid the failure modes that dominate real estates.
What you should do: use this list in design reviews; each item includes a decision rule.

15.1 Shared database across services

Definition: Two or more services write (or even heavily share write-coupled reads of) the same tables. Why it matters: schemas become a released contract across teams; invariants cannot be enforced in one place. Example: Orders and Inventory both UPDATE stock. Failure: lost updates, coupled migrations, “who broke checkout?” ambiguity. Decision rule: one writer service per table; others use APIs/events/read models.

15.2 Entity-oriented CRUD services

Definition: Services named after nouns in the Entity-Relationship Diagram (ERD) (UserService, OrderService as pure Create/Read/Update/Delete — CRUD) rather than business capabilities and aggregates. Why it matters: workflows still span many sync calls; no home for invariants. Example: Checkout is an orchestration of CreateOrder + UpdateUser + CreatePayment with no owner. Failure: nanoservice mesh of repositories. Decision rule: name and shape services around capabilities and aggregates that own invariants (Checkout/Orders, Payments, Inventory).

15.3 Chatty fine-grained services

Definition: Client or service requires dozens of RPCs to render one screen or complete one command. Why it matters: latency and partial failure explode. Example: Mobile calls 15 endpoints to open cart. Failure: timeouts; over-fetching; BFFs reinvented poorly. Decision rule: coalesce by use-case (BFF or richer service APIs); prefer events for fan-out side effects.

15.4 Distributed monolith (lockstep deploys)

Definition: Separated processes that must ship together. Why it matters: you pay distribution cost without autonomy. Example: Shared library of domain entities forces fleet rebuild. Failure: release trains return. Decision rule: version contracts; ban releasing shared domain models as required lockstep libraries—share only stable kernels (logging, metrics) carefully.

15.5 Synchronous “saga” across many services without timeouts/compensations

Definition: A long sync call graph treated as a transaction, without saga state or compensations. Why it matters: holds resources; unclear recovery. Example: HTTP checkout waits on five services; on timeout, stock reserved and card charged inconsistently. Failure: manual finance cleanup. Decision rule: async saga with persisted state machine, deadlines, and tested compensations.

15.6 Missing idempotency on money/stock commands

Definition: Handlers assume exactly-once delivery. Why it matters: at-least-once is the norm. Example: Double capture after client retry. Failure: customer double-charged; ledger imbalance. Decision rule: idempotency keys on all money/stock/ledger commands; unique constraints in the database.

15.7 Dual-write without outbox

Definition: Update database and publish message in two non-atomic steps. Why it matters: either side can succeed alone. Example: Order confirmed but OrderConfirmed never published. Failure: silent stall of downstream fulfillment. Decision rule: transactional outbox (or CDC (Change Data Capture) from the write store) for integration events.

15.8 Ignoring Conway / no owning team

Definition: Services exist without a durable product owner and on-call. Why it matters: architecture mirrors communication paths. Example: “Platform” team owns twenty domain services. Failure: abandoned queues; slow incident response. Decision rule: no service without a named owning team and SLO.

15.9 Big-bang rewrite

Definition: Rebuild everything as microservices before cutting over. Why it matters: long dual-roadmaps; requirements drift. Example: Two-year rewrite; monolith still serves prod. Failure: rewrite abandoned or rushed incomplete. Decision rule: strangler/extract with production traffic and delete criteria.

15.10 Premature Kubernetes as “architecture”

Definition: Equating orchestration tooling with sound boundaries. Why it matters: Kubernetes (K8s) does not fix a shared database or missing sagas. Example: Monolith on Kubernetes called “migration done.” Failure: ops complexity without design progress. Decision rule: choose runtime for density/ops needs after boundaries and data ownership are clear.

15.11 Shared libraries that force coupled releases

Definition: A “common” package containing domain entities, SQL, or API DTOs that all services must upgrade together. Why it matters: invisible distributed monolith. Example: company-domain-1.2.3 required everywhere. Failure: cannot deploy Payments alone. Decision rule: share only truly generic utilities; duplicate DTOs or generate from contract schemas; never share persistence models.

15.12 Leaking domain models across APIs

Definition: Publishing internal aggregates/ORM graphs as the public contract. Why it matters: every internal refactor breaks consumers. Example: REST returns nested Hibernate entities with lazy-load surprises. Failure: brittle clients; security over-exposure. Decision rule: explicit request/response and event DTOs; anti-corruption layers at boundaries.

15.13 No correlation IDs / unobservable estate

Definition: Logs and messages lack a shared request/workflow identifier. Why it matters: cannot reconstruct a checkout across services. Example: Support sees “payment failed” with no link to order saga step. Failure: long MTTR; unverifiable money trails. Decision rule: propagate correlation and causation IDs on every hop; index them in logs and traces.

15.14 Treating eventual consistency as optional in UI/support

Definition: Assuming users and agents always see strongly consistent global state. Why it matters: races are normal after split. Example: UI shows “paid” while fulfillment still “awaiting payment event.” Failure: duplicate support tickets; agents issue manual refunds incorrectly. Decision rule: model workflow states in UI; expose reason codes; give support tools that read saga state, not ad-hoc database guesses.

15.15 Unversioned or orphan event catalogs

Definition: Events renamed casually; consumers unknown. Why it matters: integration becomes folklore. Example: order_updated means five different things. Failure: poison messages; silent wrong automations. Decision rule: maintain an event catalog (producer, consumers, payload intent, compatibility rules).

16. End-to-end design: transactional commerce system

What this part is for: apply the guide to a complex money-moving system you could design and stand up.
What you should do: treat this as a reference architecture; adapt names to your ubiquitous language.

16.1 Domain map and bounded contexts

Bounded contextResponsibilityKey language
IdentityAccounts, sessions, rolessubject, credential, customerId
CatalogProduct presentation, search docslisting, SKU reference, publish
Pricing (optional)Price lists, promotions enginequote, discount, tax basis
CartPre-order selectionsline, quantity, cart snapshot
InventoryStock truth, reservationson-hand, reserved, reservationId
OrdersCheckout workflow / order lifecycleorder, line, saga state
PaymentsAuthorize, capture, refund with PSPpaymentIntent, capture, refund
Ledger / AccountingImmutable financial journaljournal entry, debit/credit, account
FulfillmentPick/pack/shipshipment, tracking
NotificationsEmail/SMS/pushtemplate, delivery attempt
READ — Context map. Do not paste into a terminal.
  Identity ----customerId----> Cart ----checkoutCommand----> Orders
  Catalog --SKU--> Cart / Inventory
  Pricing --quoteId--> Cart / Orders (optional)
  Orders --reserve--> Inventory
  Orders --pay--> Payments --journal--> Ledger
  Orders --fulfill--> Fulfillment --notify--> Notifications
  Payments --notify--> Notifications

16.2 Aggregates and invariants

Money truth: Payments (PSP-facing) + Ledger (books). Orders holds business workflow state, not the books of record.
Failure if ignored: Storing “balance” only on the Order row without a ledger makes refunds, partial captures, and audits unreliable.

16.3 Sequence: checkout, capture, refund

READ — Checkout happy path (orchestrated saga). Do not paste into a terminal.
  1. Client -> Orders: StartCheckout(cartSnapshot, idempotencyKey)
  2. Orders: create Order(Pending), outbox OrderCheckoutStarted
  3. Orders -> Inventory: ReserveStock(orderId, lines, ttl)  [sync or command]
  4. Inventory: reserve locally; emit InventoryReserved / fail InventoryReserveFailed
  5. Orders -> Payments: CreateAndAuthorize(orderId, amount, currency, key)
  6. Payments: PSP authorize; emit PaymentAuthorized / PaymentAuthFailed
  7. On auth OK: Orders -> Payments: Capture(...)  (or capture at fulfill—policy choice)
  8. Payments: capture; outbox PaymentCaptured
  9. Ledger consumer: post balanced journal (cash/receivable vs revenue clearing)
 10. Orders: status Confirmed; outbox OrderConfirmed
 11. Fulfillment + Notifications consume OrderConfirmed

  Compensations:
  - Auth fail after reserve -> ReleaseStock; Order Failed
  - Capture fail -> void/cancel auth if needed; ReleaseStock; Order Failed
  - Post-confirm refund -> Payments Refund + Ledger reversing entry + Inventory restock policy

Policy choice: authorize-at-checkout / capture-at-ship vs capture-at-checkout. Capture-at-ship reduces refund volume but prolongs authorization holds; document the policy in the saga.

16.4 Event catalog (integration events)

EventProducerConsumersPayload intent
ProductPublishedCatalogSearch indexer, Inventory (SKU upsert)SKU identity + salable flag
OrderCheckoutStartedOrdersObservability, analyticsorderId, amount, lines
InventoryReservedInventoryOrdersreservationId, orderId, lines, expiresAt
InventoryReserveFailedInventoryOrdersorderId, reason
PaymentAuthorizedPaymentsOrderspaymentId, orderId, amount
PaymentCapturedPaymentsOrders, Ledger, NotificationspaymentId, orderId, amount, pspRef
PaymentCaptureFailedPaymentsOrdersorderId, reason
LedgerEntryPostedLedgerReportingjournalId, accounts, amounts
OrderConfirmedOrdersFulfillment, Notifications, Cart clearorderId, shipTo, lines
OrderFailedOrdersNotifications, analyticsorderId, reasonCode
RefundSucceededPaymentsOrders, Ledger, Inventory policyrefundId, paymentId, amount
ShipmentDispatchedFulfillmentNotifications, OrdersshipmentId, tracking

16.5 Sync vs async choices per edge

EdgeModeRationale
Client → Orders StartCheckoutSyncNeed orderId and acceptance/validation errors
Orders → Inventory ReserveSync command or request/replyStock gate often must finish before pay; still short timeout + saga resume
Orders → Payments Authorize/CaptureSync to Payments API (Payments owns PSP async webhooks)Payments may return pending; Orders stores state
Payments → LedgerAsync eventLedger must not block PSP path; idempotent posting
OrderConfirmed → Fulfillment/NotifyAsyncSide effects; retries OK
PSP webhooks → PaymentsSync endpoint + inboxVerify signature; idempotent by psp event id

16.6 Consistency strategy

Local ACID inside Inventory, Payments, Ledger, Orders. Cross-service: orchestrated saga on Orders (or a dedicated Checkout orchestrator in the Orders context) + transactional outbox on every producer + inbox idempotency on consumers. Reconciliation jobs: expired reservations without payment; captures without ledger posts; orders stuck in non-terminal states beyond SLO.

16.7 Suggested service APIs (outline)

READ — API outline. Do not paste into a terminal.
  Orders
    POST /orders/checkout          Idempotency-Key required
    GET  /orders/{id}
    POST /orders/{id}/cancel

  Inventory
    POST /reservations             {orderId, lines, ttl}
    POST /reservations/{id}/release
    GET  /availability?sku=

  Payments
    POST /payment-intents          {orderId, amount, currency, customer}
    POST /payment-intents/{id}/capture
    POST /payment-intents/{id}/refund
    POST /webhooks/psp

  Ledger
    POST /journals                 (usually internal consumer; admin post rare)
    GET  /accounts/{id}/entries

  Fulfillment
    POST /shipments
    POST /shipments/{id}/dispatch

  Notifications
    (consume events; optional POST /admin/preview)

16.8 Failure scenarios and compensations

FailureDetectionCompensation / action
Insufficient stockReserve rejectedOrderFailed; notify user; no payment
Auth declinedPayments responseRelease reservation; OrderFailed
Capture timeout unclearTimeout / unknownIdempotent capture retry; query PSP; do not double-reserve
Ledger post failsInbox retry exhaustionAlert; replay; Orders may stay Confirmed—books must catch up
Duplicate webhookInbox hitNo-op success
Customer refundSupport/APIRefund PSP → reversing ledger → restock policy → Order state
Fulfillment downConsumer lagOrder remains Confirmed; backlog drains; UX shows processing

16.9 Minimum deploy topology (conceptual)

READ — Compose-level topology (conceptual). Do not paste into a terminal.
  [clients]
      |
  [api-gateway / BFF]
      |
  +---+---+----+------+----------+------------+
  |       |    |      |          |            |
orders inventory payments ledger fulfillment notifications identity catalog cart
  |       |    |      |          |            |
  db-o   db-i  db-p  db-l      db-f          db-n  ...

  [message-broker] <-- outbox publishers / consumers

  [otel-collector] -> traces/metrics/logs backends
  [secrets] [config]

This is deliberately vendor-neutral: any managed Postgres/MySQL, any broker (NATS/Kafka/RabbitMQ), any gateway. The architecture is the boundaries, contracts, and saga—not the brand of orchestrator.

Decision rule

Stand up commerce microservices only when Ledger + Payments idempotency + Inventory reservations are designed together. Extracting “Payments” without books and stock compensation is an incomplete cut.

16.10 Ledger design notes (money invariants)

A minimal chart of accounts for the first cut:

READ — Capture journal (illustrative). Do not paste into a terminal.
  Journal entry (paymentId=pay_9, orderId=ord_9, amount=100.00 USD):
    DR  PSP Clearing          100.00
    CR  Revenue - Product     100.00
  Constraints:
    - unique (paymentId, entryType=CapturePost) for idempotency
    - sum(DR) == sum(CR) enforced in Ledger aggregate transaction
    - no UPDATE of posted lines; refund creates a new balanced entry

Refund sequence: Support/API → Orders validates refundability → Payments Refund with idempotency key → PSP → RefundSucceeded → Ledger reversing entry → Inventory restock command (policy) → Notifications. Orders stores refund workflow state; it does not invent balances.

16.11 Support and UI under eventual consistency

Expose to operators: saga state, last reason code, paymentId, reservationId, correlation ID, and links to PSP dashboard. Never instruct agents to “just UPDATE the orders table.” Build a support command API that reuses the same idempotent domain commands.

Example: Agent sees AwaitingCapture aged 8 minutes. Playbook: query Payments by orderId; if PSP captured, replay post; if PSP declined, run compensation release. Correlation ID pastes into the trace UI.

17. From zero to a running first cut

What this part is for: a detailed, actionable learning and build path—not a shallow bullet list.
What you should do: execute phases in order; meet each definition of done before advancing.

17.1 Prerequisites

Skills

Environment

17.2 Week / phase plan

Phase A — Modular monolith (weeks 1–3)

Goal: one deployable commerce app with module boundaries and separate schemas (or schema namespaces) for Catalog, Cart, Orders, Inventory, Payments (facade), Notifications.

Work: define ubiquitous language glossary; implement checkout as in-process ports; architecture tests forbidding illegal imports; single database OK but no cross-module table writes.

Definition of done: place-order test passes in one local ACID transaction where still in-process; module dependency graph is acyclic; glossary reviewed with a domain partner.

Measure: build time; test time; count of illegal dependency violations (should be zero).

Phase B — Add broker + outbox in-process (weeks 3–5)

Goal: publish integration events via transactional outbox even while consumers may still live in-process or as a worker.

Work: outbox table; publisher relay; at-least-once consumer with inbox; correlation IDs on messages.

Definition of done: kill the process after database commit but before publish; relay recovers event; duplicate delivery does not double-send email in a test.

Measure: outbox lag; duplicate delivery test green.

Phase C — Extract Notifications (weeks 5–7)

Goal: first independently deployable service; lowest money risk.

Work: Notifications service owns its database; consumes OrderConfirmed / PaymentCaptured; strangler delete of monolith mailer.

Definition of done: monolith contains no email-sending code path; deploy Notifications without redeploying monolith; traces show correlation across both.

Measure: notification success rate; deploy frequency of Notifications vs monolith.

Phase D — Extract Payments with ledger constraints (weeks 7–12)

Goal: money path with idempotency and ledger posting.

Work: Payments service + Ledger service (or Ledger module extracted with Payments if staffing is thin—prefer separate write models even if co-deployed initially, then split); saga compensation with Inventory still in monolith or extracted; PSP sandbox; webhook inbox.

Definition of done: duplicate capture and duplicate webhook tests pass; journal always balanced; reconciliation report for “captured but not posted” is empty in staging soak; compensations covered by automated tests.

Measure: payment success, saga completion time, reconciliation exceptions count, refund correctness sample.

Phase E — Gateway / BFF + harden (weeks 12–16)

Goal: edge composition, resilience, operability.

Work: API gateway or BFF; timeouts/bulkheads/circuit breakers on sync edges; contract tests; load test checkout; runbooks.

Definition of done: dependency failure (Notifications down) does not fail checkout; Payments down fails checkout gracefully with reason codes; dashboards for RED metrics per service; on-call owners named.

Measure: error budget burn; p95 checkout; MTTR with correlation-ID drills.

Decision rule

Do not start Phase D until Phase B’s outbox/inbox discipline is proven. Do not expand service count while reconciliation exceptions are unexplained.

17.2.1 Detailed activities inside each phase

Phase A daily practice: Write three architecture tests (Orders cannot import Inventory persistence; Payments cannot import Orders entities; Catalog cannot write stock). Pair with a domain expert for one hour on glossary conflicts (“status,” “reserve,” “capture”). Deliver a thin vertical checkout that still uses in-process adapters.

Phase B daily practice: Implement outbox polling or log-tail publisher; add consumer that writes inbox + side effect in one database transaction (TX); run chaos: kill publisher, duplicate message, poison payload → Dead-Letter Queue (DLQ). Document message headers: correlationId, causationId, messageId, contentType, schemaVersion.

Phase C cutover: Shadow-send emails from both monolith and service for 48h; compare; route 10% → 50% → 100%; delete monolith mailer; remove feature flag.

Phase D rehearsal script (staging):

  1. Checkout happy path; assert ledger balanced.
  2. Replay identical idempotency key; assert no second capture.
  3. Fail reserve; assert no payment object in Captured.
  4. Authorize then kill Payments before capture response; recover via PSP query + resume.
  5. Refund partial amount; assert ledger + order state.
  6. Expire reservation TTL; assert stock returns.

Phase E resilience drills: Block Notifications network; checkout must succeed. Inject 500ms–2s latency on Inventory; verify timeouts and user-visible errors, not thread exhaustion. Break the broker; verify outbox lag alerts fire before customer impact on already-accepted commands.

17.3 What to measure (ongoing)

17.4 Skills ladder (what to study alongside the build)

  1. Relational transactions and isolation levels (why idempotency unique keys matter).
  2. HTTP timeouts and retry semantics.
  3. One broker deep enough to configure a Dead-Letter Queue (DLQ) and consumer acknowledgement.
  4. OpenTelemetry traces across HTTP and messaging.
  5. DDD tactical patterns (aggregate, domain event) from a primary text—then apply here.
  6. Payment provider sandbox webhooks and signature verification.

Avoid starting with service mesh, multi-cluster Kubernetes, or polyglot rewrites. Those amplify mistakes before you have boundaries.

18. Conway’s law and organization

What this part is for: align team topology with service topology.
What you should do: refuse service cuts that have no owner.

Conway’s law — Organizations design systems that mirror their communication structure. If all domain changes require a central committee, your “microservices” will behave like a distributed monolith. Inverse Conway maneuver: deliberately shape teams to the bounded contexts you want (Orders team, Payments team), with a platform team providing paved roads—not owning business invariants.

Example: Two developers “owning” twelve services guarantees neglect. Prefer fewer services than teams cannot operate.

Decision rule

Service count ≤ what you can on-call and product-own. Platform multiplies efficiency; it does not replace domain ownership.

19. Closing checklist

What this part is for: a design-review checklist you can reuse.
What you should do: copy into design RFCs (Request for Comments / proposal docs); require evidence for each yes.

20. Wrapping up

Senior review note: This article is intentionally opinionated on transactional systems: prefer local ACID (Atomicity, Consistency, Isolation, Durability) plus saga plus outbox over distributed Two-Phase Commit (2PC); prefer capability-aligned services over Create/Read/Update/Delete (CRUD) entity services; prefer a modular monolith until release coordination or isolation pain is measured. Treat every diagram as a starting design—validate aggregates and compensations with domain experts before production money moves. Abbreviations used throughout are expanded on first use and listed in the glossary above.
What this part is for: close the mental model.
What you should do: state your default architecture choice and next extraction seam in one paragraph each.

Software architecture is the set of hard-to-reverse decisions about boundaries, contracts, data ownership, deployment, and failure. Monoliths and modular monoliths remain excellent defaults. Microservices are justified when independent deployability, heterogeneous scale, or org isolation outweigh the tax of partial failure, eventual consistency, and operability.

For complex transactional systems, competence means designing aggregates and invariants, choosing sync versus async edges deliberately, coordinating with sagas rather than Two-Phase Commit (2PC), publishing via outbox, and keeping a ledger for money. Migration succeeds as an incremental strangler with exit criteria—not as a fashion rewrite.

If you can draw the commerce context map, name the saga states, and operate the first extracted service with correlation-quality observability, you are ready to deepen the cut. If you cannot, modularize first; the network will not invent the missing design.