@jepemo
EXIT

CI/CD, Migrations, and Release Checks

Learning / Architecture And Platforms / Backend Development Foundations / 014

LESSON

Backend Development Foundations

014 25 min beginner

CI/CD, Migrations, and Release Checks

By the end of this lesson, you will be able to...

  • Trace a backend release from commit to production traffic.

  • Explain why database migrations need compatibility windows during rollout.

  • Choose release checks that reduce blast radius without pretending deployment is risk-free.

Idea in one sentence: CI/CD moves a backend through controlled state transitions: build one artifact, verify evidence, coordinate code and data changes, roll out gradually, and stop when signals show user harm.

Core Insight

The previous lesson gave us a deployable artifact:

orders-api:<commit-sha>

Now we need to move that artifact into production.

Imagine the orders API adds idempotency support for checkout. The database migration adds a required column:

ALTER TABLE orders ADD COLUMN idempotency_key text NOT NULL;

The new app writes idempotency_key. The tests pass. The image builds. The release starts a rolling deploy across ten instances.

Then checkout begins failing.

Some old app instances are still running. Those old instances do not write idempotency_key. A background worker can also still create orders without the new field. Existing rows do not have values.

No single person typed "break production."

The failure came from treating production like one instant switch:

old world -> new world

Production is usually a transition:

old app + old database
old app + expanded database
old app + new app + expanded database
new app + expanded database
new app + enforced database rule

The beginner mistake is:

CI is green, so the release is safe.

A green build is evidence. It is not the whole release.

CI/CD should help the team answer:

What state transition are we making?
What evidence says it is safe enough to continue?
What signal says we should stop?
What recovery path remains if this fails?

The Small Situation

Use one change:

POST /orders now requires idempotency_key.

The intended release touches several boundaries:

application:
  require idempotency_key in checkout requests
  store idempotency_key with order
  reject duplicate key safely

database:
  add orders.idempotency_key
  eventually enforce uniqueness or non-null behavior

artifact:
  build orders-api:abc123
  test the artifact or equivalent output

deployment:
  apply migration
  roll out new instances
  watch checkout and database signals

The visible pieces are:

Plain meaning:

CI checks whether a change can join the codebase with evidence.

In this scenario:

Unit, integration, contract, migration, and artifact-startup checks run before production traffic sees the change.

Technical name:

That is continuous integration.

Plain meaning:

CD moves a verified artifact through environments toward users.

In this scenario:

The same orders-api:abc123 image is promoted, configured for production, rolled out, and watched.

Technical name:

That is continuous delivery or continuous deployment, depending on whether the final production step is manual or automatic.

A Release Trace

Trace one careful release:

Input:
  commit abc123
  migration 20260701_add_idempotency_key
  image build recipe

Step 1: build artifact
  create orders-api:abc123
  attach image digest

Intermediate state:
  artifact is identifiable

Step 2: run checks
  unit tests for duplicate-key logic
  contract tests for required field and error shape
  integration tests with database
  migration test from previous schema
  artifact startup check

Step 3: expand database
  add nullable idempotency_key column
  old app can still write orders

Intermediate state:
  old and new code can coexist

Step 4: deploy compatible app code
  new app writes idempotency_key
  old instances may still run during rolling deploy

Step 5: observe rollout
  checkout success rate
  validation failure rate
  duplicate-key conflict rate
  database errors
  worker failures
  container restarts

Step 6: finish migration after evidence
  backfill if needed
  enforce uniqueness or non-null only when all writers comply

Output:
  production reaches the new intended state

Now compare the broken release:

Step 1:
  add NOT NULL column immediately

Step 2:
  rolling deploy starts

Intermediate state:
  old instances still write orders without idempotency_key
  database rejects writes

Output:
  checkout fails during rollout

The release failed because the intermediate state was not compatible.

Check: Why is adding orders.idempotency_key as NOT NULL immediately risky during a rolling deploy?

Think first, then reveal.

Answer: Old app instances, workers, or scripts may still create orders without the new field, and existing rows may not have values. The database rule becomes stricter before every writer is ready.

Expand and Contract

The safer migration shape is often called expand and contract.

Plain meaning:

First make the system accept both old and new shapes. Then move traffic and writers to the new shape. Only later remove or enforce the old shape.

In this scenario:

The database should temporarily allow orders with and without idempotency_key while old and new code coexist.

Technical name:

This is an expand-and-contract migration.

For the idempotency release:

Step 1: Expand
  add nullable idempotency_key column
  add supporting index if safe
  old code still writes orders

Step 2: Deploy compatible code
  new API requires idempotency_key for new clients
  new app writes idempotency_key
  old workers tolerate the extra column

Step 3: Backfill or classify old rows
  decide what historical orders should contain
  fill safe values only if the rule needs them

Step 4: Enforce
  add uniqueness or NOT NULL only after evidence
  confirm no old writers remain
  remove compatibility code later

This pattern buys recovery options.

If new app code fails after the expand step, old code can often keep running because the nullable column does not break old writes. If the database rule was enforced immediately, rollback becomes much harder.

Check: After adding a nullable column, the new app fails for an unrelated reason. Can old app instances usually keep running?

Think first, then reveal.

Answer: Often yes, if old code ignores the extra nullable column and the migration did not break existing reads or writes. That is the point of the expand step.

Evidence Before Release

CI should produce evidence that matches the release risk.

For this release, useful checks include:

A migration test that creates only a fresh database from the final schema is not enough.

Production does not start fresh. It has existing data, current indexes, previous schema state, long-running queries, workers, and old app versions during rollout.

A better migration test asks:

Given:
  previous production schema
  representative existing orders

When:
  migration 20260701_add_idempotency_key runs

Expect:
  old writes still work after expand
  new writes work after app deploy
  enforcement waits until data and writers are ready

The artifact matters too. If CI tests source code but production deploys a newly built image, evidence is weaker. A stronger path builds once, tests the artifact or equivalent output, and promotes the same artifact through environments with different runtime configuration.

Rollout Gates and Blast Radius

Rollout strategy controls how much production traffic sees the new release before the team trusts it broadly.

Common shapes:

The names are less important than the decision.

For checkout idempotency, a useful gate watches:

checkout success rate
validation_failed rate
duplicate idempotency conflict rate
payment authorization error rate
database error rate
worker failure rate
latency percentiles
container restart count

If those signals move badly, the rollout should stop.

A gate is not a ritual. It is a decision point with an owner, a threshold, and a next action.

Bad gate:

Someone clicked approve because the button was there.

Better gate:

Continue canary only if checkout success stays above threshold,
database errors do not increase,
and worker failures remain normal for 20 minutes.

Rollback Is Not Always Undo

Rollback means returning service behavior to a safer state.

It does not always mean reversing every change.

Code rollback may be straightforward:

redeploy previous image

Data rollback is harder.

If a migration deleted a column, rewrote data destructively, added an incompatible constraint, or created values old code cannot understand, redeploying old code may not work.

For the idempotency release, ask:

If new app code fails:
  can old code still run with the expanded schema?

If migration succeeds but rollout fails:
  can the nullable column remain in place?

If new code wrote idempotency_key:
  does old code safely ignore that column?

If uniqueness enforcement causes errors:
  can enforcement be paused without losing data?

Sometimes the safer path is roll forward: ship a small fix that works with the new schema instead of trying to reverse data. The release plan should say which recovery path is expected before the incident starts.

Release Review Table

Before a backend release, write a small table that connects risk to evidence and action.

Risk                         Evidence to check                  Action if bad
migration breaks old writers migration test from previous schema stop before deploy
new image cannot start        artifact startup check             do not promote image
checkout starts failing       checkout success and error rate     pause rollout
workers fail old messages     worker failure and retry metrics    pause or roll forward
constraint causes db errors   database error and lock signals     pause enforcement
rollback may not work         compatibility review                choose roll-forward plan

This table is useful because it prevents vague confidence.

Instead of saying:

The release looks fine.

the team can say:

The artifact starts.
The migration works from the previous schema.
Old writers still work after expand.
The canary has normal checkout success.
If enforcement fails, we know how to pause it.

That does not make the release risk-free. It makes the release inspectable. The next lesson will give names to many of the signals in this table: logs, metrics, traces, release metadata, and alerts.

A release check also needs an owner. If checkout success drops, someone or some automation must know whether to pause, roll back, roll forward, or keep watching. A gate without an owner is only a dashboard with a dramatic button.

Trade-offs and Limits

The main trade-off is release speed versus blast-radius control.

Fast releases reduce batch size. Smaller changes are easier to reason about, and fixes can reach users quickly.

Controls reduce risk. Migration checks, staged rollout, manual approval, and release gates can catch problems before every user is affected.

Controls also cost time and attention. Too many gates slow harmless changes and encourage bypasses. Too few gates let risky changes affect everyone.

The practical goal is not maximum ceremony. It is matching release control to risk:

README typo -> light checks
checkout schema migration -> migration test, staged rollout, focused signals
security fix -> fast path with strong verification and monitoring

CI/CD does not remove judgment. Automation can build, test, deploy, pause, and surface signals. Engineers still design compatible states and choose meaningful thresholds.

You can see the boundary is weak when:

Practice

Review this release plan:

Change:
  Add orders.idempotency_key as NOT NULL.
  Deploy new app code that writes idempotency_key.
  Rolling deploy across 10 instances.

Current reality:
  Old app instances will run during rollout.
  Background worker can still create orders.
  Existing orders do not have idempotency_key.

A good answer should mention:

Model answer:

Do not add NOT NULL immediately. Expand first by adding a nullable idempotency_key, then deploy compatible API and worker code. Use a migration test that starts from the previous schema with existing orders. Watch checkout success rate, database errors, duplicate conflicts, and worker failures during rollout. Only enforce stricter constraints after old writers are gone and existing data has a safe plan. Rolling back to old code after the nullable expand step is usually acceptable if old code ignores the extra column.

Connections

Lesson 013 gave us a deployable artifact and the need to test the runtime package. This lesson adds the release path around that artifact: migrations, gates, rollout, and recovery.

The next lesson shows how observability supplies the signals that release gates and incident response depend on.

Resources

Key Takeaways

PREVIOUS Containers, Local Development, and Deployment Artifacts NEXT Observability Basics for Backend Services

← Back to Backend Development Foundations

← Back to Architecture And Platforms

← Back to Learning Hub