Day 157: CI/CD Pipelines - GitOps & Automated Deployments

CI/CD matters because software delivery only becomes reliable when a code change moves through a repeatable system that builds, tests, packages, and deploys with explicit gates instead of tribal procedure.

Today's "Aha!" Moment

At this point in the month the team already has cloud infrastructure, orchestration, pods, and maybe a service mesh. The next question is not "can we deploy?" but "can we deploy repeatedly without turning every release into a bespoke operation?"

A good CI/CD pipeline is not just automation for automation's sake. It is a system that reduces uncertainty in stages. Source code becomes a built artifact. The artifact is tested, scanned, and tagged. A deployment decision is then made against something concrete, not against a branch name and a hope. GitOps pushes the idea one step further: even the desired deployed state lives in version control, and controllers reconcile the cluster toward that state.

That is the aha. Delivery is a control system over change.

Once you see it this way, the pieces line up. CI is about validating and packaging change. CD is about moving approved change through environments safely. GitOps is about making deployment intent explicit and reconciled rather than pushed manually from a privileged machine.

Why This Matters

Suppose the warehouse team needs to ship a new image-processing model and a small API change in the same week. Without a disciplined delivery system, several bad patterns appear immediately:

• different engineers deploy in different ways
• the artifact running in production is not clearly tied to a commit
• environment differences appear because changes are applied manually
• rollback becomes guesswork instead of a known path

CI/CD exists to make those failure modes less likely. The pipeline is the mechanism that answers operational questions such as:

• what exactly are we deploying?
• what evidence says it is safe enough to continue?
• who or what is allowed to move this change forward?
• how does the running environment learn the new desired state?

This matters because modern cloud platforms make deployment faster, but speed without delivery discipline just turns mistakes into faster mistakes. A good pipeline increases delivery speed by making change legible, auditable, and reversible.

Learning Objectives

By the end of this session, you will be able to:

1. Explain what CI, CD, and GitOps each solve - Distinguish validation, release movement, and desired-state reconciliation.
2. Describe the core delivery flow - Follow change from commit to artifact to deployment state.
3. Reason about delivery trade-offs - Evaluate speed, safety, auditability, and blast radius when designing pipeline systems.

Core Concepts Explained

Concept 1: CI Turns Source Code Into a Verified Artifact

Continuous Integration is easiest to understand as a machine that answers one question: "What do we know about this change right now?"

A typical CI flow looks like this:

commit / merge request
   -> build
   -> unit/integration tests
   -> lint / static analysis
   -> security or dependency checks
   -> artifact image/package produced

The important output is not only green checks. It is a versioned artifact tied to a specific source revision. That artifact becomes the thing that later environments promote.

This matters because teams often accidentally deploy "whatever is on main right now" rather than a known built output. CI reduces that ambiguity. It turns code into something more concrete: an artifact with evidence attached.

Concept 2: CD Is About Moving a Known Artifact Through Environments Safely

Continuous Delivery or Continuous Deployment starts after artifact creation. The key question now becomes: how does the change move toward production?

The flow often looks like this:

verified artifact
   -> environment config selection
   -> deploy to staging
   -> smoke / integration / policy checks
   -> approval or automated promotion
   -> production rollout

The mechanism matters because rollout is rarely all-or-nothing in serious systems. Teams may use:

• staged environments
• manual approval gates
• automated canaries
• progressive delivery
• rollback or rollback-like redeploy paths

This is where delivery speed and delivery safety negotiate with each other. A faster pipeline is not automatically better if it makes blast radius harder to control. A slower pipeline is not automatically safer if it adds human inconsistency and hidden manual steps.

Concept 3: GitOps Makes Desired Deployment State Explicit and Reconciled

GitOps adds a very specific idea: the desired runtime state of the system should be stored in Git, and a controller should reconcile the live environment toward that declared state.

That changes the model from:

CI system pushes directly into cluster

to something more like:

change committed to desired-state repo
   -> controller notices new desired state
   -> cluster reconciles toward it
   -> drift becomes visible

This buys a few important properties:

• deployment intent is auditable in version control
• rollback can often be expressed as another Git change
• cluster state is less dependent on one privileged deployment runner
• live drift is easier to reason about because desired state has a durable home

But GitOps is not free either. It adds another reconciliation layer and requires careful repository design, promotion flow, secret handling, and ownership boundaries. It is worth it when teams want clearer deployment auditability and stronger alignment between infrastructure, application config, and cluster state.

Troubleshooting

Issue: The team says it has CI/CD, but production deploys still depend on manual shell sessions.

Why it happens / is confusing: Build automation exists, so people assume delivery is solved.

Clarification / Fix: Separate CI from CD. Ask whether there is a repeatable path from verified artifact to declared deployment state, not only a build pipeline.

Issue: A production issue appears, but nobody can say which exact build is running.

Why it happens / is confusing: Artifacts were not versioned clearly or environments deploy from mutable references such as floating branches or tags.

Clarification / Fix: Promote immutable artifacts tied to commits and surface that identity in deployment metadata and observability.

Issue: GitOps is introduced, but teams still edit the cluster directly during routine work.

Why it happens / is confusing: Emergency habits and old workflows remain stronger than the new desired-state model.

Clarification / Fix: Decide what the authoritative path is. If Git is the source of truth, reconcile direct cluster changes back into Git or stop using them except for true emergencies.

Advanced Connections

Connection 1: CI/CD ↔ Kubernetes Reconciliation

The parallel: GitOps fits naturally with Kubernetes because both are built around declared desired state and convergence toward it.

Real-world case: A commit to deployment manifests can become the trigger for cluster reconciliation rather than a direct imperative deploy command.

Connection 2: CI/CD ↔ Reliability Engineering

The parallel: Delivery systems are part of reliability because they control change rate, rollback path, evidence quality, and blast radius.

Real-world case: Canary rollouts, promotion gates, and deployment freeze decisions all live at the boundary between software delivery and SRE practice.

Resources

Optional Deepening Resources

• [SITE] Continuous Delivery
  • Link: https://continuousdelivery.com/
  • Focus: Use it for primary-source material on deployment pipelines, release risk, and delivery design.
• [BOOK] Continuous Delivery
  • Link: https://martinfowler.com/books/continuousDelivery.html
  • Focus: Anchor the lesson in the classic framing of build pipelines, deployment automation, and release discipline.
• [DOCS] Argo CD Documentation
  • Link: https://argo-cd.readthedocs.io/en/stable/
  • Focus: See a concrete GitOps controller model for Kubernetes reconciliation.
• [DOCS] Flux Documentation
  • Link: https://fluxcd.io/flux/
  • Focus: Compare another GitOps approach and how desired-state delivery is reconciled into clusters.

Key Insights

1. CI reduces uncertainty by producing validated artifacts - It turns source changes into something testable, traceable, and promotable.
2. CD is a release-control system, not just a deploy script - Its job is to move known artifacts through environments with controlled risk.
3. GitOps makes deployment intent declarative and auditable - Desired runtime state lives in version control and is reconciled, not pushed ad hoc.

Knowledge Check (Test Questions)

1. What is the main purpose of CI in a serious delivery system?

   • A) To deploy directly to production after every commit no matter what.
   • B) To turn source changes into validated, versioned artifacts with evidence attached.
   • C) To remove the need for testing.

2. What is the key GitOps idea?

   • A) Developers SSH into production less often.
   • B) Desired deployment state lives in Git and a controller reconciles the live system toward that state.
   • C) All infrastructure must be written in YAML.

3. Why can a team have "automation" and still have weak CD?

   • A) Because build automation alone does not guarantee repeatable promotion, artifact traceability, or safe rollout into production.
   • B) Because pipelines never help reliability.
   • C) Because CD only matters for frontend applications.

Answers

1. B: CI's job is to create a trusted artifact pipeline, not merely to run a few commands after commits.

2. B: GitOps is fundamentally about desired state in version control plus reconciliation into the live environment.

3. A: Automation can exist at the build stage while delivery into real environments remains ambiguous, manual, or hard to audit.

← Back to Learning