Process Lifecycles and Service Lifecycles

By the end of this lesson, you will be able to...

• Trace a worker from creation through startup, readiness, work, draining, failure, and restart.

• Separate evidence that a process exists from evidence that it can safely receive useful work.

• Choose the next operational observation when a restart policy is recovering capacity or amplifying a persistent fault.

Idea in one sentence: Lifecycle thinking turns “up or down” into visible states, so routing, restarting, and stopping work can follow evidence rather than a binary label.

Core Insight

Imagine a background worker that reads jobs from a queue. A deploy starts a new version. The container starts, the process appears in the process table, logs say "server started," and the platform marks the instance as running. A few seconds later the worker crashes because a credential is missing. The supervisor restarts it. It crashes again. Meanwhile the queue grows and a dashboard still shows that new instances are being created.

If you describe that system as simply "up" or "down," most of the useful information disappears. The worker was created. It started. It never became ready for useful work. It failed. It entered a restart loop. The old instances may or may not have drained their in-flight jobs. Those are different states, and each state asks for a different operational response.

Operating systems already teach this shape. A process is not just a program that is "running." It can be created, ready to run, running on a CPU, blocked on I/O, stopped by a signal, exiting, or waiting to be reaped by its parent. Service platforms add more visible policy around the same idea: scheduling, startup probes, readiness, traffic routing, termination grace periods, and replacement.

The useful mental model is not that a local process and a distributed service instance are identical. They are not. A Linux process state says something about execution on a host; a service lifecycle adds admission, routing, replacement, and shutdown policy. Both are supervised execution units, and their states must agree enough for the system to make a safe decision. Once you see that distinction, failures become easier to locate: is the unit unable to start, unable to become ready, blocked while doing work, failing after it receives traffic, or unable to stop cleanly?

From Binary Status to State

A binary status compresses too much:

worker: running

That line hides the operational question you actually care about: can this worker safely accept and complete jobs now? A lifecycle view keeps the important transitions visible:

created
-> starting
-> ready
-> running
-> draining
-> terminated

starting/running -> failed -> restart/backoff -> starting
running -> blocked_on_dependency -> running

The exact state names vary by operating system, init system, runtime, or orchestrator, but the diagnostic value is stable. Each state tells you what the unit can do and what evidence you should expect. created means something exists but may not have initialized. starting means setup is in progress. ready means useful work can be admitted. running means work is executing. blocked means the unit is alive but waiting on some resource or dependency. draining means no new work should be admitted while old work finishes. terminated means execution has ended.

The process-level analogy matters because it prevents magical thinking. A service instance in a cluster is still a supervised execution unit. It uses processes, file descriptors, memory, signals, sockets, and exit codes. The service layer adds network routing and replacement policy, but it does not remove the host-level lifecycle underneath.

The trade-off is observability surface. More explicit states give better control and safer automation, but only if the signals are maintained. A fake readiness check is worse than no readiness check because it gives automation confidence at the wrong moment.

Three Questions: Alive, Ready, or Useful?

The most common lifecycle bug is confusing existence with usefulness. A process can be alive and still not safe to use. A service can respond to a shallow ping while its database pool is empty. A worker can be running but blocked because the queue is unreachable.

Separate the questions:

Consider the worker at startup:

process exists: yes
configuration loaded: yes
queue connection: no
database migration check: unknown
ready for jobs: no

A liveness-only system might route jobs because the process exists. A lifecycle-aware system refuses new work until readiness is true. This is not pedantry. It prevents a half-initialized unit from converting startup delay into user-visible failure.

The same distinction matters during shutdown. A worker that receives a termination signal should usually stop accepting new jobs before it exits. It may need to finish or return the job it already owns. If the platform kills it immediately, the system may duplicate work, lose work, or leave locks and leases in awkward states.

Check: An instance is alive but not ready for 20 seconds. Should the supervisor restart it immediately?

Think first, then reveal.

Answer: Not from that state alone. During normal startup, “alive but not ready” can be expected. During a persistent dependency failure, repeated restart may not help. Inspect the configured startup window, dependency errors, and whether any readiness or progress evidence is improving before deciding.

Worked Path: A Deployment With a Bad Credential

Trace a rollout for a worker service. The platform wants to replace version A with version B.

initial state:
  old worker A: running, ready, processing jobs
  new worker B: not created
  queue backlog: stable

Step 1: the platform creates B.

B state: created
evidence: container allocated, process not useful yet
decision: do not route jobs to B

Step 2: B starts its process and loads configuration.

B state: starting
evidence: process exists, logs are appearing
decision: liveness may pass; readiness should still fail

Step 3: B tries to connect to the queue and database. The queue credential is wrong.

B process state: it may still be running, or it may exit after the failed initialization
B service state: not ready in either case
evidence: authentication errors, no completed jobs, readiness remains false
decision: do not route work to B and do not drain all of A yet

Step 4: assume B exits after the failed initialization. The supervisor applies restart policy.

B state: failed -> restart_backoff -> starting
evidence: repeated non-zero exits, increasing restart count
decision: classify transient vs persistent; alert if backoff loop continues

Step 5: A receives a termination signal too early because the rollout policy trusted creation rather than readiness.

A state: draining or terminated
evidence: fewer ready workers, backlog rising
decision: pause rollout; keep old capacity if possible

The values in this trace are illustrative. The lifecycle trace shows the incident more clearly than “new deploy broken.” The failure is not only the bad credential. It is also a policy failure if the rollout removed ready capacity before replacement capacity proved useful. The correct response is different at each state: fix the secret, keep B unready, preserve A, watch restart backoff, and use backlog growth as evidence that useful capacity is falling.

So far: lifecycle states connect host-level facts, service-level routing, and operational policy. A process table can tell you that something exists. Readiness and progress signals tell you whether the system should rely on it.

Restart Policy Is a Design Choice

Automatic restart is valuable because many failures are transient. A process may crash because a dependency briefly timed out, a file descriptor was exhausted and later released, or the host replaced a failed unit. In those cases, restart with backoff can restore capacity without a human doing manual cleanup.

But restart is not repair. If every new process reads the same bad credential, every restart repeats the same failure. If a dependency is down for ten minutes, aggressive restart can burn CPU, flood logs, and hide the useful signal inside noise. If startup performs expensive migrations or cache warmups, restart loops can harm the rest of the system.

Design restart policy with three pieces:

exit or failure signal
-> restart decision
-> backoff and visibility
-> readiness gate before useful work

The backoff matters because it limits how quickly a persistent fault can amplify. Visibility matters because a restart loop is a state, not background cleanup. Readiness matters because a restarted unit should earn traffic only after it has proved it can do useful work. In Kubernetes specifically, failed readiness removes a Pod from matching Service endpoints, whereas repeated liveness failure can restart the container; the exact machinery differs outside Kubernetes, but the operational distinction is general.

There is a trade-off between fast recovery and controlled diagnosis. Restart too slowly and transient failures last longer than necessary. Restart too aggressively and persistent faults become noisy, expensive, and harder to reason about. A good policy does not try to guess perfectly; it exposes the evidence needed to tell the two cases apart.

Failure Modes and Design Checks

Failure: treating running as ready. This is tempting because process existence is easy to observe. The fix is to make readiness depend on the minimum useful path: configuration loaded, required dependencies reachable, and the unit able to accept work without immediately failing.

Failure: draining without ownership rules. During shutdown, the system needs to know what happens to in-flight work. Does the worker finish it, return it to the queue, extend a lease, or let another worker retry? Without a rule, termination turns into duplicate or lost work.

Failure: restart loops mistaken for healing. A rising restart count with no readiness and no completed work is not resilience. It is a persistent state that deserves alerting and a rollback or configuration fix.

Failure: no progress signal. Liveness and readiness are not enough for long-running workers. A unit can be alive and ready while no offsets advance, no jobs complete, or latency grows. Add progress evidence for the work the service actually owns.

Try this design review:

An API instance reports healthy if /health returns 200.
The endpoint only checks that the HTTP server thread is alive.
The load balancer routes traffic as soon as /health passes.
Database connection setup happens lazily on the first user request.

What is the lifecycle bug? The health signal is being used as readiness even though it only proves liveness. A stronger design separates /live from /ready, keeps readiness false until the database path can be used, and watches request success or dependency errors after traffic starts. If startup is intentionally allowed without the database, the routing policy should make that degraded mode explicit rather than accidentally discovering it on the first real request.

Practice: Review a Degraded Worker Policy

A payment worker uses these rules:

liveness: HTTP server answers /health
readiness: queue connection authenticated and the worker can reserve a test job
shutdown: readiness becomes false, then the worker has 30 seconds to finish its leased job
restart: exponential backoff after non-zero exit

The database is down for eight minutes. The worker stays alive, readiness is false, no new jobs are assigned, and the existing leased job reaches its deadline.

What should the supervisor, router, and worker do next? What evidence distinguishes this from a code deadlock?

A good answer should mention:

• Keep the worker out of new work while readiness is false; liveness alone is not permission to route jobs.
• Apply the job's explicit ownership rule at the deadline: finish only if it can be done safely, otherwise return or retry it according to the queue's semantics.
• Avoid treating a reachable HTTP endpoint as proof of useful progress; inspect dependency errors, completed jobs, lease expiry, and restart count.
• A deadlock may justify a liveness-triggered restart under a deliberate policy; a confirmed external database outage usually calls for controlled waiting, backoff, and capacity protection rather than a restart storm.

Resources

• [BOOK] Operating Systems: Three Easy Pieces — Focus: Review process states, scheduling, and the OS view of blocked versus runnable work.
• [ARTICLE] systemd.service — Focus: Study how local service supervision models startup, stop behavior, restart policy, and notification.
• [ARTICLE] Kubernetes Liveness, Readiness, and Startup Probes — Focus: Compare container restart, traffic admission, startup, and termination behavior.
• [ARTICLE] Linux signal(7) Manual Page — Focus: Connect graceful shutdown and termination handling back to host-level process mechanics.

Key Takeaways

• A process state and a service-admission state answer different questions; neither replaces the other.
• Liveness, readiness, progress, draining, and termination each support a different operational decision.
• Restart policy improves resilience only when it is paired with backoff, visibility, and readiness gates.
• A failed readiness check should withhold work; it does not by itself establish that the process must be restarted.
• Lifecycle traces turn deployment and crash-loop incidents into inspectable state transitions with concrete evidence.