Services, Logs, and Boot

By the end of this lesson, you will be able to...

• explain why a program that works in an interactive shell can fail when systemd starts it;

• trace a service failure from status and unit definition to the first discriminating journal entry;

• make a small service change with an explicit runtime context and a verification step that survives a reboot.

Idea in one sentence: A systemd service is not “your terminal command, but earlier”; it is a process launched from a declared unit with its own user, working directory, environment, dependencies, and logs.

Core Insight

Maya runs a local note-preview server while preparing a presentation:

cd ~/notes
note-preview --config config/preview.toml

The preview opens in her browser. She packages it as note-preview.service so it starts with the workstation. After reboot, the browser cannot connect. A login and the same command make the preview work again.

The tempting explanation is: “systemd failed to run my program.” That is reasonable when a command is truly self-contained: absolute executable path, absolute configuration paths, explicit variables, correct identity, and no missing prerequisites. It fails when the command quietly relies on Maya's current directory, HOME, PATH, exported variables, mounted files, or login session.

The stronger model is:

A service is a process plus a runtime contract. The unit describes what systemd should start, under which identity and context, how it relates to other units, and where the evidence of its attempt appears.

This is why “it works in my shell” is an observation, not a deployment test. The shell and the service manager may legitimately create different process contexts.

What the User Sees and What the System Knows

Maya sees one symptom: http://127.0.0.1:9070 refuses the connection after boot. The system has more specific evidence, but it is spread across two places:

journalctl filters records by systemd unit with --unit= and by boot with --boot; output from systemd units is normally connected to the journal. This makes a unit-and-boot query more useful than scrolling through every message since the machine started. journalctl documentation

The commands are inspection tools. They do not prove that the service's purpose is correct, and they do not replace a backup before changing important configuration. They narrow the next question.

The Runtime Contract

Here is the synthetic unit Maya first installed. It is deliberately small so that the failure is visible.

[Unit]
Description=Preview local notes
After=network.target

[Service]
User=preview
ExecStart=/usr/local/bin/note-preview --config config/preview.toml
Restart=on-failure

[Install]
WantedBy=multi-user.target

The unit does not reproduce her terminal invocation. The terminal supplied a current directory of /home/maya/notes; this unit supplies no WorkingDirectory= and passes a relative configuration path. The service's identity is also preview, not Maya. Those are two different conditions that must be examined, not excuses to run the service as root.

Read a unit as a compact contract:

After= is an ordering statement: it says when a job should run relative to another unit. It does not by itself arrange for that other unit to be pulled in, and it does not prove that a remote service is usable. Conversely, WantedBy= in the install section describes how enablement can attach this unit to a target; it does not repair a bad ExecStart=. Unit dependencies and ordering are separate pieces of the systemd unit model. systemd.unit documentation

Investigation Path: One Failure, One First Cause

Start with the smallest safe question: did the service fail, remain inactive, or never become enabled? systemctl status usually gives the active state, the main process result, and recent related log lines. It is a summary, not the whole explanation.

For this lesson, suppose its output ends with this synthetic result:

Active: failed (Result: exit-code)
Process: 812 ExecStart=/usr/local/bin/note-preview --config config/preview.toml
         (code=exited, status=1/FAILURE)

The result rules out “the browser is the only problem,” but it still leaves several causes. Now query the journal for exactly the failed boot and unit:

journalctl -b -u note-preview.service --no-pager

The relevant synthetic line is:

note-preview[812]: error: cannot open config/preview.toml: No such file or directory

This is the first discriminating signal. A missing configuration path calls for inspecting the declared working directory and arguments, not changing network settings or adding retries.

systemctl cat note-preview.service
systemctl show note-preview.service \
  --property=User --property=Group --property=WorkingDirectory --property=ExecStart

cat includes the main unit and relevant drop-ins, so it protects against debugging a remembered copy while an override is actually in force. show reports manager properties in a machine-friendly form. The unit's execution settings, such as the executable, user, working directory, and environment, are part of the service's declared context. systemd.service documentation

The correction makes the required context explicit:

[Service]
User=preview
WorkingDirectory=/srv/notes
EnvironmentFile=/etc/note-preview.env
ExecStart=/usr/local/bin/note-preview --config /srv/notes/config/preview.toml
Restart=on-failure

This example assumes /srv/notes, its configuration, and the environment file were deliberately prepared for the preview account. That is an assumption to verify with the access trace from the previous lesson; it is not established merely because the unit looks tidy.

After editing a unit file, reload systemd's unit definitions, then perform a controlled test when interrupting the preview is acceptable:

sudo systemctl daemon-reload
sudo systemctl restart note-preview.service
systemctl status note-preview.service
journalctl -u note-preview.service -n 30 --no-pager

The sequence matters. Editing a file changes disk state; daemon-reload tells the manager to reread unit definitions; restarting creates a new process using that definition; status and journal output test the prediction. A green active state is encouraging, but the real acceptance check is that the preview serves one known note under the preview account.

Boot Is Another Test Case

The restart test removes Maya's interactive shell from the equation. A reboot adds more pressure: filesystems, network configuration, user sessions, and other units are becoming available over time.

Before declaring the service boot-ready, verify the link that should start it:

systemctl is-enabled note-preview.service
systemctl list-dependencies --reverse note-preview.service

If it must wait for a local mount, a secret supplied by another mechanism, or a network resource, name that prerequisite precisely. “After the network” is too vague when the actual requirement is “this directory is mounted” or “this remote endpoint is reachable.” A service that needs a network peer also needs application-level handling for the peer remaining unavailable after systemd has started the process.

After the next planned reboot, use the same focused evidence:

systemctl status note-preview.service
journalctl -b -u note-preview.service --no-pager

The first command answers the current state. The second ties the current boot to the start attempt. If persistent journal storage is not configured, older boot logs may be unavailable; that is a logging-retention boundary, not evidence that no failure occurred.

So far, we have replaced a vague contradiction—“manual works, boot fails”—with a small trace: a unit requests a process, systemd starts it in a declared context, the process emits a specific error, and a targeted correction changes the next start attempt.

Failure Mechanisms That Look Similar

Several failures can produce “the service is down,” but their next checks differ.

The point is not to memorize a table. It is to choose the next observation that separates competing explanations. Restarting repeatedly is sometimes useful after a corrected configuration; before that, it can overwrite a clean causal story with a pile of identical failures.

Trade-offs, Boundaries, and Signals

Service managers provide structure: named processes, explicit identities, dependency relationships, restart policy, and a common journal. That structure costs configuration work and forces you to state assumptions that a shell previously supplied for free.

This approach improves repairability when a process should outlive a terminal or start predictably. It does not make a program correct, make an unavailable network peer available, supply secrets safely, or preserve logs indefinitely. A service can be active while returning wrong results; it can also be failed for a small, recoverable configuration mistake.

The key signals are therefore concrete: the main process exit code, a unit-scoped log entry, an unexpected identity or working directory, disabled enablement, a missing prerequisite, or a restart count that rises without a successful request. Watch those before changing unrelated parts of the machine.

Check Your Understanding

Check: Maya's command works after cd ~/notes, but the unit passes --config config/preview.toml and has no WorkingDirectory=. What is the most likely first hypothesis?

Think first, then reveal.

Answer: The service may resolve the relative configuration path from a different directory than Maya's shell. Inspect the unit and the unit-scoped journal before changing permissions or network settings.

Check: A unit contains After=network.target, but the required remote API is still unavailable when the process starts. What did After= fail to guarantee?

Think first, then reveal.

Answer: It supplied an ordering relation, not proof that a particular remote API is reachable or ready. The service still needs an explicit prerequisite model and application-level handling for an unavailable peer.

Practice: Write a Service Failure Note

Choose one personal service—or use the synthetic preview service. Write a six-line failure note before making a change:

1. user-visible symptom and one known-good request;
2. unit name and whether it is active, failed, or disabled;
3. exact unit-scoped journal line from the relevant boot;
4. declared executable, user, working directory, and needed environment;
5. one hypothesis that the evidence distinguishes from at least one alternative;
6. a reversible change and an acceptance check after restart or a planned reboot.

A good note does not say only “systemd is broken.” It connects a symptom to a unit, an observed signal, a concrete runtime condition, and a check that could prove the hypothesis wrong. If the change affects a login, a network-facing service, or important data, include the recovery path before applying it.

Resources

• [DOC] systemd.service — Focus: inspect service-unit execution and lifecycle settings as a process contract.
• [DOC] systemd.unit — Focus: distinguish requirements, ordering, install-time enablement, and unit relationships.
• [DOC] journalctl — Focus: filter evidence by unit and boot instead of treating the journal as an undifferentiated stream.

Key Takeaways

• A successful shell command is evidence about that shell's context, not proof that a system service has the same context.
• Debug a service from a focused trace: status, actual unit definition, unit-scoped journal, then one explicit runtime condition.
• After= expresses ordering; it does not prove enablement, a mounted path, or a reachable remote dependency.
• Make executable paths, working directory, identity, environment, and recovery checks explicit when they matter to the service.
• Repeated restarts are not diagnosis. The first discriminating log line should guide the next change.