systemd and journald

systemd starts services at boot, restarts them when they crash, captures their output, and orders them against each other. You describe a service in a unit file. Its built-in journal, read with journalctl, keeps each service's stdout and stderr, indexed by unit, time and priority. Together they are how a single VM runs a service without a container platform.

Walking through it

Here is a unit file for a small API. It is a sketch you could adapt. The path and user are examples.

; /etc/systemd/system/orders-api.service
[Unit]
Description=Orders API
After=network-online.target
Wants=network-online.target

[Service]
User=orders
EnvironmentFile=/etc/orders-api/env
ExecStart=/opt/orders-api/bin/server --port 8080
Restart=on-failure
RestartSec=2
TimeoutStopSec=30
LimitNOFILE=65536
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/orders-api

[Install]
WantedBy=multi-user.target

Three lines carry most of the value:

• Restart=on-failure brings the process back after a crash, RestartSec=2 waits two seconds between tries.
• TimeoutStopSec=30 is the grace period before SIGKILL, the same idea as in graceful shutdown.
• LimitNOFILE=65536 raises the open file limit. The default soft limit of 1,024 is far too low for a busy server.

The hardening lines run the service as an unprivileged user, stop it gaining privileges, and make most of the filesystem read-only.

Daily commands:

sudo systemctl daemon-reload            # after editing a unit file
sudo systemctl enable --now orders-api  # start now and at boot
systemctl status orders-api             # state, PID, last log lines
systemctl list-units --failed           # what is broken on this box?

Reading logs:

journalctl -u orders-api -f                  # follow live
journalctl -u orders-api --since "10 min ago"
journalctl -u orders-api -p err -b           # errors since this boot
journalctl -k | grep -i "out of memory"      # kernel OOM kills

Logs for every host should ship to a central system, so you rarely need journalctl in normal operation. It becomes essential when the shipper itself is broken.

Common mistakes

• Editing a unit file and forgetting daemon-reload. systemd keeps using the old settings until you reload.
• Running the service as root to avoid permission problems. Fix the permission, or grant one capability. See permissions and capabilities.
• Trusting status alone. Its last few lines are short. Use journalctl -u for the full story.

Check yourself

1. You changed the unit file and restarted the service, but the old limit is still active. What did you skip? systemctl daemon-reload, which makes systemd read the edited file.
2. Which command shows only errors for orders-api since the last boot? journalctl -u orders-api -p err -b.

Key takeaways

• A unit file defines the command, the user, the restart policy and the stop timeout.
• Reload after every edit, then restart the service.
• journalctl -u <unit> is the first place to look when a service fails, filtered by time, priority or boot.

Further learning

For the systemd walkthrough, this video shows systemctl and journalctl on a real machine.

Next, see how to tell whether a connectivity problem is the network or the service, in debugging connectivity.