System Services and systemd¶
On nearly every modern Linux distribution, systemd is the first process that starts after the kernel boots. It manages services, handles logging, mounts filesystems, and coordinates the entire startup sequence. If you administer a Linux server, you interact with systemd daily - starting services, reading logs, and troubleshooting boot problems all go through it.
What an Init System Does¶
When the Linux kernel finishes its own initialization, it starts a single userspace process: the init system (PID 1). This process is responsible for bringing up everything else - mounting filesystems, starting network services, launching login prompts.
The original Unix approach was SysVinit, which ran shell scripts in sequence from /etc/init.d/. It worked, but startup was slow because services launched one at a time, and writing init scripts was tedious and error-prone.
Upstart (Ubuntu, 2006) introduced event-driven service startup, and systemd (Fedora, 2011) took that further with parallel startup, dependency tracking, socket activation, and a unified logging system. By 2015, Debian, Ubuntu, RHEL, Fedora, Arch, and SUSE had all adopted systemd as their default init system.
Managing Services with systemctl¶
systemctl is the primary command for interacting with systemd. Every long-running daemon on the system - web servers, databases, SSH, cron - is managed through it.
Starting and Stopping¶
# Start a service immediately
sudo systemctl start nginx
# Stop a service
sudo systemctl stop nginx
# Restart a service (stop, then start)
sudo systemctl restart nginx
# Reload configuration without restarting (not all services support this)
sudo systemctl reload nginx
# Restart only if the service is already running
sudo systemctl try-restart nginx
The difference between restart and reload matters in production. A restart kills the process and starts a new one, dropping all active connections. A reload signals the running process to re-read its configuration files without interrupting existing connections. Not every service supports reload - if it doesn't, systemd returns an error.
Checking Status¶
# Show detailed status of a service
systemctl status nginx
# Check if a service is running (returns 0 or non-zero exit code)
systemctl is-active nginx
# Check if a service is enabled at boot
systemctl is-enabled nginx
# Check if a service has failed
systemctl is-failed nginx
The status output includes the service state, PID, memory usage, and the most recent log lines - it's usually the first thing you check when something goes wrong. For practical examples of managing a production service like Nginx, see the Nginx Configuration guide.
Enabling and Disabling¶
Starting a service makes it run now. Enabling it makes it start automatically at boot:
# Enable a service to start at boot
sudo systemctl enable nginx
# Enable AND start it immediately
sudo systemctl enable --now nginx
# Disable a service from starting at boot (does not stop it now)
sudo systemctl disable nginx
# Disable AND stop it immediately
sudo systemctl disable --now nginx
Masking¶
Masking goes further than disabling - it prevents a service from being started at all, even manually:
# Mask a service (symlinks it to /dev/null)
sudo systemctl mask nginx
# Unmask a service
sudo systemctl unmask nginx
This is useful when you want to guarantee a service never runs - for example, masking a firewall service during migration to prevent accidental lockouts.
Listing Services¶
# List all active services
systemctl list-units --type=service
# List all services (including inactive)
systemctl list-units --type=service --all
# List enabled/disabled status of all services
systemctl list-unit-files --type=service
# Show only failed services
systemctl --failed
Understanding Unit Files¶
Every service, socket, timer, and mount point that systemd manages is defined in a unit file. Unit files are plain-text INI-style configuration files.
Where Unit Files Live¶
| Location | Purpose |
|---|---|
/usr/lib/systemd/system/ |
Shipped by packages (distribution defaults) |
/etc/systemd/system/ |
Administrator overrides (highest priority) |
/run/systemd/system/ |
Runtime units (transient, lost on reboot) |
Files in /etc/systemd/system/ take priority over files with the same name in /usr/lib/systemd/system/. This means you can override a package-provided unit without modifying the original.
Anatomy of a Service Unit¶
Here's a typical service unit file:
[Unit]
Description=My Application Server
Documentation=https://example.com/docs
After=network.target postgresql.service
Wants=postgresql.service
[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
The [Unit] Section¶
Defines metadata and ordering:
| Directive | Purpose |
|---|---|
Description= |
Human-readable name shown in systemctl status |
Documentation= |
Links to documentation (man pages, URLs) |
After= |
Start this unit after the listed units (ordering only) |
Before= |
Start this unit before the listed units |
Wants= |
Soft dependency - try to start the listed units, but don't fail if they can't |
Requires= |
Hard dependency - if the listed unit fails, this unit fails too |
After vs Requires
After= controls order. Requires= controls dependency. They're independent. If you need a service to start after PostgreSQL AND fail if PostgreSQL isn't running, you need both: After=postgresql.service and Requires=postgresql.service.
The [Service] Section¶
Defines how the service runs:
| Directive | Purpose |
|---|---|
Type=simple |
Default. systemd considers the service started as soon as ExecStart runs |
Type=forking |
For daemons that fork into the background. systemd waits for the parent to exit |
Type=oneshot |
For tasks that run once and exit. systemd waits for completion |
Type=notify |
The service signals readiness via sd_notify(). Most reliable for complex services |
ExecStart= |
The command to run |
ExecReload= |
The command to reload configuration |
Restart= |
When to restart: no, on-failure, on-abnormal, on-abort, always |
RestartSec= |
Delay before restarting |
User= / Group= |
Run the process as this user/group |
WorkingDirectory= |
Set the working directory before starting |
Environment= |
Set environment variables: Environment=NODE_ENV=production |
EnvironmentFile= |
Read environment variables from a file |
The [Install] Section¶
Defines what happens when you run systemctl enable:
| Directive | Purpose |
|---|---|
WantedBy=multi-user.target |
Enable this service in the standard multi-user boot (most services) |
WantedBy=graphical.target |
Enable for graphical desktop environments |
RequiredBy= |
Like WantedBy but creates a hard dependency |
Editing Unit Files Safely¶
Never edit files in /usr/lib/systemd/system/ directly - package updates will overwrite your changes. Instead, use drop-in overrides:
This opens an editor for /etc/systemd/system/nginx.service.d/override.conf. Directives here merge with (and override) the original unit file.
To replace the entire unit file instead of just adding overrides:
After editing any unit files, reload the systemd daemon:
Writing a Custom Service¶
Here's a practical example: you have a Python web application that should start at boot, restart on failure, and run as a dedicated user.
Create the unit file at /etc/systemd/system/myapp.service:
[Unit]
Description=My Python Web Application
After=network.target
[Service]
Type=simple
User=www-data
Group=www-data
WorkingDirectory=/opt/myapp
Environment=PYTHONUNBUFFERED=1
ExecStart=/opt/myapp/venv/bin/python -m uvicorn app:main --host 0.0.0.0 --port 8000
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
Then enable and start it:
# Reload systemd to pick up the new unit file
sudo systemctl daemon-reload
# Enable the service to start at boot and start it now
sudo systemctl enable --now myapp
# Verify it's running
systemctl status myapp
ExecStart must use absolute paths
systemd does not use the shell's PATH variable. Every binary in ExecStart must be a full path (/usr/bin/python, not just python). If you need shell features like pipes or variable expansion, use ExecStart=/bin/bash -c 'your command here'.
Viewing Logs with journalctl¶
systemd includes its own logging system, the journal, managed by systemd-journald. Every service's stdout and stderr, plus kernel messages and syslog output, flow into the journal.
journalctl is the tool for reading it.
Filtering by Unit¶
# Show all logs for a specific service
journalctl -u nginx
# Follow logs in real time (like tail -f)
journalctl -u nginx -f
# Show logs from multiple units
journalctl -u nginx -u php-fpm
Filtering by Time¶
# Logs since a specific timestamp
journalctl --since "2026-03-25 14:00:00"
# Logs from the last hour
journalctl --since "1 hour ago"
# Logs between two times
journalctl --since "2026-03-25 14:00" --until "2026-03-25 15:00"
# Logs from the current boot only
journalctl -b
# Logs from the previous boot
journalctl -b -1
Filtering by Priority¶
Journal entries have syslog priority levels:
| Level | Name | Meaning |
|---|---|---|
| 0 | emerg | System is unusable |
| 1 | alert | Immediate action required |
| 2 | crit | Critical conditions |
| 3 | err | Error conditions |
| 4 | warning | Warning conditions |
| 5 | notice | Normal but significant |
| 6 | info | Informational |
| 7 | debug | Debug-level messages |
# Show only errors and above
journalctl -p err
# Show only errors from a specific service
journalctl -u nginx -p err
# Show warnings and above since boot
journalctl -b -p warning
Output Formatting¶
# Show full output without paging
journalctl --no-pager
# JSON output (useful for piping to jq)
journalctl -u nginx -o json-pretty --since "1 hour ago"
# Show only the message field (no timestamps or metadata)
journalctl -u nginx -o cat
Persistent Journal Storage¶
By default on some distributions, the journal only keeps logs in memory (/run/log/journal/), which means logs are lost on reboot. To make them persistent:
# Create the persistent journal directory
sudo mkdir -p /var/log/journal
# Restart journald to pick up the change
sudo systemctl restart systemd-journald
You can also control journal size in /etc/systemd/journald.conf:
After editing, restart journald:
Targets¶
Targets are systemd's replacement for SysVinit runlevels. A target is a group of units that represents a system state.
| Target | SysVinit Equivalent | Purpose |
|---|---|---|
poweroff.target |
Runlevel 0 | System shutdown |
rescue.target |
Runlevel 1 | Single-user rescue mode |
multi-user.target |
Runlevel 3 | Multi-user, no GUI (servers) |
graphical.target |
Runlevel 5 | Multi-user with GUI (desktops) |
reboot.target |
Runlevel 6 | System reboot |
emergency.target |
- | Minimal emergency shell |
# Check the default target
systemctl get-default
# Set the default target (takes effect on next boot)
sudo systemctl set-default multi-user.target
# Switch to a target immediately
sudo systemctl isolate rescue.target
# List all available targets
systemctl list-units --type=target
Isolating targets
systemctl isolate stops all services that aren't part of the target. Running systemctl isolate rescue.target on a remote server kills your SSH session. Only use it with console access or in rescue.target/emergency.target scenarios where you're already at the physical console.
Timers¶
systemd timers are a modern alternative to cron jobs. They offer better logging (through journalctl), dependency management, and randomized delays to prevent thundering herd problems.
A timer requires two unit files: a .timer file and a matching .service file.
Example: Daily Cleanup Task¶
Create /etc/systemd/system/cleanup.service:
[Unit]
Description=Daily cleanup of temporary files
[Service]
Type=oneshot
ExecStart=/usr/local/bin/cleanup.sh
Create /etc/systemd/system/cleanup.timer:
[Unit]
Description=Run cleanup daily at 3am
[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=900
Persistent=true
[Install]
WantedBy=timers.target
Enable the timer (not the service):
Timer Types¶
| Directive | Type | Example |
|---|---|---|
OnCalendar= |
Realtime (calendar-based) | OnCalendar=Mon *-*-* 09:00:00 |
OnBootSec= |
Monotonic (relative to boot) | OnBootSec=15min |
OnUnitActiveSec= |
Monotonic (relative to last activation) | OnUnitActiveSec=1h |
OnStartupSec= |
Monotonic (relative to systemd start) | OnStartupSec=5min |
OnCalendar Syntax¶
The calendar format is DayOfWeek Year-Month-Day Hour:Minute:Second:
# Every day at midnight
OnCalendar=*-*-* 00:00:00
# Shorthand
OnCalendar=daily
# Every Monday at 9am
OnCalendar=Mon *-*-* 09:00:00
# Every 15 minutes
OnCalendar=*:0/15
# First day of every month at noon
OnCalendar=*-*-01 12:00:00
# Validate a calendar expression
systemd-analyze calendar "Mon *-*-* 09:00:00"
Persistent=true means if the system was off when the timer would have fired, it runs immediately on next boot. RandomizedDelaySec= adds jitter so multiple machines with the same timer don't all fire at the exact same second.
Listing Timers¶
# List all active timers with their next trigger time
systemctl list-timers
# Include inactive timers
systemctl list-timers --all
Debugging and Analysis¶
Boot Performance¶
# Show how long each service took to start
systemd-analyze blame
# Show the critical chain (the longest dependency path)
systemd-analyze critical-chain
# Show total boot time breakdown
systemd-analyze
# Generate an SVG chart of the boot process
systemd-analyze plot > boot-chart.svg
systemd-analyze blame lists services sorted by startup time. If a service is slow, check whether it's blocking other services with systemd-analyze critical-chain unitname.
Dependency Trees¶
# Show what a service depends on
systemctl list-dependencies nginx
# Show reverse dependencies (what depends on a service)
systemctl list-dependencies nginx --reverse
# Show the full dependency tree for the default target
systemctl list-dependencies default.target
Troubleshooting Failed Services¶
# List all failed units
systemctl --failed
# Show the status and recent logs of a failed service
systemctl status failed-service
# View the full logs for troubleshooting
journalctl -u failed-service --since "10 minutes ago"
# Reset the failed state after fixing the issue
sudo systemctl reset-failed failed-service
A common pattern: a service fails, you check systemctl status for the error, dig deeper with journalctl -u, fix the config or code, run daemon-reload if you changed the unit file, then start it again.
Further Reading¶
- systemd Documentation - official project page with specifications and design documents
- systemctl man page - complete command reference
- journalctl man page - journal query tool reference
- systemd.service man page - service unit file format
- systemd.timer man page - timer unit file format
- systemd-analyze man page - boot performance and debugging tools
- Arch Wiki: systemd - comprehensive practical reference with examples
Previous: Package Management | Next: User and Group Management | Back to Index