A service that will not start is rarely mysterious once you look in the right places in the right order. The trouble is that under pressure people jump straight to restarting things repeatedly, which destroys evidence and sometimes makes matters worse. This checklist keeps you methodical. It assumes a distribution that uses systemd, which covers most mainstream Linux server distributions today.

Step One: Read the Status Before You Touch Anything

Run systemctl status your-service first. The output tells you the current state (active, failed, activating), the main process ID, the exit code or signal of the last run and the final few log lines. Note the wording of the state. “failed” with an exit code points at the program itself. “activating” that never finishes suggests a start-up timeout or a service type mismatch. “inactive (dead)” may simply mean nobody started it, or that it exited cleanly because it had nothing to do.

Also check systemctl is-enabled, which tells you whether the service will start at boot. A service that works when started by hand but is missing after a reboot is often just not enabled.

Step Two: Pull the Full Logs

The status screen truncates. Use journalctl -u your-service to see the full history, add -b to limit to the current boot and -e to jump to the end. When a failure just happened, journalctl -u your-service --since "10 minutes ago" keeps the output manageable. Read upwards from the failure for the first error, not the last one, because later messages are often consequences.

Many applications also write their own log files, and the journal may only hold their start-up banner. If the journal looks empty, check the application’s own log location and configuration.

Step Three: Validate the Unit File

Run systemctl cat your-service to see the effective unit including any drop-in overrides, then check the common culprits:

• ExecStart path: it must be an absolute path to an executable file with execute permission.
• User and Group: the named account must exist and be able to read the files the program needs.
• WorkingDirectory and environment files: missing directories or an unreadable environment file will stop the service immediately.
• Type: a program that forks into the background but is declared as a simple type, or the reverse, can look like a failure even though it ran.
• After and Requires: a service that needs a database or network mount must be ordered after it.

After editing any unit file, run systemctl daemon-reload, otherwise systemd keeps using the old definition. This forgotten step explains a surprising number of “my fix did nothing” moments. The systemd-analyze verify command can also catch syntax mistakes in a unit file.

Step Four: Reproduce the Command by Hand

Run the same command that ExecStart runs, as the same user. For example, use sudo -u appsvc followed by the command. Errors that were swallowed by systemd, such as missing libraries, bad configuration values or ports already in use, usually appear immediately in your terminal. Differences between your interactive shell and the service environment, especially the PATH and environment variables, are a classic cause of “works for me, fails under systemd”.

Step Five: Check Resources and Dependencies

If the program starts and then dies, look beyond the unit. Is the disk full or out of inodes? Is memory exhausted so the kernel’s out-of-memory killer ended the process? The kernel log (journalctl -k) will mention it. Is another process holding the port? Tools such as ss -ltnp show which process listens where. Are SELinux or AppArmor denying access? Their denial messages appear in the audit or kernel logs, and the fix is a correct policy rather than disabling the security layer.

Failure Pattern Reference

Symptom	Likely area	First thing to check
Fails instantly with exit code	Command or configuration	Run ExecStart manually as the service user
Stuck in “activating”	Timeout or wrong Type	Unit Type and start timeout settings
Restarts in a loop	Crash after start-up	Journal entries before each restart
Works by hand, fails at boot	Ordering or environment	After/Requires lines and environment files
Starts, then killed	Resource limits	Kernel log for out-of-memory messages
“Permission denied”	Ownership or mandatory access control	File permissions and security module logs

A Hypothetical Example

Suppose a web application service fails right after a deployment. The status shows “failed” with a non-zero exit code. The journal’s first error line says a configuration file could not be opened. Running the command by hand as the service user reproduces the same message. A look at the file shows the deployment process created it owned by the deploy account with permissions that deny the service user. Fixing the group ownership solves it in one step. Nobody needed to change the unit file, and nothing needed restarting more than once.

Handle Restart Loops Carefully

A unit with Restart=always can hide a crash by recovering it instantly, while flooding logs. Systemd applies rate limits and will eventually mark the unit failed after repeated rapid restarts. When diagnosing, it can help to stop the service, reset its failed state with systemctl reset-failed, and start it once while watching the journal live with journalctl -fu your-service. Decide deliberately whether automatic restarts are appropriate: they suit stateless workers, but a database that crashes repeatedly deserves human attention.

Frequently Asked Questions

What is the difference between restart and reload? Restart stops and starts the process. Reload asks the running process to reread its configuration, if the service supports it, without dropping connections. Check the application’s documentation to see whether reload is safe.

Why does my service stop when I log out? It may be a user-level process rather than a system service, or it may have been started from your session. Make it a proper unit so systemd manages its lifetime independently.

Should I edit files in /usr/lib/systemd/system directly? Generally no. Package updates can overwrite them. Use systemctl edit your-service to create a drop-in override that survives upgrades.

How do I see why a service started at boot but later failed? Use journalctl -u your-service -b to see all messages from the current boot in order, and look at the first warning rather than the final error.

Is it safe to mask a service? Masking prevents a unit from being started at all, manually or as a dependency. It is useful for deliberately disabling something, but remember it when a service later “refuses” to start.

Next Step

Pick one service on a test machine, break it on purpose by misspelling the ExecStart path, and walk through the five steps above. Practicing the sequence when nothing is on fire makes it automatic when production needs it.