systemctl show failed services: A Practical Monitoring Guide

Updated: 2026-05-21T19:37:39+00:00

A service can look fine in a dashboard and still be broken underneath. One restart loop, one bad dependency, or one failed socket activation can leave operators with a green screen and an angry on-call phone. That is where systemctl show failed services earns its place in the monitoring toolkit.

In practice, systemctl show failed services is less about “seeing red” and more about understanding state precisely. It helps you inspect the fields systemd exposes, separate expected exits from real failures, and build checks that do not wake people for normal behavior. This guide shows how to read those fields, how to use them in uptime and monitoring workflows, and how to avoid the common mistakes that create noisy alerts.

We will also cover the difference between active and failed states, what to verify before alerting, and how to design monitoring that respects service behavior instead of guessing. For reference, systemd state concepts are documented in systemd, Linux service management uses systemctl, and journaling behavior is closely tied to journald.

What Is Systemd Service State Inspection

Systemd service state inspection is the process of reading a unit’s state fields to determine whether it is healthy, stopped, failed, or in a transitional state.

For systemctl show failed services, that usually means querying fields such as ActiveState, SubState, Result, and timestamps. A service can report inactive without being broken, while another can sit in failed after a crash or dependency error.

In practice, this matters for anything you monitor from outside the host. A backup job, VPN tunnel, agent, mail relay, or queue worker may “exit cleanly” by design. If you only check “running,” you will confuse normal exits with incidents.

The key difference is this: systemctl status is operator-friendly, while systemctl show is machine-friendly. That makes systemctl show failed services useful for scripts, alert rules, and maintenance automation.

How Systemd Service State Inspection Works

Read the unit fields.
systemctl show returns named properties such as ActiveState and SubState. That gives you structured output. If you skip this, you end up parsing human text that changes between versions.
Compare state to your expected baseline.
A service may be healthy when SubState=exited or SubState=running, depending on its type. If you skip this, you will treat normal shutdown behavior as failure.
Check whether the unit is actually failed.
systemctl --failed and systemctl list-units --state=failed show failed units. If you skip this, you may miss units that are dead for unrelated reasons.
Inspect timestamps and recent transitions.
ActiveEnterTimestamp and InactiveEnterTimestamp help you tell whether the state changed recently. If you skip this, you lose timing context and cannot separate a new crash from an old, cleared event.
Read the journal for root cause.
journalctl -u <unit> usually explains why the unit failed. Without it, you only know the symptom, not the cause.
Decide whether to alert, retry, or ignore.
Some units should be restarted automatically, while others should page immediately. If you skip this, your monitoring will either be too noisy or too slow.

A realistic case: a wg-quick@ tunnel exits after a configuration reload. That can be fine. A second later, the same unit might flip to failed because a DNS lookup broke or a route was invalid. systemctl show failed services helps you tell those apart before you escalate.

For more practical service handling, see the related notes on Linux server monitoring best practices and server performance monitoring.

Features That Matter Most

When teams build monitoring around systemctl show failed services, they usually need a small set of capabilities that make the signal trustworthy.

Feature	Why It Matters	What to Configure
`SubState` inspection	Distinguishes normal exits from real failures	Compare against the known-good value per unit
`ActiveState` inspection	Shows whether the unit is active, inactive, or failed	Alert only on unexpected transitions
`Result` field	Explains the reason systemd marked the unit failed	Log the result alongside the alert
Timestamp fields	Shows when the state changed	Record `ActiveEnterTimestamp` and `InactiveEnterTimestamp`
Journal lookup	Provides root-cause detail	Pull `journalctl -u <unit>` on alert
Dependency awareness	Reveals failures caused by another unit	Check the dependency tree before restarting
Reset behavior	Clears stale failure records after recovery	Use `reset-failed` only after confirming repair

A useful rule: monitor state, but alert on meaning. A unit that is “dead” may be fine if it is a one-shot job. A unit that is “failed” is not always the original problem, especially when a dependency broke first.

The official systemctl manual is worth keeping nearby: systemctl documentation. For log analysis patterns, the journalctl manual is equally useful.

Who Should Use This (and Who Shouldn't)

This approach is best for teams that need service-level accuracy, not just host-level uptime.

It fits:

SREs monitoring critical daemons on Linux hosts
DevOps teams running workers, queues, VPNs, and cron-like units
MSPs and agencies that need clean signal across many servers
Platform teams that must separate expected exits from genuine failures
Security teams watching services that should never stop unexpectedly
Right for you if you need to detect failed units before users notice.
Right for you if your services have different healthy states.
Right for you if false positives are wasting on-call time.
Right for you if you already use systemd on production Linux hosts.
Right for you if you want scripts that can run unattended.
Right for you if you need structured data rather than terminal output.
Right for you if you monitor many hosts with different service types.

This is NOT the right fit if:

Your environment is not systemd-based.
You only need a simple “website up or down” check and nothing else.

Benefits and Measurable Outcomes

The real value of systemctl show failed services is not the command itself. It is the operational discipline it creates.

Cleaner alerts
You reduce noise by alerting on unexpected state, not just any state change. In a busy environment, that usually means fewer false pages for one-shot or timer-driven units.
Faster triage
When a failure happens, you can map the state change to a timestamp and log trail. That shortens the time from alert to cause.
Better service ownership
Teams can define one healthy state per unit instead of using a generic check. That matters for professionals and businesses in the uptime and monitoring space because not every service should look “running.”
Safer automation
Scripts can decide whether to restart, ignore, or escalate based on SubState and Result. That avoids blind restarts after dependency failures.
Improved incident context
You can correlate service failure with deploys, maintenance windows, or host changes. That makes post-incident review much better.
Less downtime caused by misread states
Monitoring that understands unit type is less likely to miss a real outage or trigger on healthy exit behavior. For uptime teams, that directly affects response quality.
Better multi-service visibility
If you manage websites, databases, background workers, and VPNs, one state model will not fit all. This method gives you a consistent inspection pattern.

How to Evaluate and Choose

When you choose a monitoring approach around systemctl show failed services, judge it against the actual jobs you run, not marketing claims.

Criterion	What to Look For	Red Flags
State accuracy	Reads `SubState`, not only `ActiveState`	Treats all `inactive` units as bad
Failure clarity	Surfaces `Result`, logs, and timestamps	Shows only “down” with no context
Unit-type awareness	Handles services, timers, sockets, and one-shots differently	Uses one rule for every unit
Alert routing	Sends alerts to the right people and channels	Pages everyone for every failure
Multi-check coverage	Supports HTTP, ping, port, and process checks together	Forces separate tools for each signal
Maintenance handling	Can suppress alerts during planned work	Spams during deploys and reboots
API or script access	Lets you automate checks and recovery	Requires manual terminal work every time

This is where many uptime teams struggle. A website monitor, a port check, and a systemd state check all answer different questions. If you already track server uptime monitoring and CPU behavior, the service-state check fills a missing layer.

Recommended Configuration

Setting	Recommended Value	Why
`SubState` baseline	Match the known-good state per unit	Some services should be `running`, others `exited`
Check interval	Short enough to catch incidents, long enough to avoid churn	Overly aggressive polling creates noise
Retry count	2-3 checks before alerting	Filters transient startup and stop events
Log enrichment	Include `Result`, `ActiveState`, and recent journal lines	Speeds triage
Maintenance suppression	Enable during planned changes	Prevents alert storms during deploys
Dependency checks	Inspect upstream units when one fails	Finds the real root cause faster

A solid production setup typically includes a per-unit baseline, a retry window, and a clear action path. For example, if a VPN unit is expected to exit after setup, do not alert on exited. If a database unit should stay live, page immediately when SubState=failed.

For broader context, the RFC 5424 syslog format is still useful when you forward state events into a central log pipeline.

Reliability, Verification, and False Positives

False positives usually come from misunderstanding service behavior, not from systemd itself. The most common sources are short restart windows, one-shot jobs, dependency failures, and units that intentionally stop.

Prevention starts with the baseline. Record the healthy SubState on a known-good host before you build alert logic. Then test the check across a restart, a deploy, and a maintenance window.

Multi-source checks help a lot. For example, pair systemctl show failed services with a port check, a local process check, or an HTTP endpoint check. If systemd says “active” but the app port refuses connections, the service may be up while the application is still broken.

Retry logic matters too. Use a short delay and confirm the failure more than once. That reduces alerts from transient races during boot or reloads.

Alert thresholds should reflect business impact. A critical login service deserves immediate paging. A nightly report worker may only need a ticket or email. For uptime teams, that difference keeps the incident queue useful.

If you need a broader monitoring design, this pairs well with server monitoring workflows and how the platform works.

Implementation Checklist

Identify which units are long-running, one-shot, timer-based, or socket-activated.
Record the healthy SubState for each unit on a known-good host.
Decide which units should trigger alerts and which should only log.
Define a retry window for transient startup and reload events.
Add journal collection for every failed alert.
Confirm how dependency failures should be handled.
Test behavior during a planned restart and a planned maintenance window.
Verify that reset-failed is only used after the cause is fixed.
Document who owns each unit and where alerts should go.
Review the rules after each incident and adjust the baseline if needed.

Common Mistakes and How to Fix Them

Mistake: Treating every inactive unit as broken.
Consequence: You get false alerts for one-shot, timer, and exit-style services.
Fix: Baseline on SubState and unit type.

Mistake: Alerting only on ActiveState=failed.
Consequence: Some failures remain hidden behind other state values or dependency issues.
Fix: Check Result, SubState, and recent journal entries too.

Mistake: Ignoring the journal.
Consequence: You know something failed, but not why.
Fix: Pull journalctl -u <unit> into the investigation flow.

Mistake: Restarting the service before checking dependencies.
Consequence: The restart may fail again, and the real issue stays open.
Fix: Inspect the dependency tree first.

Mistake: Using one rule for every service.
Consequence: Monitoring becomes noisy and brittle.
Fix: Define per-unit behavior and expected states.

Mistake: Clearing failure state too early.
Consequence: You lose incident history before confirming recovery.
Fix: Reset failed state only after repair and verification.

Best Practices

Define the expected state for every important unit.
Use systemctl show for automation and systemctl status for human review.
Pair state checks with logs and at least one secondary signal.
Keep maintenance suppression in place for planned changes.
Treat dependency failures as first-class incidents.
Review one-shot and timer units separately from always-on services.

A simple workflow for a failed service usually looks like this:

Check systemctl --failed.
Read systemctl show for ActiveState, SubState, and Result.
Inspect journalctl -u <unit>.
Validate dependencies and recent changes.
Restart only if the root cause is understood.

That workflow is what makes systemctl show failed services practical in the real world. It turns a vague failure into a traceable event.

FAQ

What does systemctl show failed services tell me?

It tells you which systemd units are in a failed or unexpected state. More importantly, it exposes structured fields so you can script around them. That makes it much better than reading terminal output by hand.

Is systemctl show better than systemctl status for monitoring?

Yes, for automation and alert logic. systemctl status is great for humans, while systemctl show gives you machine-readable fields. Use both together when investigating incidents.

How do I know whether a service is really failed?

Check SubState, ActiveState, and Result, then confirm with logs. A service can look inactive and still be healthy if it is designed to exit. That is why systemctl show failed services is useful in monitoring.

Should I use systemctl --failed instead?

Use it as a quick overview, yes. It is excellent for a fast operator check. For alerting and scripting, though, systemctl show gives you more precise handling.

How do I avoid false positives?

Baseline the expected state for each unit, then add retries and maintenance suppression. Also check whether a unit is a timer, one-shot, or long-running service. Those behave differently.

Can I monitor dependencies too?

Yes, and you should for critical services. A unit may fail because a required dependency failed first. That is one of the biggest blind spots in naive checks.

When should I reset failed state?

Only after you have fixed the underlying issue and confirmed the service is healthy again. Resetting too early hides useful incident context. In practice, systemctl show failed services should guide the recovery, not replace it.

Conclusion

The best service monitoring is precise, not loud. It respects unit type, reads structured fields, and checks the journal before it pages a human.

The three takeaways are simple: define the healthy baseline per unit, inspect SubState and Result, and confirm the cause before you restart or reset anything. That is the difference between noisy monitoring and useful monitoring.

If you manage Linux services for uptime-sensitive systems, systemctl show failed services gives you a better operating model than “is it running?” alone. If you are looking for a reliable uptime and monitoring solution, visit zuzia.app to learn more.