There is a point in every monitoring setup where the alerts stop being read. Messages arrive, people glance at them and mentally file them under “probably nothing.” Then one of them is something, and the response is slow. That is alert fatigue, and it is a design problem rather than a discipline problem.

The Test Every Alert Should Pass

An alert that interrupts a human should satisfy three conditions:

1. It is actionable. There is something a person can do about it right now.
2. It is urgent. It cannot wait until the next working day without harm.
3. It is real. It reflects an actual problem, not a threshold that happens to be crossed every afternoon.

If an alert fails any one of these, it does not belong in a pager. It might belong in a daily report, a dashboard or a ticket, but not in someone’s pocket at night.

Start With an Audit, Not Opinions

For a few weeks, keep a simple record of every alert that fires. For each, note: Was action required? What action? How long did it take? Was it a duplicate of another alert? Many monitoring tools can export alert history, which makes this far easier than relying on memory.

Then group the alerts. You will almost always find a small number of rules produce most of the noise. Handling the top handful typically removes the bulk of the volume without touching anything important.

Common Causes of Noise and Their Fixes

Thresholds set on causes instead of symptoms. High CPU on a server is not inherently a problem; slow or failing user requests are. Where possible, alert on what users experience (error rates, latency, failed health checks) and use resource metrics for dashboards and diagnosis. Cause-based alerts remain useful for things that predictably lead to failure, such as a disk about to fill.

No duration requirement. A momentary spike that clears in seconds does not need a human. Require the condition to persist for a sensible window before firing.

Flapping. An alert that flips between firing and resolved around a threshold generates repeated messages. Use hysteresis, with a lower threshold for clearing than for triggering, or smooth the metric over a short window.

Duplicates from dependent failures. When a network device fails, every server behind it may alert. Group related alerts, or suppress downstream alerts when an upstream dependency is already down.

Fixed thresholds for variable systems. A batch server that is busy every night at the same time should not alert on that load. Use schedules, maintenance windows or thresholds that account for normal patterns.

Alerts with nobody to fix them. If the owner cannot act, the alert is not theirs. Route by service ownership.

Stale rules. Alerts for decommissioned hosts or retired services keep firing, or worse, stay silent because the target is gone. Review rules whenever infrastructure changes.

Tiers: Page, Ticket, Dashboard

A tiered approach keeps urgent things loud and everything else quiet but visible.

Tier	Delivered by	Use for	Example
Page	Phone or pager, any hour	Active or imminent user impact	Service down, error rate sustained high
Ticket	Queue, handled in working hours	Needs a fix but not tonight	Disk trending toward full over days
Dashboard or report	Reviewed periodically	Trends, context, capacity	CPU averages, memory patterns

Each time an alert fires, you should be able to say why it landed in its tier. If people routinely downgrade a page to a ticket in their heads, move it to a ticket in the system.

Write Alerts So They Help at 3 a.m.

A good alert message names what is wrong, where, how bad, and links to a short runbook. “Disk usage on the database volume at 92 percent and rising; see the cleanup steps” is a different experience from “Warning: threshold exceeded.” Even a three-line runbook, covering how to check, the first safe fix, and whom to escalate to, dramatically lowers stress and response time.

Treat the Alert Configuration as Living

Hold a short, regular review, perhaps monthly, of the alerts that fired. For each noisy rule, decide: tune it, demote it, delete it or fix the underlying problem it keeps flagging. Do not ignore the last option. Sometimes an alert is noisy because a real problem is chronic and nobody has fixed it, such as a service that restarts weekly. Silencing the alert hides a defect.

After any incident, ask two questions: did we get alerted, and was it early enough? Where monitoring missed something, add the minimum new alert to catch it, resisting the urge to add ten.

A Realistic Example

Imagine a small team that receives dozens of notifications a day from a shared channel. An audit over two weeks shows that a few rules are responsible for most messages: a CPU threshold that fires during a nightly job, a memory alert that triggers on normal cache growth, and several disk warnings for the same volume. The team makes targeted changes. The CPU rule gets a duration requirement and is excluded during the batch window. The memory alert is replaced with one on swap activity, which better reflects real pressure. The duplicate disk alerts are collapsed into one, with a ticket-level warning early and a page only when the projected time to full is short. They add a health-check-based page for the customer-facing service. The next review shows far fewer notifications, and people trust the remaining ones.

Frequently Asked Questions

How many alerts is too many? There is no magic number. The practical test is whether people respond to every page with action. If pages are routinely ignored or acknowledged and dismissed, you have too many.

Should I alert on CPU usage? Usually as a lower-tier signal or dashboard item, unless sustained high CPU reliably precedes user impact in your environment. Prefer alerts on error rates, latency and availability for paging.

What is alert flapping? An alert that repeatedly toggles between firing and resolved because the metric hovers around the threshold. Add a duration requirement or separate trigger and clear thresholds.

Is it okay to just mute noisy alerts? Temporarily, with a plan. Permanent muting without investigation can hide a real chronic problem. Decide whether to fix, tune, demote or delete.

Who should receive which alerts? The team that owns the service and can act on it. Routing everything to everyone teaches people to ignore all of it.

Next Step

Export the last month of alerts, count them by rule, and take the three noisiest. For each, decide whether to tune, demote to a ticket or delete, and make those three changes this week.