Alert quality decays by default — thresholds set for last year’s load, checks for retired services, duplicates from overlapping monitors. The countermeasure is a small recurring review, not a big annual cleanup.
The weekly method (30 minutes)
- Pull last week’s alerts grouped by rule, sorted by count
- For the top 10 rules, classify each: actionable (someone did something), informational (nobody acted, nobody needed to), or noise (nobody acted, and acting was never plausible)
- Apply one fix per noisy rule per week — small, reviewable changes
The four fixes
- Retune — the threshold no longer matches reality (CPU 80% on a box that always runs at 85%)
- Redirect — real signal, wrong audience; route it to the owning team
- Aggregate — 40 per-instance alerts should be one summary alert on the fleet
- Retire — the condition no longer matters; delete the rule and note why
The metric that matters
Track actionable percentage: alerts that led to action divided by total alerts. Healthy teams sit above 60%. Below 30%, on-call has already learned to ignore the stream, and the next real incident will be acknowledged late — the cost of noise is paid during the incident, not during the week of noise.
TagsMonitoringAIOpsOn-callBest Practice
Was this article helpful?
98% of 811 readers found it useful