At two in the morning, a switch fails in one data hall. Within minutes, hundreds of alerts arrive: unreachable servers, unreachable databases, failing applications, response times out of bounds. The on-call engineer receives notification after notification and has to find what matters. What is the cause and what is merely a symptom? Which services are actually affected? And is this perhaps related to a change running at that very moment?
Monitoring noise is not just a technical nuisance. On-call engineers gradually learn that most alerts mean nothing and start to tune them out. Then the one alert that does matter arrives and gets lost. Alert fatigue is one of the quietest risks in operations.
Why adding more rules does not help
The usual response is to tune thresholds and add exceptions in the monitoring tool itself. That helps for a while, but monitoring only sees its own slice of the world. It does not know that the database runs on a server connected to the switch that just failed. Nor does it know that an approved change is currently running on that server.
We often see the same patterns:
- several monitoring tools report the same problem at once,
- a service “flaps” between states, and every state change becomes a new alert,
- planned maintenance generates dozens of alerts nobody needs,
- every alert becomes an incident and the queue fills with duplicates.
How to approach it
Noise cannot be removed in one place. It has to be compressed step by step, like a funnel. First, remove duplicates and flapping. Then suppress what is expected, such as alerts during an approved change. Finally, group the remaining alerts according to how they relate to each other in the infrastructure.
Topology is essential for that last step. When we know what depends on what, we can estimate which alert in a group is most likely the cause. We also know which services are affected.
How Eywo Stratum handles it
Eywo Stratum ingests events from your monitoring, whether that is Zabbix, Prometheus/Alertmanager, Checkmk, Grafana or another source through a generic API. Each event is matched to a CMDB item.
Several layers then act on the incoming stream:
- deduplication – repeated reports of the same problem are merged into a single alert,
- flapping detection – a state that keeps switching back and forth is recognised and not reported on every change,
- filtering and suppression rules – what is known and harmless does not need to reach people,
- suppression during changes – alerts from items with an approved change in progress are suppressed.
The remaining alerts are grouped into situations based on CMDB topology, services and timing. Each situation shows a probable cause, meaning the item the others depend on, along with related changes. Instead of a hundred alerts, on-call sees one situation with a clear impact.
An incident can be created from a situation in a single step, or automatically. It comes with the affected service and resolver team suggested from the CMDB. If the problem returns after being resolved, the incident is reopened rather than a new one created. The whole noise-reduction funnel, from events to alerts to situations to incidents, is visible in one place, so it is clear where noise originates and where it is filtered out.
Less noise, more attention
The goal is not for monitoring to report less. The goal is for people to receive only what needs their attention, with the context they need to decide. An on-call team that trusts its alerts responds faster and more calmly. And after a night shift, they can go home knowing they spent their time on what mattered.
Event management with CMDB-based correlation and noise reduction is included in the Business edition.
Eywo Stratum Business
For operations that want to connect monitoring, security and vendors.