Back to the blog · Oct 12, 2026

Monitoring that actually pages me

I almost never write about my day job here. Not because nothing happens, but because it involves customer systems I'm not allowed to write about publicly. But part of it translates: I've been running virtualized environments for years – vSphere clusters, Linux hosts, with Checkmk for monitoring. And the lessons apply to anyone who watches over their own server, a homelab, or a small infrastructure. So here's a post that reveals nothing about my customers, but a lot about the craft.

The first year was loud

When I started with Checkmk, I built the classic: every available rule enabled, every host monitored with every plugin, thresholds taken over generously. The result was a dashboard full of green checkmarks – and an inbox full of alerts nobody cared about.

The turning point was a night I got woken up for a warning that meant nothing. A filesystem was at 81 percent, the threshold was at 80, and the value had been creeping up slowly for three months. It wasn't a problem that had to be solved immediately. It was a problem that could be solved in ten minutes on Monday. But monitoring didn't know the difference – it reported both equally loud.

That's when I understood: monitoring is not collecting metrics. Monitoring is deciding what is allowed to wake you up.

The real work is the pruning

The hard part of monitoring isn't the setup. A Checkmk agent installs on a Linux host in minutes, a vCenter can be connected as a data source. The hard part is questioning every rule afterwards:

Which state is really an alarm? A CPU that briefly hits 95 percent is often not a problem. A CPU that has been stuck at 95 percent for an hour is.
What is your reaction to the alert? If the answer is "I'll look at it sometime", it's not an alarm, it's noise. Noise belongs on the dashboard, not in the inbox.
Who does the alarm wake up? At night, only what the on-call person can actually fix counts. Everything else floods the on-call rotation with alerts nobody there can resolve.

In practice that means: I strictly separate warnings, which someone sees during the day, from critical states, which escalate immediately. The number of critical rules is small – and that's exactly the point. If everything is critical, nothing is.

Derive thresholds from real data, don't guess. The biggest mistake is setting a CPU limit at 80 percent because it sounds reasonable. Better: watch the host for two weeks, look at the actual maximum, and set the threshold above it – with buffer. A host that normally runs at 30 percent and only jumps to 85 during backup doesn't need an 80 alarm, it needs a 90 warning that knows the backup window.

Trends instead of snapshots. Disk space is the classic example. Instead of alerting when "full", it's worth looking at the growth rate. df tells you 82 percent today – but the actual information is how fast it went from 70 to 82. A value that has been climbing for weeks is no surprise when it crosses the line. Checkmk has forecast functions for exactly this – most people just never use them.

What Checkmk gets right for me

I don't want to advertise here, but a few things are genuinely well solved. The configuration is manageable for me: hosts, services, rules, all in a structure that can be queried via API. In day-to-day work that's worth more than any colorful dashboard:

# View a host including its services
cmk -v my-host

# List all rules for a host
cmk -L my-host

# What would be red right now?
cmk --list-hosts | while read h; do cmk -v "$h" 2>&1 | grep -E 'CRIT|WARN'; done

That last one-liner is my favorite tool: a sober look at what would be red right now – no dashboard, no clicking, straight in the shell. If I can only look at one number a day, it's that one.

For vSphere the same principle applies with a different tool: A cluster doesn't interest me as a collection of VMs, but as three questions – is the storage healthy, are the hosts fine, and does the load still have headroom? If you can't put that into three sentences, you have too many metrics and too few answers.

The core message

The best monitoring setup isn't the one with the most checks, it's the one where you immediately know what to do for every alert. Everything else is noise – and noise eventually makes you ignore even the important alerts.

It's the same lesson as with backups and the static generator: the best X isn't the one with the most features, it's the one you actually run. Monitoring that wakes you at night over an 81-percent warning isn't something you run – you suffer through it. The difference between the two is a handful of well-thought-out rules.