Escalation, schedules and silences
How an alert becomes a person, and how to stop it becoming one at 3am for no reason.
Monitoring that tells you something is wrong and stops there has done the easy half. The gap between an alert firing and somebody with the right access acting on it is where outages actually live.
Escalation chains
A chain is matched by severity and label, so a disk filling on a staging box and a mail node refusing connections do not wake the same person at the same hour.
If the first person does not acknowledge within the step’s window, it moves on. It does not wait, and it does not quietly give up.
On-call schedules
Who is on, when. The chain resolves against the schedule at the moment it fires — so a rota change does not require rewriting the chains that point at it.
Silences
For maintenance you already know about. A silence suppresses alerting without suppressing recording, so the history still shows what happened during the window. A maintenance window also holds a peer’s outbound alerts rather than dropping them.
Heartbeats
The failure that never alerts is the one where the thing that would have alerted is the thing that died. A heartbeat expects to hear from something on a schedule and fires when it does not — which covers the cron job that stopped running and the agent on a machine that is now off.