Subnet Supply Co. The 3am runbook: what sysadmins actually Google at night
A survey of one very tired engineer's search history — and the checklists that stop the 3am scramble before it starts.
Nobody schedules an outage at 2pm on a Tuesday. Incidents cluster at night, and the searches tell the story.
“bgp neighbor flapping cause” — Half the time it’s a timer mismatch, a quarter of the time it’s MTU, and the rest is a spicy mix of route churn and hope. The runbook answer: check hold-time config before touching anything else.
“dig +trace not resolving” — The classic. Is it the authoritative server, the resolver, or the client cache? The order you check matters more than the tools. Our Network Ops Runbook puts the decision tree on one page, because at 3am you don’t want a decision tree, you want a checklist.
“what port is this again” — 179 is BGP, 161 is SNMP, and no, you don’t need to memorize them. You need them on the wall. That’s literally why the Subnet & Ports Cheat Sheet exists.
“how to roll back calmly” — The most valuable skill in operations is the pre-agreed exit. Change windows get a rollback plan in writing, or they don’t happen.
The pattern across all of these: the fix is rarely the hard part. Remembering the order of operations while tired is the hard part. Write it down when you’re calm. Buy it printed if you have to — that’s the whole idea behind this store.