Fourteen years at five nines. Yes, we patched them.

If you hear "fourteen years at five nines" and picture fifty servers nobody dared touch, I have some maintenance work to tell you about.

At Sprint[1]Ref. 1Sprint en.wikipedia.org, I deployed, configured, upgraded, and sustained fifty bare-metal and virtualized servers running RHEL[2]Ref. 2RHEL redhat.com, CentOS[3]Ref. 3CentOS centos.org, and Ubuntu[4]Ref. 4Ubuntu ubuntu.com for billing mediation. Availability: 99.999% over fourteen years. That is a looooong time to keep the work moving while you keep changing the machines doing it.

An unpatched server can stay up a damned long time. Somebody outside the company may get to choose its next maintenance window for you, though.

Patching has a visible risk. You change something, something might break. Leaving a vulnerability alone feels easier because that risk doesn't need a change ticket this afternoon. I still have to deal with it, even if it isn't on the calendar yet.

Before a change, I want to know what depends on the host, what stops during a restart, what should recover by itself, and what I need to check deliberately. The application can start and still fail to reach its database, and the operating system looks quite pleased with itself while your service is broken.

I want to see the work moving again before I'm satisfied. A daemon with a PID[5]Ref. 5PID en.wikipedia.org doesn't tell me whether the damned thing is doing its job.

Is the affected component installed? Is it reachable in the relevant way? Did the distribution backport a fix while leaving the upstream-looking version number alone? And what does the vendor support?

Sometimes the finding doesn't apply, and I want the evidence written down. Otherwise it needs fixing, with exposure reduced while that gets arranged. I get frustrated seeing a report just sit in a directory while the work is still waiting.