Change management before DevOps: did billing get the damned records?

You can screw up a billing mediation deployment without bringing a single damned process down. Everything is running except the records have stopped reaching billing, and now you have unbilled revenue and an uptime report that thinks you are doing great.

At Sprint[1]Ref. 1Sprint en.wikipedia.org I established a production change-management system using Ansible[2]Ref. 2Ansible ansible.com and GitLab[3]Ref. 3GitLab about.gitlab.com. Deployment errors decreased by 30%, and release speed improved by 40%.

When I say change management before DevOps[4]Ref. 4DevOps en.wikipedia.org, I don't mean we invented this before somebody invented the term. The operational problem came first. We needed to know what was changing and repeat the change reliably, then establish that the billing path still worked afterward. That's what we put Ansible and GitLab to work on.

A valid configuration can still be wrong. A mediation rule can parse beautifully and send records to the wrong destination, and a service can restart cleanly with a mapping that drops a field billing needs. Nothing has to crash for you to fuck this up.

Restoring the previous configuration is often possible. But then you've got records already sent downstream, and undoing those is another job entirely. Software rollback restores behavior, data recovery deals with what happened while the behavior was wrong. I want both accounted for before anybody calls the release plan finished.

A completed automation run tells you its tasks completed. A service check tells you a process is alive. I still want evidence that records are being collected and processed, and that they're accepted where they belong.

I'd keep the change open until the expected records reach billing. Now where the fuck are the records?