Reconciliation as a job: where's the damned file?

You can have a mediation[1]Ref. 1mediation en.wikipedia.org server running, accepting files, processing perfectly good records, and still be losing money. Nothing crashed, and the records that arrived are fine, but I want to know about the ones that didn't arrive.

I spent fourteen years at Sprint[2]Ref. 2Sprint en.wikipedia.org doing end-to-end reconciliation between network elements, mediation and billing. Repairing corrupted and mangled CDR[3]Ref. 3CDR en.wikipedia.org files recovered tens of millions of dollars that would otherwise have gone unbilled. That was accumulated over the fourteen years. Mostly I was trying to figure out where a damned file went.

A truncated file is particularly annoying. It has the right name and directory entry, possibly enough valid records to look perfectly ordinary until the parser[4]Ref. 4parser en.wikipedia.org reaches the end. Now we're unhappy.

Depending on the format, you might compare declared length against actual length or inspect a trailer count. You can also check that the last record ends on a valid boundary. You need the format's rules rather than rules inferred from whatever survived in the damaged file.

Sometimes you can recover intact records around damaged data. Sometimes you can retrieve a clean source copy. You cannot invent an unknown duration because the neighboring record has one that looks plausible. I need to recover billable usage, and making shit up just to get the parser to shut up won't do that.

I'm less excited when isolation becomes storage nobody owns. Money can sit in an unwatched error queue while every process check stays green. The software has handled its exception, but somebody still has to handle the actual goddamn problem.

And before you replay recovered records, establish whether any portion was already accepted downstream. Reprocessing an entire file because part failed can turn missing revenue into duplicate billing. The customer isn't going to thank you for recovering the same money twice.

And if the next file processed successfully, I still want to know where the missing one went. I'm going to keep asking about that damned file.