Fifty million dollars. Show me the damned rubric.

The fifty-million-dollar project is one I'm proud of, and I want to tell you about the scoring sheet. This is probably why nobody lets engineers pick the entertainment.

I led the $50 million replacement of Sprint[1]Ref. 1Sprint en.wikipedia.org's legacy billing mediation[2]Ref. 2billing mediation en.wikipedia.org platform, Marconi CEOS, with Openet[3]Ref. 3Openet en.wikipedia.org FusionWorks across 130 switches. I wrote the requirements and the RFP[4]Ref. 4RFP en.wikipedia.org, along with the scoring rubric, and evaluated twelve vendors through live and lab demonstrations.

A vendor arranges a demo to show its product working. I want to know what working means and how much preparation it took. Then I want to see what happens when we bring our data.

"Supports duplicate detection" looks respectable on a requirements sheet. I'd have a lot of questions before I could score that.

Same filename? Same bytes with a different filename? A record sent again because the receiver accepted it but the sender missed the acknowledgment? How long does detection remember anything? Does it still remember after a restart? You can spend a looooong time agreeing about duplicate detection without agreeing about a damned thing.

I'd want those examples in a rubric. I don't have Sprint's original scoring sheet open beside me, so please don't read this as a reproduction of it. I remember the method rather than the paperwork.

Weighted scores are useful when products make different tradeoffs. They're also perfectly happy to let points for something harmless cancel out a serious failure. The arithmetic has no idea what you're buying.

If a product can't preserve required billing information, a lovely reporting interface doesn't fix it. I'd make that a gate, and decide which failures disqualify a product before I knew which vendor had them. I'd have a hard time trusting a gate we moved after the demo.

An attractive screen can eat half an hour. Recovery behavior takes longer to explain and might look like nothing but records moving again. It may be boring to watch, but it may also be the thing that lets you go home.

For fifty million dollars, I should fucking hope everybody liked the demo. I'd still like to see the sheet.