Base64 and signed chars: four wrong damned slashes

The corkscrew[1]Ref. 1corkscrew github.com Base64[2]Ref. 2Base64 rfc-editor.org test takes three bytes, hexadecimal 75 3a ff, and the audit harness spits out //// where Python[3]Ref. 3Python python.org's Base64 encoder produces dTr/. UndefinedBehaviorSanitizer[4]Ref. 4UndefinedBehaviorSanitizer clang.llvm.org also reports a left shift of negative value -1.

With undefined behavior we have lost the portable guarantee C gives us. Change the compiler or optimization setting, or run on another target, and we may get something else wrong entirely. This is one observed outcome, with a sanitizer diagnostic explaining why the operation is invalid. It doesn't tell us what negative shifts will always produce.

Meanwhile, the input pointer is char *, and the shift happens before the result gets to the accumulator. Here is the relevant shape from the reviewed source:

unsigned int tmp;
/* src is char * */
tmp |= *src++ << (24 - 8 * i);

Declaring the destination unsigned cannot change the expression that was already evaluated. Sorry, the variable has no time machine. The input undergoes integer promotion before shifting, and on the audited platform that high-bit byte was read as a negative char value. Left-shifting that negative signed value is undefined behavior under the C rules relevant to this code.

Plain char is a distinct type, and its signedness is implementation-defined. It is not shorthand for an unsigned octet, however much we'd enjoy that convenience. Read the arithmetic-type reference[5]Ref. 5the arithmetic-type reference en.cppreference.com alongside the shift rules[6]Ref. 6shift rules en.cppreference.com. The declaration can look perfectly respectable while the expression is doing something goddamn invalid.

The repair needs unsigned input bytes and explicitly unsigned intermediate arithmetic. Changing src to unsigned char * doesn't finish the explanation: narrow unsigned types can still promote to signed int before the shift. So I still have to check the promoted type, even after I've changed the pointer. It's a small expression, and there's quite a bit in there I need to get right.