Weekly Padgett

No. 0361 min read

Reading hex dumps: quit guessing at the damned bytes

Corrupt how, folks? Protect the original, establish the format, follow record lengths, check byte order, and find the first boundary that actually breaks.

A person holds a pink strip across a vinyl record beneath a magnifying lamp on a cluttered workbench.
File photoDramatization. No unix was harmed.

Your parser[1]Ref. 1parser en.wikipedia.org says "corrupt file," and you would like it to be a little more fucking specific.

I'd protect the original and make a working copy before poking around. I'd record the file size and a cryptographic hash[2]Ref. 2cryptographic hash en.wikipedia.org, and keep the source filename and transfer context somewhere outside the copy I'm examining. I want to know what arrived and where it came from before deciding what's wrong with it.

The editor has a save button, which is also a very convenient way to help the damage along.

od -Ax -tx1 -v -N 64 suspect.bin

In od(1)[3]Ref. 3od(1) man7.org, -Ax prints hexadecimal[4]Ref. 4hexadecimal en.wikipedia.org offsets and -tx1 prints individual bytes in hexadecimal. The -v keeps repeated lines from disappearing behind an asterisk. -N 64 limits this first look. That is 64 bytes. There's a whole file I haven't seen yet.

I follow the offset column rather than those convincing little display rows. Sixteen bytes across means somebody chose a sixteen-byte display width. It doesn't tell us that somebody else designed sixteen-byte records, however neatly they seem to line up on the screen.

A parser out of alignment can make everything afterward look broken. I'd look for the first boundary where the file and format disagree. The last error in a thousand-line report has had plenty of opportunity to inherit somebody else's problem.

If recovery is justified, I'd preserve the rejected suffix and document exactly which complete records I accepted. I'd keep what I could establish separate from what I couldn't. Padding missing payload with zeros would give me a longer file, sure, but I'd still be missing the fact that was supposed to be there.

References

  1. parser en.wikipedia.org ↩
  2. cryptographic hash en.wikipedia.org ↩
  3. od(1) man7.org ↩
  4. hexadecimal en.wikipedia.org ↩
unixbinary-formatsdebuggingtelecom

More on this beat