Achieving hexdump-like format, that includes binary strings, on the command line?

Question

I really like hexdump, especially because you can define a custom format; say:

$ echo -e '\x00\x01\x02\x03' | hexdump -v -e '1/1 "%_ad: "' -e '4/1 "%02X "' -e '1/1 " : "' -e '4/1 "%_p"' -e '1/1 "\n"'
0: 00 01 02 03 : ....
4: 0A          : .

So, I can choose to have, say, 4 bytes per line, written as hexadecimal first, then as characters. But, what I'm missing here, is a "binary string" (or "bit string") formatting character; e.g. I'd like to write something like -e '4/1 "%08b "' somewhere in that command line, and get, e.g.:

0: 00 01 02 03 : 00000000 00000001 00000010 00000011 : ....
4: 0A          : 00001010 : .

Of course, then probably one would have to specify endianness (if groups of more than one byte should be formatted) etc... But in any case, this kind of formatting doesn't exist, as far as I can see in the hexdump manual.

So my question is - what alternatives do I have on a Linux command line, so that I could obtain a formatted dump that includes binary strings as above, and yet to the greatest extent possible preserves the customizability of the hexdump program (in terms of byte grouping) when using its -e option?

I don't know how Endianess plays a role here?! Groups of more than one byte are still just groups of bytes. Otherwise you should specify as what you want to interpret it - unsigned Xbit integer, signed ... float ... many possibilities. Have you checked out `xxd`? — 0xC0000022L, Jun 12 '14 at 18:39
Thanks for the comment, @0xC0000022L - yeah, I didn't fully think through that with the endianness `:)`. I did check `xxd` out, and there is a `-b: Switch to bits (binary digits) dump,`, but I cannot find an example where that output would be mixed with hex (as I gave an example for in the OP); if someone can post an example for that with `xxd`, that would be an acceptable answer. Cheers! — sdaau, Jun 12 '14 at 18:41
@0xC0000022L - Just checked `xxd` with: `echo -e '\x00\x01\x02\x03' | xxd -c 2 -b`; the presence of the `-b` switch changes *everything* to binary string, and so apparently it cannot be "mixed" with hexadecimal. — sdaau, Jun 12 '14 at 18:44
Now I see what you mean. I am not certain how to achieve that without a more complex script. From the command line I'd think this will become rather complex as a "one-liner". — 0xC0000022L, Jun 12 '14 at 18:52

score 5 · Accepted Answer · edited Jun 12 '14 at 19:07

5

Failing a dump program with suitable dump options, you can always cobble something together by using both hexdump and xdd and then joining the output with paste. Its not pretty, but using a shell that supports process substitution (bash will do):

mkfifo fifo
echo -e '\x00\x01\x02\x03' |
  tee fifo |
  paste -d' ' \
    <(hexdump -v -e '1/1 "%_ad: "' -e '4/1 "%02X "' -e '1/1 " :\n"') \
    <(xxd -b -c 4 fifo | cut -d' ' -f 2-)

Output:

0: 00 01 02 03 : 00000000 00000001 00000010 00000011  ....
4: 0A          : 00001010                             .

edited Jun 12 '14 at 19:07

sdaau

6,668
12
57
69

answered Jun 12 '14 at 19:00

Graeme

33,607
8
85
110

2

Neat. Thanks. Every day with something new learned is a good day. Simply hadn't thought of `paste`. – 0xC0000022L Jun 12 '14 at 19:04
Many thanks for the answer, @Graeme - I edited your post and added the output; am accepting this for now - but if anyone posts a more convenient approach, I might eventually move the accept. Cheers! – sdaau Jun 12 '14 at 19:09

sdaau · Answer 2 · 2014-06-12T21:38:50.563

Here is my suggestion using Perl, utilizing its formatting specifiers for the pack()/unpack() function; the test call would be like:

$ echo -e '\x00\x01\x02\x03' | perl hexdump-00.pl --offset 120 --group 4 --add '(H2)*' --add '(B8)*' 
Opening '' STDIN
Cannot seek!
0
00000000: 00 01 02 03 00000000 00000001 00000010 00000011  '....'
00000004: 0a 00001010  '.'

It's kinda tough to insert string markers in-between - but the nice thing is, you can still "group" bytes within - e.g. you can group two bytes and interpret them as signed (short) integer, example:

$ perl -e 'print pack("s*\n", (-124))' | hexdump -C
00000000  84 ff                                             |..|
00000002

$ echo -e '\x00\x01\x84\xff' | perl hexdump.pl \
  --offset 120 --group 4 \
  --add '(H2)*' \
  --add '(B8)*' \
  --add '(s2)*'
Opening '' STDIN
Cannot seek!
0
00000000: 00 01 84 ff 00000000 00000001 10000100 11111111 256 -124  '....'
00000004: 0a 00001010  '.'

Here is hexdump-00.pl:

#!/usr/bin/perl

# perl hexdump-00.pl --offset 120 --group 4 --add '(H2)*' --add '(B8)*' test.file

use strict;
use warnings;
use Getopt::Long;
use Fcntl qw(SEEK_CUR SEEK_SET);
my $offset = 0;
my $groupsize = 1;
my $length = 128;
my @list=();
my $result = GetOptions (
  "offset=i" => \$offset,
  "group=i"   => \$groupsize,
  "length=i"   => \$length,
  "add=s" => \@list,
);
my $inputfname="";
my $inputfh;
$inputfname = $ARGV[0] if defined $ARGV[0];
if (($inputfname eq "") || ($inputfname eq "-")) {
  printf(STDERR "Opening '%s' STDIN\n", $inputfname);
  $inputfh = *STDIN;
} else {
  printf(STDERR "Opening '%s'\n", $inputfname);
  open ($inputfh, "<$inputfname");
}

binmode($inputfh);
my $startaddr=0;
if( not(defined($startaddr = sysseek($inputfh, $offset-1, SEEK_SET))) ) {
  printf(STDERR "Cannot seek!\n");
  #~ $startaddr = sysseek($inputfh, 0, 0); // cannot reset like this
  $startaddr = 0; # just avoid errors
}
print(STDERR $startaddr . "\n");

my $buffer=undef;
my $nread;
my $total=0;
while (($nread=sysread($inputfh, $buffer, $groupsize)) > 0) { # , $startaddr
  #~ printf("%08X: nr: %d, buf '%s'\n",$startaddr,$nread,$buffer);
  printf("%08X: ", $startaddr);
  foreach my $tformat (@list) {
    foreach my $tentry (unpack($tformat, $buffer)) {
      printf("%s ", $tentry);
    }
  }
  (my $newbuf = $buffer) =~ s/[^[:print:]]/./g; # make non-printable into '.'
  printf(" '%s'", $newbuf);
  print("\n");
  $startaddr += $nread;
  $total += $nread;
  if ($total > $length) { last; }
}

close($inputfh);

mikeserv · Answer 3 · 2014-06-14T21:07:03.470

Here's some sed to coax dc to translate od's output to base 2:

od -t d1z -w4 -v -N12 </dev/urandom |
sed -e '1i2o' -e 's/.*/[&]p/p;$d
    s/>/]n [>/;s/[^ ]*/&]n [/;h;s/>.*//;
    s/ -/ _/g;s/ [^] [][^ ]*/ ]n&n [ /g;G
    s/\n[^>]*//' | 
dc

It's a little simpler now - which is not to mention faster - but it's still no beauty queen. It also prints the decimal and base 2 values of all the bytes.

When I run it I get:

0000000  -43  125 -117  -39  >.}..<
0000000  -101011   1111101  -1110101   -100111   >.}..<
0000004   62   28   80   61  >>.P=<
0000004   111110    11100    1010000    111101   >>.P=<
0000010    6   14  120  -16  >..x.<
0000010    110    1110   1111000   -10000   >..x.<
0000014

Or...

echo aBcD | od -t d1z -w4 -v | sed ... | dc


0000000   97   66   99   68  >aBcD<
0000000   1100001    1000010    1100011    1000100   >aBcD<
0000004   10                 >.<
0000004   1010                  >.<
0000005

The field widths could use a little work, but it's all yours. You don't need the -N12 option - I just used that so I didn't choke on an endless pipe of pseudo-random data. And the -w4 specifies 4 bytes per line but you should be able to use any number of bytes. Also the 1i2o sed command is a dc instruction regarding its output base - 2 for binary - but any base between 2 and 16 should work just as well. If wish to see, for instance, hexadecimal and base 2 output you'll need to add '16i' to that first sed statement and change od's -t d1z option to t x1z.

Other options include...

printf does this:

printf '%o\n%x\n' 128 128

200

80

...even...

printf '%o\n%x\n' "'a" "'a"

141

61

Binary isn't quite as simple, but bc can do all of it if you set its obase= to your specifications:

printf 'obase=2;%d
        obase=8;%d
        obase=16;%d
        obase=2;%d
        ' 64 128 "'A" "'a" |
bc

OUTPUT

dc isn't quite as chatty:

printf '%do%dn10P' 2 64 8 128 16 "'A" 2 "'a" |dc

OUTPUT

Do man dc bc for more info.

And then again, for file streams you can always use od:

for o in o d x ; do
    echo aBcD | 
    od -A n -t ${o}1z -v -w4
done

OUTPUT

 141 102 143 104  >aBcD<
 012              >.<
  97  66  99  68  >aBcD<
  10              >.<
  61  42  63  44  >aBcD<
  0a              >.<

With ^that^ one I tell od not to print offsets - which I'm now second-guessing - that I want outputs of -type o, d, or x one byte at a time and that I want the ASCII representationz of each byte appended to the end of the line, -verbosely (so it doesn't just print me an 0* for 0000) at -w4 bytes per line.

Without -A n it prints:

0000000 141 102 143 104  >aBcD<
0000004 012              >.<
0000005
0000000  97  66  99  68  >aBcD<
0000004  10              >.<
0000005
0000000  61  42  63  44  >aBcD<
0000004  0a              >.<
0000005

And any combination of dc bc od is of course possible in a |pipeline.

Thanks for that , @mikeserv - nice to keep this in mind, but it still takes some extra work to get it formatted per line (with offset address into the file etc) as with `hexdump`. Cheers! — sdaau, Jun 12 '14 at 21:18
@sddau - not much. I can show that... In fact - I explicitly told `od` not to output offsets with `-A n` to avoid confusing the output. Sorry. I'll show it again. — mikeserv, Jun 12 '14 at 21:20

Achieving hexdump-like format, that includes binary strings, on the command line?

3 Answers3

OUTPUT

OUTPUT

OUTPUT