8

I'm using a Mac. In Bash, I'm trying to decode a base64 string and then try to print the Hex value. I'm using base64 -d command and then assigning it to a variable.

myText='YYN29+2wV2XRAHymIyhgytWuqY4atgHnIUFfXA7FPOA='
myTextBytes=$(echo -n "$myText" | base64 --decode)
echo -n $myTextBytes | xxd -p -c 99999

The output of the above script is:

618376f7edb05765d17ca6232860cad5aea98e1ab601e721415f5c0ec53ce0

However, if i were to run the following command directly:

echo -n "$myText" | base64 --decode | xxd -p -c 100000

I get:

618376f7edb05765d1007ca6232860cad5aea98e1ab601e721415f5c0ec53ce0

I even tried using openssl enc -base64 and I get the same result. That is, 00 is getting deleted when assigning to a variable. How do I preserve the 00 when I'm assigning it to a variable?

SilleBille
  • 191
  • 5
  • 6
    Did you get "bash: warning: command substitution: ignored null byte in input" as part of the output of your 3-line script? What does 'bash --version' tell you? The bash provided with a Mac is an older version, I think. My version 4.4.20(1) gives me that warning. – Wastrel Aug 26 '23 at 15:51
  • I'm on `GNU bash, version 3.2.57(1)-release (arm64-apple-darwin22)`. That's why i didn't see the warning. Now, it makes sense why it went missing without any warning – SilleBille Aug 28 '23 at 17:56

1 Answers1

14

That's not "00" as in a string of two 0x30 characters (ASCII "0"), but a NUL character, 0x00. You can see this if you pipe the output of the decode command into od -a or od -x.

Bash's handling of command substitutions (like var1=$( command )) strips out NUL/0x00 characters. However, since Bash 4.4, it also prints out a warning. Inability to handle NUL bytes is a limitation of most shells, zsh being an exception. However, there are issues with passing 0x00 characters embedded within arguments to commands that zsh is still subject to (because it's not an issue within zsh but rather in the exec() family of calls to invoke commands).

If you don't want to switch to another language like Perl, Python, Ruby, etc., then I suggest trying zsh.

Sotto Voce
  • 3,664
  • 1
  • 8
  • 21
  • 4
    At least (modern) bash gives a warning for that case; dash also strips and ksh93 truncates. While a shell can be written so that a var can contain NUL when used inside the shell, like zsh, shell vars are often (mostly?) used to pass as an argument and/or environment variable to program(s) you run -- and the OS calls that do that are defined to use C-language strings which are _terminated_ (truncated) by NUL. – dave_thompson_085 Aug 26 '23 at 04:47
  • 4
    Beware that `$(...)` also strips trailing newline characters (0x0A bytes), so even in zsh, you'd have to work around it (with the usual `var=$(print -r - $encoded | base64 -d; print .); var=${var%.}`) – Stéphane Chazelas Aug 26 '23 at 05:33
  • 17
    One of the reasons base64 is commonly used in the first place is to transmit and protect binary data through non-binary safe paths. Probably your best bet (assuming you stick with bash) is to keep it as base64 until you are ready to use it and then decode it and use it in a pipe (as worked in your second command). – user10489 Aug 26 '23 at 07:35
  • 2
    zsh handles the NUL here fine, the commands they gave work directly and give the output with the tell-tale `00` in the middle. No need to doubt that part, though the other caveats mentioned above do apply. In particular, `echo $x` works with NUL bytes with zsh's built-in `echo`, but any external implementation of `echo` won't and can't work like that. – ilkkachu Aug 26 '23 at 21:25