bash can't store hexvalue 0x00 in variable

Question

I'm trying to do some tricks with dd. I thought it would be possible to store some hexvalues in a variable called "header" to pipe it into dd.

My first step without a variable was this:

$ echo -ne "\x36\xc9\xda\x00\xb4" |dd of=hex
$ hd hex

00000000  36 c9 da 00 b4                                    |6....|
00000005

After that I tried this:

$ header=$(echo -ne "\x36\xc9\xda\x00\xb4") 
$ echo -n $header | hd

00000000  36 c9 da b4                                       |6...|
00000004

As you can see I lost my \x00 value in the $header variable. Does anyone have an explanation for this behavior? This is driving me crazy.

Answer 1

You can't store a null byte in a string because Bash uses C-style strings, which reserve the null byte for terminators. So you need to rewrite your script to simply pipe the sequence that contains the null byte without Bash needing to store it in the middle. For example, you can do this:

printf "\x36\xc9\xda\x00\xb4" | hd

Notice, by the way, that you don't need echo; you can use Bash's printf for this an many other simple tasks.

Or instead of chaining, you can use a temporary file:

printf "\x36\xc9\xda\x00\xb4" > /tmp/mysequence
hd /tmp/mysequence

Of course, this has the problem that the file /tmp/mysequence may already exist. And now you need to keep creating temporary files and saving their paths in strings.

Or you can avoid that by using process substitution:

hd <(printf "\x36\xc9\xda\x00\xb4")

The <(command) operator creates a named pipe in the file system, which will receive the output of command. hd will receive, as its first argument, the path to that pipe—which it will open and read almost like any file. You can read more about it here: https://unix.stackexchange.com/a/17117/136742.

Answer 2

You can use zsh instead which is the only shell that can store the NUL character in its variables. That character even happens to be in the default value of $IFS in zsh.

nul=$'\0'

Or:

nul=$'\x0'

Or

nul=$'\u0000'

Or

nul=$(printf '\0')

However note that you can't pass such a variable as an argument or environment variable to a command that is executed as the arguments and environment variables are NUL-delimited strings passed to the execve() system call (a limitation of the system's API, not the shell). In zsh, you can however pass NUL bytes as arguments to functions or builtin commands.

echo $'\0' # works
/bin/echo $'\0' # doesn't

Answer 3

Bash uses C strings internally which cannot store the null byte. Store the value in a temporary file like this:

    zHex=$(mktemp --tmpdir "$(basename "$0")-XXXX")
    trap "rm -f ${zHex@Q}" EXIT

The variable zHex now contains a unique file name. The file referenced by $zHex can be deleted manually, but the file will be automatically deleted when the program terminates for any reason.

Then use the variable like this:

    echo -ne "\x36\xc9\xda\x00\xb4" > "$zHex"
    hd "$zHex"

This does NOT store the value with null bytes into a variable. Instead, it uses a variable to store the name of a file. The file, like any other file, may contain null bytes and can be used over and over. The file itself will most likely never be physically written to the disk.

Via a trap, bash deletes the file automatically, so you need not worry about removing it manually unless you are creating an a crazy array of garbage. Due to RAM buffering, this technique is decently fast.

Answer 4

Since a carriage return can be part of a file name, I like to use null terminated lists. But I cannot store the sting with null bytes because bash because Bash stores strings as simple C strings where the null byte is the string terminator and therefore cannot be part of the string itself.

To get around the problem, I create an array of strings where the null byte is assumed to exist after each element. The list itself obviously contains null bytes. I store the value containing null bytes in an array like this...

    readarray -d $'\0' zArray < <(null_terminated_list_maker)

Then, I can reproduce the value with the null bytes like this...

    [[ "${zArray[*]}" ]] && printf '%s\0' "${zArray[@]}"

In this manner, a bash array can be used to store any value containing null bytes.

The purpose of the [[ "${zArray[*]}" ]] test is to see if the array has any values at all (the null string is a value). The test solves the problem where if an empty array passed to printf like this, then printf will print one null byte, which is wrong. It should print nothing.

When representing completely arbitrary data, there is a problem: Did your input actually terminate with the null byte? This method needs expanded to handle data which may or may not end with the null byte.