Bash option to make command substitution preserve trailing newlines

Question

I'm hoping that bash comes with an option that stops command substitution from stripping trailing newline characters. Is there? If there isn't, then do any shells exist that are bash-like, and for which standard—POSIX, I guess—sh code would run in, that has such an option or perhaps some special syntax for command substitution that doesn't strip newlines (the latter would be preferable)?

Answer 1

There's no option to do this directly (that I know of), but you can fake it by adding a protective non-newline character in the command substitution, and then removing that afterward:

var=$(somecommand; echo x)    # Add an "x" (and one newline that command
                              # substitution will remove) *after* the
                              # newline(s) we want to protect

var=${var%x}    # Remove the "x" from the end

Note that the exit status of somecommand is lost in the process, since both echo x and var=${var%x} set the status (normally to 0). If you need to preserve it until the end, you need to add a little juggling (w/ credit to Kusalananda):

var=$(somecommand; err=$?; echo x; exit "$err")    # Make the subshell
                                                   # exit with somecommand's
                                                   # status
commandstatus=$?    # Preserve the exit status of the subshell
var=${var%x}
# You can now test $commandstatus to see if the command succeeded

Note that the err and commandstatus variables both store the command's exit status, but they aren't redundant, since err only exists in the subshell created by $( ), and commandstatus only exists in the parent shell. You could use the same name for both, but that might add confusion.

BTW, if you don't need to preserve the status until after the "x" is trimmed, you can skip the commandstatus part:

if var=$(somecommand; err=$?; echo x; exit "$err"); then
    # somecommand succeeded; proceed with processing
    var=${var%x}
    dosomethingwith "$var"
else
    echo "somecommand failed with exit status $?" >&2
fi

Answer 2

The only shells that I know that have a command substitution that can be told not to remove trailing newline characters are rc (the shell of Unix V10 and plan9, and Byron Rakitzis's public domain clone and its derivatives such as es or akanga) and fish.

In rc-like shells,

var = `{somecommand}

Stores the $ifs delimited words in the output of somecommand into the $var variables (and variables are arrays/lists in rc).

With an empty $ifs, (ifs = ()), that stores the output as is. The separators can also be specified with the ``(separators){somecommand}, so:

var = ``(){somecommand}

But rc/es/akanga are not very Bourne-like.

Also, you generally want to remove one newline character.

For instance in:

dirname=$(dirname -- "$file")

You do want to remove the newline character added by dirname, but not the ones that may be at the end of $file's directory.

rc has no builtin operator for that. You'd need:

dirname = ``(){dirname -- $file | head -c -1}

(not in standard head). In es, you can use the ~~ pattern extraction operator:

nl = '
'
dirname = <={ ~~ ``(){dirname -- $file} *$nl }

Which is hardly more legible (though could be made into a function).

The fish shell command substitution splits the output into its constituent lines.

set var (somecommand)

Stores the lines of somecommand's output into the $var array (and empty lines are preserved even those at the end of the output).

If you set $IFS to the empty list or one empty string, that splitting is disabled and up to one newline character is removed from the end of the output (at least in current versions of fish, IIRC there have been several changes on that front over the years). So there,

begin; set -l IFS; set dir (dirname -- $file); end

is reliable.

I'm not aware of any Bourne-like shell that can be told not to strip all trailing newline characters. That behaviour is required by POSIX and all Bourne-like shells do it even when not in posix mode (for those that have one like bash or zsh).

In bash 4.4+, in place of command substitution, you can also use readarray combined with process substitution (or a pipe with the lastpipe option and in non-interactive instances):

readarray -td '' var < <(somecommand)

Here -d '' is to splits on NULs. bash doesn't support storing NUL in its variables anyway, so for a replacement of command substitution that would be enough.

readarray -t lines < <(somecommand)

Stores that lines of the output of somecommand into the $lines array. You could then join them with newline to reconstruct that output without one trailine newline:

IFS=$'\n'
output="${lines[*]}"

But in all those approaches, you lose the exit status of somecommand.

To work around that misfeature of command substitution removing all newline characters, a common idiom is to add a non-newline character and stripping it (along with one newline character) afterwards as Gordon said.

But to do it without losing the exist status, you'd so something like:

cmdsubst() { # args: var cmd args
  eval "$1"'=$(shift; "$@"; ret=$?; printf .; exit "$ret")'
  eval "$1=\${$1%.}; $1=\${$1%'
'}; return $?"
}

Which is like command substitution except that only up to one newline character is removed.

To be used as:

cmdsubst dir dirname -- "$file"

in place of:

dir=$(dirname -- "$file")

Or for more complex commands:

cmdsubst var eval 'cmd1; for i in a b; do cmd "$i"; done'