15

The following code best describes the situation.  Why is the last line not outputting the trailing newline char?  Each line's output is shown in the comment.  I'm using GNU bash, version 4.1.5 

     echo -n $'a\nb\n'                  | xxd -p  # 610a620a  
           x=$'a\nb\n'   ; echo -n "$x" | xxd -p  # 610a620a
     echo -ne "a\nb\n"                  | xxd -p  # 610a620a
x="$(echo -ne "a\nb\n")" ; echo -n "$x" | xxd -p  # 610a62
Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
Peter.O
  • 32,426
  • 28
  • 115
  • 163
  • 5
    Workaround for the rare times when it's needed: `tmp=$(somecommand; echo a); tmp=${tmp%a}` – Gilles 'SO- stop being evil' Jul 31 '11 at 21:08
  • See also [Why does shell Command Substitution gobble up a trailing newline char?](http://unix.stackexchange.com/questions/17747/why-does-shell-command-substitution-gobble-up-a-trailing-newline-char) – Gilles 'SO- stop being evil' Jul 31 '11 at 21:11
  • 2
    @Gilles: Re your example, above: `tmp=$(somecommand; echo a)` ... This has certainly driven the point home... Until I saw the example, my tendancy would still have been to use `echo -n a`... but, of course!, there is no need of the `-n`, because Command Substitution will remove the introduced trailing newline in any case! ... thanks... – Peter.O Jul 31 '11 at 22:23
  • https://stackoverflow.com/questions/613572/capturing-multiple-line-output-into-a-bash-variable – sancho.s ReinstateMonicaCellio Sep 14 '18 at 12:40

2 Answers2

19

The command substitution function $() (and its cousin the backtick) specifically removes trailing newlines. This is the documented behavior, and you should always be aware of it when using the construct.

Newlines inside the text body are not removed by the substitution operator, but they may also be removed when doing word splitting on the shell, so how that turns out depends on whether you used quotes or not. Note the difference between these two usages:

$ echo -n "$(echo -n 'a\nb')"
a
b

$ !! | xxd -p
610a62

$ echo -n  $(echo -n 'a\nb')
a b

$ !! | xxd -p   
612062

In the second example, the output wasn't quoted and the newline was interpreted as a word-split, making it show up in the output as a space!

Caleb
  • 69,278
  • 18
  • 196
  • 226
  • 1
    Thanks Caleb.. I was aware of word-splitting changing whitespace to a single space when *unquoted*... That is why I was most surprised to see my trailing newline disappear even though I had quoted it... Now I'm aware that it is because of the 'normal' behaviour of *Command Substitution* dropping a trailing newline... Oh well, c'est la vie.. and thanks for the link – Peter.O Jul 31 '11 at 11:21
6

When using command substitution, the shell executes the commands in a subshell, returning their stdout. in this process, the IFS characters loss their significance (if they are not quoted), as the commend returns plain split words, so the trailing ones are removed. For example:

$ echo "$(echo -e '\n')" | wc
 1       0       1

$ echo -e '\n' | wc
2       0       2

and more practically, pwd will work even if your directory name has a newline in between, but $(pwd) will not.

The usual workaround is to add something at the end of your command and strip it thereafter.

Philomath
  • 2,827
  • 1
  • 22
  • 21
  • 1
    Thanks Philomath... by the way, you first example is syntactically wrong ('wc' doesn't accept a string as an arg).. For you examples to make sense, the first one could be `echo "$(echo -e '\n')" | wc`, which outputs `1   0   1`, compared to the `2   0   2` – Peter.O Jul 31 '11 at 10:58
  • @fred: Oops, just a typo. – Philomath Jul 31 '11 at 11:00
  • 1
    "converted to spaces" is not the appropriate explanation, there is just some splitting involved. In particular you can remove space from IFS and those will not act as separators. More over, your example falls into a special case category, and there is a difference between `$(pwd)` and `"$(pwd)"`, see Caleb's answer. – Stéphane Gimenez Jul 31 '11 at 11:08
  • PS..Your suggestion of the usual workaround is just what I needed to clear this one up.. – Peter.O Jul 31 '11 at 11:19
  • Re "converted to spaces", see [Word Splitting](http://www.gnu.org/software/bash/manual/bashref.html#Word-Splitting) – Peter.O Jul 31 '11 at 12:13
  • I knew that. maybe it was inaccurate, but you can't call that incorrect or non-useful. – Philomath Jul 31 '11 at 13:10