16

I realize that ! has special significance on the commandline in the context of the commandline history, but aside from that, in a runing script the exclamation mark can sometimes cause a parsing error.
I think it has something to do with an event, but I have no idea what an event is or what it does.  Even so, the same command can behave differently in different situations.
The last example, below, causes an error; but why, when the same code worked outside of the command substitution? .. using GNU bash 4.1.5 

# This works, with or without a space between ! and p
  { echo -e "foo\nbar" | sed -nre '/foo/! p'
    echo -e "foo\nbar" | sed -nre '/foo/!p'; }
# bar
# bar

# This works, works when there is a space between ! and p
  var="$(echo -e "foo\nbar" | sed -nre '/foo/! p')"; echo "$var"
# bar

# This causes an ERROR, with NO space between ! and p
  var="$(echo -e "foo\nbar" | sed -nre '/foo/!p')"; echo "$var"
# bash: !p': event not found
Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
Peter.O
  • 32,426
  • 28
  • 115
  • 163
  • 1
    See http://unix.stackexchange.com/questions/3747/understanding-the-exclamation-mark-in-bash/3748#3748 – Warren Young Aug 23 '11 at 12:37
  • @Warren.. Thanks. I had seen that QA, but it really only talks about how to escape the backslash... My issue relates more to why the *seemingly* already escaped code works in one situation and not another... – Peter.O Aug 23 '11 at 12:51
  • @fred: "seemingly already escaped"? I don't see any escapes at all and you are using double quotes. See my (revised) answer. What part do you think is escaped? – Caleb Aug 23 '11 at 13:07
  • @Caleb. Yes, I used the wrong term.. `protected` would have been more appropriate. (protected by 'single quotes') – Peter.O Aug 23 '11 at 13:17
  • If you are only concerned with simple assignments, then you can use `var=$(…)` (no double quotes), and it will work like (I think) you expect. This is still “safe” because the value part of a simple assignment is not subject to word splitting or globbing (though this may not be true of assignments done through builtins (e.g. `export`, `local`, etc.) under all shells). Unfortunately, this does not extend beyond simple assignments since the double quotes are the way to protect against word splitting and globbing while still getting other types of expansion in other contexts. – Chris Johnsen Aug 24 '11 at 03:37

3 Answers3

13

The ! character invokes bash's history substitution (enabled by default in interactive shells). When followed by a string (as in your failing example) it tries to expand to the last history event that began with that string. Just like $var gets expanded to the value of that string, !echo would expand to the last echo command in your history.

Space is a breaking character in such expansions. First note how this would work with variables:

# var="like"
# echo "$var"
like
# echo "$"
$
# echo "Do you $var frogs?"
Do you like frogs?       <- as expected, variable name broken at space
# echo "Do you $varfrogs?"
Do you?                  <- $varfrogs not defined, replaced with blank
# echo "Do you $ var frogs?"
Do you $ var frogs?      <- $ not a valid variable name, ignored

The same thing will happen for history expansion. The bang character (!) starts off a history replacement sequence, but only if followed by a string. Following it with a space make it literal bang instead of part of a replace sequence.

You can avoid this kind of replacement for both variable and history expantion by using single quotes. Your first examples used single quotes and so ran fine. Your last examples are in double quotes and thus bash scanned them for expantion sequences before it did anything else. The only reason the first one didn't trip is that that the space is a break character as shown above.

ilkkachu
  • 133,243
  • 15
  • 236
  • 397
Caleb
  • 69,278
  • 18
  • 196
  • 226
  • Thanks Caleb.. Another of my pre-conceptions dismissed... I thought that bash parsing was done from the innermost bracket or brace, and then worked outwards... It seems that bash parses differently to my assumption. – Peter.O Aug 23 '11 at 13:01
  • 1
    Quoting is confusing enough in bash without it changing in nested strings. As it is, the replacements happen very early on in the process. Consider this example: `var=word; echo "test '$var'"; echo 'test "$var"'` – Caleb Aug 23 '11 at 13:12
  • .. Yes undestood. I was aware of nesting quotes within quotes... My misunderstanding was that I thought that the code within the brackets of the command substitution would be parsed seperately, to what surrounds those brackets; but apparently not.. thanks. – Peter.O Aug 23 '11 at 13:30
6

As already said by Caleb, ! is used to invoke bash's history substitution.

If like me you feel you don't need such a feature, you can disable it inserting the following line in ~/.bashrc:

set +H

I don't need it because the history can be recovered by the up arrow and Ctrl-r incremental reverse search. See bash's manual page, section Commands for Manipulating the History for a detailed list of shortcuts.

Community
  • 1
enzotib
  • 50,671
  • 14
  • 120
  • 105
  • 2
    How do you live without `!!`? – Caleb Aug 23 '11 at 12:54
  • Thanks., I would think that could be a problem, portability-wise, but using `set +H` in the script works just as well :) +1 – Peter.O Aug 23 '11 at 12:57
  • 2
    @fred: strange, generally history expansion is "on" only for interactive shells. – enzotib Aug 23 '11 at 13:03
  • @enzo.. Thanks again.. I had tested it from the commandline.. Ah! If learning wasn't so much fun, it would be tedious... did I mention coffee? it helps too :) – Peter.O Aug 23 '11 at 13:08
  • Ya, that's a gotcha. I've copy pasted code from scripts that failed on the command line for just this reason. History expansion wasn't a concern in the script but it is on an interactive shell. – Caleb Aug 23 '11 at 13:16
3

your first example:

{ echo -e "foo\nbar" | sed -nre '/foo/! p'
    echo -e "foo\nbar" | sed -nre '/foo/!p'; }

could be reduced to 

echo '! p' 
echo '!p'

Within single quotes, all characters preserve their literal values. Thus ! has lost its special meaning and history expansion is not preformed.

your second and third examples:

var="$(echo -e "foo\nbar" | sed -nre '/foo/! p')"; echo "$var"

var="$(echo -e "foo\nbar" | sed -nre '/foo/!p')"; echo "$var"

could be reduced to

echo "'! p'"

echo "'!p'"

'! p' and '!p' are essentially parts of the double quoted strings.

Within double quotes, all characters preserve their literal values except $, `, \ and !.

That implies the single quotes of '! p' and '!p' have lost their special meanings (i.e.: unable to escape !) but ! still retains its special meaning thus history expansion is performed.

However, when ! is followed by a space character, history expansion is not performed.

Quoting from man bash:

QUOTING

[...]

Enclosing characters in single quotes preserves the literal value of each character within the quotes. [...]

Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of $, `, \, and, when history expansion is enabled, !. [...] If enabled, history expansion will be performed unless an ! appearing in double quotes is escaped using a backslash. The backslash preceding the ! is not removed.

HISTORY EXPANSION

[...]

History expansions are introduced by the appearance of the history expansion character, which is ! by default. Only backslash (\) and single quotes can quote the history expansion character.

Several characters inhibit history expansion if found immediately following the history expansion character, even if it is unquoted: space, tab, newline, carriage return, and =. If the extglob shell option is enabled, ( will also inhibit expansion.

cychoi
  • 435
  • 3
  • 13