20

Does anybody know why bash still has history substitution enabled by default? My .bashrc has included set +H for many many years but some other people are still getting bitten by this feature.

Given that pretty much everybody are using terminals with copy-paste features and bash compiled with readline library and history substitution is enabled by default only in interactive shells, is there really any reason to have the feature at all? None of the existing scripts would be broken even if this was disabled by default for all shells.

Try this if you do not know why history substitution is broken:

$ set +H # disable feature history substitution
$ echo "WTF???!?!!?"
WTF???!?!!?
$ set -H # enable feature history substitution
$ echo "WTF???!?!!?"
echo WTF???echo WTF???!?!!?
WTF???echo WTF???!?!!?

(Clearly the feature has major issues if it's disabled by default for all scripting and a feature exists to verify the results before executing: shopt -s histverify.)

See also:

Philippos
  • 13,237
  • 2
  • 37
  • 76
Mikko Rantalainen
  • 3,899
  • 1
  • 25
  • 32
  • 16
    You seem to think that people don't use history substitution. I use it every day. I find to quicker to follow `ls -l foo/bar/baz/weeble.cpp` with `less !$` than recall the command and edit it. – Martin Bonner supports Monica Apr 09 '18 at 12:55
  • 3
    @MartinBonner So you can enable it. The question is about why it's enabled by *default*, not why it still exists. – Barmar Apr 09 '18 at 15:35
  • 5
    @Barmar : Sure. But my point was to challenge the assumption that the question is predicated on. – Martin Bonner supports Monica Apr 09 '18 at 15:38
  • 5
    If we follow your reasoning, anything that requires escaping or quoting should be disabled by default. Why should I need to escape or quote an URL containing `&`? Why should I need to escape or quote a filename containing a `?`? There are UIs for people who think this is a problem. – jcaron Apr 09 '18 at 16:23
  • @jcaron I think the distinction being made is which of these features are more well known, as well as having more common alternatives. `!` history substitution is a historical relic from the days before interactive command editing, relatively few contemporary users know about it. Backgrounding and wildcards are used more frequently and don't have more popular replacements. – Barmar Apr 09 '18 at 16:33
  • 3
    I use `!$` several times a day, and `!!` quite often as well. I have to admit I don't use other history substitutions as much, but I would definitely be unhappy if the default behaviour of the shell I have been using for years suddenly changed. – jcaron Apr 09 '18 at 16:35
  • 2
    "Given that pretty much everybody are using..." This statement is almost never true. – barbecue Apr 09 '18 at 20:24
  • 1
    IMO the real question is why history substitution happens within (double) quotation marks. – Kevin Apr 09 '18 at 21:38
  • 1
    @Kevin it happens because bash implements the substitution on such a low level that it does not even work with multi-line input (e.g. have an open quotation mark and press enter and the substitution does not work on any later line). It's pure raw string replacement with no smarts of any kind. – Mikko Rantalainen Apr 10 '18 at 12:21

4 Answers4

32

If you are already familiar with bash, then dealing with history substitution patterns is not much more likely to bite you than handling any other characters that are special to this shell. However, if one is unfamiliar with the shell or just never used its history substitution features, it will obviously be a surprise when seemingly innocuous unquoted or double-quoted strings triggers it.

In an interactive shell with history substitutions enabled, the ! character is special in pretty much the same way as the $ character is special, i.e. everywhere unless escaped with \ or in single-quoted strings.

As opposed to $ through, history substitutions do not expand in here-documents, and since they are line-oriented, they additionally will happen on lines where the substitution falls within an unquoted context or a double quoted context (in that line when scanned separately). See this bug report for more info.

History substitution is disabled in non-interactive shells (scripts) because the shell's command history capability is not needed there, not because the feature has "major issues". In a script, saving every command to $HISTFILE makes no sense, and history substitution likewise is not something you'd want to rely on in a script.

Whether or not it should be enabled by default or not in interactive shells can be debated (though I'm not entirely convinced that a debate here would matter much to the bash developers). You seem to think that most bash users are having problems with history expansions, but neither one of you and me know how common it is to use them.

Unix shells allow one to modify the shell's behaviour to fit one's personal needs and taste. If you want to turn off history substitutions for all your interactive shells, continue doing what you are doing with using set +H in your ~/.bashrc file, or lobby the bash developers to change the default (which, I believe, would upset and confuse more people than it would help).

Kusalananda
  • 320,670
  • 36
  • 633
  • 936
  • 1
    The major difference between history substitution and `noglob` is that the former does not honor quotation whereas latter does. Compare `echo !!` vs `echo "!!"` and `echo *` vs `echo "*"`. I find it problematic that the exact same commands written on interactive shell do not work the same as in a shell script. – Mikko Rantalainen Apr 09 '18 at 11:39
  • 1
    @MikkoRantalainen That's a fair comparison. So, you seem to accept that $ is special but not that ! is special? – Kusalananda Apr 09 '18 at 11:49
  • 1
    Yeah, my point is that `!` is so rarely used special character that only works with interactive shells that it should be enabled only if requested by the user. IMHO, those that really want to use history substitution know a way to enable that feature, the rest of the world is better served with keeping `!` as a regular character (which is the way it already behaves in bash scripts by default!). – Mikko Rantalainen Apr 09 '18 at 11:53
  • 2
    @MikkoRantalainen You seem to think that history substitution is not something that majority of `bash` users are using. I don't think either of us knows how common it is. I'd suggest that if you feel strongly about this, you submit a feature/bug request to the `bash` mailing list. See https://savannah.gnu.org/mail/?group=bash – Kusalananda Apr 09 '18 at 12:05
  • 8
    Right. The problem is not actually about history substitution at all, but about the fact that double “quotes” in Bash aren't really quotes in the sense other languages use the term, but rather work like a special kind of parentheses. @MikkoRantalainen, make it a habit to put anything that you don't want Bash to interpret in any special way in _single_ quotes! – leftaroundabout Apr 09 '18 at 13:39
  • 1
    @leftaroundabout What other languages? – Kusalananda Apr 09 '18 at 13:43
  • @Kusalananda off the top of my head: C, Fortran, BASIC, Python, Java, Haskell, Scheme, Common Lisp, Rust... _most_ languages. None of these just go interpreting stuff in quotes as syntax, at least not unless you explicitly request some kind of `eval`. – leftaroundabout Apr 09 '18 at 13:46
  • 1
    Use simple quotes `'` if you want to avoid special characters being interpreted by anything. Double quotes `"` have double meaning, where simple quotes are so simple that they don't allow interpreting the contents of the quote. – Mio Rin Apr 09 '18 at 14:02
  • 5
    @leftaroundabout It depends on what you mean by interpreting. Most languages interpret control characters in strings for output (but granted, not in the same way as in the shell when just shuffling strings around). Perl's quoting rules additionally works pretty much like the shell's (variable interpolation). As with any programming language, it helps if one is able to remember what language one is currently working in. – Kusalananda Apr 09 '18 at 14:05
  • 5
    "_my point is that ! is so rarely used **special** character_" I'd say it's the other way around... the `!` is so rarely used as a **normal** character that – echoing WTFs aside – the number of people getting caught out is far outweighed by the number of people who still use it to substitute history. – TripeHound Apr 09 '18 at 15:46
  • 3
    -1 for deflecting the blame onto improper quoting. Since `!` is *not* special to *most* Bourne-like shells, nor in `bash` scripts, there are many paths by which people can reasonably learn that `"$my_var some text!!"` will expand the `$my_var` but not the `!!`. I got familiar with Bourne-like shell on `busybox ash`, and my first encounter with `bash` history substitution *was* me being bitten by it when I was trying to do some quick one-liner in an interactive shell. People proficient in portable and scripting use of Bourne-like shells are often bitten by it at least once when using `bash`. – mtraceur Apr 09 '18 at 17:14
  • @mtraceur I'm sorry, but I believe this was specifically tagged with `bash`. I would not go off on a rant about C++ when I discover that I can't use a variable called `this` or `new`, just because I'm used to C and "C and C++ are similar" and "I learnt C first". – Kusalananda Apr 09 '18 at 17:21
  • 1
    +1. There's also `!n` where `n` is the line number of a history entry. I use this all the time after `history | grep ` - useful when bash's fixed-string Ctrl-R recall isn't good enough (also when the exact command i want is a few thousand history entries in the past). and `!-n` for the nth most recent command. – cas Apr 09 '18 at 17:49
  • 1
    Also: `!` where `` is the start of a command line. This is another one i use routinely. e.g. somewhere in root's history on all my machines is the command line `dselect update ; apt-get -V -d -u dist-upgrade`. Since `dselect` is the only command I ever use that begins with `dsel`, typing `!dsel` is guaranteed to run that command-line. That's reason enough to keep `dselect` installed even though it's obsolete and has been for at least 10 or 15 years. I could maybe use `!apt` but I run a lot of commands beginning with `apt`, so I'd never be sure what I was about to run – cas Apr 09 '18 at 18:05
  • 1
    @Kusalananda Didn't my comment apply equally to `bash`-only users who got familiar with what-character-is-special rules in *scripting* rather than *interactive* use cases? (Responding to the rest of the reasoning implicit within your response would be outside the intended scope of stackexchange comments, so I'll just say this: is it constructive to dismissively/disdainfully tell people "you weren't really bitten by [counter-intuitive and non-obvious feature that you could reasonably have not expected], you were bitten by your failure to rtfm"?) – mtraceur Apr 09 '18 at 18:55
  • @mtraceur If you can give me some constructive critique that would help me improve my answer, that would be welcomed. Is it the too generic language used in the first paragraph? – Kusalananda Apr 09 '18 at 19:07
  • 1
    @Kusalananda Well, if I understand you correctly, the *essence* of your answer's first sentence is: "If you are already familiar with `bash`, this feature is not much more likely to bite you than any other characters that are special to the `bash` shell." So what do you think about that wording or something like it? I think it would help convey the core *reason* for why the feature persists (because once you do know about it, you don't usually have issues with it, and it *is* useful), without coming off as dismissive of the surprising frustration of invoking the feature accidentally. – mtraceur Apr 10 '18 at 16:41
  • @mtraceur Thanks! Yes, I will modify the answer to include a similar formulation. – Kusalananda Apr 10 '18 at 16:44
  • @Kusalananda Great! I've reversed my previous downvote in light of your edit. – mtraceur Apr 10 '18 at 16:54
  • I agree that discussion here will not help, but I strongly disagree to compare history expansion in bash to other special characters, as it is (maybe the only) line-oriented feature, surprising even experienced shell users: https://unix.stackexchange.com/questions/390931/bash-history-expansion-inside-single-quotes-after-a-double-quote-inside-the-sam – Philippos Apr 27 '18 at 11:09
  • @Philippos I saw your edit to the question and your answer to the other question, and I agree. I would regard the behaviour as a bug though, but I will nevertheless make an adjustment to my answer in light of this (in a few hours). – Kusalananda Apr 27 '18 at 11:20
9

History substitution is useful. Take for example

% make-me-a-sandwich
make-me-a-sandwich: Permission denied
% sudo !!
Ok.
Johan Myréen
  • 12,862
  • 1
  • 32
  • 33
  • 4
    So? If you find it useful you can enable it in your `.bashrc`. – Barmar Apr 09 '18 at 15:37
  • 3
    https://xkcd.com/149/ – Walter Tross Apr 09 '18 at 17:03
  • 3
    @Barmar and if you find it un-useful you can disable it. Funny how symmetry works. – hobbs Apr 09 '18 at 19:39
  • 3
    @hobbs If you don't know that it exists, how would you know to disable it? – Barmar Apr 09 '18 at 19:48
  • 2
    This answer does a good job of explaining one reason why the feature is useful. I think OP wants to understand *why* this usefulness is *sufficient* to justify this behavior, even though OP (and others, such as the people asking several of the related questions on this stackexchange) find this behavior surprising/unexpected. – mtraceur Apr 10 '18 at 16:46
  • 2
    ctrl-P ctrl-A and typing `sudo ` does the same with the main difference that you see what you are doing. And without the need to remember a special feature with a broken implementation. – Philippos Apr 27 '18 at 11:15
4

Social/cultural inertia.

This question is in the how-humans-work problem space, so I'm going to answer from that angle, without putting forth any opinion about whether or not the feature ought to be on by default.

To start, to make sure you understand the other side, consider that the annoyance you feel about having to go out of your way to turn off the feature, is the annoyance they would feel if they had to go out of their way to turn on the feature.

Combine the above with the fact that enough bash users do use the feature, suggestions of removing it or turning it off by default are met with resistance from the people who are already comfortable with it being there by default.

Also, bash is the default shell for many people (not just in the default login or system shell sense, but in a psychological sense). If your reference frame for shell quoting is bash, if that's the shell you learned first, the fact that ! is a special shell character will feel natural and automatic to you (or at least, when you first learn it, it'll be just part of the way the shell is, just one quirk to accept among many).

And if you think about it, a lot of bash users probably encounter the history substitution syntax in a positive context: they read about it or someone shows it to them and they see the possible usefulness of it, when they're first learning bash.

It's only coming from the peripheral world of other Bourne-like shells, that you'd be bitten by ! being special and thus be inclined to view it negatively: Because if you're used to shells where you never had the feature, then your first exposure to it will be when it screws you when you're trying to get something done in a hurry.

TL;DR: Most users probably don't strongly care what the default is either way, some users like the feature and have the strong advantage of it already being that way, and there hasn't been enough people actively advocating against the feature to overcome that.

mtraceur
  • 1,146
  • 9
  • 14
  • 2
    No, it's not only when coming from other shells. I've known this feature for years in `bash` and still get bitten by surprise, as single quoting [doesn't always work](https://unix.stackexchange.com/questions/390931/bash-history-expansion-inside-single-quotes-after-a-double-quote-inside-the-sam) on `!` because of the broken `bash`implementation. – Philippos Apr 27 '18 at 11:19
  • @Philippos That's terrifying (well, not exactly terrifying... but some sort of distressing negative qualia). Thanks for pointing this out. I will try to revisit this answer later to work your point into my answer once I've thought of a good way to integrate it. – mtraceur Apr 27 '18 at 20:32
  • This answer is really well written to avoid annoying anybody. My problem with this feature is the fact that `!` means different thing in interactive shell vs scripting. As far as I know, this is the only feature which has different behavior for the same commands in interactive mode vs in scripts. – Mikko Rantalainen Feb 17 '22 at 19:10
2

Why is bash history substitution still enabled by default?

Because many people use it, and people using an interactive bash shell probably should know the rules to avoid problems and will generally find it helps more than it hurts.

My .bashrc has included set +H for many many years but some other people are still getting bitten by this feature.

So it's a feature you don't use, that doesn't mean a majority of users don't use it. You can petition for the default to be changed, but you have to look at A) the portion of people that care, and B) the ratio of people that prefer it your way. The people that care and don't like it probably already have it disabled. Changing it would help when they get an account on a new computer. People that care and do like it would have to update their settings on every computer they use and every account they get in the future.

Given that pretty much everybody are using terminals with copy-paste features

A clunkier option in my opinion...

is there really any reason to have the feature at all? None of the existing scripts would be broken even if this was disabled by default for all shells.

Yes, people find it useful. What's the use in having a remote control given that you used to be able to just get up and change the channel?

(Clearly the feature has major issues if it's disabled by default for all scripting and a feature exists to verify the results before executing: shopt -s histverify.)

History doesn't really make sense in scripts, but more importantly it could cause security issues. In your case you could avoid the problem by using single quotes. I don't remember this ever causing an issue for me, so I don't know how you can say it has 'major issues'. Has this caused an actual problem for you, or were you annoyed at having to set your defaults on a new computer?

I don't see how it is any different than having to escape or use single quotes in this if you actually want to get some money:

$ echo "Give me $50 or the cat gets it"
Give me $0 or the cat gets it
Jason Goemaat
  • 121
  • 3
  • 2
    See the related questions (right column); most questions are hitting this feature by accident and assume that the shell is broken. I personally learned about this feature a long time ago due the fact that I accidentally used character sequence that caused wrong output. I would have been saved by `shopt -s histverify` but *that* was not on by default. In addition, figuring the cause by oneself is pretty hard because the error message is cryptic ("event not found") and the character sequence that triggers it is hard to google for. – Mikko Rantalainen Apr 10 '18 at 12:32
  • 1
    People get genuinely bitten by this feature because the "universal" shell syntax features like `$` being special inside `"..."` are one of the first things people learn about shell, while the fact that `!` is special inside `"..."` (but only in interactive mode) is less broadly known, typically not taught as immediately/prominently, and is `bash`-specific. Also, Bourne-like shells mostly have a symmetry between text-in-a-script and text-on-the-interactive-shell, so you can generally copaste from one to the other and have it work the same - and this feature creates one of the only exceptions. – mtraceur Apr 10 '18 at 16:19
  • 1
    *I don't see how it is any different than having to escape or use single quotes* True for `zsh`, when single-quoting `!` always works, but not for `bash`, where the implementation produces [unexpected results](https://unix.stackexchange.com/questions/390931/bash-history-expansion-inside-single-quotes-after-a-double-quote-inside-the-sam). You say, we *should know the rules*, well, did you know that linked rule? – Philippos Apr 27 '18 at 11:28