Why is sed giving me an error about an unterminated `s'?

Question

I have a set of sed replacements in a bash script and am getting an error about an unterminated `s' command. Here's what the sed line looks like:

sed -n -e "s/TMPFOO1/$FOO1/" -e "s/TMPFOO2/$FOO2/" -e "s/TMPFOO3/$FOO3/" -e "s/TMPFOO4/$FOO4/" -e "s/TMPFOO5/$FOO5/" /home/foo/template > /home/foo/template/finishedresult

For some reason though, bash doesn't like this and I get an error about

sed: -e expression #4, char 69: unterminated `s' command

What am I missing here? How can I have SED input the variable? It looks to me like they are all terminated.

$FOO4 contains something weird (ex: "\" or "\\" ?) that makes the sed s command not terminated. (That's why I think it's a trailing "\", making the next "/" be treated as a character part of the replace string, instead of as the terminating character for the s command) — Olivier Dulac, May 10 '13 at 07:42

Gilles 'SO- stop being evil' · Accepted Answer · 2014-02-06T10:12:36.413

15

You cannot safely interpolate a variable in a sed command, because the substitution is performed by the shell, not by sed. The value of the variable becomes sed syntax. For example, in "s/TMPFOO1/$FOO1/", if $FOO1 contains a newline, this will cause a syntax error like the one you observed. If $FOO1 contains a /, this terminates the s command and may cause an error or may cause other commands to be executed (if what's after the / happens to be valid sed syntax).

While you can do a first pass of substitution of FOO1 to quote its special characters for inclusion in that sed command, it's a lot simpler to use awk. Awk has a notion of variable, and a command line syntax to set the initial value of a variable.

awk -v FOO1="$FOO1" -v FOO2="$FOO2" -v FOO3="$FOO3" -v FOO4="$FOO4" -v FOO5="$FOO5" '{
    sub(/TMPFOO1/, FOO1);
    sub(/TMPFOO2/, FOO2);
    sub(/TMPFOO3/, FOO3);
    sub(/TMPFOO4/, FOO4);
    sub(/TMPFOO5/, FOO5);
    print;
}' /home/foo/template > /home/foo/template/finishedresult

edited Feb 06 '14 at 10:12

answered May 13 '13 at 01:21

Gilles 'SO- stop being evil'

807,993
194
1,674
2,175

Trivial note: - a `-v` needs to precede each variable declaration. – Mike B Feb 06 '14 at 06:01
@MikeB Thanks for the bug report. In general, please [edit](http://unix.stackexchange.com/posts/75581/edit) the post directly when you find typos like this. – Gilles 'SO- stop being evil' Feb 06 '14 at 10:13
@Gilles Edits from non-authors must be at least six characters. – NobleUplift Feb 06 '14 at 22:09
What does it help if the $FOO1 has a \n in it. Init it will not remove the \n. I read line by line in bash script and replace some lines like `sed "s/$li/- [x]${li:5}/" $dat` .Should I stay with the script and use awk, or find out how to work with \n in sed or better move to e.g.python? – Timo Nov 18 '20 at 18:42
1

@Timo I don't understand the first two sentences of your comment. If you want to replace strings containing newlines, sed is very hard to work with. Awk is easier: set a value of `FS` that doesn't match anything that `$FOO1` matches. – Gilles 'SO- stop being evil' Nov 18 '20 at 20:11
Thanks Gilles, [here](https://github.com/tik9/ml/blob/master/solu.sh) is the sh code with sed line 42, [here](https://github.com/tik9/ml/blob/master/solu.py) is the py code. The basis [here](https://github.com/tik9/pluralsight-skill-test/blob/master/angular.md). As you see, text is read that can have special characters as it is code (angular). Maybe not use awk but stay with py? – Timo Nov 19 '20 at 08:08
1

@Timo I can't answer that in a comment. Ask a new question. – Gilles 'SO- stop being evil' Nov 19 '20 at 08:34
Bewary of sed's special characters that need to be escaped. See also: https://unix.stackexchange.com/questions/32907/what-characters-do-i-need-to-escape-when-using-sed-in-a-sh-script – randomScott Nov 23 '21 at 01:05

score 6 · Answer 2 · answered May 10 '13 at 05:16

Most likely one of your $FOO variables contains special characters that are interpreted by sed.

I have another version of sed which generates other error messages but here is an example of a similar problem:

$ VAR=a
$ echo i | sed -e "s/i/"$VAR"/"
a
$ tmp> VAR=/
$ echo i | sed -e "s/i/"$VAR"/"
sed: 1: "s/i///
": bad flag in substitute command: '/'

In this case $VAR contains a character which is interpreted by sed as the trailing slash.

Rany Albeg Wein · Answer 3 · 2013-05-10T19:10:51.043

As others have mentioned here, it depends on the content of your FOO* variables. In your case, using sed is the wrong choice because it probably contain some special characters.

Take a look at this link and see the function gsub_literal

Since gsub_literal reads from stdin and writes to stdout, the usage will be:

gsub_literal "$search" "$replace" < /home/foo/template > /home/foo/template/finishedresult

Sample output:

rany$ cat > foo.txt
a'
a'

rany$ gsub_literal a\' b < foo.txt 
b
b

score 1 · Answer 4 · edited Sep 29 '15 at 20:48

The problem has been resolved for me by trying additional backslashes within the sed 's/.../.../g' command. This command

sed 's/\"a/ä/g' input_file

gave me this error message

sed: -e expression #1, char 9: unterminated `s' command

when it was executed as part of a c-shell script, and it did so only with a recent version of sed, not with an older version. (Strangely, the same command works fine when it is given on the command line (in a tc-shell)).

The problem with the c-shell script has been remedied by preceding the ä character by a backslash:

sed 's/\"a/\ä/g' input_file

Why is sed giving me an error about an unterminated `s'?

4 Answers4

Linked