12

I am learning shell-scripting and for that I am using HackerRank. There is a question related to sed on the same site: 'Sed' command #1:

For each line in a given input file, transform the first occurrence of the word 'the' with 'this'. The search and transformation should be strictly case sensitive.

First of all I tried,

sed 's/the/this/'

but in that sample test case failed. Then I tried 

sed 's/the /this /'

and it worked. So, the question arises what difference did the whitespaces created? Am I missing something here?

Kusalananda
  • 320,670
  • 36
  • 633
  • 936
JHA
  • 127
  • 1
  • 5
  • I assume the first version also "worked", but not as you expected. It should have replaced the first occurrence of the letter sequence "the", but you probably looked at the first occurrence of the word " the ". – Dubu Apr 01 '19 at 08:40
  • Well, in thiseory, yes, in practice, no. – Rolf Apr 02 '19 at 10:40

3 Answers3

20

It's a cheap and error-prone way of doing word matching.

Note that the with a space after it does not match the word thereby, so matching with a space after the avoids matching that string at the start of words. However, it still does match bathe (if followed by a space), and it does not match the at the end of a line.

To match the word the properly (or any other word), you should not use spaces around the word, as that would prevent you from matching it at the start or end of lines or if it's flanked by any other non-word character, such as any punctuation or tab character, for example.

Instead, use a zero-width word boundary pattern:

sed 's/\<the\>/this/'

The \< and \> matches the boundaries before and after the word, i.e. the space between a word character and a non-word character. A word character is generally any character matching [[:alnum:]_] (or [A-Za-z0-9_] in the POSIX locale).

With GNU sed, you could also use \b in place of \< and \>:

sed 's/\bthe\b/this/'
Kusalananda
  • 320,670
  • 36
  • 633
  • 936
7

The difference is whether there is a space after the in the input text.
For instance:

With a sentence without a space, no replacement:

$ echo 'theman' | sed 's/the /this /'
theman

With a sentence with a space, works as expected:

$ echo 'the man' | sed 's/the /this /'
this man

With a sentence with another whitespace character, no replacement will occur:

$ echo -e 'the\tman' | sed 's/the /this /'
the     man
G-Man Says 'Reinstate Monica'
  • 22,130
  • 27
  • 68
  • 117
BDR
  • 266
  • 3
  • 7
  • I missed that. I had to take "the" as a string. Not a substring. – JHA Mar 31 '19 at 20:53
  • 1
    @JHA: It also matters at the end of a line. e.g. the word "the" could appear at the end of a line as part of a file with line wrapping, but still be in the middle of a paragraph and thus still be a normal word in an English sentence. `the( |$)` might be closer to working, if that Extended regex works. Anyway, IDK what you mean "as a string" vs. substring. In both cases it's a substring of the whole line, and your testcases are insufficient to detect the cases where `"the "` fails. Kusalanada's answer is significantly better, I'd recommend accepting it. – Peter Cordes Apr 01 '19 at 16:27
7

sed works with regular expressions. Using sed 's/the /this /' you just make the space after the part of the matched pattern.

Using sed 's/the/this/' you replace all occurrences of the with this no matter if a space exists after the.

In the HackerRank exercise, the result is the same because to replace the with this is logical... you replace just a pro-noun which by default is followed by space (grammar rules).

You can see the difference if you try for example to capitalize the in the word the theater:

echo 'the theater' |sed 's/the /THE /g'
THE theater                              
#theater is ignored since the is not followed by space

echo 'the theater' |sed 's/the/THE/g'
THE THEater
#both the are capitalized.
JHA
  • 127
  • 1
  • 5
George Vasiliou
  • 7,803
  • 3
  • 18
  • 42
  • Thank you for the answer. Appreciated :) – JHA Mar 31 '19 at 21:02
  • _"you replace all occurrences"_ To be clear: Without the `g` after the replacement text, you replace only the _first_ occurrence. – Dubu Apr 01 '19 at 08:41