sed : have a range finishing with the last occurrence of a pattern (greedy range)

Question

Take the following file :

$ cat f1
stu vwx yza
uvw xyz abc
abc def ghi
def ghi jkl
ghi jkl mno
jkl mno pqr
mno pqr stu
pqr stu vwx
stu vwx yza

To print all lines from the first one containing abc to the first one containing mno with GNU sed :

$ sed -n '/abc/,/mno/p' f1
uvw xyz abc
abc def ghi
def ghi jkl
ghi jkl mno

How could I print all lines until the last one containing mno, e.g. how could I get the following result :

uvw xyz abc
abc def ghi
def ghi jkl
ghi jkl mno
jkl mno pqr
mno pqr stu

In other words, is there a way to make GNU sed's range selection greedy ?

Update

In my setting :

• If mno is missing, it should print out everything until the end of the file.
• mno cannot occur before the first abc.
• There's always at least one abc, and abc and mno are never on the same line

EDIT I just added a dummy stu vwx yza line at the start, so that the file doesn't start with a line including abc (to avoid solutions that start from the first line - they should start from the first line having abc in it)

Answer 1

You could use awk if it is an option. You could mark the lines for pattern start and pattern stop and print those lines in one-pass of the file (involves storing lines starting from the first line containing abc till the last line in a buffer)

awk '/abc/ && !start {
  start = NR
}
/mno/ {
  stop = NR
}
start { line[NR] = $0 }
END {
  if ( !stop ) {
    stop = NR
  }
  for ( s = start; s <= stop; s++ )
    print line[s]
}' file

Note that this will not work, when the start pattern is not present, printing only a series of blank lines.

Answer 2

sed '/abc/,$!d;0,/mno/b;:1;/mno/b;$d;N;b1' file

Work algorithm:
Two address ranges are used.
The first /abc/,$!d; removes everything up to the first pattern match.
The second 0,/mno/b; up to a match with the pattern /mno/, sends each line buffer(pattern space) to the output bypassing the remaining script, thereby preventing the deletion if the pattern is not found in the file.
The rest of the script :1;/mno/b;$d;N;b1 works in a loop. In the editor buffer, lines are appended until a pattern match occurs. If a /mno/ pattern is encountered, the entire buffer is sent to the output, bypassing the rest of the script. If no match occurs, the buffer is deleted at the last line.

Answer 3

Another awk solution with less buffering:

awk '!f&&/abc/{f=1} f==1; f==2{buf=buf $0 ORS} f&&/mno/{f=2; printf "%s",buf; buf=""}' input.txt

• This will print everything starting with the first occurence of abc (where it sets a flag f to 1) up to and including the first occurence of mno. The f==1 statement outside of the rule blocks instructs awk to print the current line as long as f is set to 1.
• Then, the content of all lines after each occurence of mno (where f now has the value 2) is stored in a buffer buf, which is printed and cleared at the next occurence of mno. To ensure we correctly cope with situations where the first mno occurs before the first abc, we demand that f be set at least to 1 before applying that logic.

It will therefore store at most the text between two occurences occurence of mno, or the last occurence of mno and end-of-file (only that the latter part will never be printed).

If you want to exchange speed with memory efficiency, the following two-pass method won't rely on buffering at all:

awk 'FNR==NR{if (/abc/&&!start) {start=FNR} else if (/mno/) {end=FNR}; next} FNR>=start&&(!end||FNR<=end)' input.txt input.txt

This will process the file twice (hence it is specified twice as argument).

• The first time, when FNR, the per-file line-counter is equal to NR, the global line counter, we look for the first occurence of abc and the last occurence of mno, and store their line numbers in start and end, respectively.
• In the second pass, we print lines as long as the FNR counter is between (and including) start end end (or simply if it is greater/equal than start if end is unset).

Answer 4

I don't think sed can be made greedy, no. A possible workaround, simple but inefficient, would be to process the file twice. Once to get the line range and another to print. For instance:

$ perl -lne '$s||=$. if /abc/; $e=$. if /mno/; }{ print "$s $e"' file | 
    while read start end; do sed -n "$start,${end}p" file; done
abc def ghi
def ghi jkl
ghi jkl mno
jkl mno pqr
mno pqr stu

Alternatively, if you want to handle cases where either one or both patterns are missing:

perl -lne '$s||=$. if /abc/; $e=$. if /mno/; }{ $s||=1; $e||=$.; print "$s $e"' file | 
    while read start end; do sed -n "$start,${end}p" file; done

If abc isn't found, it will print from the beginning of the file. If mnc is not found, it will print from abc (or the beginning, if abc isn't there) until the end. If neither pattern is found it will, of course, print nothing.

Answer 5

Collect lines until a mno sequence makes them ready to print:

sed -e '/abc/,$!d;:loop' -e'/mno/{p;d;}' -e '$d;N;bloop'

• /abc/,$!d deletes everything except for the range from first abc line to the end. This also handles the case when there is no abc at all.
• Then we need to :loop.
• /mno/{p;d;} if there is mno in the pattern space, print and start over.
• $d if we reach the last line without mno, delete everything in the buffer. Unfortunately, this means no output, if there is no mno at all.
• Otherwise append the Next line and continue the loop.

Answer 6

You can collect all lines, starting from the abc line in hold space and then use the greedy nature of .* to delete everything after the last mno:

sed '/abc/,$!d;H;$!d;x;s/\n//;s/\(.*mno[^\n]*\).*/\1/'

• /abc/,$!d is to delete everything before the first abc line (or the whole file, if there is no abc line at all)
• H;$!d is the classical pattern to collect the whole file in the hold space (note that this can be a problem for very big files)
• we xchange buffers instead of using g to avoid copying a large buffer
• s/\n// removes the wrong newline at the beginning, produced by appending to the empty hold space
• s/\(.*mno[^\n]*\n\).*/\1/ removes everything after the last mno line (or prints the whole remaining file, if there is no mno line, as requested). Note that [^\n] is not POSIX and will work only on some versions like GNU sed.

Answer 7

Using Raku (formerly known as Perl_6)

raku -e '(S:g/ <( ^ .*? $$ \n )> ^^ .*? abc .*? $$ // andthen S:g/ ^^ .* mno .*? $$  <( .*? $)> //).put for lines.join("\n");'

Sample Input:

1. stu vwx yza
2. uvw xyz abc
3. abc def ghi
4. def ghi jkl
5. ghi jkl mno
6. jkl mno pqr
7. mno pqr stu
8. pqr stu vwx
9. stu vwx yza
10. mno pqr stu
11. xyz xyz xyz

Sample Output:

2. uvw xyz abc
3. abc def ghi
4. def ghi jkl
5. ghi jkl mno
6. jkl mno pqr
7. mno pqr stu
8. pqr stu vwx
9. stu vwx yza
10. mno pqr stu

Note above that Sample Output is a return of lines 2-through-10 when fed the 11 line Sample Input. Also, when the Sample Input is truncated to only lines 1-through-10 (i.e. with mno on the last line), the Raku code above still (correctly) returns lines 2-through-10.

Thanks to @ImHere and @ChennyStar for prodding me in the comments to come up with a more robust Raku solution.

https://raku.org

Answer 8

Using GNU sed Note: first abc line does not have mno as per OP so we can exploit that fact in the Below sed code.

sed -e '
  /abc/,$!d
  /mno/{h;b;}
  $!{N;s/^/\n/;D;}
  x;/./d;x
' file

---

In this method we use the slurp mode -z to read in the full file in the pattern space. Then we delete till before the first line containing abc. After that reach the last mno line using the greediness of regex.

sed -Ez '
  s/abc/\x0&/
  s/.*\n(.*)\x0/\1/
  s/(.*mno[^\n]*\n).*/\1/
' file

---

Yet another way is a two-pass approach where we record the line numbers of the first abc line and the last mno line. In case no mno present we fill in $ in it's place. Then using these two numbers we construct a sed command begin,endp;endq

sed -n '/abc/{=;:a;n;/mno/=;ba}' file |
sed -En '
  1{h;$s/.*/$/;}
  ${x;G;}
  s/\n(.*)/,\1p&q/p
' | sed -nf - file

---

We can use perl to slurp the file and then the whole file is one long string which we burn from both ends and stop when our conditions are met.

perl -0777 -pe '
  s/^.*\n// until /^.*abc/;    /mno/||next;
  s/.*\n$// until /mno.*$/;
' file

Answer 9

Here are more ways using the sed editor to get the desired output.

sed -n '
  /\n/{/mno/!d;P;D;}
  /abc/,$H;$!d
  z;x;G;/mno/D
  s/.//;s/.$//p
' file

• Save the file from the first /abc/ till eof in the hold space.
• print the top of pattern space while we still can see /mno/ anywhere in it.
• Then clip the top of pattern space and repeat previous step.
• Stop when /mno/ is no longer visible.
• Alternative ly, before beginning this P;D cycle if there is no /mno/ then just print the whole of hold space.

---

Another method where we store lines in hold only until /mno/ is seen. At which point we flip and print what was in hold.

sed -n '
  /abc/,$!d
  /mno/!{H;ba;}
  x;p;:a
  ${x;//P;//!p}
' file | sed 1d

---

Here is the Python way using the versatile itertools module groupby method to get the job done.

python3 -c 'import sys, itertools as it
ifile,start,stop = sys.argv[1:]
G,K,F = [],[],lambda x: x.find(stop)
with open(ifile) as f:
  for _ in f:
    if not _.find(start): continue
    for t in it.groupby(f,F):
      G.append(list(t[1]))
      K += [t[0] > -1]
if len(K) > 1 and not K[-1]: G.pop()
print(*[e for L in G for e in L], sep="",end="")
' file "abc" "mno"