Print everything between two patterns, then delete first and last line of the resulting output

Question

otherdata
otherdata
start_data
one
two
three
four
end_data
otherdata
otherdata

The resulting output should just be:

one
two
three
four

This looked like a job for sed to me:

sed -n '/start_data/,/end_data/{1d;$d;p}' myfile

Did not work. First line was deleted, but not the last line! (for no reason that I could explain by logic so far)

OK, so let's try the ugly way:

sed -n '/start_data/,/end_data/{/start_data\|end_data/!p}' myfile

Fair enough, this works. But I'd like to make the shorter method work as well, as the resulting output will always contain the two patterns on first and last line, since we're only extracting the data in between.

Why does sed choke at the attempt of combining the 1d and $d statements in curly braces?

Answer 1

You can reverse the logic:

sed '1,/start_data/d;/end_data/,$d'

That assumes start_data is not on the first line. To work around that, if you have GNU sed, you can make it instead:

sed '0,/start_data/d;/end_data/Q'

That 0 and Q are GNU-specific. Q quits sed without printing the pattern space, so that would also make it more efficient as it wouldn't keep reading and discarding the rest of the file as with the first solution.

Answer 2

awk seems to be a good fit to this problem:

$ awk '/end_data/{f=0;};f{print;};/start_data/{f=1;}' myfile
one
two
three
four

The above uses the flag f to decide if a line should be printed. When start_data, the flag is set to true (1). When end_data is found, the flag is set to false (0). When f is true, the line is printed.

Why does sed choke at the attempt of combining the 1d and $d statements in curly braces?

It is not "choking." It is just that 1d and $d refer to the first and last lines in the file, not the first and last lines in the pattern.

Answer 3

Well, this works:

sed -ne/start_data/!d\;:n -e'n;/end_data/q;p;bn' <in

It doesn't even attempt to print until it encounters /start_pattern/ and from that address on through to the last line, it will replace the current line w/ the next, quit input entirely if the newline pulled in matches /end_data/, or else print. And that's all.The output is, given your sample data:

one
two
three
four

It won't recognize a line as an end_data match if it also matches the first start_data line which occurs in input.

Answer 4

You have an answer to your question already; I'll throw in another way of doing this using Perl.

< inputfile perl -0777 -pe 's/^(.*\n)*?start_data.*\n((.*\n)*?)end_data(.*\n)*/$2/'

• -0777: slurps the whole file at once instead of one line at the time
• -p: places a while (<>) {[...]} loop around the script and prints the processed file
• -e: reads the script from the arguments

Perl command breakdown:

• s: asserts to perform a substitution
• /: starts the pattern
• ^: matches the start of the file
• (.*\n)*?: matches any number of any character greedily within the current line and a newline, zero or more times lazily within the current file (i.e. it matches the least times as possible, stopping when the following pattern starts to match)
• start_data.*\n: matches a start_data string, any number of any character greedily within the current line and a newline
• ((.*\n)*?): groups and matches any number of any character greedily within the current line and a newline, zero or more times lazily within the current file (i.e. it matches the least times as possible, stopping when the following pattern starts to match)
• end_data: matches an end_data string
• (.*\n)*: matches any number of any character greedily within the current line and a newline, zero or more times greedily within the current file (i.e. it matches the the most times as possible)
• /: stops the pattern / starts the replacement string
• $2: replaces with the second captured group
• /: stops the replacement string / starts the modifiers

Answer 5

Here, let me make a trivial, cosmetic modification to the input file provided in the question:

% cat myfile
red
orange
start_data
one
two
three
four
end_data
yellow
green

I have simply replaced the otherdata lines with distinct other data, so we can refer to every line in the input file uniquely, by content, without having to say “the first line”, since that is apparently subject to misinterpretation, or “the first otherdata line”, which is a little verbose (and, for all I know, also maybe subject to misinterpretation).

Now, probably the closest thing you're going to find to your first attempt is

% sed -n '/start_data/,/end_data/p' myfile | sed '1d;$d'
one
two
three
four

Your first attempt (sed -n '/start_data/,/end_data/{1d;$d;p}' myfile) "chokes" because (as John1024 said) line 1 is the red line^* and line $ is the green line^**. The 1d;$d; has no effect because those lines (along with, in fact, all of the otherdata/colordata lines) are already excluded by the /start_data/,/end_data/ range.
__________
^* i.e., the first line in the entire input file, not just the matched range
^** i.e., the last line in the entire input file, not just the matched range

---

By the way, are you saying that your command produced the following output?

one
two
three
four
end_data

Because that doesn't make sense, unless start_data was line 1 (i.e., if red and orange were absent).