What does POSIX sed require for `1d;1,2d` where an address range starts from an already-deleted line?

Question

In the comments to this question a case came up where various sed implementations disagreed on a fairly simple program, and we (or at least I) weren't able to determine what the specification actually requires for it.

The issue is the behaviour of a range beginning at a deleted line:

1d;1,2d

Should line 2 be deleted even though the start of the range was removed before reaching that command? My initial expectation was "no" in line with BSD sed, while GNU sed says "yes", and checking the specification text doesn't entirely resolve the matter.

Matching my expectation are (at least) macOS and Solaris sed, and BSD sed. Disagreeing are (at least) GNU and Busybox sed, and numerous people here. The first two are SUS-certified while the others are likely more widespread. Which behaviour is correct?

---

The specification text for two-address ranges says:

The sed utility shall then apply in sequence all commands whose addresses select that pattern space, until a command starts the next cycle or quits.

and

An editing command with two addresses shall select the inclusive range from the first pattern space that matches the first address through the next pattern space that matches the second. [...] Starting at the first line following the selected range, sed shall look again for the first address. Thereafter, the process shall be repeated.

Arguably, line 2 is within "the inclusive range from the first pattern space that matches the first address through the next pattern space that matches the second", regardless of whether the start point has been deleted. On the other hand, I expected the first d to move on to the next cycle and not give the range a chance to start. The UNIX™-certified implementations do what I expected, but potentially not what the specification mandates.

Some illustrative experiments follow, but the key question is: what should sed do when a range begins on a deleted line?

---

Experiments and examples

A simplified demonstration of the issue is this, which prints extra copies of lines rather than deleting them:

printf 'a\nb\n' | sed -e '1d;1,2p'

This provides sed with two lines of input, a and b. The program does two things:

1. Deletes the first line with 1d. The d command will

   Delete the pattern space and start the next cycle. and

2. Select the range of lines from 1 to 2 and explicitly prints them out, in addition to the automatic printing every line receives. A line included in the range should thus appear twice.

My expectation was that this should print

b

only, with the range not applying because 1,2 is never reached during line 1 (because d jumped to the next cycle/line already) and so range inclusion never begins, while a has been deleted. The conformant Unix seds of macOS and Solaris 10 produce this output, as does the non-POSIX sed in Solaris and BSD sed in general.

GNU sed, on the other hand, prints

b
b

indicating that it has interpreted the range. This occurs both in POSIX mode and not. Busybox's sed has the same behaviour (but not identical behaviour always, so it doesn't seem to be a result of shared code).

Further experimentation with

printf 'a\nb\nc\nd\ne\n' | sed -e '2d;2,/c/p'
printf 'a\nb\nc\nd\ne\n' | sed -e '2d;2,/d/p'

finds that it appears to treat a range starting at a deleted line as though it starts on the following line. This is visible because /c/ does not match to end the range. Using /b/ to start the range does not behave the same as 2.

---

The initial working example I was using was

printf '%s\n' a b c d e | sed -e '1{/a/d;};1,//d'

as a way to delete all lines up to the first /a/ match, even if that is on the first line (what GNU sed would use 0,/a/d for — this was an attempted POSIX-compatible rendition of that).

It has been suggested that this should instead delete up to the second match of /a/ if the first line matches (or the whole file if there's no second match), which seems plausible - but again, only GNU sed does that. Both macOS sed and Solaris's sed produce

b
c
d
e

for that, as I expected (GNU sed produces the empty output from removing the unterminated range; Busybox sed prints just d and e, which is clearly wrong no matter what). Generally I'd assume that their having passed the certification conformance tests means that their behaviour is correct, but enough people have suggested otherwise that I'm not sure, the specification text isn't completely convincing, and the test suite can't be perfectly comprehensive.

Clearly it isn't practically portable to write that code today given the inconsistency, but theoretically it should be equivalent everywhere with one meaning or the other. I think this is a bug, but I don't know against which implementation(s) to report it. My view currently is that GNU and Busybox sed's behaviour is inconsistent with the specification, but I could be mistaken on that.

What does POSIX require here?

Answer 1

That was raised on the Austin group mailing list in March 2012. Here's the final message on that (by Geoff Clare of the Austin Group (the body that maintains POSIX), who is also the one who raised the issue in the first place). Here copied from the gmane NNTP interface:

Date: Fri, 16 Mar 2012 17:09:42 +0000
From: Geoff Clare <gwc-7882/[email protected]>
To: austin-group-l-7882/[email protected]
Newsgroups: gmane.comp.standards.posix.austin.general
Subject: Re: Strange addressing issue in sed

Stephane Chazelas <[email protected]> wrote, on 16 Mar 2012:
>
> 2012-03-16 15:44:35 +0000, Geoff Clare:
> > I've been alerted to an odd behaviour of sed on certified UNIX
> > systems that doesn't seem to match the requirements of the
> > standard.  It concerns an interaction between the 'n' command
> > and address matching.
> > 
> > According to the standard, this command:
> > 
> > printf 'A\nB\nC\nD\n' | sed '1,3s/A/B/;1,3n;1,3s/B/C/'
> > 
> > should produce the output:
> > 
> > B
> > C
> > C
> > D
> > 
> > GNU sed does produce this, but certified UNIX systems produce this:
> > 
> > B
> > B
> > C
> > D
> > 
> > However, if I change the 1,3s/B/C/ to 2,3s/B/C/ then they produce
> > the expected output (tested on Solaris and HP-UX).
> > 
> > Is this just an obscure bug from common ancestor code, or is there
> > some legitimate reason why this address change alters the behaviour?
> [...]
> 
> I suppose the idea is that for the second 1,3cmd, line "1" has
> not been seen, so the 1,3 range is not entered.

Ah yes, now it makes sense, and it looks like the standard does
require this slightly strange behaviour, given how the processing
of the "two addresses" case is specified:

    An editing command with two addresses shall select the inclusive
    range from the first pattern space that matches the first address
    through the next pattern space that matches the second.  (If the
    second address is a number less than or equal to the line number
    first selected, only one line shall be selected.) Starting at the
    first line following the selected range, sed shall look again for
    the first address. Thereafter, the process shall be repeated.

It's specified this way because the addresses can be BREs, but if
the same matching process is applied to the line numbers (even though
they can only match at most once), then the 1,3 range on that last
command is never entered.

-- 
Geoff Clare <g.clare-7882/[email protected]>
The Open Group, Apex Plaza, Forbury Road, Reading, RG1 1AX, England

And here's the relevant part of the rest of the message (by me) that Geoff was quoting:

I suppose the idea is that for the second 1,3cmd, line "1" has
not been seen, so the 1,3 range is not entered.

Same idea as in

printf '%s\n' A B C | sed -n '1d;1,2p'

whose behavior differ in traditional (heirloom toolchest at
least) and GNU.

It's unclear to me whether POSIX wants one behavior or the
other.

So, (according to Geoff) POSIX is clear that the GNU behaviour is non-compliant.

And it's true it's less consistent (compare seq 10 | sed -n '1d;1,2p' with seq 10 | sed -n '1d;/^1$/,2p') even if potentially less surprising to people who don't realise how ranges are processed (even Geoff initially found the conforming behaviour "strange").

Nobody bothered reporting it as a bug to the GNU folks. I'm not sure I'd qualify it as a bug. Probably the best option would be for the POSIX specification to be updated to allow both behaviours to make it clear that one cannot rely on either.

Edit. I've now had a look at the original sed implementation in Unix V7 from the late 70s, and it looks pretty much like that behaviour for numeric addresses was not intended or at least not thought through completely there.

With Geoff's reading of the spec (and my original interpretation of why it happens), conversely, in:

seq 5 | sed -n '3d;1,3p'

lines 1, 2, 4 and 5 should be output, because this time, it's the end address that is never encountered by the 1,3p ranged command, like in seq 5 | sed -n '3d;/1/,/3/p'

Yet, that doesn't happen in the original implementation, nor any other implementation I tried (busybox sed returns lines 1, 2 and 4 which looks more like a bug).

If you look at the UNIX v7 code, it does check for the case where the current line number is greater than the (numerical) end address, and gets out of the range then. The fact that it doesn't do it for the start address looks more like an oversight then than an intentional design.

What that means is that there's no implementation that is actually compliant to that interpretation of the POSIX spec in that regard at the moment.

Another confusing behaviour with the GNU implementation is:

$ seq 5 | sed -n '2d;2,/3/p'
3
4
5

Since line 2 was skipped, the 2,/3/ is entered upon line 3 (the first line whose number is >= 2). But as it's the line that made us enter the range, it's not checked for the end address. It gets worse with busybox sed in:

$ seq 10 | busybox sed -n '2,7d; 2,3p'
8

Since lines 2 to 7 were deleted, line 8 is the first one that is >= 2 so the 2,3 range is entered then!