Why can't `paste` print stdin next to stderr?

Question

Usually paste prints two named (or equivalent) files in adjacent columns like this:

paste <(printf '%s\n' a b) <(seq 2)

Output:

a   1
b   2

But when the two files are /dev/stdin and /dev/stderr, it doesn't seem to work the same way.

Suppose we have blackbbox program which outputs two lines on standard output and two lines on standard error. For illustration purposes, this can be simulated with a function:

bb() { seq 2 | tee >(sed 's/^/e/' > /dev/stderr) ; }

Now run annotate-output, (in the devscripts package on Debian/Ubuntu/etc.), to show that it works:

annotate-output bash -c 'bb() { seq 2 | tee >(sed 's/^/e/' > /dev/stderr) ; }; bb'
22:06:17 I: Started bash -c bb() { seq 2 | tee >(sed s/^/e/ > /dev/stderr) ; }; bb
22:06:17 O: 1
22:06:17 E: e1
22:06:17 O: 2
22:06:17 E: e2
22:06:17 I: Finished with exitcode 0

So it works. Feed bb to paste:

bb | paste /dev/stdin /dev/stderr

Output:

1   e1
e2
^C

It hangs -- ^C means pressing Control-C to quit.

Changing the | to a ; also doesn't work:

bb ; paste /dev/stdin /dev/stderr

Output:

1
2
e1
e2
^C

Also hangs -- ^C means pressing Control-C to quit.

Desired output:

1    e1
2    e2

Can it be done using paste? If not, why not?

Answer 1

Why you can't use /dev/stderr as a pipeline

The problem isn't with paste, and neither is it with /dev/stdin. It's with /dev/stderr.

All commands are created with one open input descriptor (0: standard input) and two outputs (1: standard output and 2: standard error). Those can typically be accessed with the names /dev/stdin, /dev/stdout and /dev/stderr respectively, but see How portable are /dev/stdin, /dev/stdout and /dev/stderr?. Many commands, including paste, will also interpret the filename - to mean STDIN.

When you run bb on its own, both STDOUT and STDERR are the console, where command output usually appears. The lines go through different descriptors (as shown by your annotate-output) but ultimately end up in the same place.

When you add a | and a second command, making a pipeline...

bb | paste /dev/stdin /dev/stderr

the | tells the shell to connect the output of bb to the input of paste. paste first tries to read from /dev/stdin, which (via some symlinks) resolves to its own standard input descriptor (which the shell just connected up) so the line 1 comes through.

But the shell/pipeline does nothing to STDERR. bb still sends that (e1 e2 etc.) to the console. Meanwhile, paste attempts to read from the same console, which hangs (until you type something).

Your link Why can't I read /dev/stdout with a text editor? is still relevant here because those same restrictions apply to /dev/stderr.

How to make a second pipeline

You have a command that produces both standard output and standard error, and you want to paste those two lines next to each other. That means two concurrent pipes, one for each column. The shell pipeline ... | ... provides one of those, and you're going to need to create the second yourself, and redirect STDERR into that using 2>filename.

mkfifo RHS
bb 2>RHS | paste /dev/stdin RHS

If this is for use in a script, you may prefer to make that FIFO in a temporary directory, and remove it after use.

Answer 2

annotate-output is able to do that because it's doing something special (namely it redirects the stderr of a command to a fifo), something that paste has absolutely no way of doing -- simply because paste is not running itself the command(s) it gets its input from, and it has no way of redirecting their input or their output.

But you could write a wrapper that use exactly the same trick annotate-output is using:

pasteout(){
  f=$(mktemp -u) || return
  mkfifo -m 600 -- "$f" || return
  "$@" 2>"$f" | paste -- - "$f"
  rm -f -- "$f"
}
pasteout bb

Note however that it's prone to dead-locks. If for instance bb produces more standard output than can fit in the pipe plus the extra amount initially read by paste but doesn't produce any error output, paste will be blocked waiting for input on the fifo and will not empty the pipe bb is feeding its stdout to, causing bb's write()s to the pipe to hang as well.

Answer 3

There are a couple of problems with the whole line that we need to analyze, that is:

seq 2 | tee >(sed 's/^/e/' > /dev/stderr) | paste /dev/stdin /dev/stderr

stderr

First, the last command. Only stdout can pass through a pipe:

$ seq2 | paste -
1
2

$ seq2 | paste - -
1 2

There is nothing to read from stderr:

$ seq 2 | paste - /dev/stderr 
1   ^C

You need to ^C it because it blocks, there is nothing to read from stderr.
Even if you create some output to stderr it doesn't travel through a pipe:

$ { seq 2; seq 3 4 >/dev/stderr; } | paste - /dev/stderr
1   3
4

Exactly as before, the 1 gets printed and the paste blocks waiting for stderr.
The other 2 numbers went directly to the console and got (independently) printed.

You could give some input to stderr in the last command of the pipe:

$ { seq 2; seq 3 4 >/dev/stderr; } | paste - /dev/stderr 2</dev/null
1
2
3
4

Which is exactly the same as 2>/dev/null by the way, to avoid blocking the second file descriptor used in the paste command. But the values printed come directly from the seq 3 4 redirected to the console, not from paste. This does the same:

$ { seq 2; seq 3 4 >/dev/tty; } | paste - /dev/stderr 2</dev/null
1   
2   
3
4

And this doesn't block:

$ seq 2 | tee >(sed 's/^/e/' > /dev/stderr) | 
  paste /dev/stdin /dev/stderr 2</dev/null
1   
2   
e1
e2

order

Second, the output of tee doesn't have to be "in order". `tee` and `bash` process substitution order

And, in fact: The output of a process substitution doesn't have to be "in order": The process substitution output is out of the order

$ echo one; echo two > >(cat); echo three;
one
three
two

In fact, on some examples, if you try several times, you could get different orders. non-deterministic output from independent processes run concurrently by process substitution

$ printf '%s\n' {0..1000} | tee >(head -n2) >(sort -grk1,1 | head -n3) >/dev/null
1000
999
998
0
1

So, no, it could not be done with process substitution and paste.
You need to give some order to the execution:

$ seq 2 | { while read a; do printf "%s %s\n" "$a" "e$a" ; done; }
1 e1
2 e2

bb

So, your bb function, which (basically) contains:

| tee >(sed 's/^/e/')

Which could be tested with :

$ printf '%s\n' {0..1000} | tee >(sort -grk1,1 | head -n3 >&2) | head -n 2
0
1
291
290
289

Should print 0, 1, 1000, 999, 998, In that order, but many times it doesn't.
That is: It is intrinsically in-stable.

Stable real Solution.

The only safe solution for bb is to avoid any process substitution.
And, taking advantage that the {…} capture both stdout and stderr, example:

$ bash -c '{ echo test-str >/dev/stderr; }' 2>/dev/null

No output, remove the 2 to confirm.

This will work for bb:

$ bb() { seq 5 | tee /dev/stderr | sed 's/^/e/'; }

And use a fifo for paste:

$ mkfifo out2
$ bb 2>out2  | paste out2 -
1   e1
2   e2
3   e3
4   e4
5   e5

You will need to set a trap to remove the fifo file and test if the fifo file exist before creating it.

Seems to work portably on all shells (compatible with Almquist syntax) I tested. Not fully tested, ask for confirmation from other users, there may be some yet unknown surprises.