Time required to do pipe output to head/tail

Question

There are lots of txt files in a directory.

If I do time wc -l *.txt | head it takes

real    0m0.032s
user    0m0.020s
sys     0m0.008s

If I do time wc -l *.txt | tail it takes

real    0m0.156s
user    0m0.076s
sys     0m0.088s

Does this mean that wc will know beforehand that it is piping to head and will only count for first 10 files and save time? In other words, is it aware of the pipe? And is this somethign special about wc or does it apply to all standard/built-in commands?

score 2 · Answer 1 · answered Oct 08 '14 at 04:20

2

Any process that does not block SIGPIPE will be killed if its output goes to the write end of a pipe that no one is reading from.

So as soon as head closes its input (i.e. terminates), wc dies, which takes less time than finishing all the work.

answered Oct 08 '14 at 04:20

o11c

1,408
7
13

score 2 · Accepted Answer · answered Oct 08 '14 at 07:08

I did a strace on both commands. The interessting thing is that when you pipe the output to head there are only 123 system calls. On the other hand when pipeing to tail there are 245 system calls (or more when there are more *.txt files).

Case: head

Here are the last few lines when pipeing to head:

open("file12.txt", O_RDONLY)            = 3
fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
read(3, "", 16384)                      = 0
write(1, "0 file12.txt\n", 13)          = -1 EPIPE (Broken pipe)
--- SIGPIPE (Broken pipe) @ 0 (0) ---
+++ killed by SIGPIPE +++

When wc tries to write the output of the 12th file it gets an error EPIPE. Thats why head exited after getting the 11th line. When head exits, wc gets SIGPIPE. As seen in the strace output above, wc first tries to write to that pipe (where head no longer reads from) and gets an error that the pipe is broken.

The SIGPIPE signal is sent to a process when it attempts to write to a pipe without a process connected to the other end. -- from Wikipedia

Case: tail

When pipeing to tail, there is nothing like that above. wc ends gracefully after writingall the output to the pipe where tail needs to be connected to for the whole time. tail needs all lines before it can print the last 10 of them. When there is no more output to read, tail prints the lines and exits gracefully too

score 0 · Answer 3 · answered Oct 08 '14 at 02:16

0

You can do it for disappear your files:

time wc -l *.txt > tee   | tail

But a bit you add time for tee command to time .

With tee command :

root@debian:/home/mohsen/test# time wc -l *.txt > tee   | tail 

real    0m0.005s
user    0m0.000s
sys 0m0.000s

Without tee command:

root@debian:/home/mohsen/test# time wc -l *.txt | tail 
   8 f3.txt
   7 fi.txt
   5 mydata.txt
   4 newfile.txt
   4 t1.txt
   4 t2.txt
   5 test.txt
   4 text.txt
   0 t.txt
  49 total

real    0m0.004s
user    0m0.000s
sys 0m0.000s

answered Oct 08 '14 at 02:16

PersianGulf

10,728
8
51
78

I cannot understand this answer, what exactly do you want to say by including `tee` into the problem? – jimmij Oct 08 '14 at 04:39
@jimmij it's exposing the thing I mentioned in my problem. Since tee needs to consume all of its output anyway for the copy it writes to the file, it is forced to block `SIGPIPE` and thus `wc` has to run for the whole time. – o11c Oct 08 '14 at 04:42
@o11c ok, I see. – jimmij Oct 08 '14 at 04:49

Time required to do pipe output to head/tail

3 Answers3