Unix and Linux Stack Exchange
Unix and Linux Stack Exchange

Questions tagged [pipe]

A Unix pipe connects file descriptors of two processes. A pipe is created with the POSIX pipe() function declared in . Shells provide pipe creation between processes using "|".

Pipes and data streams

Every command or program executed by the shell has 3 data streams associated with it:

  1. standard input (stdin, with a file descriptor of 0) – where commands get their input from (by default, keyboard input provided by the terminal).
  2. standard output (stdout, file descriptor 1) – where commands send their output (by default, the terminal display).
  3. standard error (stderr, file descriptor 2) – where commands send their error and warning messages (by default, the terminal display).

There are ways to connect streams between programs and files called piping and redirections.

Piping is a mechanism for sending data from one program to another using the "|" operator in most shells. The operator feeds the output from the program on the left as input to the program on the right.

Example:

$ cat two_columns
column1:cloth
column2:strawberries
column3:fish
column4:chocolate
column5:punch cards

$ cat two_columns | awk -F: '{print $1}'
column1
column2
column3
column4
column5

$ cat two_columns | awk -F: '{print "HAS: " $2}'
HAS: cloth
HAS: strawberries
HAS: fish
HAS: chocolate
HAS: punch cards

Further reading

Related tags

Useful links

1819 questions
523
votes
15 answers

Turn off buffering in pipe

I have a script which calls two commands: long_running_command | print_progress The long_running_command prints progress but I'm unhappy with it. I'm using print_progress to make it nicer (namely, I print the progress in a single line). The…
Aaron Digulla
  • 5,918
  • 4
  • 19
  • 17
490
votes
19 answers

Get exit status of process that's piped to another

I have two processes foo and bar, connected with a pipe: $ foo | bar bar always exits 0; I'm interested in the exit code of foo. Is there any way to get at it?
Michael Mrozek
  • 91,316
  • 38
  • 238
  • 232
361
votes
21 answers

How do I trim leading and trailing whitespace from each line of some output?

I would like to remove all leading and trailing spaces and tabs from each line in an output. Is there a simple tool like trim I could pipe my output into? Example file: test space at back test space at front TAB at end TAB at front sequence…
rubo77
  • 27,777
  • 43
  • 130
  • 199
221
votes
7 answers

Pass the output of previous command to next as an argument

I've a command that outputs data to stdout (command1 -p=aaa -v=bbb -i=4). The output line can have the following value: rate (10%) - name: value - 10Kbps I want to grep that output in order to store that 'rate' (I guess pipe will be useful…
Paul
  • 2,211
  • 2
  • 12
  • 3
198
votes
7 answers

How to use watch command with a piped chain of commands/programs

I usually use watch Linux utility to watch the output of a command repeatedly every n seconds, like in watch df -h /some_volume/. But I seem not to be able to use watch with a piped series of command like: $ watch ls -ltr|tail -n 1 If I do that,…
Tulains Córdova
  • 2,926
  • 4
  • 18
  • 26
181
votes
7 answers

How big is the pipe buffer?

As a comment in I'm confused as to why "| true" in a makefile has the same effect as "|| true" user cjm wrote: Another reason to avoid | true is that if the command produced enough output to fill up the pipe buffer, it would block waiting for true…
Kit Sunde
  • 4,394
  • 10
  • 30
  • 34
170
votes
1 answer

Bash: What does ">|" do?

I have just seen this written down; $ some-command >| /tmp/output.txt Vertical pipes are used in standard redirects "piping" the output of one command to another, is >| in fact completely useless as it would be the same as just > in this scenario?
jwbensley
  • 7,402
  • 11
  • 35
  • 43
140
votes
4 answers

How do I pass a list of files to grep

I am using find and getting a list of files I want to grep through. How do I pipe that list to grep?
Tegra Detra
  • 4,908
  • 11
  • 39
  • 53
136
votes
3 answers

Pseudo files for temporary data

I often want to feed relatively short string data (could be several lines though) to commandline programs which accept only input from files (e.g. wdiff) in a repeated fashion. Sure I can create one or more temporary files, save the string there and…
highsciguy
  • 2,534
  • 4
  • 21
  • 29
125
votes
6 answers

Process substitution and pipe

I was wondering how to understand the following: Piping the stdout of a command into the stdin of another is a powerful technique. But, what if you need to pipe the stdout of multiple commands? This is where process substitution comes in. In…
Tim
  • 98,580
  • 191
  • 570
  • 977
123
votes
5 answers

In what order do piped commands run?

I've never really thought about how the shell actually executes piped commands. I've always been told that the "stdout of one program gets piped into the stdin of another," as a way of thinking about pipes. So naturally, I thought that in the case…
action_potato
  • 2,051
  • 3
  • 16
  • 7
119
votes
6 answers

How to download an archive and extract it without saving the archive to disk?

I'd like to download, and extract an archive under a given directory. Here is how I've been doing it so far: wget http://downloads.mysql.com/source/dbt2-0.37.50.3.tar.gz tar zxf dbt2-0.37.50.3.tar.gz mv dbt2-0.37.50.3 dbt2 I'd like instead to…
BenMorel
  • 4,447
  • 8
  • 36
  • 46
111
votes
4 answers

Can't pipe into diff?

I wanted to be clever and compare a remote file to a local file without first manually downloading it. I can get the contents of the remote file by ssh user@remote-host "cat path/file.name" However, piping that to diff ssh user@remote-host "cat…
user394
  • 14,194
  • 21
  • 66
  • 93
110
votes
4 answers

What is the difference between "cat file | ./binary" and "./binary < file"?

I have a binary (that I can't modify) and I can do: ./binary < file I also can do: ./binary << EOF > "line 1 of file" > "line 2 of file" ... > "last line of file" > EOF But cat file | ./binary gives me an error. I don't know why it doesn't work…
Boris
  • 1,073
  • 2
  • 8
  • 7
106
votes
9 answers

Bash: Assign output of pipe to a variable

I am trying to get the output of a pipe into a variable. I tried the following things: echo foo | myvar=$(
Parckwart
  • 1,061
  • 2
  • 8
  • 5
1
2 3
…
99 100