How to download an archive and extract it without saving the archive to disk?

Question

I'd like to download, and extract an archive under a given directory. Here is how I've been doing it so far:

wget http://downloads.mysql.com/source/dbt2-0.37.50.3.tar.gz
tar zxf dbt2-0.37.50.3.tar.gz
mv dbt2-0.37.50.3 dbt2

I'd like instead to download and extract the archive on the fly, without having the tar.gz written to the disk. I think this is possible by piping the output of wget to tar, and giving tar a target, but in practice I don't know how to put the pieces together.

score 157 · Accepted Answer · edited Apr 28 '22 at 12:20

157

You can do it by telling wget to output its payload to stdout (with the flag -O-) and suppress its own output (with the flag -q):

wget -qO- your_link_here | tar xvz

To specify a target directory:

wget -qO- your_link_here | tar xvz -C /target/directory

If you happen to have GNU tar, you can also rename the output dir:

wget -qO- your_link_here | tar --transform 's/^dbt2-0.37.50.3/dbt2/' -xvz

edited Apr 28 '22 at 12:20

Stephen Kitt

411,918
54
1,065
1,164

answered Aug 01 '13 at 14:23

Joseph R.

38,849
7
107
143

To specified path should be: `wget -qO- your_link_here | tar xvz - -C /target/directory` – Marslo Sep 12 '18 at 12:10
maybe just tell people to use tar instead wget then? – Alexander Mills May 19 '19 at 04:29
4

`wget -qO- | tar -xvz -C ` worked on gnu tar. – AlikElzin-kilaka Jun 22 '19 at 14:22
will this require less than double the space of the archive on my disk? right now I'm having to unpack a huge tar and I'm looking for a way to optimize the operation space-wise – doplumi Jun 17 '21 at 17:28
2

You don’t have to specify the stdin `-`. – Константин Ван Jun 22 '21 at 10:08

score 40 · Answer 2 · edited Mar 13 '23 at 17:32

40

Another option is to use curl which writes to stdout by default:

curl -s -L https://example.com/archive.tar.gz | tar xvz - -C /tmp

edited Mar 13 '23 at 17:32

Paweł Prażak

205
2
6

answered Mar 12 '18 at 21:46

Zlemini

591
5
10

3

I like your option more than others but ```curl -s some_url | tar xvz - -C /tmp``` – FiftiN Mar 18 '19 at 17:44
4

as [FiftiN](https://unix.stackexchange.com/users/342396/fiftin) suggested -> e.g. to view a filtered list of files inside repository one could use: `$ curl -L https://api.github.com/repos/repo_owner/repo_name/tarball | tar tvfz - -C /tmp --wildcards *.py` – Alex Glukhovtsev Apr 24 '19 at 09:45
7

Better curl with "-L" to follow redirects – rfmoz Mar 27 '20 at 15:37
works by default on a Mac too – ElFik May 11 '21 at 22:03

score 13 · Answer 3 · edited Aug 25 '14 at 19:57

13

This oneliner does the trick:

tar xvzf -C /tmp/ < <(wget -q -O - http://foo.com/myfile.tar.gz)

short explanation: the right side in the parenthesis is executed first (-q tells wget to do it quietly, -O - is used to write the output to stdout).

Then we create a named pipe using the process substitution operator from Bash <( to create a named pipe. This way we create a temporary file descriptor and then direct the contents of that descriptor to tar using the < file redirection operator.

edited Aug 25 '14 at 19:57

Daniel Serodio

1,123
1
9
14

answered Aug 01 '13 at 14:49

ItsMe

314
1
5

This would need `-f -` (for stdin) or `-f <(wget...` to work. – Jan Kyu Peblik Nov 20 '19 at 17:33
This should be something like ```tar zxvf - < <(wget -q -O - https://github.com/peak/s5cmd/releases/download/v2.1.0/s5cmd_2.1.0_Linux-64bit.tar.gz)``` – Nasir Mahmood Jul 03 '23 at 06:34

score 2 · Answer 4 · answered Sep 02 '21 at 15:35

2

Named pipe with stdin solution and really mind the flags for tar's -xvz

tar -xvz -C /tmp/ -f <(wget -q -O - https://github.com/user/repo/release/download/v/v.tar.gz)

answered Sep 02 '21 at 15:35

Khalil Gharbaoui

121
3

score 2 · Answer 5 · answered Jan 16 '22 at 03:19

2

One liner that handles redirects and can extract tar.bz2 files. Use xzfor extracting gzip files.

curl -L https://downloads.getmonero.org/cli/linux64 | tar xj

answered Jan 16 '22 at 03:19

Elijah

121
1

score 0 · Answer 6 · answered Aug 02 '22 at 16:53

The extraction part should take input from STDOUT. We may need tar -xzvf - -C <output_dir>

Example:


# this may not work
# It might complain 
# tar (child): -C: Cannot open: No such file or directory
wget -qO - https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3-scala2.13.tgz | tar -xzvf -C /opt/spark --strip-component 1


# this should work. 
wget -qO - https://dlcdn.apache.org/spark/spark-3.3.0/spark-3.3.0-bin-hadoop3-scala2.13.tgz | tar -xzvf - -C /opt/spark --strip-component 1

How would one go about this using the wget -N flag? So only do this if the downloaded file has changed? I would imagine I would need to save the existing file for that? — fred, Feb 13 '23 at 21:43

How to download an archive and extract it without saving the archive to disk?

6 Answers6

Linked