How to use multi-threading for creating and extracting tar.xz

Question

I use

tar -cJvf resultfile.tar.xz files_to_compress

to create tar.xz and

tar -xzvf resultfile.tar.xz

to extract the archive in current directory. How to use multi threading in both cases? I don't want to install any utilities.

score 34 · Accepted Answer · edited Nov 24 '20 at 22:38

34

tar -c -I 'xz -9 -T0' -f archive.tar.xz [list of files and folders]

This compresses a list of files and directories into an .tar.xz archive. It does so by specifying the arguments to be passed to the xz subprocess, which compresses the tar archive.

This is done using the -I argument to tar, which tells tar what program to use to compress the tar archive, and what arguments to pass to it. The -9 tells xz to use maximum compression. The -T0 tells xz to use as many threads as you have CPUs.

edited Nov 24 '20 at 22:38

Alexander Batischev

2,363
11
19

answered Sep 06 '20 at 23:25

Artem S. Tashkinov

26,392
4
33
64

1

On my TAR though, `-I` won't accept arguments, and expects a program only. Maybe you can make a simple shell script that is `exec xz -9 -T0 $*`? – ZiggyTheHamster Oct 14 '20 at 01:51
This program is used for piping, so `$*` is redundant. Also, not sure why you want to use `exec`. – Artem S. Tashkinov Oct 14 '20 at 09:39
it is freaking fast and very less filesize – Thamaraiselvam Oct 28 '20 at 06:53
1

Warning: If you have many CPUs, this will use a _lot_ of RAM..... – starbeamrainbowlabs Nov 24 '20 at 17:54
1

This returns `tar: Couldn't open xz -9 -T0: No such file or directory` here (macOS). The following from another answer worked: `tar -cf - list of files and folders| xz -9 -T0 >| archive.tar.T.xz` – aaronk6 Jan 07 '22 at 12:55
@aaronk6 older versions of tar don't support passing arguments to compressors. Probably MacOS has the same limitation. – Artem S. Tashkinov Jan 07 '22 at 13:45

score 28 · Answer 2 · edited Jun 30 '21 at 10:32

You can use XZ_DEFAULTS or XZ_OPT environment variables:

XZ_DEFAULTS is recommended to be used as a system wide configuration, typically set in a shell initialization script.

XZ_OPT is for passing options to xz when run by a script or tool, e.g. GNU tar. See man xz.

Example: using multiple threads (-T0) and max compression level (-9):

XZ_OPT='-T0 -9' tar -cJf resultfile.tar.xz files_to_compress

Reference with a recent GNU tar on bash or derived shell, also see the xz man page

score 4 · Answer 3 · edited Nov 24 '20 at 08:38

4

For older tars, this works:

tar -cf -  list of files and folders| xz -9 -T0 >| archive.tar.T.xz

edited Nov 24 '20 at 08:38

AdminBee

21,637
21
47
71

answered Nov 23 '20 at 15:16

Will Fischer

41
2

How to use multi-threading for creating and extracting tar.xz

3 Answers3

Linked