5

What I am doing:

XZ_OPT='-T0 -9 -vv' tar -vvcJf ~/backup.tar.xz ...FILES

I doubt if tar really pass the given options, I have tried those things:

  1. I used -vv in XZ_OPT, but there is no message from xz in the output, neither --verbose
  2. I also useps aux | grep xz to see if tar will spawn another process for xz, but I didn't see that tar create any process.

Questions

  1. Does XZ_OPT environment really work? How to verify it?
  2. Why can't I find xz processes during tar execution? Does tar really spawn process to compress files?

Environment

$ xz --version
xz (XZ Utils) 5.2.5
liblzma 5.2.5
$ tar --version
bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6
Weihang Jian
  • 1,149
  • 1
  • 9
  • 15

2 Answers2

11
  1. Does XZ_OPT environment really work? How to verify it?

Pass something invalid to it:

% XZ_OPT='--this-wont-work' tar -cJf foo.tar.xz foo
xz: unrecognized option '--this-wont-work'
xz: Try `xz --help' for more information.
tar: foo.tar.xz: Cannot write: Broken pipe
tar: Child returned status 1
tar: Error is not recoverable: exiting now
  1. Why can't I find xz processes during tar execution? Does tar really spawn process to compress files?

From the output above, it looks like it does. Does your archive take long enough to create for the process to last? ps aux | grep xz and pgrep -fa xz both show xz processes for me.

In all likelihood, tar won't show output from the programs it calls unless they fail. Otherwise, they could add uncontrolled noise to the output which wasn't asked for from tar itself.

muru
  • 69,900
  • 13
  • 192
  • 292
  • That's weired. I get exit code 0 after run `XZ_OPT='--this-wont-work' tar -cJf backup.tar.xz backup.txt`, and output nothing. – Weihang Jian Mar 11 '21 at 11:18
  • 1
    https://i.imgur.com/Fzu309V.png – Weihang Jian Mar 11 '21 at 11:18
  • 1
    @Quasímodo Still the same https://i.imgur.com/SHV6Ue2.png no luck :( – Weihang Jian Mar 11 '21 at 11:25
  • @Weihang what version of Fish are you using? – Stephen Kitt Mar 11 '21 at 11:30
  • 1
    @WeihangJian seems you're using bsdtar. My tests used GNU tar. I'll test bsdtar and see what comes up. Bsdtar might be using the xz library directly. – muru Mar 11 '21 at 11:31
  • With `--use-compress-program`, it works! https://i.imgur.com/y5SesQV.png – Weihang Jian Mar 11 '21 at 11:52
  • 2
    Not quite sure why it works but I think without `--use-compress-program`, bsdtar would try to use it's built-in compressor and it won't even concern XZ_OPT. Need more time for investigation. – Weihang Jian Mar 11 '21 at 11:53
  • 4
    @WeihangJian yes, my [`man bsdtar`](https://man.archlinux.org/man/core/libarchive/bsdtar.1.en#BUGS) says "The compression and decompression is implemented internally, so there may be insignificant differences between the compressed output generated by `tar -czf - file` and that generated by `tar -cf - file | gzip`" [and](https://man.archlinux.org/man/core/libarchive/bsdtar.1.en#EXAMPLES) "you can set the compression level used by gzip or xz compression: `tar -czf file.tar --options='compression-level=9'`. – muru Mar 11 '21 at 15:47
  • @WeihangJian: Another way to be sure what's happening is to trace system calls, like `strace -f -efork,execve tar ...`. (Different kernels have different names for their trace tools; `strace` works on Linux, and maybe some other OSes.) – Peter Cordes Mar 12 '21 at 07:15
0

The syntax that you used, in most shells, will create that environment variable and "export" it. After that, unless tar explicitly deletes it from the environment, every child program will see it.

I do see it having some effect. Contrast this:

$ XZ_OPT='-T0 -9 -vv' tar -cJf junk.tar.xz ~/.config
tar: Removing leading `/' from member names
xz: Filter chain: --lzma2=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0
xz: Using up to 4 threads.
xz: 4,997 MiB of memory is required. The limiter is disabled.
xz: Decompression will need 65 MiB of memory.
  100 %   2,541.8 KiB / 5,930.0 KiB = 0.429                   0:01

with this (where I removed the -vv):

$ XZ_OPT='-T0 -9' tar -cJf junk.tar.xz ~/.config
tar: Removing leading `/' from member names