How to compress all files in the current directory using lz4 using GNU parallel?

Question

I try ls | parallel lz4 but it outputs to stdout. Why? How to fix?

score 2 · Answer 1 · edited Nov 11 '22 at 14:03

2

Use the -m option for multiple input files (implies automatic output filenames).

ls | parallel lz4 -m

edited Nov 11 '22 at 14:03

AdminBee

21,637
21
47
71

answered Nov 11 '22 at 12:53

Anthony Kelly

294
1
4

1) Is there any difference between `ls | parallel lz4 -m` and `parallel lz4 -m ::: *`? 2) I don't understand: why there is a need of `-m` if lz4 is called N times in parallel, each time with a single file name? What do I miss? – pmor Nov 11 '22 at 13:15
1) Yes, they're equivalent. 2) I'm not entirely certain but I don't think parallel is necessarily running one file at a time. – Anthony Kelly Nov 11 '22 at 13:49
2) Both gzip (for example) and lz4 can take a single argument "input file". Then why `ls | parallel gzip` works (as expected), while `ls | parallel lz4` doesn't? Can you guess / explain? – pmor Nov 14 '22 at 08:35

score 0 · Answer 2 · answered Nov 14 '22 at 19:07

but it outputs to stdout. Why? How to fix?

The answers are in the man page...
Why it outputs to stdout:

  •   When  no  destination is specified, result is sent on implicit output, which depends on stdout status. When stdout is Not the
       console, it becomes the implicit output. Otherwise, if stdout is the console, the implicit output is filename.lz4.

In this particular case (parallel runs lz4) the stdout is not the console (see steeldriver's answer HERE); also, if you check your output it contains warning(s):

Warning : using stdout as default output. Do not rely on this behavior: use explicit `-c` instead !

How to fix (next paragraph in the man page):

   •   It is considered bad practice to rely on implicit output in scripts. because the script's environment may change. Always  use
       explicit  output in scripts. -c ensures that output will be stdout. Conversely, providing a destination name, or using -m en‐
       sures that the output will be either the specified name, or filename.lz4 respectively.

So the proper way would be to either provide a destination name

parallel lz4 {} {}.lz4 ::: ./*

or use the -m switch (same result but no verbosity):

parallel lz4 -m ::: ./*

How to compress all files in the current directory using lz4 using GNU parallel?

2 Answers2