5

I'm using cat largeFile.iso | lz4 -10 - and it's going quite slowly (30mb/s?).

Using htop I see it using only a single core to 100%? Are there any additional parameters I need to specify? I can't seem to find any relevant info about this in lz4 --help.

I expect it to use multiple cores, and in turn compress this file faster.

Chris Stryczynski
  • 5,178
  • 5
  • 40
  • 80

1 Answers1

7

I don't believe the normal lz4 program uses more than one core. It would be possible to split the input into several parts, compress these parts in parallel, and then concatenate the compressed parts. This has been done for other compressors for example with the pigz program.

It is not clear that there is much advantage in doing this for lz4 as the compressor is pretty fast, and so I/O speed is likely to be the thing limiting the overall compression speed.

icarus
  • 17,420
  • 1
  • 37
  • 54
  • 1
    This seems to contradict the github page though: "LZ4 is lossless compression algorithm, providing compression speed > 500 MB/s per core, scalable with multi-cores CPU." How is it "scalabale with multi-cores" then? – Chris Stryczynski Nov 04 '19 at 10:56
  • Looking in the source on a debian machine, there are no references to either pthread nor fork, which are the likely ways of creating additional processes. – icarus Nov 04 '19 at 15:07
  • 3
    You're correct - the lz4 cli does not currently offer this functionality by default (at least for now). https://github.com/lz4/lz4/issues/807 – Chris Stryczynski Nov 04 '19 at 18:33
  • For followers, that github issue says it's "multi-threadable" but the default CLI does not do that...and links to a few that do. – rogerdpack Mar 24 '21 at 20:32
  • 1
    "It is not clear that there is much advantage in doing this for lz4 as the compressor is pretty fast" I disagree, when updating my initramfs, the update uses lz4 quite a bit and this process contributes a noticeable amount of time on my servers and even my i9 machine with a Samsung 980 Pro NVMe (7GB/s Read 5GB/s Write) SSD... LZ4 being single threaded is disadvantageous without question in the modern day. – Tmanok Jan 12 '22 at 05:50