0

I have >100 lz4 files named like this:

-rw-r--r-- 1 root root 210M Apr 11 10:11 compressedfile.1-0.lz4
-rw-r--r-- 1 root root 208M Apr 11 11:35 compressedfile.1-1.lz4
-rw-r--r-- 1 root root 185M Apr 11 12:49 compressedfile.2-0.lz4
-rw-r--r-- 1 root root 193M Apr 11 13:06 compressedfile.2-1.lz4
-rw-r--r-- 1 root root 201M Apr 11 14:28 compressedfile.3-0.lz4
-rw-r--r-- 1 root root 236M Apr 11 15:02 compressedfile.3-1.lz4
....

The files are huge csv files that look like this:

10.27.221.233,11,TCP,SSL,66,8578,0,,(null),510-12
10.133.205.134,10,UDP,ICMP,26,3470,1,,(null),515-10
10.92.160.173,10,TCP,SSL,66,8578,0,,(null),510-15
10.132.81.71,11,TCP,SSL,0,2,0,,(null),511-10

I need to filter out IP addresses that use SSL. My method is like this:

lz4 -dc compressedfile.1-0.lz4 | awk -F, '{if ($4=="SSL") print $1}'

that's only for one file. I tried to process multiple files with wildcards, like this one:

lz4 -dc compressedfile.*.lz4 | awk -F, '{if ($4=="SSL") print $1}'

Warning : compressedfile.1-1.lz4 won't be used ! Do you want multiple input files (-m) ? 
Warning : compressedfile.2-0.lz4 won't be used ! Do you want multiple input files (-m) ?
Warning : compressedfile.2-1.lz4 won't be used ! Do you want multiple input files (-m) ?
....
10.27.221.233
10.92.160.173
10.132.81.71
10.140.81.238
10.92.5.90
....
<it ends with the IP (with SSL) on compressedfile.1-0.lz4>

then I tried adding -m option to lz4:

lz4 -mdc compressedfile.*.lz4 | awk -F, '{if ($4=="SSL") print $1}'

there are compressedfile.* uncompressed files created :(

I need your advice on using wildcards on lz4. I'm trying to avoid using a for loop if possible.

Zanna
  • 3,491
  • 18
  • 28
dragon21century
  • 3
  • 3
  • `lz4 -mdc *.lz4 | whatever...` works fine here with `lz4` `v 1.7.4` - it decompresses multiple files to _stdout_ and does _not_ create any file. – don_crissti Apr 11 '17 at 13:11
  • @don_crissti that’s interesting; are you using Fedora or RHEL by any chance? The Fedora version works as you describe, whereas the Debian version which I tested before answering exhibits the behaviour described by the OP. – Stephen Kitt Apr 12 '17 at 08:30
  • yes, i use RHEL, it's freezed (￣︿￣) – dragon21century Apr 12 '17 at 10:40

1 Answers1

1

lz4 files can be concatenated and processed as a unit, so this will work:

cat compressedfile.*.lz4 | lz4 -dc | awk -F, '{if ($4=="SSL") print $1}'
Stephen Kitt
  • 411,918
  • 54
  • 1,065
  • 1,164