2

I have zipped a file recursively in a directory. But what I notice with these last few zips is that the files do not get compressed.

  adding: 1.3.12.2.1107.5.1.4.64517.30000014091005511462300042406.dcm (deflated 0%)
  adding: 1.3.12.2.1107.5.1.4.64517.30000014091005511462300042279.dcm (deflated 0%)
  adding: 1.3.12.2.1107.5.1.4.64517.30000014091005511462300042466.dcm (deflated 0%)
  adding: 1.3.12.2.1107.5.1.4.64517.30000014091005511462300042200.dcm (deflated 0%)
  adding: 1.3.12.2.1107.5.1.4.64517.30000014091005511462300042227.dcm (deflated 0%)
  adding: 1.3.12.2.1107.5.1.4.64517.30000014091005511462300042372.dcm (deflated 0%)
  adding: 1.3.12.2.1107.5.1.4.64517.30000014091005511462300042245.dcm (deflated 0%)
  adding: 1.3.12.2.1107.5.1.4.64517.30000014091005511462300042282.dcm (deflated 0%)

Can anyone explain why this would occur when using zip?

ryekayo
  • 4,705
  • 9
  • 41
  • 66
  • 1
    There are several reasons for the zip to not compress as discussed [here](http://stackoverflow.com/questions/13309012/zip-deflated-0-why-no-compression). – Ramesh Sep 11 '14 at 15:19
  • Thanks, one thing in that link stood out that may explain the reasoning when I am zipping files at work. And it makes plenty of sense since the files are encrypted. – ryekayo Sep 11 '14 at 15:20

1 Answers1

7

As mentioned in the comment, the SO question had this pretty much covered. Now, I wanted to experiment on how this deflation actually works. So, I did the below testing.

What is Entropy?

Entropy is a measure of the unpredictability of an information stream. A perfectly consistent stream of bits (all zeroes, or all ones) is totally predictable (has no entropy). A stream of completely unpredictable bits has maximum entropy. The idea of entropy of information is credited to Claude Shannon who gave a formula to express it.

Now, I created a file just with y or n as below.

perl -e 'my $y; $y .= int(rand(100))>90 ? "y" : "n" for (0..999); print $y;' > f1

Now, I ran the command, zip f1.zip f1 and got the output as,

zip f1.zip f1
  adding: f1 (deflated 89%)

Now, in the above command we have predictable bytes y or n which is why we have a deflated percentage of 89.

Now, I am carrying out the experiment as below. 

 dd if=/dev/urandom of=./f2 bs=1M count=1

If I do the command zip f2.zip f2, this is what I got as output. 

zip f2.zip f2
  adding: f2 (deflated 0%)

Since, the /dev/urandom is completely unpredictable, we are getting an deflation rate of 0%. The reference link that I have provided below has very good explanation on how to calculate entropy for predictable bytes.

Also, there is this tool ent to calculate a file's entropy in debian based systems. You could just do an apt get install ent and calculate the entropy rate as ent filename and figure out what's really happening.

You could read from here about this command.

References

http://troydhanson.github.io/misc/Entropy.html

Community
  • 1
Ramesh
  • 38,687
  • 43
  • 140
  • 215
  • The tool sounds interesting enough but I am not working on a debian based system. I am using an rpm based. – ryekayo Sep 11 '14 at 15:48
  • @ryekayo, yeah me too. I tried installing but it is not there. Moreover, as far as I read, the tool can give just give some upper bound and it won't be a correct calculation. – Ramesh Sep 11 '14 at 15:50
  • That's okay. I just needed a brief explanation of what I was seeing because at my job we do a lot of zipping and file transfers and I notice every so often that some files would not compress when zipping and it puzzled me – ryekayo Sep 11 '14 at 15:51
  • Another thing you cannot compress by zipping: A file that has already been compressed. Its redundancy has already been squeezed out. – alexis Sep 11 '14 at 21:24
  • entropy is a property of data _generation method_, not the specific produced bytes. – Display Name Apr 16 '17 at 05:03