2

I have a folder with ~22k subfolders on it, and each subfolder can have more subfolders, up to a depth of ~7.

So I wanted to make it a single file, but not a rar or similar file, because decompressing it could take a long time. I wanted a .iso because I could mount it and access only what I need, without having to extract everything.

I used this command genisoimage --iso-level 4 -o folder.iso "folder"

The folder size was 21.7 GB but after turning it in a iso it's 43.2 GB? Is this expected?

schily
  • 18,806
  • 5
  • 38
  • 60
Freedo
  • 1,205
  • 6
  • 31
  • 57
  • 2
    If you have many small files you will lose lots of space due to padding. If I remember correctly, the sector size is 2048 bytes, so a file of 2049 bytes will take 2 complete sectors, double the size needed. You should consider squashfs, as it is intended for fast direct access, and you have options to not compress data etc, though you may find the defaults give the best throughput as these days cpu is not usually a limiter, but i/o is. – meuh Nov 03 '18 at 09:01
  • 1
    @Fox: `tar` does not really give better results. With the historic `tar` format the overhead is 512 bytes and with the POSIX.1-2001 format the overhead is 1536 bytes per file. – schily Nov 03 '18 at 15:23
  • @meuh with squashfs the single file is the same size as the folder...using the defaults – Freedo Nov 05 '18 at 02:55

3 Answers3

3

First, since genisoimage is full of bugs and creates filesystems with structral defects, it is not recommended to use genisoimage.

If you like to create an ISO-9660 filesystem, you should use the well maintained original mkisofs. Make sure that you really use the original and not just a link to genisoimage. It is easy to verify whether you are using the original software, call mkisofs -version and verify that you are using a recent version. The buggy genisoimage does not give a date and the original uses a version > 3.0 and the current version prints a version date from May 2018 and is from September 2018 when using the latest version of my own schilytools. You may download recent source versions here: http://sourceforge.net/projects/schilytools/files/ this is updated with a 1-3 week schedule.

But since the sector size used by ISO-9660 is 2048 bytes, archiving many small files may easily result in doubling the size of the achive compared to the sum of the size of the files.

It is recommended to use at least Rock Ridge attributes in the ISO image since that includes more attributes but does not really enhance the size.

BTW: if you have many small files and create a tar archive with the POSIX.1-2001 extensions usually named as pax, you get a tar archive with a similar sze as the ISO image.

The historic tar archive has an overhead of 512 bytes per file, the POSIX.1-2001 format has an overhead if 1536 bytes per file.

terdon
  • 234,489
  • 66
  • 447
  • 667
schily
  • 18,806
  • 5
  • 38
  • 60
  • It seems I can't install mkisofs anymore on ubuntu. It just defaults to genisoimage... – Freedo Nov 03 '18 at 23:57
  • 1
    You always can compile it yourself and install the compiled binary (this is easier than with other OSS, just type `make` as user and then `make install` as root) and there are binary packages from Ubuntu users. BTW: `mkisofs` has approx. twice as many features than `ganisoimage`. – schily Nov 04 '18 at 10:53
  • Thanks, and I'm not doubting you. But do you have a source that genisoimage is bad? If it's bad, why it's used in ubuntu packages by default? – Freedo Nov 04 '18 at 22:24
  • This is easy to prove. Even if you don't believe me that Debian added own new bugs to `genisoimage`, it is based on a `mkisofs` source from May 2004 and since `mkisofs`fixed dozen of bugs in August 2006 and as there never have been bug fixes in `genisoimage`, it should be obvious that these bugs are still in `genisoimage`. Ubuntu is based on Debian and the creation of a `cdrtools` fork in May 2004 was anti social act from a Debian packetizer who was in hope to get visibility from that act and wo has been in hope to get a job at Nero from this visibility. – schily Nov 05 '18 at 11:40
  • Lol just saw you are the author of mkisofs lol...you should try to add your image to ubuntu again...maybe a PPA or something :P thanks! – Freedo Nov 06 '18 at 03:22
  • Check for Brandon Snider – schily Nov 06 '18 at 09:57
  • Thanks @schilly - genisoimage 1.1.11 on Debian Buster just created a corrputed ISO for me, your answer put me on the right track. Unbelievable. – spawn Dec 24 '21 at 01:20
2

So I wanted to make it a single file, but not a rar or similar file, because decompressing it could take a long time. I wanted a .iso because I could mount it and access only what I need, without having to extract everything.

This looks like a job for SquashFS: SquashFS lets you compress a directory into a single file. As opposed to standard compressed files (tar.gz etc.), you access the content by mounting the file instead of actually extracting it to the disk.

  • Read access is very efficient: sometimes it's faster than accessing a regular uncompressed filesystem (especially if the directory contains many files and subdirectories).
  • The file is much smaller than an ISO image since the content is compressed.
Erwan
  • 219
  • 1
  • 5
  • Does this work for windows? I need my file to be accessed by windows too – Freedo Nov 04 '18 at 22:23
  • Apparently it's possible to use squashFS files on Windows but I didn't try: https://stackoverflow.com/questions/36478351/how-to-handle-squashfs-in-windows – Erwan Nov 04 '18 at 23:55
  • I've confirmed that 7-zip can open it on windows 7, but not winrar. And the squashFS file is same size as the folder...impressive! but I think schily anwser the question better so I'll accept his answer but no offense and thanks! – Freedo Nov 05 '18 at 02:55
  • 1
    In general, if you like to archive data for a longer time, use a standard archive format. ISO-9660 and tar are both standard and widely available. – schily Nov 05 '18 at 11:33
  • @freedo no worries, I suggested SquashFS as a possible alternative but schily is right that it's not a standard like ISO so it might not be be appropriate for your case. – Erwan Nov 05 '18 at 14:50
0

I had ran into a similar issue, but it was a case where I was regenerating and iso image of RHEL 7 installation media including a kickstart configuration and other supporting files. I mention this because I was beginning to exceed the allowable size that would fit on a single layer DVD. I never noticed it before, but I assume that the RHEL Packages directory had grew over subsequent releases. The mkisofs tool, I had found was making the image larger than expected. After some research I discoverd that I could just edit the iso in place with xorriso and this somehow allowed me to keep it to a size that would fit on a single layer DVD.

Here is the command that I used.

xorriso -boot_image any keep -dev rhel-server-7.8-x86_64-dvd_ks-v2r7.iso -cpr ../rhel-ks/* / --

In this specific case, you could try to create an iso image with an empty iso9660 filesystem using xorrisofs and then use xorriso to add the files you need to the image.

Example for a data cd/dvd image that doesn't need to be bootable.

xorrisofs -U -r -v -J -joliet-long -V "MyData" -volset "MyData" -A "MyData" -o ./mydata.iso 

xorriso -dev ./mydata.iso -cpr ./data_dir / --
Billy Hill
  • 1
  • 1