5

Quite simply, when a tar file is made on disk, would the extents be deduplicatable with extents inside and/or outside of the tar? I am asking in the theoretical sense, so if the extents of data are identical inside the tar (no shifting, or splitting within the extents in order to compact) then theoretically, the extents will match those outside the tar equally and so would be deduplicatable.

For example, if I were to tar a directory, then use block level deduplication, would the effective size of the tar, be the size of the additional headers, metadata and the end of archive marker.

Obviously I am talking about uncompressed tar, specifically GNU tar. I have looked at the GNU tar standard and it does seem to maintain the origional block data from what I have read, but maybe I have mis-interpreted what I have read.

lcd047
  • 7,160
  • 1
  • 22
  • 33
flungo
  • 341
  • 1
  • 2
  • 11
  • 1
    I suspect that if you were to use a dedup system on a disk containing both a tree and an uncompressed archive of that tree then, yes, you could very easily end up with a sparsed archive or tree. The member files of a `tar` archive are usually one-for-one identical to their originals. – mikeserv Jun 10 '15 at 13:14
  • That is what it seems like to me, that all the tar really does is wrap the data so that the file system treats it as one file. Hoping someone can confirm this. – flungo Jun 10 '15 at 14:31
  • [that is what happens with `tar`](http://unix.stackexchange.com/questions/208482/view-a-file-in-a-tar-archive-without-extracting-it/208523#208523) but I doubt if anyone can confirm it for you without your clarifying the dedup/fs systems you're using. They're not all one and the same, and if someone wanted to test it, how could they? – mikeserv Jun 10 '15 at 15:22
  • I am using btrfs with duperemove for deduplication. If its a case of testing then I can (and probably will) do some tests once I get my backup server organised. But in terms of how tar works, if it results in a data extent that is the same (but just part of the larger tar file), then deduplication should be possible. My concern was if it did anything weird like shifting files closer together or splitting extents. – flungo Jun 10 '15 at 15:56
  • @mikeserv I have updated my question to clarify I am looking for an answer that confirms how the `tar` is made and that a tar would have extents that match the untarred files. – flungo Jun 10 '15 at 16:00
  • did you look at that link? It discusses deconstructing a `tar` archive according to spec in something like passable detail. There's another link at the top of that answer where I use a few other GNU tools to pull apart and afterwards to put together my own `tar` archive from a pipeline. Also, `info tar`'s appendices get into some pretty nitty-gritty about the file-format - specifically sections on standards compliance and sparse files as I remember it. It maybe though that `btrfs` *wouldn't* dedup from a `tar` if you've enabled compression on the disk as a whole, I think. – mikeserv Jun 10 '15 at 16:58
  • The answer given here does not apply to `btrfs` volume, unfortunately, as it only discusses *in-band* dedup anf which is not a stable `btrfs` offering. The dedup offered by the `btrfs` user-space tools is one which scans and hashes extents and then *deallocating* duplicated blocks - it has nothing to do with inline CRC checks because the inline systems in develeopment consume too much memory, but *[Patches are being worked on*](https://btrfs.wiki.kernel.org/index.php/Deduplication) – mikeserv Jun 11 '15 at 00:45

1 Answers1

5

Generally, no. It would be possible to design a filesystem that provides this kind of deduplication, but it would be very costly, for very little practical benefit, so I doubt that it's been done. The issue is that deduplication only looks at aligned extents.

Deduplicating filesystems generally work at a block level. When the filesystem driver is about to store a block, it calculates a checksum for the block content and looks up this checksum in a table. If the table says that no block with this checksum exists, the block is stored and the checksum is added to the table. If the checksum is present in the table, the driver checks whether any of the blocks with that checksum is identical to the block that's about to be stored; if there is one, a new reference to this block is created, and if they aren't then the block is added.

As you can see, there's a cost to be paid whenever writing a block. But at least this cost is only paid once per write of a block. If file 1 contains aaaabbbbcccc, file 2 contains aabbbbcccc and the block size is 4, then the files do not contain any identical block, so no deduplication will take place. Detecting that file 2 is included in file 1 would require computing checksums for blocks at any alignment, at a prohibitive cost.

In general, the blocks of a file in a tar file are not aligned with the blocks of the filesystem. A file in a tar archive can start at any offset that's a multiple of 512 (the tar block size), but most filesystems use a larger block size. If the start of a file inside the archive happens to be aligned with the start of a filesystem block, then that file will be deduplicated if the opportunity presents itself. Typical filesystem block sizes are larger than that, though since they are a multiple of 512, deduplication will occasionally happen, e.g. about 1 in 8 for 4096-byte blocks assuming a uniform distribution of file sizes modulo 4096 (which isn't quite true, so the probability is in fact somewhat less).

The typical use case for deduplication is files that are identical or mostly identical: backup copies, old versions of a file, etc. Transformed files are not typical. Uncompressed archives are especially not typical.

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175