2

I create a .tgz/tarball with $(npm pack).

I then run:

sha1sum oresoftware-npp-0.0.1001.tgz

and I get:

77c58da68593dcdcd14bb16a37f5f63ef42bab63  oresoftware-npp-0.0.1001.tgz

I want to compare that shasum against another tarball on a remote server. I can query for a shasum for a tarball on the NPM registry, with:

 npm view @oresoftware/npp@latest dist.shasum

which yields:

3c2e7328110ba57e530c9938708b35bde941c419

this shasum is different than the other one above, but that's expected, since I changed the contents of the .tgz tarball file.

my question is 3 fold:

  1. When I generated a sha1sum of the .tgz file resulting from npm pack, is that the right way to do it? To generate the sha1sum after the tar file is created?

  2. I assume that the the sha1sum would be identical if the tarballs had identical contents? Would they differ if the files were created/modified at different times even if they have otherwise the same contents?

  3. Is there a better way to check if two tarballs have the same contents? That's all I am trying to do.

Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
Alexander Mills
  • 9,330
  • 19
  • 95
  • 180
  • A `tar` archive contains information about ownership and timestamps on files, so to compare two *archives* generated on two separate machines would likely not work. You would be better off extracting the content of the two archives and comparing that. I'm not writing an answer because I don't know, off the top of my head, of a simple way of doing that. – Kusalananda Aug 04 '18 at 18:41
  • For comparing two archives, there is no software sulution. Since 25 years, `star` however supports to compare an archive with the filesystem and allows to configure what to compare while doing that. – schily Aug 04 '18 at 21:39
  • @schily what about this: `tar -xOzf foo.tgz | sort | sha1sum` – Alexander Mills Aug 05 '18 at 02:00
  • Since `-O` usually tells `tar` to create archives compatible to the 1977 format, you seem to seem to use a different program. Let me assume that you have GNU tar in mind. With that tar clone and that command, the content of the files is extracted to stdout. If you did omit `sort`, you would get a definite value to compare tar archives with the same order of files inside. With the `sort` call, the result does not help you as you could get the same final result even in case there are differences. – schily Aug 05 '18 at 08:20

2 Answers2

2

The checksums available from the NPM registry provide two features: they allow you to verify that your download hasn’t been corrupted, and if you can verify the checksums out of band, that the downloaded files haven’t been altered. Unless NPM archives are built reproducibly, the checksums don’t allow you to verify that an archive you build yourself using npm pack contains what it’s supposed to.

The issue with tarballs is that they contain metadata: the ownership, permissions and timestamps of the stored files, as stored by tar, and on top of that, compression metadata. If values for all of these are pre-agreed, they can be specified to override the values obtained from the file system, but that requires pre-agreement.

To compare the contents of two arbitrary tarballs, the only reliable way is to extract their contents and compare that.

Stephen Kitt
  • 411,918
  • 54
  • 1,065
  • 1,164
  • 1
    Thanks Stephen, so yes say I go through all the files in the tarball and compare them - name of file and size of file, can you maybe explain a good place to start with that? I don't care about file modification date, but I do care about file permissions. I figure if the file count across the tarballs is the same, the file names are the same, and all the file sizes are the same, then probably good enough. – Alexander Mills Aug 04 '18 at 19:10
  • 1
    I reckon you should also compare the files’ contents; I wouldn’t consider comparing file names and sizes as sufficient. – Stephen Kitt Aug 04 '18 at 21:00
  • In my testing, the shasum thing seems to work. I publish the tarball to NPM, then I retrieve the shasum from NPM. Then I copied the files with rsync to another local folder, and then generated a new tarball with npm pack and got the shasum, and it was the same as the one from the remote registry. Not sure how meaningful that is, but maybe the shasum works in this case. I have to read more about how sha1sum works. – Alexander Mills Aug 04 '18 at 22:05
  • As an example, if you have a .tgz file in a Git repo, Git can tell you if that .tgz file changed, I just am not sure how exactly. – Alexander Mills Aug 04 '18 at 22:07
  • something like this perhaps: https://stackoverflow.com/questions/51689992/how-to-use-sha1sum-to-verify-contents-of-a-tarball – Alexander Mills Aug 04 '18 at 22:28
  • git knows that a file changed because its SHA-1 sum changed. In your `npm` test, `rsync` ensures the metadata of your copied files is identical to the originals’, so the tarball ends up with the same contents (if the files are stored in the same order, which appears to be the case); if compression doesn’t add its own timestamp, the resulting compressed tarball will be identical. – Stephen Kitt Aug 05 '18 at 21:16
0

Whether this checksum comparison is useful for you depends on what tar command you are using and that tar archive type is used.

star e.g. includes all three time stamps since 1986 and this causes tar archives to differ even with the same content, as the last acess time differs.

With the new POSIX tar enhancements from 2001, there are also all three time stamps - depending on whether the tar implementation implements the original standard or the later change.

If you don't care about time stamps, comparing archive checksums is not the right way to go.

Comparing two tar archives and making statements from that is thus usually impossible.

If you however may unpack one of the two archives, there is a nice way to compare content and meta data with a configurable set of meta data that is used for the comparison. Use star -diff, see http://schilytools.sourceforge.net/man/man1/star.1.html

If you e.g. call:

star -diff -v diffopts=!times < archive.tar.gz

the file content and all meta data except for all time stamps is compared.

schily
  • 18,806
  • 5
  • 38
  • 60