Is it possible to re-create a directory structure that creates a binary identical bsdcpio output stream?

Question

Building initramfs files from script results in binary not comparable cmp files even though diff -r /rootfs1/ /rootfs2/ exits with 0 thus diff finds no differences.

The differences I can find are the Access and Change dates. For example:

# stat /rootfs1/config
  File: /rootfs1/config
  Size: 275         Blocks: 8          IO Block: 4096   regular file
Device: 28h/40d Inode: 119120      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-12-12 01:31:03.800453381 +0100
Modify: 2017-12-05 20:11:38.000000000 +0100
Change: 2017-12-12 01:02:38.607238288 +0100
 Birth: -
# stat /rootfs2/config
  File: /rootfs2/config
  Size: 275         Blocks: 8          IO Block: 4096   regular file
Device: 28h/40d Inode: 119881      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-12-12 01:28:02.913799257 +0100
Modify: 2017-12-05 20:11:38.000000000 +0100
Change: 2017-12-12 01:20:44.920496295 +0100
 Birth: -

Regression

1. I have checked that the find command returns all directory items in exactly the same order, OK.

2. I tried to touch reference every entry in the directory structures to an identical source file. Still with no luck for creating an identical byte stream.

3. The lz4 -l bsdcpio resulting initramfs images already differ at byte 5:

   $ cmp /tmp/1.img /tmp/2.img
/tmp/rootfs1/initramfs.img /tmp/rootfs2/initramfs.img differ: char 5, line 1

4. mount -o remount,noatime /tmp does still update "time of last status change" according to stat %z

The bsdcpio scripting code is like:

pushd "$BUILDROOT" >/dev/null
find -mindepth 1 -printf '%P\0' | LANG=C bsdcpio -0 -o -H newc | lz4 -l > "$out"

Is there any chance in having an identical output byte stream for consecutive script runs where $BUILDROOT is a new temporary folder on each run?

Answer 1

Access time and change time are not stored in the cpio newc structure. What is though is the inode number and the filesystem's device number.

           struct cpio_newc_header {
                   char    c_magic[6];
                   char    c_ino[8];
                   char    c_mode[8];
                   char    c_uid[8];
                   char    c_gid[8];
                   char    c_nlink[8];
                   char    c_mtime[8];
                   char    c_filesize[8];
                   char    c_devmajor[8];
                   char    c_devminor[8];
                   char    c_rdevmajor[8];
                   char    c_rdevminor[8];
                   char    c_namesize[8];
                   char    c_check[8];
           };

You could replicate inode numbers consistently by extracting the files in a new file system each time, but you won't be able to have two separate directory structures that exist at the same point in time produce the same cpio archive as the combination of dev+ino is by design unique on a given system.

To produce a consistent cpio archive, you'd need to create a new file system each time, and make sure it gets assigned the same device number, and make sure each file is assigned the same inode number each time.

Or alternatively, you could post-process the resulting cpio archive to replace the device and inode numbers. Something like:

printf '%s\0' **/*(D) | bsdcpio -0 -o -H newc | perl -0777 -pe '
  $dev = sprintf("%016x", 1);
  while (substr($_, 0, 6) eq "070701") {
    ($inode, $size, $nsize) = map hex, unpack("x6a8x40a8x32a8", $_);
    $nsize += (2-$nsize) % 4; $size += (-$size) % 4;
    $inode{$inode} ||= ++$n;
    substr($_, 6, 8) = sprintf("%08x", $inode{$inode});
    substr($_, 62, 16) = $dev;
    print substr($_, 0, 110+$size+$nsize, "")
  }'

Where the dev is hardcoded as 1 and inode is incrementing from 1 for each entry (but still equal for hardlinks).

Using zsh's **/*(D) to get a sorted list of file paths instead of find's whose order is not guaranteed to be consistent.