Generate SFV file, optimizing hard links

Question

I'm currently running cfv -C -rr to generate a SFV file that contains the CRC32 checksums of all files in all subdirectories. The issue is that because there are so many hardlinks, this process is taking much longer than it should take. Is there any way to get cfv to ignore hardlinks?

cfv can be found here. The man page doesn't mention hardlinks. Are there any alternative applications?

Do you mean you want `cfv` to ignore files that are multiply linked (which I think it what you've actually asked), or that you want `cfv` to ignore the second and subsequent times it finds a file? — roaima, May 17 '15 at 22:15
I want cfv to ignore the second and subsequent times it finds a file so that there is no wasted time. I want all content checksums to only be calculated once. — user115746, May 17 '15 at 22:16

score 0 · Answer 1 · answered May 18 '15 at 22:09

I assume that file names don't contain newlines or initial or trailing whitespace (the SFV format doesn't support those anyway) and that you have GNU utilities (non-embedded Linux, Cygwin).

First generate a mapping from file names to inode numbers, and sort it by inode numbers.

tmpdir=$(mktemp -d)
find .* * -xdev -name .. -prune -o -type f -printf '%i %p\n' |
sort -k 1n > "$tmpdir/inodes.txt"

Generate a checksum file for the first file with each inode number.

<"$tmpdir/inodes.txt" awk '$1 != previous {previous = $1; sub(/^[0-9]+ /,""); print}' |
xargs cfv -C - >"$tmpdir/1.cfv"

Now complete the checksum file taking advantage of the hard links. I don't know if cfv prints the files in the order in which they are passed on the command line, if it does you can save memory at this step because the temporary CFV file and the inode file will be in ordered in the same way.

awk '
    FNR==1 {
        match($0, / +[[:alnum:]] *$/);
        cksums[substr($0, 1, RSTART-1)] = substr($0, RSTART);
        next
    }
    { inode = $1; sub(/^[0-9]+ /, ""); }
    previous != inode { cksum = cksums[$0] }
    { print $0, cksum }
' "$tmpdir/1.cfv" "$tmpdir/inodes.txt" >full.cfv

You can then remove the temporary files.

Warning: untested code.

Generate SFV file, optimizing hard links

1 Answers1