2

Perhaps I'm doing something very stupid here, but Google hasn't been very helpful in resolving the problem.

I have an archive of files I made for backup purposes. I've generated an SHA1 checksum file from this archive:

sha1sum myarchive.tar > myarchive.tar.sha1

The contents of this file are as follows:

6f5d7bdd71fe25ed8e881265fdb8a8bbcdaa41c1  myarchive.tar

I have also run the SHA1 process in the terminal without piping to a file:

sha1sum myarchive.tar

This gives me the output:

6f5d7bdd71fe25ed8e881265fdb8a8bbcdaa41c1  myarchive.tar

Clearly these checksums are the same. However, when I run the verification command, with the archive and its SHA1 file next to each other in the same directory:

sha1sum -c myarchive.tar.sha1

I get an error message saying that the checksum does not match:

myarchive.tar: FAILED
sha1sum: WARNING: 1 computed checksum did NOT match

Obviously something is wrong here, but I have no idea what it could be. Can anyone enlighten me?

EDIT: Interestingly, doing two consecutive MD5s on the file comes up with two different checksums. Now I'm very confused.

$ md5sum myarchive.tar
9a15036eed341613bbcf2c4b53a09859  myarchive.tar
$ md5sum myarchive.tar
a662d6b469627c62f2b03ee0df067436  myarchive.tar

EDIT2: Additional context:

  • This is on real hardware (my Ubuntu MATE 19.10 desktop machine).
  • The archive I've made was destined for a Blu Ray backup disc. It's 22.6GB in size.
  • Performing an SHA1 verification of the file as burned to the Blu Ray disc ends up being successful.

EDIT3: In response to requests to see the output of dmesg, there appear to be a few errors like the following:

[ 7102.039819] perf: interrupt took too long (2502 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
[ 8278.017874] sr 4:0:0:0: [sr0] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 8278.017876] sr 4:0:0:0: [sr0] tag#0 Sense Key : Blank Check [current] 
[ 8278.017877] sr 4:0:0:0: [sr0] tag#0 Add. Sense: No additional sense information
[ 8278.017878] sr 4:0:0:0: [sr0] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
[ 8278.017879] blk_update_request: critical target error, dev sr0, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 8278.019391] sr 4:0:0:0: [sr0] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 8278.019392] sr 4:0:0:0: [sr0] tag#0 Sense Key : Blank Check [current] 
[ 8278.019392] sr 4:0:0:0: [sr0] tag#0 Add. Sense: No additional sense information
[ 8278.019393] sr 4:0:0:0: [sr0] tag#0 CDB: Read(10) 28 00 00 00 00 00 00 00 01 00
[ 8278.019394] blk_update_request: critical target error, dev sr0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 8278.019396] Buffer I/O error on dev sr0, logical block 0, async page read

I think this is to do with my USB Blu Ray drive, though someone may be able to correct me.

Vlastimil Burián
  • 27,586
  • 56
  • 179
  • 309
NoodleCollie
  • 183
  • 1
  • 7
  • is `sha1um` a typo or an alias? Does `sha1sum -c <(sha1sum myarchive.tar)` work? It's a bit unfortunate that `sha1sum` does not print out which result it expected vs. which one it got, so you can only guess... – frostschutz Jul 19 '20 at 14:02
  • Sorry, `sha1um` was a typo on my part. `sha1sum -c <(sha1sum myarchive.tar)` validates correctly. – NoodleCollie Jul 19 '20 at 14:07
  • Are you in different directories (containing the same file names) or have you modified `myarchive.tar` between creating the checksum file and the test of it? – sudodus Jul 19 '20 at 14:08
  • Same directory, same terminal session, commands executed one after another. No modifications whatsoever. – NoodleCollie Jul 19 '20 at 14:09
  • @x6herbius then either the checksum really changed or something happened to your `myarchive.tar.sha1` file... make a new one then compare them with `cmp -l` or `diff -u`. or run it as `sha1sum -c <(sha1sum myarchive.tar | tee myarchive.tar.tee)` and see what's in the .tee file compared to .sha1 file – frostschutz Jul 19 '20 at 14:13
  • There might be a read error. Please check the file system (with `fsck`) and the drive's S.M.A.R.T. information. Or is some process adding data to the archive? Is the file size constant, or is it changing? – sudodus Jul 19 '20 at 14:13
  • I've added to the original post: performing an SHA1 verification of the file as burned onto the Blu Ray comes out OK, and it definitely cannot be being modified in that case. Additionally, performing an SHA1 on a sample text file also validates correctly on my hard drive. Could it be to do with the size of the archive? If it's larger than my RAM size, could that impact how it's being read? – NoodleCollie Jul 19 '20 at 15:18
  • Nothing's overclocked, I've barely touched the BIOS on this system at all. Given things appear to check out once I've burned the disc, I think resetting the BIOS is going to be more effort to solve this problem than I really need. – NoodleCollie Jul 19 '20 at 16:32

1 Answers1

2

There are 2 things that need some scrupulous investigation in your case.

Blu-Ray disc

As proven with your dmesg output snippet, your Blu-Ray disc has reached end-of-life. There is no way to fix that disc, unless the Blu-Ray drive manages to read the data on the Blu-Ray disc 100% fine, which is improbable at this point.

Personal recommendation regarding Blu-Ray discs: If ever I have to use for whatever reason my Blu-Ray Drive to burn something, I sure use the M-Disc (wiki) Blu-Ray discs. For highly sensitive small data it's close to perfect storage.

Storage of your computer

The varying consecutive checksum read from your computer storage would bother me more, if that was read from your storage, that is. You could in theory have bad RAM, but that is not much probable. My advice is to watch the output of dmesg (man page) for a long period of time, example of such command follows:

\dmesg --human --color=auto --ctime --level=err,warn --follow

That could help you identify if your storage, be it a classic SATA HDD, or modern M.2 NVMe SSD, it does not matter, is failing. If you find there actually is a problem with your storage, better have backups prepared.

I hope this answer helps someone in similar position.

Vlastimil Burián
  • 27,586
  • 56
  • 179
  • 309
  • It's a little disturbing that the Blu Ray drive/disc has reached the end of its life, since I unpacked both from Amazon brand new yesterday. It was also the SHA1s from the files on the BD that actually *do* validate, as opposed to when the files are on my desktop. I will check my hard disks, though, and continue backing everything up. Thanks for your help. – NoodleCollie Jul 19 '20 at 17:56
  • @x6herbius Happy that we got somewhere and didn't reset your BIOS in the meantime :) If there is something missing, just comment, and once I get up in the morning, I'll fill in the gaps. Cheers and good luck! – Vlastimil Burián Jul 19 '20 at 18:00
  • "You could in theory have bad RAM, but that is not much probable." Funnily enough, for the last year or so I'd been getting random crashes on my machine (namely Firefox and GCC, which should be rock solid). Turns out my ram was bad after all. This could have had something to do with it! – NoodleCollie Feb 16 '23 at 08:28