8

Well, the task is simple: a part of my script has to compute both md5 and sha1 hashes. The input is a file - big file - and hashes have to be put into MD and SH variables for later output composition.

While the processed files are realy big (hundreds of GB) I try to use some kind of multiple use of data once read. I found something called process substitution what I adopted in the next way: 

$ dd if=big.tgz 2>/dev/null |tee >(sha1sum ) > >(md5sum ) ;

instead of:

$ SH=$(sha1sum big.tgz); MD=$(md5sum big.tgz);

But I found the next:

  • there is apparently no resource neither time saving as both takes aprox. 40s (for 4.776 GB file)

  • I have no idea how to save the result of the subprocess >(md5sum ) into the variable MD to use it later in the script

I tried to understand the pipexec but even the nice color illustrations no success until yet.

Is there some other way to redirect the output to a vriable, other than VAR=$(command) ?

schweik
  • 1,160
  • 8
  • 16
  • Of course it won't go faster if instead of two processes freely eating off the same data you have made some 6 or 7, swapping it in small chunks between them ;-) Try this `set $(cut -d' ' -f1 <(md5sum $big) <(sha1sum $big)); md5sum=$1; sha1sum=$2`. –  Jun 04 '19 at 22:36
  • btw, you have some crazily fast rig there which is able to md5sum a ~5TB file in 40 seconds;-) –  Jun 04 '19 at 22:47
  • Sorry, mosvy, decimal point disappeared from the number while editing. Of course, 40 sec is for 4.776 GB; if I were stand with 5GB , it could be more correct. – schweik Jun 05 '19 at 08:24
  • just in case that isn't clear, the process substitutions (eg. the `a`, `b`, `c` in `cmd <(a) <(b) <(c)`) are always run in parallel, not sequentially. Also, modern systems use (a lot of) caching, so if two completely different processes are reading the same file at the same time and their speed doesn't differ that much, the data will be fetched just once from the backing storage. –  Jun 05 '19 at 15:27

3 Answers3

4

On the subject of performance, you may be limited by CPU. Actually 4.7TB in 40 seconds for both MD5 and sha1sum feels fast. So even if you work this way. For what it's worth you will have reduced disk IO.

You really don't need to dd for this. You can also just write the output of sha1sum and md5sum direct to a file for later use

tee < big.tgz  >(sha1sum > big.tgz.sha1 ) > >(md5sum > big.tgz.md5 )
sha1=`cat big.tgz.sha1`
md5=`cat big.tgz.md5`

I'm suggesting using temp files like this (big.tgz.sha1 and big.tgz.md5) because AFAIK there's no way to simultaneously set two variables with different values. You can capture one straight into a variable but not both. And allowing both md5sum and sha1sum to write to the same stdout at the same time might cause unpredictable problems.

Philip Couling
  • 17,591
  • 5
  • 42
  • 82
  • Well, the `dd` was used to indicate that I may get the data from another process (carving, compressing); If I understand it - I may rely on the kernel´s buffering that the data won´t be read twice. – schweik Jun 05 '19 at 08:38
2

Well, you can just add another redirect inside:

tee < big.tgz >(sha1sum > big.tgz.sha1sum) >(md5sum > big.tgz.md5sum)

You could also take the output as is, considering it's easy to differentiate between sha1 and md5 (different length so there is no confusion which is which).

There are also utilities that calculate multiple checksums on their own, without jumping through hoops with tee.

Actually the above can also be written without tee:

sha1sum big.tgz > big.tgz.sha1sum &
md5sum big.tgz > big.tgz.md5sum
wait # for sha1sum

In theory this is bad as data is being read from disk twice.

In practice, running both readers in parallel (& background) should allow the disk cache to handle it so that data is effectively still read only once. This assumes the hash calculation is fast and I/O is slow so no process can run away from the other.

( I previously posted about reading twice in another context here: Using pv with md5sum - although it usually works, there are some risks involved so tee is still the more reliable method. )

frostschutz
  • 47,228
  • 5
  • 112
  • 159
2

parset from GNU Parallel is made for setting variables in parallel, and --tee will tee the input to multiple commands:

parset md5,sha1,sha256 --pipe --tee {} ::: md5sum sha1sum sha256sum < bigfile
echo $sha1

parset sumarr --pipe --tee {} ::: md5sum sha1sum sha256sum < bigfile
echo ${sumarr[1]}
Ole Tange
  • 33,591
  • 31
  • 102
  • 198