bitwise-OR 2 binary files

Question

I did 2 rescue attempts on a dying HDD a while back; I ran (GNU) ddrescue first, and then straight dd with manual seeking. I want to get the best of both images. Since any empty stretches in the files will just be 0's, bitwise-AND should be sufficient to merge the two files.

Is there a utility that allows me to create a file that is the OR of two input files?

(I am using ArchLinux, but I'm happy to install from source if it's not in the repos)

Answer 1

I don't know of a utility that does this, but it should be pretty easy to write a program to do this. Here's a skeletal example in python:

#!/usr/bin/env python
f=open("/path/to/image1","rb")
g=open("/path/to/image2","rb")
h=open("/path/to/imageMerge","wb") #Output file
while True:
     data1=f.read(1) #Read a byte
     data2=g.read(1) #Read a byte
     if (data1 and data2): #Check that neither file has ended
          h.write(chr(ord(data1) | ord(data2))) #Or the bytes
     elif (data1): #If image1 is longer, clean up
          h.write(data1) 
          data1=f.read()
          h.write(data1)
          break
     elif (data2): #If image2 is longer, clean up
          h.write(data2)
          data2=g.read()
          h.write(data2)
          break
     else: #No cleanup needed if images are same length
          break
f.close()
g.close() 
h.close()

Or a C program that should run faster (but is significantly more likely to have an unnoticed bug):

#include <stdio.h>
#include <string.h>

#define BS 1024

int main() {
    FILE *f1,*f2,*fout;
    size_t bs1,bs2;
    f1=fopen("image1","r");
    f2=fopen("image2","r");
    fout=fopen("imageMerge","w");
    if(!(f1 && f2 && fout))
        return 1;
    char buffer1[BS];
    char buffer2[BS];
    char bufferout[BS];
    while(1) {
        bs1=fread(buffer1,1,BS,f1); //Read files to buffers, BS bytes at a time
        bs2=fread(buffer2,1,BS,f2);
        size_t x;
        for(x=0;bs1 && bs2;--bs1,--bs2,++x) //If we have data in both, 
            bufferout[x]=buffer1[x] | buffer2[x]; //write OR of the two to output buffer
        memcpy(bufferout+x,buffer1+x,bs1); //If bs1 is longer, copy the rest to the output buffer
        memcpy(bufferout+x,buffer2+x,bs2); //If bs2 is longer, copy the rest to the output buffer
        x+=bs1+bs2;
        fwrite(bufferout,1,x,fout);
        if(x!=BS)
            break;
    }
}

Answer 2

Python

with open('file1', 'rb') as in1, open('file2', 'rb') as in2, open('outfile', 'wb') as out:
    while True:
        bytes1 = in1.read(1024)
        bytes2 = in2.read(1024)
        if not bytes1 or not bytes2:
            break
        out.write(bytes(b1 | b2 for (b1, b2) in zip(bytes1, bytes2)))

This is roughly 10× faster than Chris's Python solution due to reading 1024 bytes at a time. It also uses the with open pattern as this is more reliable at closing files (e.g. in case of error).

This seems to work for me with Python 3.6.3 (successfully merging 2 partial torrent files), but has not been thoroughly tested.

Perhaps the if ...: break pattern could be removed and instead use while in1 or in2:

Answer 3

Preliminary notes:

• The title is kinda XY problem. It's good the question body clarifies your specific case. I mean you don't exactly want to "bitwise-OR 2 binary files", arbitrary files. You want to merge two HDD images, one of them was created by ddrescue.

• dd is not the best tool for reading from faulty devices. conv=sync,noerr is often advised but it's not enough. Please read this. Unless you also used iflag=fullblock, the image generated by dd cannot be trusted. Having ddrescue I would not use dd at all. The right way to retry is to run ddrescue again with the same mapfile.

  My answer assumes your image generated by dd can be trusted.

---

If you have the mapfile from the ddrescue run, then use --fill-mode.

When ddrescue is invoked with the --fill-mode option it operates in "fill mode", which is different from the default "rescue mode". That is, if you use the --fill-mode option, ddrescue does not rescue anything. It only fills with data read from infile the blocks of outfile whose status character from mapfile coincides with one of the type characters specified as argument to the --fill-mode option.

^(source)

In your case it will be like:

ddrescue --OPTIONS --fill-mode='?*/-' image_from_dd image_from_ddrescue mapfile_from_ddrescue

where ?*/- mean:

? non-tried block
* failed block non-trimmed
/ failed block non-scraped
- failed block bad-sector(s)

And there's + you don't want in your command:

+ finished block

--OPTIONS denotes some options you originally used; options that, if changed, would make the tool misinterpret the mapfile. The set contains --sector-size for sure. Hint: the original command line may be stored in the mapfile as a comment.

If you used --input-position and/or --output-position (and/or skip= and/or seek= with dd) then some recalculations are required, I think. Without testing I cannot give you the right formula. You probably did not use these options anyway.

Thanks to --fill-mode all sectors that are not considered rescued by your original ddrescue will be filled with data taken from respective fragments of image_from_dd.

---

If the mapfile from the original ddrescue run is unavailable or if you really want to overwrite all sectors filled with zeros (even ones where ddrescue has actually read zeros without errors), then you should ask ddrescue to generate a mapfile:

When ddrescue is invoked with the --generate-mode option it operates in "generate mode", which is different from the default "rescue mode". That is, if you use the --generate-mode option, ddrescue does not rescue anything. It only tries to generate a mapfile for later use.

[…]

ddrescue can in some cases generate an approximate mapfile, from infile and the (partial) copy in outfile, that is almost as good as an exact mapfile. It makes this by simply assuming that sectors containing all zeros were not rescued.

[…]

Note that you must keep the original offset between --input-position and --output-position of the original rescue run.

^(source)

The manual gives this example:

ddrescue --generate-mode infile outfile mapfile

but my tests indicate infile is only formally checked for existence and general sanity, then it doesn't matter; only outfile is used to generate mapfile. So you don't need the original HDD at all, the command may be:

ddrescue --generate-mode /dev/zero image_from_ddrescue new_mapfile

After it finishes you will have the new_mapfile. Use it with --fill-mode, like described above (--OPTIONS should match the relevant options used with --generate-mode).

Note you can work in the opposite direction: generate a (different) mapfile from image_from_dd and use --fill-mode to write parts of image_from_ddrescue into image_from_dd.

Answer 4

This is not bitwise or, but it works for (whole blocks of) zeroes:

dd conv=sparse,notrunc if=foo of=baz
dd conv=sparse,notrunc if=bar of=baz

Due to sparse it skips writing anything that is zero in the source file.

So baz will look like bar, plus whatever was zero in bar but not zero in foo.

In other words if there's non-zero data that's not identical in foo and bar, bar wins.