9

There is bdiff(1) command in Solaris, which allow you to diff(1) files with size bigger than your RAM size (documentation).

Is there something like that in Linux? I tried googling but I don't find which package has bdiff in Ubuntu.

AntonioK
  • 1,151
  • 2
  • 15
  • 28
  • Related: http://superuser.com/questions/174283/how-to-diff-large-files-on-linux –  May 27 '13 at 12:01
  • possible duplicate: http://unix.stackexchange.com/questions/2150/diffing-two-big-text-files – slm May 27 '13 at 12:02

1 Answers1

14

bdiff appears to be available on Linux (at least as part of the Heirloom Toolchest).

diff

I would probably just use regular old diff with this switch however:

diff --speed-large-files bigfileA bigfileB

Why it doesn't work?

See comment by @EvanTeitelman, --speed-large-files doesn't affect how files are loaded into memory.

Can be demonstrated/confirmed not to work using the following command:

fallocate -l 10G testa; fallocate -l 10G testb && \
        diff --speed-large-files -a testa testb

bsdiff

Hard to confirm this but I found a tool called bsdiff which derives from bdiff. I've confirmed that this tool is in Ubuntu, simply apt-get install bsdiff.

Why it might work?

Again thanks @EvanTeitelman in the comments, the bsdiff above is a diff tool for binary files. bsdiff is a binary diff tool and can deal with large files. It's unclear just how large. See the following links to a thread which discuss it's use.

rdiff

I think you could also use rdiff to do this as well. Rdiff is able to deal with very large files.

  1. Create a signature of one file:

    rdiff signature A sigs.txt

  2. Use generated signature file sigs.txt and the other big file B to create the delta:

    rdiff delta sigs.txt B deltaAB.txt

  3. Delta contains all the info you need to recreate file B when you have just A and the delta file deltaAB.txt.

    To recreate B, run:

    rdiff patch A deltaAB.txt B

Why it works?

I found this blog post titled: A Better diff Or What To Do When GNU diff Runs Out Of Memory ("diff: memory exhausted"), which reports that a rdiff of 4.5GB files only consumed ~66MB of RAM.

lfhex

lfhex is an application for viewing and editing files in hex, octal, binary, or ascii text. The main strength of lfhex is it's ability to work with files much larger than system memory. It's a GUI tool however.

screenshot

                  ss of lfhex]![ss of lfhex

References

slm
  • 363,520
  • 117
  • 767
  • 871
  • Although this isn't always documented, GNU diff allows you to use `-H` as a synonym for `--speed-large-files`. –  May 27 '13 at 12:10
  • 2
    `bsdiff` is a binary diff tool, not a large-file diff tool. –  May 27 '13 at 12:17
  • I read that as well on several forums but it isn't documented in `man diff`. – slm May 27 '13 at 12:18
  • 3
    Unfortunately, it seems as though the `--speed-large-files` flag doesn't affect the manner in which GNU diff loads files into memory. Try running `fallocate -l 10G testa; fallocate -l 10G testb && diff --speed-large-files -a testa testb` to confirm this. (Or take a look at the source code.) –  May 27 '13 at 12:25
  • Is it possible to turn the output of rdiff or bsdiff into a unified diff format? I've been searching for a solution to no avail :( – unhammer Mar 05 '14 at 09:18
  • 2
    I managed to build `bdiff` from the Heirloom Toolchest after replacing `/sbin/sh` by `/bin/sh` in the makefiles. Now when I try to execute it in place, I get `bdiff: Can not execute '/usr/5bin/diff'`. Sorry, I do not want to install anything to `/usr/5bin/`. This is not a viable solution. The other options mentioned here do not work for me because I want to eyeball the differences as text. – Christian Pietsch Feb 02 '15 at 17:57
  • 1
    PS: Surprisingly, `lfhex -c file1 file2` works well for me after setting View –> Editing base –> ASCII for both panes. – Christian Pietsch Feb 02 '15 at 18:13