Is there MD5 (or similar) to a folder? How to verify if two folders are equal?

Question

I'd like to find an md5sum (or similar calculation) of a folder without compressing it into an archive.

For example, if in the folder MyFolder we have the files 1.txt, 2.txt and 3.txt, containing:

---

1.txt

The text into 1.txt

2.txt

The text into 2.txt

3.txt

The text into 3.txt

---

How can I calculate the md5sum of this entire MyFolder? Is there a way? I want to use this to verify if two folders have the same contents.

This can be util to verify if two cds or folders are equal. I'd like a easy way to do it.

Answer 1

The md5deep tool was developed for precisely this purpose. Many Linux distributions offer it in package form.

Answer 2

If you don't want to archive it, maybe you could do something like this

diff <(find folder1) <(find folder2)

You may have to adapt the find commands to be more accurate.

EDIT You could add -exec to your find call to compare the content of files. Something similar to this:

diff <(find folder1 -type f -exec md5sum {} \; | sort) <(find folder2 -type f -exec md5sum {} \; | sort)

Remember that you may want to adapt this.

Answer 3

One way to test could be to generate an md5sum based on the concatenation of all of the files in the folder and its subfolders. Bear in mind that this also requires that the files have the same names (as they must be in the same sort order). The following code should work:

#!/bin/bash

shopt -s nullglob
shopt -s globstar || { printf '%s\n' 'Bash 4 is required for globstar.' ; exit 1 ; }
(( $# == 2 )) || { printf '%s\n' "Usage: ${0##*/} olddir newdir" ; exit 2 ; }

for _file in "$1"/**/*; do [[ -f ${_file} && ! -L ${_file} ]] && _files_in_old_dir+=( "${_file}" ); done
for _file in "$2"/**/*; do [[ -f ${_file} && ! -L ${_file} ]] && _files_in_new_dir+=( "${_file}" ); done

(( ${#_files_in_old_dir[@]} )) || { printf '%s\n' 'No files in old dir.' ; exit 3 ; }
(( ${#_files_in_new_dir[@]} )) || { printf '%s\n' 'No files in new dir.' ; exit 4 ; }

_md5_old_dir=$(cat "${_files_in_old_dir[@]}" | md5sum)
_md5_new_dir=$(cat "${_files_in_new_dir[@]}" | md5sum)

{ [[ ${_md5_old_dir} == "${_md5_new_dir}" ]] && (( ${#_files_in_old_dir[@]} == ${#_files_in_new_dir[@]} )) ; } && printf '%s\n' 'Folders are identical.' || { printf '%s\n' 'Folders are not identical.' ; exit 3 ; }

If you truly care about the file names, etc, you could use a loop to compare what is in ${_files_in_old_dir} and ${_files_in_new_dir}. This should work for most cases (it at least checks the number of files in the dir and its subdirectories).

Answer 4

I noticed the original post is rather old, however, I think this information can still be valuable to those looking for a solution to verify that files are copied correctly. Rsync might be the best method to copy data and answers given in this thread are good, however for those not that experienced with Linux, I will try to give a more detailed explanation.

Scenario: You just copied data from a disk to another, with lots of sub-directories and files. You want to verify that all the data is copied correctly.

First check that md5deep is installed by issuing the command md5deep -v.

If you get a message saying something like 'command not found', then install md5deep by apt-get install md5deep.

It's assumed you only want to deal with regular files. If you want to deal with other types of files, refer to the -o flag in the md5deep manual. (man md5deep)

Now you are good to go, and we assume that you copied files from /mnt/orginal to /mnt/backup, substitute these for any directories you are using.

First change to the source directory, this is the original source for the files you copied or backed up:

cd /mnt/orginal

Then make a checksum of each file:

md5deep -rel -o f . >> /tmp/checksums.md5

This command explained:

-r enables recursive mode

-e displays progress indicator

-l enables relative file paths.

-o f only work on regular files (not block devices, named pipes etc.)

. tells md5deep to start in the current directory.

>> /tmp/checksums.md5 tells md5deep to redirect all output to /tmp/checksums.md5.

Note, if you want to overwrite content in previous versions of /tmp/checksums.md5, use > and not >>

Note that this command could take quite a while, depending on the io-speed and the size of the data. You could experiment with nice and/or ionice to increase performance of md5deep, but that's outside the scope of this answer.

When the creation of the check sums has finished, you now have a file that has entries similar to:

69c0a826b29c8f40b7ca5e56e53d7f83 ./oldconfig-11-09-2013/etc2/apm/event.d/20hdparm 651f3c7f79a14332f9fa7bb368039210 ./oldconfig-11-09-2013/etc2/apm/event.d/anacron 50d89784c1e201f68ff978b95ff4bdfb ./oldconfig-11-09-2013/etc2/apm/scripts.d/alsa e9b9131660a8013983bc5e19d7d669eb ./oldconfig-11-09-2013/etc2/ld.so.cache

The first column is the md5 check sum, and the second column is the relative path to the file the checksum belongs to.

If you want to see how many files exists in the checksum file, issue the command:

wc /tmp/checksums.md5 -l

Now, you want to check that the copied data is correct:

cd /mnt/backup

md5deep -o f -reX /tmp/checksums.md5 . >> /tmp/compare.result

The only difference from when we created the checksums is -X which displays the current hash of a file if the entry in the checksums.md5 file does not match. So by the end of the test, if /tmp/compare.result is blank, you can trust that all the files are copied correctly since the checksums matches.

Note that only files listed in the /tmp/checksums.md5 file will be checked for a correct checksum, if there's any additional files in the /mnt/backup directory, md5deep will not notify you about these.

Notes:

• You don't necessarily have to use redirection to store output files. Refer to the md5deep manual for further information.

• You might have to run md5deep commands as root, depending on the permissions of the files you're handling.

Answer 5

If you want to check recursively the differences between two directories /path1 and /path2 without using md5deep:

diff <(cd /path1 && find . -type f |xargs md5) <(cd /path2 && find . -type f |xargs md5)

Explanation:

• got into path1 to make all lines printed by find be relative to path1 (cd /path1)
• lists all files recursively in the current path (&& find . -type f)
• use each line output by find as input for md5 (| xargs md5)

The output will look like this if they are any differences:

< MD5 (./index.html) = 36b01762f0329b2c12a5186520c7d78a
< MD5 (./inline.js) = ce99823a4b2c24839a727c5781f59a36
< MD5 (./main.js) = 3a597404d3ba7f0a6e3cb093ef57ebb2
---
> MD5 (./index.html) = 3a3d7663a7b2871ff37b9081a53593f9
> MD5 (./inline.js) = 1bbd0ecfc75b578413105c6b9009f9b3
> MD5 (./main.js) = 0f44abe5084add3cabdc39feec0c699878c78