How can I do a "copy if changed" operation?

Question

I would like to copy a set of files from directory A to directory B, with the caveat that if a file in directory A is identical to a file in directory B, that file should not be copied (and thus its modification time should not be updated). Is there a way to do that with existing tools, without writing my own script to do it?

To elaborate a bit on my use-case: I am autogenerating a bunch of .c files in a temporary directory (by a method that has to generate all of them unconditionally), and when I re-generate them, I'd like to copy only the ones that have changed into the actual source directory, leaving the unchanged ones untouched (with their old creation times) so that make will know that it doesn't need to recompile them. (Not all the generated files are .c files, though, so I need to do binary comparisons rather than text comparisons.)

(As a note: This grew out of the question I asked on https://stackoverflow.com/questions/8981552/speeding-up-file-comparions-with-cmp-on-cygwin/8981762#8981762, where I was trying to speed up the script file I was using to do this operation, but it occurs to me that I really should ask if there's a a better way to do this than writing my own script -- especially since any simple way of doing this in a shell script will invoke something like cmp on every pair of files, and starting all those processes takes too long.)

Answer 1

You can use the -u switch to cp like so:

$ cp -u [source] [destination]

From the man page:

   -u, --update
       copy only when the SOURCE file is newer than the destination file or 
       when the destination file is missing

Answer 2

rsync is probably the best tool for this. There are a lot of options on this command so read man page. I think you want the --checksum option or the --ignore-times

Answer 3

While using rsync --checksum is a good general way to "copy if changed", in your particular case there is an even better solution!

If you want to avoid unnecessarily recompiling files you should use ccache which was built for exactly this purpose! In fact, not only will it avoid unnecessary recompiles of your auto-generated files, it will also speed things up whenever you do make clean and re-compile from scratch.

Next I'm sure you'll ask, "Is it safe?" Well, yes, as the website points out:

Is it safe?

Yes. The most important aspect of a compiler cache is to always produce exactly the same output that the real compiler would produce. This includes providing exactly the same object files and exactly the same compiler warnings that would be produced if you use the real compiler. The only way you should be able to tell that you are using ccache is the speed.

And it's easy to use it by just adding it as a prefix in the CC= line of your makefile (or you can use symlinks, but the makefile way is probably better).

Answer 4

This should do what you need

diff -qr ./x ./y | awk '{print $2}' | xargs -n1 -J% cp % ./y/

Where:

• x is your updated/new folder
• y is the destination you want to copy to
• awk will take the second argument of the each line from the diff command (maybe you will need some extra stuff for filenames with space - can't try it now)
• xargs -J% will insert the file name to cp at the proper place

Answer 5

I like to use unison in favor of rsync because it supports multiple masters, having already setup my ssh keys and vpn separately.

So in my crontab of only one host I let them synchronize every 15 minutes:

*/15 * * * * [ -z "$(pidof unison)" ] && (timeout 25m unison -sortbysize -ui text -batch -times /home/master ssh://192.168.1.12//home/master -path dev -logfile /tmp/sync.master.dev.log) &> /tmp/sync.master.dev.log

Then I can be developing on either side and the changes will propagate. In fact for important projects I have up to 4 servers mirroring the same tree (3 run unison from cron, pointing to the one that doesn't). In fact, Linux and Cygwin hosts mixed--except don't expect sense out of soft links in win32 outside the cygwin environment.

If you go this route, make the initial mirror on the empty side without the -batch, i.e.

unison -ui text  -times /home/master ssh://192.168.1.12//home/master -path dev

Of course there is a config to ignore backup files, archives, etc.:

 ~/.unison/default.prf :
# Unison preferences file
ignore = Name {,.}*{.sh~}
ignore = Name {,.}*{.rb~}
ignore = Name {,.}*{.bak}
ignore = Name {,.}*{.tmp}
ignore = Name {,.}*{.txt~}
ignore = Name {,.}*{.pl~}
ignore = Name {.unison.}*
ignore = Name {,.}*{.zip}

    # Use this command for displaying diffs
    diff = diff -y -W 79 --suppress-common-lines

    ignore = Name *~
    ignore = Name .*~
    ignore = Path */pilot/backup/Archive_*
    ignore = Name *.o

Answer 6

While rsync --checksum is the correct answer, note that this option is incompatible with --times, and that --archive includes --times, so if you want to rsync -a --checksum, you really need to rsync -a --no-times --checksum.

Answer 7

This is the perfect answer: numbered backup file and copy only if source (also multiple, also directories) and destination file doesn't matches, with the possibility to choose to do backup recursively or not (with the depth level of your choice or unlimited) (default: not recursively), in any case the copy (no-clobber) is always recursive (unlimited depth):

export NULLGLOB="$(shopt -p nullglob)"

copy () {
    command cp -a --no-preserve=mode,ownership --remove-destination "$@"
    return $?
}

backup-and-copy () {
    $NULLGLOB
    local exit_code
    local error_message
    local i
    local maxdepth=0
    local submaxdepth=-1
    local abs
    for (( i=1; i<=$#; i++ )); do
        if [[ ${!i} == -- ]]; then
            command set -- "${@:1:i-1}" "${@:i+1}"
            break
        fi
        if [[ ${!i,,} == -r ]]; then
            command set -- "${@:1:i-1}" "${@:i+1}"
            if [[ ${!i} =~ ^-?[0-9]+$ ]]; then
                maxdepth=${!i}
                abs=${maxdepth/#-}
                if [[ ${#abs} -gt 9 ]]; then
                    echo "backup-and-copy: $maxdepth: Numerical result out of range=[-999999999, 999999999]"
                    return 1
                fi
                command set -- "${@:1:i-1}" "${@:i+1}"
            else
                maxdepth=-1
            fi
            i=$((i-1))
        fi
    done
    if [[ $maxdepth -gt 0 ]]; then submaxdepth=$((maxdepth-1)); fi
    error_message="$(cp -n -- "$@" 2>&1)"
    exit_code=$?
    if [[ $exit_code -ne 0 ]]; then
        echo "backup-and-copy: ${error_message}"
        return $exit_code
    fi
    if [[ -d ${!#} ]]; then
        for (( i=1; i<$#; i++ )); do
            if [[ -d ${!i} ]]; then
                if [[ $maxdepth -ne 0 ]]; then
                    shopt -s nullglob
                    backup-and-copy -R $submaxdepth -- "${!i}/"{,.[^.],..?}* "${!#}/$(command basename "${!i}")"
                fi
            else
                command cmp -s -- "${!i}" "${!#}/$(command basename "${!i}")" 2> /dev/null || cp --backup=numbered -- "${!i}" "${!#}"
            fi
        done
    else
        command cmp -s -- "$@" 2> /dev/null || cp --backup=numbered -- "$@"
    fi
    return $exit_code
}