Is it true that rsync does not handle "File renames" gracefully?

Question

While using the backup tool duplicity, I noticed that if I rename a file on the source, the data will be resent again over the network to the destination, which is a bit sad. Since duplicity uses librsync internaly, I decided to have a look at rsync.

This ArchWiki page states:

Handles renames
Moved/renamed files are detected and not stored or transferred twice.  It typically means that a checksum of files or its chunks is computed.  Applications missing this functionality can be supplemented by combining with hsync^AUR, which only synchronizes renames.

rsync: Handles renames: No

Does this really mean that, when using rsync, there is no way to prevent 10 GB to be re-transferred over the network to the destination, if I rename /test/10GBfile to /test/10GBfile_newname on the source computer?

Given the long-time popularity of rsync, is there no mode in which this would be better handled?

Answer 1

There is no mechanism for rsync to track renames as it does not maintain state other than while it is running. If you rename /test/10GBfile to /test/10GBfile_newname on the source computer, then by default rsync can only see that 10GBfile has been deleted and that 10GBfile_newname has been created.

The --fuzzy parameter (-y) may help identify 10GBfile as a potential source of data for 10GBfile_newname on the target, thereby avoiding a network copy at the expense of a file copy. However, it can (mostly) only consider matches of files in the same directory so although your example would match, a rename of /test/10GBfile to /test/otherdir/10GBfile_newname would not.

Notice also that the documentation (man rsync) advises that if you want to use --delete you should instead use either --delay-updates or --delete-after so that potential matches for --fuzzy are not deleted before they can be used.

Example

# Prepare an uncompressible 100MB file
mkdir -p /tmp/test
dd bs=1M count=100 iflag=fullblock if=/dev/urandom >/tmp/test/file1

# Normal first-time copy
rsync -av --fuzzy --delete-after /tmp/test/ remote:/tmp/test

# Skip copy because unchanged
rsync -av --fuzzy --delete-after /tmp/test/ remote:/tmp/test

# Rename file (per your example)
mv /tmp/test/file1 /tmp/test/file2

# Fast copy because fuzzy match
rsync -av --fuzzy --delete-after /tmp/test/ remote:/tmp/test

Add two more -v flags (i.e. rsync -avvv …) to see the block-by-block detail of what's going on.

Answer 2

--fuzzy has already been answered but there is another interesting hack involving hard links.

After the first transfer

$ rsync -avHP --delete-after ~/family/Photos remotebox:backups

You create a hard linked work directory:

$ cd ~/family
$ cp -rlp Photos Photos-work

Then you can use

$ rsync -avHP --delete-after --no-inc-recursive ~/family/Photos ~/family/Photos-work remotebox:backups

To transfer the new structure to the remote.

Why and how this works is explained here:

https://lincolnloop.com/blog/detecting-file-moves-renames-rsync/

Answer 3

That page claims patches for --detect-renamed option were available for rsync 3.0.9. It has links to patches, link to bugzilla discussion. My rsync 3.1.3 does not have --detect-renamed option. The discussion mentioned is going on with:

elatllat 2021-01-15 14:19:12 UTC

This feature request is so old it has lost relavence because btrfs/zfs/etc are more optimal backup solutions than rsync.

Some there do not agree to "lost relevance" and I agree with those who disagree more.

That reddit page claims:

btrfs sync supports this, but you need to have btrfs at both ends.

which is I think what elatllat referred to.

Below is my workaround for now (git is overkill for me here I think).

My specifics: I rename not often and I want sync chained (not only one-to-one), not backup. So I decided to create shell command files where I would write mv commands (adding new ones at the end) to be executed additionally by the script I wrote initially to run rsync back and forth. This script has not been tested much, run at your own risk, comments for improving are welcomed. I hope rsync --detect-renamed option will go through to production soon.

Complete script that follows is that long because it separately processes when shell file is larger in size from one side of sync than from the other and includes some checks as I'm not sure (only recently started to use the script) comm program properly finds unique lines as it is stated to work on "sorted" (per man page) files. The core is:

# do renames / moves at both ends
cd $remote_path 
bash -c "$(comm --nocheck-order -3 -1 $f_RE $f_LO)"
cd $local_path 
bash -c "$(comm --nocheck-order -3 -2 $f_RE $f_LO)"

# merge files via temp file
(comm --nocheck-order -1 -2 $f_LO $f_RE) > $f_LO.3
(comm --nocheck-order -3 -1 $f_RE $f_LO) >> $f_LO.3
(comm --nocheck-order -3 -2 $f_RE $f_LO) >> $f_LO.3
cp $f_LO.3 $f_RE
mv $f_LO.3 --force $f_LO
rm $f_LO.1 $f_LO.2  

Full shell function:

#!/bin/bash
do_sync(){
    if [ -d $remote_path ]; then 
        
        # ===== workaround for renaming / moving (run manually made commands before rsync) ===== #
    
        # man comm: comm - compare two sorted files line by line
        complex_flag=0 # later set by script to 1 if changes identified from both sync directions
        to_rsync=1 # to run rsync by default
        f_RE=$remote_path/_rename_move.sh
        f_LO=$local_path/_rename_move.sh

        if [ -f $f_RE ]; then
             if [ -f $f_LO ]; then
                # if both exist, -gt greated than, stat --printf="%s" size in bytes
                if [ $(stat --printf="%s" $f_RE) -gt $(stat --printf="%s" $f_LO) ]; then
                    # small file (2nd) is fully contained in the beginning of larger file (maybe test binary mode more efficient)
                    # -1     suppress column 1 (lines unique to FILE1) : man comm
                    if [ -z "$(comm --nocheck-order -3 -1 $f_RE $f_LO)" ]; then
                        # run only additional commands 
                        cd $local_path 
                        bash -c "$(comm --nocheck-order -3 -2 $f_RE $f_LO)"
                        # overwrite small with larger one                     
                        cp $f_RE $f_LO
                    else complex_flag=1; fi
                # remote smaller than local
                elif [ $(stat --printf="%s" $f_RE) -lt $(stat --printf="%s" $f_LO) ]; then
                    # small file (1nd) is fully contained in the beginning of larger file (maybe test binary mode more efficient)
                    if [ -z "$(comm --nocheck-order -3 -2 $f_RE $f_LO)" ]; then
                        # run only additional commands
                        cd $remote_path
                        bash -c "$(comm --nocheck-order -3 -1 $f_RE $f_LO)"
                        # overwrite small with larger one                     
                        cp $f_LO $f_RE
                    else complex_flag=1; fi
                # same size but different contents
                elif [ ! $(sha256sum $f_RE | awk '{ print $1 }') = $(sha256sum $f_LO | awk '{ print $1 }') ]; then
                    complex_flag=1;
                fi
                # nothing to do if files are the same
             # if only remote exists                
             else
                cd $local_path && $f_RE
             fi

             # neither file was found to be part of another as a whole
             # expect changes (moves/renames) from both ends
             if [ $complex_flag -eq 1 ]; then

                # doing echo "$()" removes trailing empty lines compared to for some reason (TODO why?)

                # check that doing symmetrically with appending to local results in same number of lines in a file
                # and selecting matching in both and adding distinct from both too results in same number of lines in a file
                cp $f_RE $f_LO.1 && (comm --nocheck-order -3 -1 $f_RE $f_LO) >> $f_LO.1
                cp $f_LO $f_LO.2 && (comm --nocheck-order -3 -2 $f_RE $f_LO) >> $f_LO.2
                (comm --nocheck-order -1 -2 $f_LO $f_RE) > $f_LO.3
                (comm --nocheck-order -3 -1 $f_RE $f_LO) >> $f_LO.3
                (comm --nocheck-order -3 -2 $f_RE $f_LO) >> $f_LO.3
                counts_1="$(wc $f_LO.1 | awk '{ print $1,$2,$3 }')"
                counts_2="$(wc $f_LO.2 | awk '{ print $1,$2,$3 }')"
                counts_3="$(wc $f_LO.3 | awk '{ print $1,$2,$3 }')"
                # same counts, Ok
                if [ $counts_1 = $counts_2 ] && [ $counts_2 = $counts_3 ]; then
                    cd $remote_path 
                    bash -c "$(comm --nocheck-order -3 -1 $f_RE $f_LO)"
                    cd $local_path 
                    bash -c "$(comm --nocheck-order -3 -2 $f_RE $f_LO)"
                    cp $f_LO.3 $f_RE
                    mv $f_LO.3 --force $f_LO
                    rm $f_LO.1 $f_LO.2
                else
                    echo "========= manual intervention might be needed ==========="
                    echo "Results of analysis of $f_LO & $f_RE via [comm] app has not matched;"
                    echo "renaming/moving not performed;"
                    echo "rsync of $local_path & $remote_path not performed; see differences between files:"
                    echo "$f_LO.1, $f_LO.2, $f_LO.3"
                    echo "========================================================="
                    to_rsync=0
                fi
         fi   
        # if only local exists
        elif [ -f $f_LO ]; then
            cd $remote_path && $f_LO
        fi

        # ===== end of workaround ===== #

        if [ $to_rsync -eq 1 ];then
            rsync $options $local_path/ $remote_path
            rsync $options $remote_path/ $local_path
            rsync $options $local_path/ $remote_path
        fi

        # below is to move old versions away 
        find "$local_path" -path "$local_path/prevs" -prune -o -name '*.bak' -exec mv "{}" "$local_path/prevs" \;
        find "$remote_path" -path "$remote_path/prevs" -prune -o -name '*.bak' -exec mv "{}" "$remote_path/prevs" \;

# previous versions, prune works "more correct", forgot why
#        find $local_path -maxdepth 1 -name '*.bak' -exec mv "{}" $local_path/prevs \;
#        find $remote_path -maxdepth 1 -name '*.bak' -exec mv "{}" $remote_path/prevs \;
    else
        echo $remote_path is not available
    fi
}

# trailing / would prevent proper pruning in find commands
local_path=/home/$(id -un)/Documents
remote_path=/media/$(id -un)/Projects
do_sync

The answers in https://serverfault.com/questions/489289/handling-renamed-files-or-directories-in-rsync have as I understand it some drawbacks.

One with hardlinks: most systems do not support hardlinks for directories and it won't check which side has name changed (Ok for backup, not Ok for sync).

stat filename can be used to check which side has file renamed but I do not know how to merge it with rsync.