5

When rsyncing a directory to a freshly plugged-in external USB flash drive, via

rsync -av /source/ /dest/

all files get transferred (i.e. rewritten) despite no changes in the files.

Note that overwriting the files only takes place once the USB is un- and replugged. Doing the rsync command twice in a row without unplugging the drive in-between does successfully skip the whole directory contents.

Including the -u update option and explicitly adding the -t option did not change anything.

The mount point remains the same (i.e. /media/user/<UUID>, the drive is automouted by xfce, the /dev/sdxy obviously changes) The hard drive source is ext4, while the USB is vfat with utf8 character encoding.

What could be the reason for this behaviour is it the change in the /dev/ name entry? How can I make rsync run with properly recognizing file changes? My backup should just take seconds without this, while it now is always minutes due to the large amount of data being overwritten repeatedly, nor is the massive writing the best for the flash drive's life time expectancy.

FelixJN
  • 12,616
  • 2
  • 27
  • 48

3 Answers3

7

Your FAT drive can store timestamps only to two second accuracy. When you unplug and replug the drive you effectively break all the file times. See the --modify-window option for a workaround.

Secondly, you're never going to see fast backups with rsync like this, because when copying locally it behaves much like cp.

roaima
  • 107,089
  • 14
  • 139
  • 261
2

By default, rsync identifies changed files through their modification time and file size. A quick Google search for "vfat timestamp" reveals several problems related to the tz=UTC mount option.

Could it be that the timestamps on external drive differ from the internal drive and rsync therefore always considers the files to be different? If that is the case, you should probably check your mount options.

Alternatively (and actually regardless of whether my theory is right), you could try to run rsync with the -c option, which makes it check for changes through hashes of the file contents. This potentially slows down things, but I'm unsure about the practical impact.

F30
  • 519
  • 3
  • 14
-2

I found that the problem was more common with the bigger files [which are precisely the ones I want to avoid copying.]

A cheap alternative to the -c option is to use --size-only:

rsync -av --size-only  .....

this will ignore the time-stamp and copy only according if the size has changed. It is not as precise as using checksum but good enough for backing up

roaima
  • 107,089
  • 14
  • 139
  • 261
Yousef
  • 11
  • 1
    It's a terrible idea. Consider `date >file`. Backup. Then `date >file` once more. The file has changed but your backup would skip this file – roaima Jan 21 '22 at 19:10
  • Obviously there are more ways to get it wrong than to get it right... – U. Windl Jan 22 '22 at 23:51
  • It's a great idea. `--modify-window=1` even `--modify-window=999` didn't work for me, even on subsequent runs of `rsync` without changing the mount. `--ignore-times` (`-I`) didn't work either, `rsync` repeatedly insists on re-copying the whole thing from the top. Only `--size-only` made a difference and thankfully resolved this problem. Down votes here are naive. Read the problem. This is only a terrible idea if timestamps matter. If you know that a file of the same size hasn't changed then timestamps don't matter. This solution solved *this* problem for me. The accepted answer didn't. – NeilG Mar 19 '23 at 11:08