1

Let's create a test/ directory containing a random 1 GB file: head -c 1G </dev/urandom >test/1GBfile, and let's do a backup with duplicity:

duplicity test/ file:///home/www/backup/

Then /home/www/backup/ contains an encrypted archive, taking around ~1 GB.

Then let's add a new file of a few bytes: echo "hello" >test/hello.txt, and redo the backup:

duplicity test/ file:///home/www/backup/

The backup/ is still ~ 1 GB. Only a few files were created of < 1 KB, as usual in incremental backup.

Now let's rename the 1 GB file: mv test/1GBfile test/1GBfile_newname and redo the incremental backup:

duplicity test/ file:///home/www/backup/

Then backup/ is now ~ 2 GB!.

Why does duplicity not take into account the fact it's the same file content with a new name?

Here if we had used networking, we would have wasted 1 GB transfer even if the file content is exactly the same. duplicity uses rsync which usually takes care of this problem, is there an option to avoid this problem?

Log after the addition of the .txt file:

--------------[ Backup Statistics ]--------------
StartTime 1605543432.43 (Mon Nov 16 17:17:12 2020)
EndTime 1605543432.72 (Mon Nov 16 17:17:12 2020)
ElapsedTime 0.29 (0.29 seconds)
SourceFiles 3
SourceFileSize 1073745926 (1.00 GB)
NewFiles 2
NewFileSize 4102 (4.01 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 2
RawDeltaSize 6 (6 bytes)
TotalDestinationSizeChange 230 (230 bytes)
Errors 0
-------------------------------------------------

Log after the renaming of the file:

--------------[ Backup Statistics ]--------------
StartTime 1605543625.97 (Mon Nov 16 17:20:25 2020)
EndTime 1605543840.72 (Mon Nov 16 17:24:00 2020)
ElapsedTime 214.76 (3 minutes 34.76 seconds)
SourceFiles 3
SourceFileSize 1073745926 (1.00 GB)
NewFiles 2
NewFileSize 1073745920 (1.00 GB)
DeletedFiles 1
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 3
RawDeltaSize 1073741824 (1.00 GB)
TotalDestinationSizeChange 1080871987 (1.01 GB)
Errors 0
-------------------------------------------------

TotalDestinationSizeChange 1080871987 (1.01 GB), arghh! The file has just been renamed!

Basj
  • 2,351
  • 9
  • 37
  • 70
  • Consider using `restic` instead? – Kusalananda Nov 16 '20 at 16:16
  • @Kusalananda does it do encrypted storage in destination (i.e. encrypted locally and destination NEVER ever has the decrypted content)? Which option does this with `restic`? – Basj Nov 16 '20 at 16:19
  • `restic` encrypts by default. Check out the documentation https://restic.net/ – Kusalananda Nov 16 '20 at 16:23
  • For future reference, can you post an answer with an example command-line command to do this with restic @Kusalananda? – Basj Nov 16 '20 at 16:24
  • Hmm... That would not be an answer to your question though. I have no experience of using `duplicity`, so someone that has may still be able to help you with that. – Kusalananda Nov 16 '20 at 16:27

1 Answers1

0

It seems that duplicity does not "handle renames" indeed, see this column here: https://wiki.archlinux.org/index.php/Synchronization_and_backup_programs#Chunk-based_increments

Since duplicity uses librsync (that does not "handles renames"), it seems normal.

Basj
  • 2,351
  • 9
  • 37
  • 70