rdiff-backup needs too much space to recover from failure of previous backup?

Question

I have quite a lot of free space (and inodes) on this filesystem, but rdiff-backup apparently runs out of space. And hence my backups are not working at the moment :-(. Why?

(I wonder how much space I should be reserving for this in future!)

My rdiff-backup is version 1.2.8-7. My OS is Debian 9 "stretch".

# df -h /d/backup
Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/vg_brick-lv_backup  139G   95G   38G  72% /d/backup

# df -i /d/backup
Filesystem                      Inodes  IUsed   IFree IUse% Mounted on
/dev/mapper/vg_brick-lv_backup 9240576 413341 8827235    5% /d/backup

# df -h
Filesystem                      Size  Used Avail Use% Mounted on
udev                            244M     0  244M   0% /dev
tmpfs                            51M  1.9M   49M   4% /run
/dev/sda7                        15G  4.1G  9.8G  30% /
tmpfs                           251M     0  251M   0% /dev/shm
tmpfs                           5.0M     0  5.0M   0% /run/lock
tmpfs                           251M     0  251M   0% /sys/fs/cgroup
/dev/ubi0_0                     462M  225M  233M  50% /d/ubi
tmpfs                           251M  4.0K  251M   1% /tmp
/dev/mapper/vg_brick-lv_attic    44G   21G   21G  50% /d/attic
/dev/mapper/vg_brick-lv_backup  139G   95G   38G  72% /d/backup
/dev/sda6                       2.0G   81M  1.8G   5% /boot
/dev/mapper/vg_brick-lv_home    134G   91G   37G  71% /home
/dev/sda10                      9.8G  2.9G  6.9G  30% /d/xfer
tmpfs                            51M     0   51M   0% /run/user/1003

root@brick:/d/backup/jenkins-desktop# du -sh mike
5.7G    mike
root@brick:/d/backup/jenkins-desktop# du -sh ../jenkins-desktop.rdiff/mike
8.2G    ../jenkins-desktop.rdiff/mike

root@brick:/d/backup/jenkins-desktop# rdiff-backup mike ../jenkins-desktop.rdiff/mike
Previous backup seems to have failed, regressing destination now.                                                                                     
Exception '[Errno 28] No space left on device' raised of class '<type 'exceptions.IOError'>':                                                         
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/robust.py", line 32, in check_common_error                                                      
    try: return function(*args)                                                                                                                       
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/restore.py", line 468, in get_fp                                                                
    Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)                                                                                              
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Rdiff.py", line 73, in write_patched_fp                                                         
    rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp), out_fp)                                                                               
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 64, in copyfileobj                                                              
    outputfp.write(inbuf)

Exception '[Errno 28] No space left on device' raised of class '<type 'exceptions.IOError'>':                                                         
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main                                                         
    try: Main(arglist)                                                                                                                                
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 337, in Backup
    backup_final_init(rpout)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 501, in backup_final_init
    checkdest_if_necessary(rpout)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 920, in checkdest_if_necessary
    dest_rp.conn.regress.Regress(dest_rp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/regress.py", line 71, in Regress
    for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 281, in __call__
    last_branch.fast_process(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/regress.py", line 268, in fast_process
    if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/regress.py", line 290, in restore_orig_regfile
    tf.write_from_fileobj(rf.get_restore_fp())
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/restore.py", line 488, in get_restore_fp
    return robust.check_common_error(error_handler, get_fp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/robust.py", line 32, in check_common_error
    try: return function(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/restore.py", line 468, in get_fp
    Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Rdiff.py", line 73, in write_patched_fp
    rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp), out_fp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 64, in copyfileobj
    outputfp.write(inbuf)

Traceback (most recent call last):                                                                                                                    
  File "/usr/bin/rdiff-backup", line 30, in <module>                                                                                                  
    rdiff_backup.Main.error_check_Main(sys.argv[1:])                                                                                                  
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 304, in error_check_Main    
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 324, in Main
    take_action(rps)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 280, in take_action
    elif action == "backup": Backup(rps[0], rps[1])
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 337, in Backup
    backup_final_init(rpout)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 501, in backup_final_init
    checkdest_if_necessary(rpout)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Main.py", line 920, in checkdest_if_necessary
    dest_rp.conn.regress.Regress(dest_rp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/regress.py", line 71, in Regress
    for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rorpiter.py", line 281, in __call__
    last_branch.fast_process(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/regress.py", line 268, in fast_process
    if rf.metadata_rorp.isreg(): self.restore_orig_regfile(rf)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/regress.py", line 290, in restore_orig_regfile
    tf.write_from_fileobj(rf.get_restore_fp())
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/restore.py", line 488, in get_restore_fp
    return robust.check_common_error(error_handler, get_fp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/robust.py", line 32, in check_common_error
    try: return function(*args)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/restore.py", line 468, in get_fp
    Rdiff.write_patched_fp(current_fp, delta_fp, new_fp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/Rdiff.py", line 73, in write_patched_fp
    rpath.copyfileobj(librsync.PatchedFile(basis_fp, delta_fp), out_fp)
  File "/usr/lib/python2.7/dist-packages/rdiff_backup/rpath.py", line 64, in copyfileobj
    outputfp.write(inbuf)
IOError: [Errno 28] No space left on device

score 1 · Accepted Answer · answered Dec 03 '18 at 19:05

You need to either disable tmpfs on /tmp, or configure rdiff-backup to use some other temporary directory which has enough space. Possibly /var/tmp.

tmpfs                           251M  4.0K  251M   1% /tmp

When restoring or regressing, rdiff-backup needs free space in the default temp directory. Under unix systems this is usually the /tmp directory. The temp directory that rdiff-backup uses can be set using the --tempdir and --remote-tempdir options available in versions 1.1.13 and newer. See the entry for tempfile.tempdir in the Python tempfile docs for more information on the default temp directory. The amount of free space required can vary, but it usually about the size of the largest file being restored.

https://www.nongnu.org/rdiff-backup/FAQ.html

score 1 · Answer 2 · answered Apr 22 '20 at 20:39

I'm not sure this was the situation of the OP, but in my case I had truncated a 96Gb mssql "backup" file on the host I was backing up (I had discovered that mssql backups were appending to it, causing it to grow indefinitely; see this answer for details).

When my rdiff-backup master attempted to back up the new, truncated version of my mssql "backup" file, it was first generating the previous 96Gb version of this file in its /tmp directory. I assume rdiff-backup was doing this in order to perform a "diff" operation on the file. But in the process of generating this huge previous version, the master rdiff-backup process was exhausting its space in /tmp, leading to results as shown in this question.

Once I understood the problem, I followed the steps in the rdiff FAQ item entitled For some reason rdiff-backup failed while backing up. Now every time it runs it says "regressing destination" and then fails again. What should I do? to disable regression. As a result I assume rdiff-backup "gave up" on comparing files and relied on directory information instead. As a result my rdiff-backup master no longer consumed all of its /tmp space and my backup succeeded.

Le Déchaîné · Answer 3 · 2022-12-12T11:48:09.650

0

I've been using rdiff-backup for years and I definitely trust it, but if all else fails, rsync -av (source dir) (dest dir) is an alternative.

Important edit: That is if you can use another (dest dir) or drive, otherwise it might wreak havoc with your rdiff-backup history files. Thanks @sourcejedi !

edited Dec 12 '22 at 11:48

answered Dec 12 '22 at 02:49

Le Déchaîné

1
2

Beware that if `dest dir` is an existing rdiff-backup directory, this will break all the rdiff-backup history even though the `rdiff-backup` subdirectory is left in place. – sourcejedi Dec 12 '22 at 10:18
Considering that you're the third person to write on this page, and none of us have reached "if all else fails", I personally would recommend worrying that if someone found a different problem, it might be more of a sign of a problem with their system, including hardware failure, than that rdiff-backup cannot cope with their system :-). – sourcejedi Dec 12 '22 at 10:23

rdiff-backup needs too much space to recover from failure of previous backup?

3 Answers3