4

My intention is to create a "snapshot" of a certain hard drive on a server every day. How can I automatically make sure that the snapshot of today, three days ago, five days ago, 10 days ago (and so on) are being kept and cyclically replaced? I found a tool: duplicity, but I didn't find any way to achieve this using it. Should I go for a bash shell script? Is there any example I can take inspiration from?

The system is running Debian.

giovi321
  • 817
  • 3
  • 13
  • 34

3 Answers3

3

Duplicity creates snapshots, but in the form of compressed archives, not in a form that can be read directly.

There are several ways to create snapshots. Some advanced filesystems such as ZFS and Btrfs have them as a built-in feature, as do some disk layers such as LVM.

On a generic filesystem, a basic technique is to reproduce the directory tree that you want to make a snapshot of, but create hard links to each file instead of copying files. When a file is deleted and replaced by a new version, the snapshot retains the old version. Note that some applications modify files in place; this would modify the snapshot version as well since they are the same file. This is an inherent problem with hard-link-based solutions.

A variant of straight hard linking is to always make a copy when backing up a file, but if a file hasn't changed since the last backup, make a hard link to the earlier backup rather than make a new copy. Unlike the previous approach, this one is suitable for backups since it doesn't risk rewriting history by modifying files in old backups.

Rsync with the --link-dest option and Rsnapshot are the most common tools to make such snapshots. Rsnapshot is a higher-level tool: it uses rsync under the hood, but takes care of calling it with the right arguments, of managing the culling of old backups, etc.

With rsnapshot, you specify how to cull old backups by defining several intervals. The lowest interval is the frequency at which backups are taken, in your case very day. Longer intervals define what backups are retained. For each directive retain INTERVAL N in rsnapshot.conf, when you run rsnapshot INTERVAL, rsnapshot does two things:

  1. Cull the backup directories so that only N-1 snapshots remain for INTERVAL.
  2. For the lowest interval, make a backup; for other intervals, replicate the current image for the previous interval.

For example, to make daily backups and retain the backups from 3, 5 and 10 days ago, and include the following retain lines in your /etc/rsnapshot.conf:

retain daily 2
retain day3 2
retain day5 3

Run rsnapshot daily as a daily cron job. A few minutes before that job, run rsnapshot day3 and rsnapshot day5 every three days; you can approximate it like this:

04 04 5,10,15,20,25,30 * * rsnapshot day5
07 04 3,6,9,12,15,18,21,24,27,30 * * rsnapshot day3
10 04 * * * rsnapshot daily

(You can't have backups from e.g. exactly 3 days ago because you're erasing on day 2 — what you do instead is retain one daily backup in 3.) A more classical retention scheme is weekly and monthly retention as illustrated in the rsnapshot documentation.

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
  • This is very helpful, however the biggest issue for me is how to preserve backups of, let's say, five days ago, one week ago, one month ago. Why do I want this? Let's say that a file gets corrupted, if I backup it and do not preserve the old backup, I will have a backup of the corrupted file. – giovi321 Mar 28 '15 at 08:22
  • @giovi321 Sure, rsnapshot is geared for that. See my edit and the documentation for the `retain` directive. – Gilles 'SO- stop being evil' Mar 28 '15 at 15:21
  • Rsnapshot. Full ack. – user236012 Mar 28 '15 at 17:01
1

After month of research on internet, I found this: Butterfly Backup

This utility used rsync for backup, restore and other operations (archive, retention, clean, etc...)

This utility has a catalog that serve to save point of various backup and so, restore a backup from specific time. (Similar at time machine or VSS on Windows).

This is an example of backup:

$> bb backup --computer pc1 --destination /nas/backup_store --data User Config --type Unix

and restore:

$> bb restore --computer pc1 --catalog /nas/backup_store --backup-id f65e5afe-9734-11e8-b0bb-005056a664e0

to see the backup-id:

$> bb list --catalog /nas/backup_store

BUTTERFLY BACKUP CATALOG

Backup id: f65e5afe-9734-11e8-b0bb-005056a664e0
Hostname or ip: pc1
Timestamp: 2018-08-03 17:50:36

...

and detail of f65e5afe-9734-11e8-b0bb-005056a664e0:

$> bb list --catalog /nas/backup_store --backup-id f65e5afe-9734-11e8-b0bb-005056a664e0

Backup id: f65e5afe-9734-11e8-b0bb-005056a664e0
Hostname or ip: pc1
Type: Full
Timestamp: 2022-02-03 17:50:36
Start: 2022-02-03 17:50:36
Finish: 2022-02-03 18:02:32
OS: MacOS
ExitCode: 0
Path: /nas/mybackup/pc1/2022_02_03__17_50
List: etc
home

That's it!

Moreover, this tool backup Unix (Linux, BSD, etc..), MacOS and Windows (cygwin installed with rsync) machine.

wesijay864
  • 11
  • 1
-1

I have been using BackupPC to do my backups. This can use tar, SMB, rsync to access the files being backed up. In my case, I use rsync on all systems (Cygwin in Windows).

I've found backups so it is fast after the first back. Incremental backups and full backups can be retained on different schedules. Only one copy of each file version is kept so it is quite a space efficient. File restorations are done from a web page where you can browse and then download individual files, create a restoration archive or restore files directly. On most systems, I have disabled write access, so I need to restore from archives or by copying individual files.

There are a variety of other backup systems that use similar mechanisms to retain historical backups.

BillThor
  • 8,887
  • 22
  • 27