12

I am trying to take snapshots of a massive folder regularly.

I have read here: http://www.mikerubel.org/computers/rsync_snapshots/#Incremental
that cp -al takes a snapshot of a folder by simply copying the hard links.

That is all great, but the problem is that in this snapshot, if I change a file, it changes in all snapshots. What I would like instead is to have the system create a new file on-change and link to that instead. That way each snapshot would not become invalid on an edit of the first file.

How can I achieve that?

p.s. I tried rsync -a --delete --link-dest=../backup.1 source_directory/ backup.0/, but it has the same problem.

john-jones
  • 1,698
  • 4
  • 24
  • 38

4 Answers4

7

That's how hardlinks work. But, there are ways around it:

A couple of options come to mind:

  • Use a filesystem with support for copy-on-write files, like btrfs. Of course, were you using btrfs, you'd just use its native snapshots... If your filesystem supports it, you can use cp --reflink=always. Unfortunately, ext4 doesn't support this.
  • Only share hardlinks across your snapshots, not with the original. That is, the first time you see a given version of a file, copy it to the snapshot. But the next time, link it to the one in the previous snapshot. (Not sure what program I used to do this—a decade ago—but searching turns up dirvish, obnam, storebackup, and rsnapshot)
  • Depending on how your files are being changed, you might be able to guarantee that a write temp/rename is used to change them, then that will break the hardlink—so the version in the snapshot will remain pristine. This is less safe, though, as bugs could corrupt your snapshot.
  • Take LVM snapshots of the entire filesystem.

Of course, there is the other option—use a proper backup system. Most all of them can manage to only backup changed files.

tshepang
  • 64,472
  • 86
  • 223
  • 290
derobert
  • 107,579
  • 20
  • 231
  • 279
  • What do you recommend as a way to back up a massive folder? – john-jones Mar 29 '13 at 19:37
  • I was thinking using rsync to a server which has a cronjob to do cp -al regularly for snapshots.. alongside rsync-ing onwards for even more copies. How does that sound? – john-jones Mar 29 '13 at 19:52
  • @HermannIngjaldsson well, it depends on how you do your backups. Personally, I'd just add it to my Bacula setup—but I wouldn't recommend that unless you have a bunch of machines to back up, or already know Bacula. So, I guess I'd suggest you try rsnapshot first. – derobert Mar 29 '13 at 20:00
  • `rsnapshot` is good – brettwhiteman Jun 20 '15 at 02:31
4

What you're looking for is a form of copy-on-write, where multiple files that have the same content use the same space on the disk until one of them is modified. Hard links only implement copy-on-write if the application that does the writing deletes the file and creates a new file by the same name (which is usually done by creating a new file by a different name, then moving it into place). The application you're using is evidently not doing this: it's overwriting the existing file.

Some applications can be configured to use the replacement strategy. Some applications use the replacement strategy by default, but use the overwrite strategy when they see a file with multiple hard links, precisely so as not to break the hard links. Your current snapshot technique will work if you can configure your application to replace instead of overwriting.

Fl-cow modifies programs to systematically use the replacement strategy on files with multiple hard links.

Alternatively, you may store your files on a filesystem that performs copy-on-write or deduplication, or have a snapshot feature, and not worry about hard links: Btrfs or Zfs. Depending on your partitioning scheme, using LVM snapshots may be an option.

My recommendation is to use a proper snapshot tool. Making reliable backups is surprisingly difficult. You probably want rsnapshot.

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
2

The following is a ruby script that I wrote that wraps the "cp -al" and rsync into a nice script that can be run manually or via cron. Destination can be local or remote (via ssh):

Ghetto Timemachine

The basic answer to your question, as mentioned in a previous comment, the source needs to be kept apart from the hard links. Ex, assume a daily backup of your home directory:

Source:

  • /home/flakrat

Destination:

  • /data/backup/daily
    • /monday
    • /tuesday
    • /wednesday
    • /thursday
    • ...

The hard links are created by running "cp -al" against yesterday's backup. Say it's Tuesday morning when you run it:

cd /data/backup/daily

rm -rf tuesday

cp -al monday tuesday

rsync -a --delete /home/flakrat /data/backup/daily/tuesday/

Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
FlakRat
  • 19
  • 3
0

rdiff-backup seems to do what you want, check it out.

Using rsync you must first make a full backup not using hard links. The next backup can point to the previous backup and hard link to it. That way your backups are not hard linked to your working files (the ones you are modifying). Example. If my previous backup was so folder backup.01 my backup script would first increment the folders by renaming them up by one so backup.01 becomes backup.02. Then the script creates a new empty folder called backup.01. it would then rscync the new backup into the new folder and hard link to backup.02 so that only new files will take any space in the backup. The rsync command would look something like this: rsync -rlt sourcepath backuppath/backup.01 --link-dest=backuppath/backup.02

So you can see, all the hard linking is happening on the backup path. This way you do not have to worry about copy on write when modifying files in your source path.

Marty
  • 1