0

I'm planning to backup the /home folder of my Kubuntu box to an external drive.

In order to speed up the process I consider zipping large files (individually) so that writing them is faster. I also want to do the zipping/copying in multiple threads so that the resources of the box are better used. I thought of something like this:

find /home/ -exec sh -c 'bzip2 "$1" "/path/to/target/$1.bz2"&' find-sh {} \;

On the other hand I want to avoid to spawn thousands of processes, what this script most likely will do.

What is the best way to combine find and bzip2 so that the latter runs in parallel with a limited number of threads?

Timothy Truckle
  • 292
  • 2
  • 8
  • Consider the use of `rsync` and `parallel` together. – steve Nov 12 '16 at 19:16
  • @steve do you mean something like this: `find /home/ -exec sh -c 'parallel rsync --compress ... "$1" /path/to/target/$1.gz' find-sh {} \;`? – Timothy Truckle Nov 12 '16 at 20:11
  • I was thinking of the answers shown at http://unix.stackexchange.com/questions/189878/parallelise-rsync-using-gnu-parallel – steve Nov 12 '16 at 21:04
  • 1
    xz has a better compression/time tradeoff than bzip2. Consider zipping files together, you'll get better compression. And I strongly recommend using a robust tool for backups rather than cobbling up your own. Duplicity is pretty simple to use and does compressed incremental backups, optionally encrypted. – Gilles 'SO- stop being evil' Nov 13 '16 at 22:04
  • @Gilles does Duplicity also use more than one core of my cpu? and can I backut to alocal folder like /mnt/removableDisk/backup? – Timothy Truckle Nov 13 '16 at 22:23

1 Answers1

0

I finally ended up with starting a few tar jobs by hand...

Timothy Truckle
  • 292
  • 2
  • 8