4

I have a Linux server (Ubuntu 16.04) where I would like to mount an external "storage box" where to periodically backup several terabyte of server's data (both high and low-frequency changing), possibly keeping track of user's attributes (owner/group/permissions).

The provider of these storage box offers the following protocols: FTP, FTPS, SFTP, SCP, Samba/CIFS, HTTPS, WebDAV

Which backup tool should I consider? If you suggest rsync, as I would have first to mount the data locally, which protocol should I prefer for the mounting ? Would it be possible to backup user's attributes (i.e. mapping users between server and host ) ?

Antonello
  • 955
  • 2
  • 9
  • 14
  • You mean those _storage boxes_ don't have `rsync` support natively (over `ssh` or directly)? Have they got at least `ssh` support? What's the storage technology on them? If ZFS/btrfs, you could possibly use `zfs send`/`btrfs send`. – Stéphane Chazelas Sep 08 '17 at 13:00
  • 1
    `scp` would suggest they have `ssh`. I'd be surprised if you couldn't use `rsync` without mounting the FS. Network file system will always be slower than a syncing method over a TCP connection. – Stéphane Chazelas Sep 08 '17 at 13:03

3 Answers3

2

That really depends on the specifics of the network, and how the storage appliance is designed.

If you absolutely have to mount the storage directory as a filesystem, you're pretty obviously stuck using either SMB/CIFS or SSHFS (which just needs working SFTP support on the storage device). Of those two, SMB is almost always going to get better performance for anything except bulk transfers (and for bulk transfers, either one may win out). If you can get working NFSv4 support (and you absolutely should be able to if it's an Ubuntu box, regardless of what the OEM says), that will usually outperform both SMB/CIFS and SSHFS most of the time.

If you just want to throw data across the network, then your options aren't much better to be honest. SFTP and SCP are what I would probably use in your situation because regular FTP (and FTPS, which is FTP over SSL) is crap, and WebDAV (regardless of if it's over HTTP or HTTPS) was not designed for bulk storage scenarios (DAV stands for Distributed Authoring and Versioning, which pretty well describes the fact that it was designed as an alternative to FTP for pushing content to web servers). In an ideal situation for that, I would either use rsync (if I had lots of files to transfer), or possibly just netcat (if I needed to just copy one big file).

Austin Hemmelgarn
  • 11,401
  • 1
  • 24
  • 42
0

I recommend using duplicity (but i did not test it at the scale of several terabyte), which has a lot of backends and does incremental backups split up into chunks (by default 25 MB) and only needs to download the chuncks, which are actually required for a restore. I would suggest using it with scp/sftp as it will work fine.

If you need to backup as fast as your network is able to transfer files, none of these protocols are ideal. But for one big backup and then incremental daily (or even hourly) backups of only a few gb duplicity should work fine.

allo
  • 916
  • 1
  • 7
  • 13
0

Don't do anything that destroys the isomorphism between IMPLEMENTATION and SEMANTICS.

From the client OS's point of view, a file's ownership, in semantics, is metadata.

When the file is uploaded onto a backup server, the file's on-client ownership is not metadata from the server OS's point of view. The file's on-server metadata should be the file's ownership on the server OS.

So, on the server, you need another independent data structure to record the metadata of the server file's on-client original image. For example, a database.

From the backup server's point of view, a file on this server's file system is treated as such a regular file, even if this file's content has a meaning of the mirror image of the client file, since the server OS never understands the file's content.

Actually, among all attributes of a file, only "last-modified timestamp" can still be regarded as metadata after the file is uploaded to the server, because that attribute means the version of the file's content.

Zim
  • 179
  • 1
  • 5
  • Please note that this answer does not seem to address the actual question - _"which protocol to use"_ for backing up data. – AdminBee Nov 12 '21 at 08:00
  • @AdminBee It just answers the asker's last question -- "Would it be possible to backup user's attributes (i.e. mapping users between server and host ) ?" – Zim Nov 12 '21 at 17:37