1

I have been backing up my ZFS pool in Server A to Server B (backup server) via zfs send/recv, and using daily incremental snapshots.

Server B acts as a backup server, holding 2 pools to Server A and Server C respectively (zfs41 and zfs49/tank)

Due to hardware issues, the ZFS pool in Server A is now gone - and I want to restore/recover it asap.

Currently the snapshot list in my Server B is as follows :

NAME                        USED  AVAIL     REFER  MOUNTPOINT
zfs41@2021Nov301205        14.9G      -     3.74T  -
zfs41@2021Dec011205        3.87G      -     3.74T  -
zfs41@2021Dec021205        3.77G      -     3.74T  -
zfs41@2021Dec031205           0B      -     3.74T  -
zfs49/tank@2021Nov301705   368G      -     3.52T  -
zfs49/tank@2021Dec011705  65.2G      -     3.52T  -
zfs49/tank@2021Dec021705  66.4G      -     3.52T  -
zfs49/tank@2021Dec031705     0B      -     3.52T  -

where zfs49/tank@2021Dec031705 is the latest for Server B

I would like to send back the whole pool (including the snapshots) back to Server A, but I'm unsure of the exact command to run.

Question : On Server B, will doing zfs send zfs49/tank@2021Dec031705 | ssh <Server A host ip> zfs recv tank be sufficient to receive the full ZFS pool + all the snapshots (so I can continue incremental send/recv backups) on Server A?

LooseAlien123
  • 11
  • 1
  • 3

1 Answers1

4

First you'll need to create an empty pool on Server A. zfs recv cannot create a new pool. So on Server A:

zpool create -R /mnt zfs49 [ mirror diskID1 diskID2 ]

... or some other VDEV structure of your choice. The VDEV structure on Server A doesn't need to match whatever the old structure was of the pool you lost. It just needs to be large enough to hold the data. So if you've gained experience from the choices you made last time, feel free to make better choices when creating the pool this time. But yeah, if you are using 4k-native drives, you'll still want ashift=12 etc.

Also, I gather from your comment below that the pool and filesystem names will be the same, so we will restore pool zfs49 filesystem tank on server B, to pool zfs49 filesystem tank on server A.

Rather than transfer each filesystem explicitly, it is probably beneficial to illustrate how to use a recursive snapshot to transfer the entire pool, including all filesystems, and their snapshots.

After you've created the empty pool on Server A above, the next step is to make an up-to-date recursive snapshot on Server B. This snapshot will be chiefly for the purpose of this one-time restore job, and thus does not need to match your naming scheme of YYYYMonDDHHMM. In fact, insofar as this restore job snapshot is likely to be short-lived, it may help to have it stick out from the others.

I propose that you create a recursive snapshot on Server B and name it xfer, to be used for the specific purpose of this one time transfer. So on Server B:

zfs snap -r zfs49@xfer

Now, since you have a pristine (empty, blank) pool on Server A named zfs49, you can transfer that recursive snapshot from Server B to Server A, beginning from Server B:

zfs send -R zfs49@xfer | ssh <Server A ip> zfs recv -Fuv zfs49

I tend to use -Fuv options routinely on zfs recv because:

  • -u confirms that I want to leave the newly-restored filesystems unmounted (for the time being) on server A. It is easy to get into trouble transfering ZFS pools from one system to another and clobbering (overlaying) existing mountpoints. The -R /mnt option on the zfs create step also protects against this, so perhaps this is a case of "suspenders and a belt," but when transfering and restoring backups, it's difficult to have too much caution.
  • -v gives me some feedback on how the job is progressing, as each completed snapshot will be listed in turn.

As for progress indication, you may also find that zfs transfers are an excellent opportunity to make use of Andrew Wood's intrepid pipe viewer pv(1):

(From Server B:)

zfs send -R zfs49@xfer |
  pv -Wbraft |   
  ssh <Server A ip> zfs recv -Fuv zfs49/tank

Adjust the pv options to suit your taste and needs.

Once the transfer has completed, inspect the pool on Server A, with an especially close eye to mountpoints, bearing in mind that they will be prefixed with /mnt because of your zpool create ... syntax. If you wish, you may (on Server A):

zpool export zfs49
zpool import -N zfs49
zfs list

This will import the pool to show the native mountpoints, but the zpool import -N command tells zpool to NOT actually mount anything. This is helpful to see an apples-to-apples comparison of where the pool's mountpoint will actually be located. Make sure the pool's mountpoints aren't going to overlay any existing mountpoints on Server A, or you may lose access to possibly critical filesystems. This use of zpool import -N dovetails with the above use of zpool recv -u to try to minimize the risk of mounting newly-created filesystem(s) in the wrong location(s).

Finally, once you're happy with how the pool looks on Server A, you can delete the xfer snapshot from both systems. From Server A:

zfs destroy -rv zfs49@xfer
ssh <Server B> zfs destroy -rv zfs49@xfer

One caveat that I don't have a good solution for is that it is not trivial to backup and restore properties of the pool itself, that is, options that must be read and written via zpool rather than zfs. This includes properties like bootfs, and all the other tidbits that zpool get all zfs49 shows you. Most of the defaults will likely be reasonable, and if you've changed any of them in the past, hopefully you'll recognize them and can manually change the properties on your Server A's new pool to suit your needs. Otherwise, the topic of how to backup and restore pool properties is probably worthy of a separate question, if one isn't out there already.

Jim L.
  • 7,188
  • 1
  • 13
  • 25
  • thanks for the detailed response! yes, i wish to restore the entire `zfs49` into the Server A (because thats the only zpool/filesystem in Server A originally) - in that case, i'll replace `zfs49/tank@2021Dec031705` with `zfs49` ? also, regarding the `zpool create` - must I create the same pool configuration (pool was originally created with `ashift=12`), or it doesnt matter as it'll just take whatever `zpool recv` receives? – LooseAlien123 Dec 05 '21 at 17:15
  • @LooseAlien123 Can you be more specific, please? Are you trying to send `zfs49/tank` back to Server A to be received as filesystem `tank` or will Server A also have the same pool/filesystem structure, like `zfs49/tank`? – Jim L. Dec 05 '21 at 17:23
  • yes, the latter - where Server A will have the same pool/filesystem structure, `zfs49/tank`. when I do `zfs list` on Server B, it listed `zfs49` and `zfs49/tank`, which arethe orignal pool/filesystem structure in Server A. so if im understanding this correct, doing `zfs send -R zfs49` back instead of `zfs send -R zfs49/tank@2021Dec031705` would be the right move? – LooseAlien123 Dec 05 '21 at 17:30
  • @LooseAlien123 No, if you want to send all the filesystem(s) as they are, the best way IMO is to create a new snapshot **of the whole pool** and send that. So `zfs snap -r zfs49@xfer` and then `zfs send -R zfs49@xfer` to Server A `zfs recv -Fuv zfs49` – Jim L. Dec 05 '21 at 17:44
  • noted on this - by sending a new snapshot `zfs49@xfer`, after Server A fully receives it, will Server A still have the original snapshots `zfs49/tank@2021Dec031705`, etc? this is because I wish to continue the daily incremental send/recv from it as backup. or do I need to start the incremental all over? – LooseAlien123 Dec 05 '21 at 17:47
  • @LooseAlien123 Yes, because you use `send -R` all underlying filesystems AND their snapshots will be transfered. And yes, you can then resume use of your `YYYYMonDDHHMM` snapshot names for your incremental backups. Try the above, and on Server A inspect the results: `zfs list -Hrt all zfs49` will show you all filesystems and all their snapshots. – Jim L. Dec 05 '21 at 17:50
  • @LooseAlien123 Ehh, maybe no `-H` in that, force of habit on my part: `zfs list -rt all zfs49` – Jim L. Dec 05 '21 at 17:59
  • @LooseAlien123 Several typos fixed. Please have another look and make sure these commands work for you. Questions and feedback appreciated! – Jim L. Dec 05 '21 at 19:49
  • thanks for the clarification. i noticed that `zfs snap` manual does not seem to be able specify the `pool` to snapshot. i forgot to mention that in Server B, it is actually holding 2 pools `zfs41` and `zfs49` (it backups Server C, `zfs41`) as well. does `zfs snap -R zfs49@xfer` only snapshot `zfs49`? sorry for the last minute details - have also updated the question – LooseAlien123 Dec 06 '21 at 00:32
  • @LooseAlien123 Little `r` on `zfs snap ...` Big `R` on `zfs send ...` On Server B, `zfs snap -r zfs49@xfer` should work fine. Then `zfs list -rt all zfs49` to view the new snapshot on Server B. – Jim L. Dec 06 '21 at 03:42