Significant advantages I see for Ceph over BTRFS:
- CRUSH: This is Ceph's object placement algorithm, and is also what handles striping, replication, and erasure coding. It's got a couple of significant advantages over BTRFS, namely that it can handle (mostly) arbitrary erasure coding levels (RAID5 and RAID6 are special cases of erasure coding), as well as allowing arbitrary levels of replication and striping.
- Easier to debug: This probably sounds odd as a selling point for a home storage system, but it's actually pretty significant. Because BTRFS (and ZFS) runs in kernel mode, it is a serious pain in the arse to debug if you run into an issue. Ceph on the other hand runs in userspace (at least, if you'r enot using RBD or the in-kernel CephFS driver, both of which I would recommend against for home usage), which means that debugging when you run into an issue is significantly easier. Easier debugging in turn usually translates to much quicker bug fixes.
Significant advantages I see for BTRFS over Ceph:
- Fewer moving parts: BTRFS runs natively in kenel mode, doesn't require multiple daemons running, and also doesn't involve a networking layer. This has two specific advantages. The first is that there's less that can crash and cause issues. The second is that you have less overhead on the server itself.
- Simpler setup: This one follows from the item above, but is significant in it's own rright. It takes a lot less effort to set up BTRFS than it does Ceph.
Beyond that, I can't comment much, I've not looked at Ceph in about 3 years, and didn't do much with it back when I did because I quickly found out that CephFS was very much not production ready at the time.
