7

How to shard file file into m files, so it can be recovered with any n of them ?

It looks like Erasure Code, preferably "optimal erasure codes". (Example of another application and proposed programming library: "You need erasure code" https://stackoverflow.com/a/28932095 ). It's like Reed-Solomon error correction (something more flexible than RAID6) style of redundancy.

Early findings: I've found rsbep so far and some modification, but they rather seem to be designed toward different use case. I've also found reed-solomon from Linux Kernel ported to userspace here, but it's not the tool for my described purpose.

Example for 3-out-of-6 level of redundancy:

split_with_redundancy -n 3 -m 6 input.dat

Producing input.dat.0..5, so any tree of those files are sufficient for recovery:

recover_using_redundancy intput.dat.{0,2,4}

I do not care for errors within given file, i.e. I do not need Forward Error Correction. I assume that I can rely on having n-out-of-m redundant parts fully correct.

Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
Grzegorz Wierzowiecki
  • 13,865
  • 23
  • 89
  • 137
  • 1
    [par2](https://github.com/Parchive/par2cmdline) –  Apr 23 '16 at 23:46
  • Just came across: https://github.com/utamaro/rsraid -> need to give it a try as description sounds promising: "You can divide a file into k files and create redundant m files, and can recover the original file from arbitrary k files from these (k+m) files." . Also I've found that many projects (rozofs, ceph...) take use of Intel's Opensource ISA-L library: https://github.com/01org/isa-l , so maybe there is also some tool based on it. – Grzegorz Wierzowiecki May 10 '16 at 21:22
  • maybe it's a bit stretch, but one can encrypt file with passphrase, and store it with BananaSplit https://bs.parity.io/ ;) – Grzegorz Wierzowiecki Oct 20 '20 at 08:52

2 Answers2

4

You would want to look at

https://pypi.org/project/zfec/

This does exactly what you are looking for. E.g. 

zfec -k3 -m6 input.dat

creates 6 files, 3 of which are necessary to reconstruct the original file.

jf1
  • 407
  • 2
  • 10
2

You can use gfshare for this: http://manpages.ubuntu.com/manpages/bionic/man7/gfshare.7.html

apt-get install libgfshare-bin

For example, with 2-of-7.

gfsplit -n 2 -m 7 myfile.txt

Only two shares can be used to reconstruct the file:

gfcombine myfile.txt.1 myfile.txt.7
Martin Monperrus
  • 1,221
  • 3
  • 12
  • 20