19

As far as I know, WinRAR archives may contain ECC (error correction codes), so if the archive is slightly damaged, then it can be fixed by itself.

For example, I can first encode archives.tar to archives.tar.ecc, and then upload it to my server. If the file is slightly damaged after downloading by the client, then it can be fixed automatically without downloading the file again by decoding archives.tar.ecc. I think it will be a great idea if the network connection is unstable.

I wonder whether there is any (open-sourced) software run on Linux that can meet my needs.

Any suggestions?

Anthon
  • 78,313
  • 42
  • 165
  • 222
Kevin Dong
  • 1,139
  • 1
  • 9
  • 18
  • to some degree, this already happens - each file's header chksum field is the sum of that header's constituent bytes - to include the size field. Many `tar`s go further - GNU `tar` encodes sparse file holes in header metadata and upon extraction by any POSIX `pax` compatible archiver generates files containing information needed to rebuild the file, or by GNU `tar` to the original sparse file. It uses primarily encoded header variables definable as [specified by POSIX for the `pax` utility](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html) w/ `-o`. – mikeserv Nov 30 '14 at 08:51

5 Answers5

8

You're interested on PAR2, which uses Reed-Solomon error correction. parchive is the ECC file format specification, for Linux you will want PyPar or par2tbb, and on Windows you use a GUI called QuickPar.

Uyghur Lives Matter
  • 1,656
  • 3
  • 22
  • 31
RAKK
  • 1,332
  • 2
  • 17
  • 35
8

I had this same problem and found that the solutions provided in the other answer were not acceptable since the par2 program/standard involves creating separate files to do recovery and verification. I, like the original asker I think, want a single file with error correcting codes.

I found the program rsbep which used to be in the package manager and then found the necessary code here: https://www.thanassis.space/rsbep.html

This allows one to create a file with Reed-Solomon error correcting codes and then retrieve the original file from the, possibly corrupted, ECC version.

Ethan
  • 180
  • 1
  • 3
  • rsbep is great! But it's a bit different: it's an entire frozen OS, so you can add files inside but you cannot modify what is inside, and it's a bit more cumbersome to move around. But it's a really interesting approach, using FUSE, and opensource. – gaborous Dec 28 '16 at 15:08
  • 1
    @gaborous, that's not true. There is a FUSE fs implementation using `rsbep` transparently, but that's only on top of the scripts that work on a file-by-file basis. – nyov Mar 25 '19 at 14:01
3

If you want to have the ecc file separate from the archive, you can use pyFileFixity, an opensource Python project that aims to provide a next-gen PAR-like protection system against bit corruption.

If you want the ecc protection to be included directly in the archive, you can use the opensource DAR, a kind of next-gen TAR, which allows to generate both non-solid (ie, allows partial extraction of corrupted archives) and ecc (using PAR2) and catalog isolation (ie, save a backup of directory tree and filesystem meta-data) directly in the .dar archives.

gaborous
  • 145
  • 4
2

Another similar 2018 project is Redupe that seems to provide both separate and embedded file error correction in two separate tools:

The first tool, redupe , is modeled after compression tools like gzip or bzip2 , but adds redundancy instead of eliminating it.

The example in the webpage (changed for brevity) looks similar to your use case:

$ redupe home-backup.tar.gz
$ ls -l
-rw------- 1 rescrv 5625162218 home-backup.tar.gz
-rw-r--r-- 1 rescrv 6433996800 home-backup.tar.gz.rd

reundupe can then be used to restore the file and possibly correct small errors.

yuri
  • 129
  • 3
1

You can use dvdisaster which works also for other file types than images. It is able to create an ECC file for an input file or use one to repair a file.

user2525536
  • 111
  • 2
  • This answer is not directly applicable to OPs issue. – number9 Jun 27 '22 at 15:52
  • 1
    dvdisaster can create ECC files for single files. You can use this ECC file to repair the original file. Of course, dvdisaster is usually used for other use cases but not limited to those. – user2525536 Oct 11 '22 at 21:05
  • That was not apparent in your initial response, you should edit it (as it was unclear, for example). Thanks – number9 Oct 12 '22 at 12:36