How to use 7z to compress via pipe?

Question

I'm using this for creating backups securely (? - is it really secure? with a good password?):

# ENCRYPT
ORIGDIR="DIRECTORYNAMEHERE"; tar cvf - "${ORIGDIR}/" 2>/dev/null | gzip -9 - 2>/dev/null | openssl aes-256-cbc -salt -out "${ORIGDIR}.tar.gz.aes"

# DECRYPT
openssl aes-256-cbc -d -salt -in "ENCDIRECTORYNAMEHERE" | tar -xz -f -

Q: But how can I do this using 7z with max compression rate?

Creating temporary files besides the only OUTPUT file is not good, because if I need to compress ~100 GByte sized files/directories on a 180 GByte FS I wouldn't have enough free space (if ex.: the compressed file would take ~60 GByte).

score 37 · Answer 1 · answered Aug 13 '13 at 13:43

37

This is covered in the man page of 7z:

-si    Read data from StdIn (eg: tar cf - directory | 7z a -si directory.tar.7z)

answered Aug 13 '13 at 13:43

Wieland

6,353
3
28
31

1

Can I put a "|" and further process it after the 7z? – gasko peter Aug 19 '13 at 19:30
7

You can use `-so` to dump the compressed data on stdout instead of a file. – Wieland Aug 19 '13 at 21:14
1

`-so` results in error `E_NOTIMPL` – phil294 May 22 '20 at 16:35
Please open a new question with an example to reproduce it. – Wieland May 23 '20 at 07:28

score 11 · Accepted Answer · answered Aug 13 '13 at 12:05

11

Is there a reason you want to use 7z specifically, or do you just want better compression than gzip?

The xz utility uses the same compression algorithm as 7z (LZMA), and allows piped compression the same as gzip.

tar cvf ... | xz -9 | openssl ...

answered Aug 13 '13 at 12:05

David Baggerman

2,154
20
11

The reason I personally needed specifically 7z is ability to split to multiple volumes – Unirgy Dec 16 '15 at 10:00
1

xz wasn't multi-threaded at the time (it may not be yet) so on a multi-core computer it could be a lot slower than 7z – Xen2050 Apr 25 '16 at 19:59

mpen · Answer 3 · 2020-05-31T22:52:03.160

Some examples of reading from stdin and writing to stdout:

echo "hello world" | 7z a .xz -si -so | xz -dc # hello world
echo "hello world" | 7z a .xz -si -so | wc -b # 64
echo "hello world" | 7z a .xz -si -so | 7z x -txz -si -so # hello world
7z x -so db.sql.xz | less # preview compressed database file

a create archive. Requires a filename; we're using just .xz to abuse it to set the archive type, since we don't actually care about a name
-si read from stdin
-so write to stdout
x extract; must also specify -t the type since we didn't create an archive w/ filename

tansy · Answer 4 · 2023-07-29T14:06:07.893

For the record as this thread is quite old.

7-zip does not allow streamed write of 7z format, it requires free seeking access as it writes, amongst the other stuff, header at the end of operation. It does allow to pipe standard *nix formats like bz2, gz, xz, though. It also allows to pipe single input without restrictions (with -si switch).

So one can compress single file into 7z format but cannot pipe it further.

$ cat archive.tar | 7z a -si archive.tar.7z

Other formats, like mentioned bz2, gz, xz can be written to stdout (with -so switch).

$ cat archive.tar | 7z a -tgzip -si -so -an > archive.tar.gz

==

$ 7z a -si -so -tgzip -an < archive.tar > archive.tar.gz

This obviously allows to pipe it further:

$ cat archive.tar | 7z a -tgzip -si -so -an | wc -c

Note -an switch that disables parsing of archive_name. What it actually does, god one knows - this program is bit magical and you have to cope with its quirks. Example of this behaviour can be found here.

score 1 · Answer 5 · answered Jun 30 '17 at 06:31

To quickly create an remote backup preparation file of all my home scripts.I use the following:

# Scripts backup
ls ~/*.sh | cpio -ov | 7z a -si ~/Documents/SCRIPT_BACKUP_30062017.cpio.7z
# Scripts restore
7z x -so ~/Documents/SCRIPT_BACKUP_30062017.cpio.7z | cpio -iv

Reason I don't backup my 'home' root is that I specifically sync and send only some directories. Notice how cpio will skip and check for newer existing files on restore. This is powerful. Your work will not be overwritten.

bud@

score -1 · Answer 6 · edited Apr 27 '18 at 14:48

-1

just got it to work, postgres pgdumpall to a 7zip file: use set pgpassword=xxxxx before issuing command:

pg_dumpall -U postgres | c:\"program files"\7-Zip\7z.exe a -si e:\directory\output.file.sqlc

it just took 4 days to get it dumped and compressed! 50201104790 bytes (47GB)

edited Apr 27 '18 at 14:48

Romeo Ninov

16,541
5
32
44

answered Apr 27 '18 at 14:06

klmbear

9

How to use 7z to compress via pipe?

6 Answers6

Linked