14

I want to tar and send a ~700GiB directory to a remote drive I don't control. I don't have the HDD space locally to create the tarball and then copy that over. The remote is also protected by rssh - you cannot ssh into the server and run arbitrary commands.

However, I can use scp, rsync, or sftp. So, my question is specific to one of those 3 commands. Is there a way for scp, rsync, or sftp to copy a pipe, without being able to run arbitrary commands on the other side?

Update: The below solutions would not have worked for us at our secondary storage. We are restricted to running certain commands. We cannot ssh - commands that use ssh are filtered/controlled with rssh.

In the end, we canceled all our secondary accounts, and paid a lot of money to double our primary storage size (our primary host could do this without migration, which was cool). We then created a temporary archive and sent it as per normal. (Which we could not do with our initial amount of storage.)

In the future, we will be using a secondary VPS with its own dedicated larger storage, so that this does not occur again. It's clear that what I was asking for is not possible.

AReddy
  • 3,122
  • 5
  • 35
  • 75
SaburoutaMishima
  • 141
  • 1
  • 1
  • 5
  • I've answered this over at the [sister site](https://superuser.com/a/1252714/117590) -- in short, `lftp` is capable of doing this over the SFTP protocol. – Bob Sep 22 '17 at 08:07

2 Answers2

6

It is not exactly "pipe", but you can basically tell scp to copy specific FD (which can be pipe) from your host to the other. Simple bash command like this:

(scp does not work as it needs a size in advance):

scp <(tar cz files to compress) host:/path/to/new.file

but it can work with pure ssh:

tar cz files to compress | ssh host "cat > /path/to/new.file"
Jakuje
  • 20,974
  • 7
  • 51
  • 70
  • 2
    Did you try this? It doesn't work for me (OpenSSH scp 6.6). The [scp protocol](https://blogs.oracle.com/janp/entry/how_the_scp_protocol_works) requires sending the filesize ahead of the contents, which isn't suited to sending a stream of an unknown length. – Kenster Jun 15 '16 at 17:48
  • @Kenster Yes, you are right. `scp` will not work. But it should work with pure `ssh`. See edited answer. I didn't try that, but the conceptually it should do the job. – Jakuje Jun 15 '16 at 17:54
  • 1
    @Jakuje but the point of this question is that `rssh` prevents invoking `cat`, it has to go through one of the three mentioned commands. – b0fh Nov 15 '16 at 09:09
0

RSYNC behavior over SFTP

Without even compressing anything, you can replicate the directory structure to the other side using LFTP (name is misleading) and the mirror subsystem of LFTP. LFTP supports SFTP and the mirror subsystem supports nearly all the capabilities of rsync. It can also split up the transfer into as many connections as you want, to make it go faster. I do this all the time.

Example

Also check out this post: https://serverfault.com/questions/135618/is-it-possible-to-use-rsync-over-sftp-without-an-ssh-shell

Community
  • 1
Aaron
  • 181
  • 1
  • 3