How to copy a file that is still being written over ssh?

Question

Here is the situation:

1. I am uploading a large file from client A to a server using sftp.
2. I also need to download this file from the server to client B over ssh.

What I would like to do is start the transfer from the server to client B when the upload is still happening from client A.

What is the best method/tool to get this done?

UPDATE:

The answers so far are interesting--I'll be sure to read and test them all. Bonus points for answers that don't depend on controlling how Client A is uploading the file. (ie. the only thing we know from client A is that the file is being written to a known filename.)

Answer 1

For a single file instead of using SFTP you could pipe the file over ssh using cat or pv at the sending side and using tee on the middle server to both send the data to a file there and send a copy over the another ssh link the other side of which just writes the data to a file. The exact voodoo required I'll leave as an exercise for the reader, as I've not got time to play right now (sorry). This method would only work if the second destination is publicly accessible via SSH which may not be the case as you describe it as a client machine.

Another approach, which is less "run and wait" but may otherwise be easier, it to use rsync between the server and client B. The first time you run this it may get a partial copy of the data, but you can just re-run it to get more data afterwards (with one final run once the Client1->Server transfer is complete). This will only work if the server puts the data direct into the right file-name during the SFTP transfer (sometimes you will see the data going into a temporary file which is then renamed once the file is completely transferred - this is done to make the file update more atomic but will render the rsync idea unusable). You could also use rsync for the C1->S transfer instead of scp (if you use the --inplace option to avoid the problem mentioned above) - using rsync would also give you protection against needing to resend everything if the C1->Server connection experiences problems during a large transfer (I tend to use rsync --inplace -a --progress <source> <dest> instead of scp/sftp when rsync is available, for this "transfer resume" behaviour).

To summarise the above, running:

rsync --inplace -a --progress <source> user@server:/<destination_file_or_folder>

on client1 then running

rsync --inplace -a --progress user@server:/<destination_file_or_folder> <destination_on_cli2>

on client2 repeatedly until the first transfer is complete (then running once more to make sure you've got everything). rsync is very good at only transferring the absolute minimum it needs to update a location instead of transferring the whole lot each time. For paranoia you might want to add the --checksum option to the rsync commands (which will take much more CPU time for large files but won't result in significantly more data being transfered unless it is needed) and for speed the --compress option will help if the data you are transferring is not already in a compressed format.

Answer 2

I can't try it at the moment, so this might well fail: My idea is this: Mount the directory where the file is arriving in client B, e.g. with sshfs to /mnt/server in the filesystem of client b. Then

tail -c +0 -f /mnt/server/thefileinquestion > ~/finalfile

Answer 3

I think this should work:

user@clientA:~$ cat file | ssh server "cat > dest"

and then

user@clientB:~$ ssh server "tail +0 -f dest" > file

Add the pv command if you want to see your throughput.

Answer 4

You could use a fifo for it. For simplicity first without ssh only involving two xterms:

At xterm A:

$ mkfifo fif
$ cat test.tar.gz | tee copy.tar.gz > fif

At xterm B:

$ cat fif > dest.tar.gz
$ cmp test.tar.gz dest.tar.gz
$ echo $?
0
$ cmp test.tar.gz copy.tar.gz
$ echo $?
0

With ssh it should be something along these lines - perhaps you have to disable the escape-character in ssh (-e none):

client A:

 $ ssh server mkfifo fif
 $ cat src.tar.gz | ssh "tee fif > copy.tar.gz"

client B:

 $ ssh server cat fif > dest.tar.gz

Answer 5

I have a situation that needs a solution like the original poster asked. I am recording a hockey game on my computer in one location and I'd like to watch it on my TV at another location. The link between the two locations allows the copy to go at about 1.3Mb/s and the recording video is about 1.5Mb/s. So, I want to copy the file as it starts recording. This way my 3 hour game will copy in about 3.5 hours. So, I copy it as it starts recording and I can start watching it 30 minutes after it starts. Then I can watch it with no interruptions, almost in real time. That is, as long as I can get it to copy as its writing the new file. The problem with tools like rsync and scp is that they look at the size of the file when you initiate the copy and once it copies that amount of data, it quits; even if the file has grown by more than double during that copy. And, if, I'm just using rsync in a loop to copy it once it stops, when the next rsync finishes it rebuilds the target file and that kills my video player and I have to restart watching it and fast forward to wherever I was in the program when it suddenly killed it. I wanted a better solution and I haven't been able to find one, so I pieced together this instead:

dd if=2031_20160514030000.mpg |
pv --size 4653819304 |
ssh -C -c arcfour,blowfish-cbc -p 5555 myserver.com 'dd of=/media/TV/2031_20160514030000.mpg'

So what does this do?

First, I use dd to copy the file as it grows. Since the file grows faster than dd can send it over the network, dd never catches up to the end of the file. Next, I pipe it to "pipe viewer (pv)" and I give it an estimate on how big the file is going to be based on how big these files usually are. This isn't necessary, but I like to see a progress meter. Then, I pipe the stream to my ssh connection. The ssh connection uses -C for compression (to reduce the network bandwidth and try to speed it up), -c arcfour,blowfish-cbc for the least expensive encryption (again to speed things up a bit), the -p is for my firewall port I'm using at the destination, and the ssh finally runs the dd command on the target to recreate the file as it receives it. I'm happy to say, this solution works great. I can watch the hockey game while the file is being created and copied with only a short delay.

Answer 6

I'm not sure that the tail -f method works ( though it probably does if the file is text ). The reason is that I don't know how tail -f, and sftp transfer and rely on meta information.

If sftp transfers the meta information first and tail -f relies on the meta information to tell it that there is no more file, then tail may bad the end with EOFs or nulls.

If you do not care about the path of upload ie Computer 1 uploads to computer 2 uploads to computer 3, then you might try top use bittorent instead of sftp. It seems that is what it was designed for.

Answer 7

You can try to read the file from the beginning but you need to be sure that you can write it as the same speed at least.