5

I have a 100gig file on a remote server, what I need to do is connect to that machine and zcat that file and pipe the output of zcat to a command on a local machine... I was hoping smbclient would help but I can't seem to find a way to run a command locally but have the left side of the pipe come from a remote source.

What I think it will kinda look like with my made up command remoteZcatCommand

remoteZcatCommand 100gigRemotefile.gz | grep findSomeStuff

This maybe somewhat misleading as grep findSomeStuff will actually be a command that requires the hardware on the local machine. Also moving the 100gig file to the local machine is not an option.

Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
Dylan
  • 1,018
  • 2
  • 10
  • 19
  • It's not an option to mount the remote SMB (CIFS) file system on the local machine and then access that file via the mounted file system? – wurtel Mar 16 '15 at 08:33
  • I can mount the remote file system but it seems that the only option I have is to copy the desired file to my local computer. This is not an option as space is very very limited and there is an ASIC on the local machine i need to make use of. No I am not bitcoin mining ;) – Dylan Mar 16 '15 at 19:26
  • if you can mount the remote file system then you should be able to run `zcat` directly on a file on that file system and then pipe the output for further processing... but if the `smbclient` solution below is OK then that's great :) – wurtel Mar 17 '15 at 12:12

1 Answers1

11
smbclient -E -U $USERNAME //server/sharename $PASSWORD -c 'get \\dir\\filename.gz /dev/fd/1' 2>/dev/null | zcat | yourcommand

The -E instructs smbclient to send all messages to standard error instead of standard out where those messages will mess up the output we actually want. I'm not interested in those messages so those are discarded by the 2>/dev/null.

The -U $USERNAME is to indicate what username should be used when connecting to the SMB server.

The //server/sharename should be obvious.

The $PASSWORD is $USERNAME's password on the SMB server.

The -c 'get \\dir\\filename.gz /dev/fd/1' is the command that should be executed: get the named file (escaping the backslashes by doubling them), and send it to the local file /dev/fd/1 which is the same as the command's standard output. The standard output is then piped through zcat to expand it before whatever further processing is necessary.

wurtel
  • 15,835
  • 1
  • 29
  • 35
  • Ok the `get` command is something that I considered however reading the man page made it seem unlikely to work. However it seems that the *nix magic you are pulling is having the data sent to the local file `/dev/fd/1` which is stdout. Is this correct? That is a great idea, impressed and disappointed I didn't think of it. Forgot everything is a file in *nix. I'll be home soon and I'll give it a whirl and assume I will be accepting it. Thanks! – Dylan Mar 16 '15 at 19:34
  • This worked like a charm!!!! Great idea sending the data to the local file `/dev/fd/1` !! Thanks again! – Dylan Mar 16 '15 at 22:21