0

I am working on a process to send data via a pipe from one server to another for processing.

Although this is not the exact command, it might look something like this:

tail -f logfile | grep "abc" | grep "def" | grep -v "ghi" | netcat -q 0 n.n.n.n 7777

I would like to wrap all those greps into a script and more importantly prepend the pipe to netcat with an identifier, so the command would look like this:

tail -f logfile | myscript.sh {id}

The script listening on the other end should receive:

{id}
[Line 1 of the logfile]
[Line 2 of the logfile]
...

Wrapping it in a script is easy:

#!/bin/sh
id=$1
grep "abc" | grep "def" | grep -v "ghi" | netcat -q 0 n.n.n.n 7777

but I cannot figure out how to inject $id at the start.

The receiving end is using

 socat -u tcp-l:7777,fork system:/dev/receivePipe

so if there is a different way I could get the id (for example somehow as a parameter to /dev/receivePipe), or via an environment variable, that would work too.

EDIT: The final answer was figured out in the comments of the accepted answer:

#!/bin/sh
{
  printf '%s\n' $1
  grep "abc" | grep "def" | grep -v "ghi" 
} | netcat -q 0 192.168.56.105 7777
Ben Holness
  • 103
  • 3

1 Answers1

3

Just do:

#! /bin/sh -
{
  printf '%s\n' "${1-default-id}"
  awk '/abc/ && /def/ && ! /ghi/'
} | socat - tcp:n.n.n.n:7777

${1-default-id} expands to the first positional parameter if specified or default-id otherwise. Replace with ${1?} to exit with an error if not passed any argument instead (or ${1?The error message} to specify an error message instead of the default).

We redirect the output of a command group that runs printf to output the ID and the filtering commands (here with your grep pipeline replaced with a single awk invocation that does the same thing a bit less clumsily) to socat/netcat.

Or to only print the ID if and when one line has been read and matches:

#! /bin/sh -
ID=${1-default-id} awk '
  /abc/ && /def/ && ! /ghi/ {
    if (!already_printed++) print ENVIRON["ID"]
    print
  }' | socat - tcp:n.n.n.n:7777

Or to prepend the ID (and a space character) to every line:

#! /bin/sh -
ID=${1-default-id} awk '
  /abc/ && /def/ && ! /ghi/ {
    print ENVIRON["ID"], $0
  }' | socat - tcp:n.n.n.n:7777

Beware awk, like grep will buffer their output when it goes to a pipe (to anything other than a tty device). With the GNU implementation of awk (aka gawk), you can add a call to fflush() after each print to force the flushing of that buffer. See also the -Winteractive of mawk. In most awk implementations, doing a system("") would also force a flush. The GNU implementation of grep has a --line-buffered option to force a flush after each line of output.

Also note that tail -f logfile is short for tail -n 10 -f logfile. Chances are you actually want either tail -n +1 -f logfile for the whole log file to be processed, and then tail carrying on following the file, or tail -n 0 -f logfile to process only the lines being added from now on.

Stéphane Chazelas
  • 522,931
  • 91
  • 1,010
  • 1,501
  • I am a little confused as to why you are piping to `socat`. Did you mean to pipe to `netcat`, or is something else going on? Regardless, I don't want to convert the pipes to other things such as `awk`. The `grep`s were just an example for this post. In reality, it might be other intermediary programs that it's getting piped through, with no `greps` at all. All I want to do is inject a line at the start of what ends up going to `netcat`. – Ben Holness Aug 01 '22 at 12:39
  • @BenHolness, seeing that you already used `socat`, I replaced your `netcat` call with the `socat` equivalent. In any case, the first approach will allow you to insert the header before any command. – Stéphane Chazelas Aug 01 '22 at 13:48
  • I used `netcat` on the sending server as `socat` is not installed there. I'm trying to understand how to use the first approach without awk and with the other pipes. I tried removing the `awk` line entirely (didn't work) and taking out all of the `/abc/` parts in the `awk` line (also didn't work). Here's an example of what I tried (the one without `awk`), with different, perhaps non-sensical, intermediary programs (and squashed because multi-line comments aren't possible): `#!/bin/sh { printf '%s\n' $1 } | grep "abc" | head -n 10 | netcat -q 0 192.168.56.105 7777` – Ben Holness Aug 01 '22 at 13:55
  • @BenHolness, replace the `awk '/abc/ && /def/ && ! /ghi/'` (which is the same as `grep abc | grep def | grep -v ghi`) with your filtering command (like `grep abc | head` or `grep -m10 abc`) – Stéphane Chazelas Aug 01 '22 at 14:10
  • Got it, with (for example): `#!/bin/sh { printf '%s\n' $1 grep "e" | head -n 3 } | netcat -q 0 192.168.56.105 7777` Thanks! – Ben Holness Aug 01 '22 at 14:15