1

I'm using the saw and jq plus some filtering to get pretty-printed JSON log streams, but what I have below has no output. It just hangs.

saw watch /aws/ecs/container-log \
| sed -e 's/^\[.*\] \(.*\)(.*)/\1/' \
| tr '\n' '\0' \
| xargs -0 -n1 jq .

>

saw watch streams the container logs, similar to tail -f log.file. I'm using sed to filter out timestamp and log IDs, with the remainder being JSON.

When I just have sed after the watch, it correctly filters out the beginning of the log stream.

saw watch /aws/ecs/container-log \
| sed -e 's/^\[.*\] \(.*\)(.*)/\1/'

> {"data": "..."}

But when I add tr back in, it hangs again.

saw watch /aws/ecs/container-log \
| sed -e 's/^\[.*\] \(.*\)(.*)/\1/' \
| tr '\n' '\0'

>

However, tr on echo works fine.

echo hey | hexdump -c

> 0000000   h   e   y  \n
> 0000004

echo hey | tr '\n' '\0' | hexdump -c

> 0000000   h   e   y  \0
> 0000004

What could be causing this, and how can I fix it?

I'm using

  • zsh 5.8 (x86_64-apple-darwin19.3.0)
  • macOS 10.15.3
  • coreutils: stable 8.31 (installed via brew)
  • tr from coreutils

I can't find version for sed, but man sed is marked BSD May 10, 2005.

Glorfindel
  • 805
  • 2
  • 10
  • 19
zshift
  • 119
  • 2
  • 1
    is that just a buffering issue? What happens if you replace the `saw watch` with something that only prints a finite amount of output and then stops, like `printf "blahblabh"`? – ilkkachu Feb 27 '20 at 19:59
  • as shown above, `echo` works fine. I'm not familiar with how `tr` buffers, but streaming log output is pretty hefty. I also tried `cat /dev/random | tr '\n' '\0'` and that ran until I force quit. – zshift Feb 27 '20 at 20:30
  • I also tried prefixing `stdbuf -o0` on all commands with no change in behavior. From https://unix.stackexchange.com/a/239230/315211 – zshift Feb 27 '20 at 20:36
  • What if you replace the sed+tr with a single call to awk? You didn't provide any sample input/output for the sed command so I'm not sure what you're doing with that and therefore can't for sure provide the awk equivalent but the whole thing would be something like `saw ... | awk -v ORS='\0' '{gsub(/^\[.*\] |\(.*\)/,"")}1' | xargs -0 ...`. – Ed Morton Feb 27 '20 at 22:51
  • 3
    Hang on. What are you doing with `xargs`? xargs executes a given command, e.g. `jq`, on a list of files provided as input but you're not providing xargs a list of files as input are you? Isn't what's coming out of `tr` just a stream of text? Why aren't you just doing `saw watch /aws/ecs/container-log | sed -e 's/^\[.*\] \(.*\)(.*)/\1/' | jq .` or equivalent? – Ed Morton Feb 27 '20 at 22:59
  • Sorry for the late reply. I tried what you suggested, but the behavior was the same. It seems to be buffering, as changing to a log stream that was far more active was giving results in large batches. I ended up reading the source for `saw` and found 2 options that work for me, `--filter` and `--expand`. However, I'm leaving the question open, as I'm not sure how to get around this if it didn't have those options. – zshift Mar 04 '20 at 16:33

0 Answers0