How to extract certain data from an MQTT stream

Question

I type mosquitto_sub -d -t +/# from the Ubuntu terminal to access the MQTT stream.

However I would like to see only a specific set of data from the MQTT stream. I've added | grep -v PING so everything still be printed except for lines with PING in them but that didn't work.

I've tried | grep -A1 PUBLISH so I can see lines from the MQTT with PUBLISH in them, nothing changed: I just get the whole MQTT stream nothing gets filtered out.

How can I see the certain thing that I want to see from an MQTT stream? Alternatively; how can I filter certain things out of the MQTT stream or how can I extract data from the MQTT stream?

My input from the ubuntu terminal is as follows $ mosquitto_sub -d -t +/# and I've tried $ mosquitto_sub -d -t +/# | grep -v PING and $ mosquitto_sub -d -t +/# | grep -A1 PUBLISH but it just streams everything, doesn't grep anything. The output is as follows:

ed@agharta:~$ mosquitto_sub -d -t +/#
Received CONNACK
Received SUBACK
Subscribed (mid: 1): 0
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/jsyd/TEST/001/d/status', ... (34 bytes))
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/jsyd/TEST/001/d/SVlts', ... (28 bytes))
Sending PINGREQ
Received PINGRESP

For example how can I only see the PUBLISH from the live stream and not the PINGs?

Please edit your question to include sample input and output... — jasonwryan, Feb 24 '15 at 02:49
@jasonwryan I've included the input and output from the terminal — 3kstc, Feb 24 '15 at 03:01
@jasonwryan I looked up http://www.grymoire.com/Unix/Sed.html#uh-15 and the format is `sed -n '/PATTERN/p' file`; I'm not too sure how to filter out specific words like "PING" and what the `/PATTERN/` and `file` refer to; Simply put I just don't get it from reading it. — 3kstc, Feb 24 '15 at 03:39
Well, it is unclear what you want to filter on: I took from your question you only wanted lines with the word "Subscribe" in them... — jasonwryan, Feb 24 '15 at 03:58
@jasonwryan I just want words with 'PUBLISH' in them so I am guessing it would be `sed -n '/PUBLISH/p' stream` however what does the `stream` component do? — 3kstc, Feb 24 '15 at 04:03
This is why I asked for sample input and output... Editing in the crucial information afterwards is not that helpful. — jasonwryan, Feb 24 '15 at 04:11
@jasonwryan this is a live feed that I am accessing, there was no data when I posted it. However in the code `sed -n '/PUBLISH/p' stream` what does the `stream` component do? — 3kstc, Feb 24 '15 at 04:48
@3kstc - Your commands are OK but note that `grep` doesn't operate on `stderr`. [Apparently](https://answers.launchpad.net/mosquitto/+question/252384) `mosquitto_sub` logs to `stderr` so (with `bash`) you could try something like `mosquitto_sub -d -t +/# 2> >(grep PUBLISH)`. For more details & options read the answers [here](http://unix.stackexchange.com/q/3514). — don_crissti, Feb 24 '15 at 12:42
@don_crissti it works like a treat, but looking at the forum provided I didn't understand what the 2> does and also `[what does this value mean]>[what does this space do?]>(grep [grepped keyword])` Additionally if I wanted to grep two or more keywords, how would I do it? — 3kstc, Feb 24 '15 at 22:04
@3kstc - redirection and process substitution... [here](http://www.catonmat.net/blog/bash-one-liners-explained-part-three/) is some stuff for you to read; as to _grep more patterns_, [here's](http://www.thegeekstuff.com/2011/10/grep-or-and-not-operators/) some more to read :) HTH. — don_crissti, Feb 25 '15 at 02:42
@don_crissti it's my second week in with bash and linux, - massive learning curve, I've [looked at dot point 22](http://www.catonmat.net/blog/bash-one-liners-explained-part-three/) I think I understand, but I've one [last question](http://unix.stackexchange.com/questions/186718/how-can-i-remove-a-carriage-return-add-a-time-stamp-and-ignore-some-data-from-a) — 3kstc, Feb 25 '15 at 03:24

score 2 · Accepted Answer · edited Apr 13 '17 at 12:36

I was in the process of writing the following answer to the other, related question here when it was deleted. I don't know if you'll be able to follow that link, but (an abbreviation of) the example data referred to below is this:

Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/SVlts', ... (28 bytes))
86,1224830,27610 27869 17565
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/status', ... (39 bytes))
86,1243000,164573,-33.836465,151.051189
Sending PINGREQ
Received PINGRESP

So I wrote the following to make it work with sed like this:

mosquitto ... 2>&1 | 
sed -u "/.*PUBLISH[^']*\([^,]*,\).*/!d 
       s//\1/;N;s/[\%]/&&/g;s/'/&\\\\&&/g
       s/.*/printf '&'/;s/\n/'&date '+[%s,]/"|  
sh -s

Now, I don't have your mosquitto program, but if I copy your whole example thing to my clipboard, and do:

xsel -bo | sed ... | sh ...

It prints:

'm/gf/TMX6BP/075/d/SVlts',[1425002404,]86,1224830,27610 27869 17565
'm/gf/TMX6BP/075/d/status',[1425002404,]86,1243000,164573,-33.836465,151.051189
'm/NWRL/TMX/098/d/SVlts',[1425002404,]806,3040421,7549 7750 3904
'm/NWRL/TMX/098/d/status',[1425002404,]806,3069000,59666,-33.836465,151.051189
'm/NWRL/TMX/098/d/SVlts',[1425002404,]810,5440995,6143 7807 4076
'm/NWRL/TMX/098/d/status',[1425002404,]810,5489000,59897,-33.836465,151.051189

...which is, I think, what you want. It works by ensuring all of the input is safely quoted for both the shell and for date and printf's format strings, and then constructing shell commands out of each line pair.

For each line pair printf first prints the contents of the first line w/out a trailing \newline, then date prints the [timestamp] in seconds since the epoch followed by the contents of the second line and a trailing \newline. sed's output looks something like this:

printf ''\''m/gf/TMX6BP/075/d/SVlts'\'','
date '+[%s,]86,1224830,27610 27869 17565'

None of the commands printed at sh, of course, are executed until sed prints them at it. So if sed were reading mosquitto's output and one line came in later than a previous one then the date line would reflect that.

For example:

(set -f; IFS='
';  for l in $(xsel -bo)
    do  printf %s\\n "$l"
        sleep 1
    done) |
sed -u "/.*PUBLISH[^']*\([^,]*,\).*/!d 
       s//\1/;N;s/[\%]/&&/g;s/'/&\\\\&&/g
       s/.*/printf '&'/;s/\n/'&date '+[%s,]/"|  
sh -s

...which waits a second before printing every output line at sed in order to more faithfully emulate a live stream. It takes advantage of the GNU sed -unbuffered switch to get sed's output flushed as often as possible, but if that is not available to you let me know and I will show you how to block out a pipe correctly w/ dd to avoid buffering issues.

Anyway, the above prints:

'm/gf/TMX6BP/075/d/SVlts',[1425002863,]86,1224830,27610 27869 17565
'm/gf/TMX6BP/075/d/status',[1425002865,]86,1243000,164573,-33.836465,151.051189
'm/NWRL/TMX/098/d/SVlts',[1425002869,]806,3040421,7549 7750 3904
'm/NWRL/TMX/098/d/status',[1425002871,]806,3069000,59666,-33.836465,151.051189
'm/NWRL/TMX/098/d/SVlts',[1425002881,]810,5440995,6143 7807 4076
'm/NWRL/TMX/098/d/status',[1425002885,]810,5489000,59897,-33.836465,151.051189

How to extract certain data from an MQTT stream

1 Answers1