Parse colon-separated value pairs (nmcli output) and convert to JSON fromat

Question

I've written a script that converts the output of nmcli --mode multiline dev wifi into JSON, but I'm finding it's inconsistent (breaks when results have a space), long, and hard to read.

I wonder if it is possible to pipe the results directly into jq.

The nmcli output (input to my script) looks like this:

*:                                       
SSID:                                   VIDEOTRON2255
MODE:                                   Infra
CHAN:                                   11
RATE:                                   54 Mbit/s
SIGNAL:                                 69
BARS:                                   ▂▄▆_
SECURITY:                               WPA1 WPA2
*:                                      * 
SSID:                                   VIDEOTRON2947
MODE:                                   Infra
CHAN:                                   6
RATE:                                   54 Mbit/s
SIGNAL:                                 49
BARS:                                   ▂▄__
SECURITY:                               WPA1 WPA2

I'm looking to generate something like this:

[{
    "network": "VIDEOTRON2255",
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "bars": "▂▄▆_",
    "security": "WPA1 WPA2"
},
{
    "network": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "bars": "▂▄__",
    "security": "WPA1 WPA2"
}]

---

I asked a related question earlier. This is the first script I wrote based on Gilles's answer. It worked for some of the values but not security or rate, which have spaces.

Answer 1

The script that you linked to is extremely inefficient - you're doing a lot of useless pre-processing...
Use nmcli in --terse mode since, per the manual, "this mode is designed and suitable for computer (script) processing", specify the desired fields and pipe the output to jq -sR e.g.

printf '%s' "$(nmcli -f ssid,mode,chan,rate,signal,bars,security -t dev wifi)" | \
jq -sR 'split("\n") | map(split(":")) | map({"network": .[0],
                                             "mode": .[1],
                                             "channel": .[2],
                                             "rate": .[3],
                                             "signal": .[4],
                                             "bars": .[5],
                                             "security": .[6]})'

Answer 2

This GNU sed code isn't jq, (it isn't a complex conversion), but it seems to work well enough, (even the bars come out OK):

nmcli --mode multiline dev wifi | 
sed    '/^*/! {s/^[A-Z]*/\L&/
               s/ssid/network/
               s/: */": "/
               s/$/"/
               {/^sec/!s/$/,/}
               s/^/\t"/}
        1     s/^\*.*/[{/
        /^\*/ s/.*/},\n{/
        $  {p;s/.*/}]/}'

Easier to read standalone pcsvp.sed script, (save to file, then run chmod +x pcsvp.sed):

#!/bin/sed -f
# Text lines (the non "*:" lines.)
/^*/! {s/^[A-Z]*/\L&/
       s/ssid/network/
       s/: */": "/
       s/$/"/
       {/^sec/!s/$/,/}
       s/^/\t"/}

# First JSON line
1     s/^\*.*/[{/

# Middle JSON lines.  If a line begins with a '*'...
/^\*/ s/.*/},\n{/

# Last line, close up the JSON.
$     {p;s/.*/}]/}

To run that do:

nmcli --mode multiline dev wifi | ./pcsvp.sed

---

Note: Since there are doubts about the input file, I've opted to use nmcli for input instead. At my location this shows about 50 networks, which makes the resulting output too long to quote here.

If the input sample typo is corrected, ./pcsvp.sed input.txt outputs:

[{
    "network": "VIDEOTRON2255",
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "bars": "▂▄▆_",
    "security": "WPA1 WPA2"
},
{
    "network": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "bars": "▂▄__",
    "security": "WEP"
}]

Answer 3

Complex jq solution (with BARS line removed as it contains irregular/non-ASCII characters):

Input file input.txt:

*:                                       
SSID:                                   VIDEOTRON2255
MODE:                                   Infra
CHAN:                                   11
RATE:                                   54 Mbit/s
SIGNAL:                                 69
SECURITY:                               WPA1 WPA2
*:                                      * 
SSID:                                   VIDEOTRON2947
MODE:                                   Infra
CHAN:                                   6
RATE:                                   54 Mbit/s
SIGNAL:                                 49
SECURITY:                               WPA1 WPA2

---

The job:

jq -sR '[ gsub("[*]: *\n| {2,}";"") | gsub("SSID";"network") | split("\n[*]:[*] +\n";"n")[] 
    | [ capture("(?<key>[^:\n]+):(?<value>[^:\n]+)";"g") | .key |= (. | ascii_downcase) ] 
    | from_entries ]' input.txt

The output:

[
  {
    "network": "VIDEOTRON2255",
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "security": "WPA1 WPA2"
  },
  {
    "network": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "security": "WPA1 WPA2"
  }
]

---

Additional approach for another particular input presented/posted on https://pastebin.com/8stHSUeu:

jq -sR '[sub("[*]: *[*]\n";"") | gsub(" {2,}";"") | gsub("SSID";"network") 
  | split("\n[*]: *\n";"n")[] 
  | [ capture("(?<key>[^:\n]+):(?<value>[^:\n]+)";"g") | .key |= (. | ascii_downcase) ] 
  | from_entries]' input.txt

Answer 4

If you can, use a tool that understands JSON back and forth. I'd use Python:

#! /usr/bin/env python3
import json
import re
import sys

objects = []
obj = {}
for line in sys.stdin:
    entry = re.split(':\s*', line.strip(), maxsplit=1) # split on first `:`
    if entry[0] == '*':
        if obj:  # skip a null entry (the first, here)
            obj['network'] = obj.pop('ssid') # rename the SSID entry
            objects.append(obj)
        obj = {} # start a new object for each `*`
        continue
    obj[entry[0].lower()] = entry[1]  # lowercase the key
obj['network'] = obj.pop('ssid') # rename the SSID entry
objects.append(obj)
json.dump(objects, sys.stdout)

Gets me:

[{"mode": "Infra", "chan": "11", "rate": "54 Mbit/s", "signal": "69", "bars": "\u2582\u2584\u2586_", "security": "WPA1 WPA2", "network": "VIDEOTRON2255"}, {"ssid": "VIDEOTRON2947", "mode": "Infra", "chan": "6", "rate": "54 Mbit/s", "signal": "49", "bars": "\u2582\u2584__", "security": "WPA1 WPA2"}]

which, when pretty-printed by jq is:

[
  {
    "mode": "Infra",
    "chan": "11",
    "rate": "54 Mbit/s",
    "signal": "69",
    "bars": "▂▄▆_",
    "security": "WPA1 WPA2",
    "network": "VIDEOTRON2255"
  },
  {
    "ssid": "VIDEOTRON2947",
    "mode": "Infra",
    "chan": "6",
    "rate": "54 Mbit/s",
    "signal": "49",
    "bars": "▂▄__",
    "security": "WPA1 WPA2"
  }
]

Answer 5

1. GNU sed and mlr method:

   nmcli dev wifi | sed 'y/*/ /;1{s/.*/\L&/;s/ssid/network/};s/   */\t/g'  | 
   mlr --p2j --fs '\t' --jvstack --jlistwrap cat
   

2. bash, text mode nmcli, (swiped from don_chrissti's answer), and mlr:

   h=ssid:mode:chan:rate:signal:bars:security
   { echo ${h/ssid/network} ; nmcli -f ${h//:/,} -t dev wifi ; } | \
   mlr --c2j --fs ':' --jvstack --jlistwrap cat
   

Answer 6

Using csvjson from the csvkit toolkit to create the JSON document directly from the nmcli command:

#!/bin/sh

set -- SSID MODE CHAN RATE SIGNAL BARS SECURITY

( IFS=:; printf '%s\n' "$*"
  IFS=,; nmcli -g "$*" device wifi ) |
csvjson -d :

The first printf creates a :-delimited CSV header for the cvsjson command created from the positional parameters set at the start of the script, and the nmcli command gets a comma-delimited list of the same parameters as the option-argument to its -g (--get-values) option. The nmcli command will create what amounts to :-delimited CSV output, and this, together with the simplistic CSV header, is parsed by csvjson to produce a JSON document.

The JSON document created will contain an array of objects like

{
  "SSID": "The wireless network",
  "MODE": "Infra",
  "CHAN": 6,
  "RATE": "195 Mbit/s",
  "SIGNAL": 45,
  "BARS": "▂▄__",
  "SECURITY": "WPA2"
}

To lowercase the keys and to change SSID into network, we may use jq:

#!/bin/sh

set -- SSID MODE CHAN RATE SIGNAL BARS SECURITY

( IFS=:; printf '%s\n' "$*"
  IFS=,; nmcli -g "$*" device wifi ) |
csvjson -d : |
jq '.[] | with_entries(if .key == "SSID" then .key |= "network" else .key |= ascii_downcase end)'

This is expected to produce an array with entries like

{
  "network": "The wireless network",
  "mode": "Infra",
  "chan": 6,
  "rate": "195 Mbit/s",
  "signal": 34,
  "bars": "▂▄__",
  "security": "WPA2"
}