parse one field from an JSON array into bash array

Question

I have a JSON output that contains a list of objects stored in a variable. (I may not be phrasing that right)

[
  {
    "item1": "value1",
    "item2": "value2",
    "sub items": [
      {
        "subitem": "subvalue"
      }
    ]
  },
  {
    "item1": "value1_2",
    "item2": "value2_2",
    "sub items_2": [
      {
        "subitem_2": "subvalue_2"
      }
    ]
  }
]

I need all the values for item2 in a array for a bash script to be run on ubuntu 14.04.1.

I have found a bunch of ways to get the entire result into an array but not just the items I need

Gilles Quénot · Accepted Answer · 2015-01-06T23:49:04.057

31

Using jq :

$ cat json
[
  {
    "item1": "value1",
    "item2": "value2",
    "sub items": [
      {
        "subitem": "subvalue"
      }
    ]
  },
  {
    "item1": "value1_2",
    "item2": "value2_2",
    "sub items_2": [
      {
        "subitem_2": "subvalue_2"
      }
    ]
  }
]

CODE:

arr=( $(jq -r '.[].item2' json) )
printf '%s\n' "${arr[@]}"

OUTPUT:

value2
value2_2

edited Jan 06 '15 at 23:49

answered Jan 06 '15 at 23:30

Gilles Quénot

31,569
7
64
82

Is it possible to do this from a variable instead of a file? I try to avoid excess filesystem access if I don't need it. Once I pull this array I am done with the json output. – JpaytonWPD Jan 07 '15 at 00:10
1

Have you tried something ? – Gilles Quénot Jan 07 '15 at 00:13
I tried replacing json with $JSON but jq is only looking for files. The json output is from a API call gathered with curl. Your answer does work if I store the output in a file. I was just hoping to avoid it. – JpaytonWPD Jan 07 '15 at 00:17
I also looked through the jq manual but theres nothing helpful. Could I pipe it through jq? – JpaytonWPD Jan 07 '15 at 00:22
2

`jq . <<< "$json"` it's shell (bash) related, non specific to `jq` – Gilles Quénot Jan 07 '15 at 00:23
I actually saved a step and did this: `curl -k https://localhost/api | jq -r '.[].item2'` I wrapped it in `arr=$(curl -k https://localhost/api | jq -r '.[].item2')` – JpaytonWPD Jan 07 '15 at 00:27
1

Missing parentheses : `arr=( $(...) )` – Gilles Quénot Jan 07 '15 at 00:35
6

Great `jq` command, but please don't parse command output into an array with `arr=( $(...) )` (even though it happens to work with the sample input): it doesn't work as intended with embedded or leading/trailing whitespace and can result in accidental globbing. – mklement0 Jul 14 '16 at 05:21
2

@GillesQuenot, ...would you an accept an edit fixing the shell end of this to either not depend on word-splitting, or explicitly setting IFS and disabling globbing to make that word-splitting reliable? – Charles Duffy Feb 01 '19 at 14:15
1

if item2 has value with space then this command separate them. – alhelal Mar 12 '20 at 09:58
Worked great ,thanks! – barakbd Jul 20 '20 at 05:30
if item2 has value with space then this command separate them. – alhelal Mar 12 '20 at 9:58 -- Is there any solution for this ??? – codelover Jan 13 '21 at 10:09
In bash (and Bourne-like shells other than zsh), problems are not limited to whitespace (Tab, Newline, Spc), but to characters of `$IFS` and glob characters as leaving a `$(...)` unquoted in list context like that is the split+glob operator. – Stéphane Chazelas Jan 21 '21 at 16:55
It would be more helpful if the array translation from jq applied to ALL formats of json, like lists. The jq -r '.[]' simple outputs a string with values separated by space and the format given here just takes the first value. Not general or translatable. – eco Oct 12 '22 at 01:04

Charles Duffy · Answer 2 · 2021-01-21T18:14:45.323

9

The following is actually buggy:

# BAD: Output line of * is replaced with list of local files; can't deal with whitespace
arr=( $( curl -k "$url" | jq -r '.[].item2' ) )

If you have bash 4.4 or newer, a best-of-all-worlds option is available:

# BEST: Supports bash 4.4+, with failure detection and newlines in data
{ readarray -t -d '' arr && wait "$!"; } < <(
  set -o pipefail
  curl --fail -k "$url" | jq -j '.[].item2 | (., "\u0000")'
)

...whereas with bash 4.0, you can have terseness at the cost of failure detection and literal newline support:

# OK (with bash 4.0), but can't detect failure and doesn't support values with newlines
readarray -t arr < <(curl -k "$url" | jq -r '.[].item2' )

...or bash 3.x compatibility and failure detection, but without newline support:

# OK: Supports bash 3.x; no support for newlines in values, but can detect failures
IFS=$'\n' read -r -d '' -a arr < <(
  set -o pipefail
  curl --fail -k "$url" | jq -r '.[].item2' && printf '\0'
)

...or bash 3.x compatibility and newline support, but without failure detection:

# OK: Supports bash 3.x and supports newlines in values; does not detect failures
arr=( )
while IFS= read -r -d '' item; do
  arr+=( "$item" )
done < <(curl --fail -k "$url" | jq -j '.[] | (.item2, "\u0000")')

edited Jan 21 '21 at 18:14

answered Oct 05 '16 at 03:02

Charles Duffy

1,651
15
22

1

But does not handle values with a newline in them :-( – jrw32982 Jan 21 '21 at 16:05
But I want the *values* in my array, not the encoded strings. See my answer below. – jrw32982 Jan 21 '21 at 16:54
@jrw32982, gotcha. You've got two choices, then, either using `"\u0000"` (NULs) as separators, or `@sh`. – Charles Duffy Jan 21 '21 at 17:05
@jrw32982, ...edited to show the former, since there are already two answers (one of them yours) with the latter. (Also, I consider using `eval` unnecessarily a nonstarter). – Charles Duffy Jan 21 '21 at 17:08
Note that `bash`'s `readarray` now supports `readarray -td ''` to read NUL-delimited data into an array. – Stéphane Chazelas Jan 21 '21 at 17:13
@StéphaneChazelas, good point. I don't suppose you'd remember which release that was added in? I'd want to change the "4.x" comment to be more specific if amending to demonstrate it. – Charles Duffy Jan 21 '21 at 17:15
That would be 4.4 – Stéphane Chazelas Jan 21 '21 at 17:34
To *detect failure*, you can do `{ readarray -td '' array && wait "$!"; } < <(...)`. – Stéphane Chazelas Jan 21 '21 at 17:39

Kusalananda · Answer 3 · 2021-01-21T16:49:11.663

4

Use jq to produce a shell statement that you evaluate:

eval "$( jq -r '@sh "arr=( \([.[].item2]) )"' file.json )"

Given the JSON document in your question, the call to jq will produce the string

arr=( 'value2' 'value2_2' )

which is then evaluated by your shell. Evaluating that string will create the named array arr with the two elements value2 and value2_2:

$ eval "$( jq -r '@sh "arr=( \([.[].item2]) )"' file.json )"
$ printf '"%s"\n' "${arr[@]}"
"value2"
"value2_2"

The @sh operator in jq takes care to properly quote the data for the shell.

Alternatively, move the arr=( ... ) part out of the jq expression:

eval "arr=( $( jq -r '@sh "\([.[].item2])"' file.json ) )"

Now, jq only generates the quoted list of elements, which is then inserted into arr=( ... ) and evaluated.

If you need to read data from a curl command, use curl ... | jq -r ... in place of jq -r ... file.json in the commands above.

edited Jan 21 '21 at 16:49

answered Jan 21 '21 at 16:31

Kusalananda

320,670
36
633
936

Somehow, this question sat here for a couple of years, without an accurate answer. I just discovered the solution (answered separately here) and now I see you've posted almost the exact same answer only a few minutes before me. If only you'd posted this before, it would have saved me figuring it out on my own! – jrw32982 Jan 21 '21 at 16:52
@jrw32982 I don't think I saw this question before, but Charles just edited [their answer](https://unix.stackexchange.com/a/314379/116858) which bumped the question to the top of the home page, where I spotted it. – Kusalananda Jan 21 '21 at 16:54
1

Yes, that's the only approach so far that handles arbitrary values. `jq` even happens to support non-text values in its strings as an extension over standard json. It seems to be using the safest form of quoting (`'...'`). I note that it transforms a NUL byte to `\0` and doesn't quote numbers nor `false`/`true` though that should be fine. As usual, note that it may transform numbers (like change `1e2` to `100` or `infinity` to `1.7976931348623157e+308`). – Stéphane Chazelas Jan 21 '21 at 17:01
@StéphaneChazelas Do you think it's an issue that some values are left unquoted? If so, one could pass the data through `tostring` before letting `@sh` format them. – Kusalananda Jan 21 '21 at 17:06
1

I don't expect it would here. There are context where not quoting numbers can be an issue (like in `echo '2'>file` vs `echo 2>file`), but no here I'd say. – Stéphane Chazelas Jan 21 '21 at 17:10
@StéphaneChazelas or Kusalananda: I could not find a way to set the positional parameters with `jq` because it separates values with newlines. So `eval "set -- $( some-jq-expression )"` doesn't work. Of course one could always set an array and then use the array to set the positional parameters. Is there another way? – jrw32982 Jan 21 '21 at 17:18
@jrw32982 You may want to ask a separate question about that. However, I have no issue setting the positional parameters with `eval "set -- $( jq -r '...' )"`. Testing with e.g. `printf '"%s"\n' "$@"`. I don't really see what the issue is. – Kusalananda Jan 21 '21 at 17:22
Try it with the example json file in my answer to this question. Replace the `eval` statement with `eval "set -- $(jq -r ' .[].item2 | @sh ' json)"`. – jrw32982 Jan 21 '21 at 17:30
@jrw32982 You're getting two separate lines from `jq`. What you want for the `jq` expression is `[.[].item2] | @sh`. Note the added `[...]`. You want to convert an array, not two individual values. – Kusalananda Jan 21 '21 at 17:39
Nice. That's what I was looking for. – jrw32982 Jan 21 '21 at 17:44

jrw32982 · Answer 4 · 2021-01-21T18:34:22.253

To handle arbitrary values:

#!/bin/bash
cat <<EOF >json
[
  {
    "item1": "value1",
    "item2": "  val'\"ue2  ",
    "sub items": [
      {
        "subitem": "subvalue"
      }
    ]
  },
  {
    "item1": "value1_2",
    "item2": "  value\n2_2  ",
    "sub items_2": [
      {
        "subitem_2": "subvalue_2"
      }
    ]
  }
]
EOF
eval "arr=( $(jq -r ' .[].item2 | @sh ' json) )"
printf '<%s>\n' "${arr[@]}"

Output:

<  val'"ue2  >
<  value
2_2  >

Putting together some of the other solutions in this question mentioned in comments and answers from @CharlesDuffy, @StéphaneChazelas, @Kusalananda:

#!/bin/bash
v1='a\nb  \n'   
v2='c'\''\"d  '   # v2 will contain <c'\"d  >  
printf '$v1=<%s>\n$v2=<%s>\n\n' "$v1" "$v2"

>json printf "%s\n" "[ \"$v1\", \"$v2\" ]" 
printf 'JSON data: '; cat json
printf '\n'

eval "arr=( $( cat json | jq -r '.[] | @sh ' ) )" 
printf '$arr[0]:<%s>\n$arr[1]:<%s>\n\n' "${arr[@]}"

set -- 
eval "set -- $( cat json | jq -r '[.[]] | @sh ' )" 
printf '$1:<%s>\n$2:<%s>\n\n' "$1" "$2"

{ readarray -td '' arr2 && wait "$!"; } < <( 
   cat json | jq -j '.[] | (., "\u0000") '
)
printf 'rc=%s\n$arr[0]:<%s>\n$arr[1]:<%s>\n\n' "$?" "${arr2[@]}"

{ readarray -td '' arr3 && wait "$!"; } < <( 
   { echo x; cat json; } | jq -j '.[] | (., "\u0000") '
)
printf 'rc=%s\n' "$?"

Output:

$v1=<a\nb  \n>
$v2=<c'\"d  >

JSON data: [ "a\nb  \n", "c'\"d  " ]

$arr[0]:<a
b  
>
$arr[1]:<c'"d  >

$1:<a
b  
>
$2:<c'"d  >

rc=0
$arr[0]:<a
b  
>
$arr[1]:<c'"d  >

parse error: Invalid numeric literal at line 2, column 0
rc=4

See also @Kusalananda's answer, posted a few minutes before mine. — jrw32982, Jan 21 '21 at 16:53
What's the purpose of the `echo x`? Is that to trigger a failure case, and thus test whether `rc` is set correctly? — Charles Duffy, Jan 21 '21 at 21:47
@CharlesDuffy Correct; to cause a failure in the call to `jq` — jrw32982, Jan 21 '21 at 23:58

score 2 · Answer 5 · answered Jan 07 '15 at 00:39

2

Thanks to sputnick I got to this:

arr=( $(curl -k https://localhost/api | jq -r '.[].item2') )

The JSON I have is the output from an API. All I needed to do wans remove the file argument and pipe | the output of curl to jq. Works great and saved some steps.

answered Jan 07 '15 at 00:39

JpaytonWPD

648
1
6
16

That code's actually a bit buggy. Look at what happens if you have a result element of `*` -- it'll get replaced with a list of files in your current directory. – Charles Duffy Oct 05 '16 at 02:55
1

Similarly, an `item2` value with whitespace in it would become more than one array element. – Charles Duffy Oct 05 '16 at 03:02

score 0 · Answer 6 · answered Nov 12 '18 at 18:00

0

as an easy alternative, look at jtc tool (at https://github.com/ldn-softdev/jtc), to achieve the same thing (as in jq's example):

bash $ arr=( $(jtc -w '<item2>l+0' file.json) )
bash $ printf '%s\n' "${arr[@]}"
"value2"
"value2_2"
bash $

explanation on -w option: angular brackets <...> specify search entire json, suffix l instructs to search labels rather than values, +0 instructs to find all occurrences (rather than just first one).

answered Nov 12 '18 at 18:00

Dmitry L.

41
2

Same bug as all the `arr=( $(jq ...) )` answers, insofar as the contents are being string-split and glob-expanded to populate the array -- meaning spaces (not just newlines) create new elements, and elements that look like a glob expression are replaced by files that expression matches. – Charles Duffy Feb 01 '19 at 14:14

parse one field from an JSON array into bash array

6 Answers6

CODE:

OUTPUT:

Linked