Getting jq error when using variable in command line

Question

I am using jq to get the tag_name index with the script:

curl \
 https://api.github.com/repos/checkstyle/checkstyle/releases \
 -H "Authorization: token $GITHUB_TOKEN" \
 -o /var/tmp/cs-releases.json

TARGET_RELEASE_NUM=$1
TARGET_RELEASE_INDEX=$(cat /var/tmp/cs-releases.json | \
    jq -r "[.[].tag_name] | to_entries | .[] | \
    select(.value==\"checkstyle-${TARGET_RELEASE_NUM}\") | .key")
echo TARGET_RELEASE_INDEX="$TARGET_RELEASE_INDEX"

I get the expected result only when I use the $GITHUB_TOKEN as a hardcoded value (directly in the script) but when I use $GITHUB_TOKEN as a command line variable, I get this output:

jq: error (at <stdin>:4): Cannot index string with string "tag_name"

This error is getting when using:

rahul@rk7:~/Desktop/opensource/checkstyle$ GITHUB_TOKEN=ghp_xxxx && ./.ci/update-github-page.sh 10.1

and Working fine with:

curl \
 https://api.github.com/repos/checkstyle/checkstyle/releases \
 -H "Authorization: token ghp_xxxx" \
 -o /var/tmp/cs-releases.json

TARGET_RELEASE_NUM=$1
TARGET_RELEASE_INDEX=$(cat /var/tmp/cs-releases.json | \
    jq -r "[.[].tag_name] | to_entries | .[] | \
    select(.value==\"checkstyle-${TARGET_RELEASE_NUM}\") | .key")
echo TARGET_RELEASE_INDEX="$TARGET_RELEASE_INDEX"

Please learn to debug. You’re generating `/var/tmp/cs-releases.json` two slightly different ways, but then reading it with *the exact same command,* and you’re getting an input error when reading the file in one of the scenarios. So obviously the file is different in the two scenarios. So the obvious next step is *to **look at the file*** in the two scenarios. That would have made it somewhat obvious that `curl` was getting invoked with different parameters, and you could have focused your question on that. … (Cont’d) — G-Man Says 'Reinstate Monica', Sep 05 '22 at 03:33
(Cont’d) … Or, better yet, you could have changed the script to say simply `echo "Authorization: token $GITHUB_TOKEN"; exit`, and carried on debugging from there. — G-Man Says 'Reinstate Monica', Sep 05 '22 at 03:33

Kusalananda · Accepted Answer · 2022-08-21T12:21:41.123

Your shell variable GITHUB_TOKEN must be set in the environment of your script. You are currently setting it in the invoking shell, and then you are running your script:

GITHUB_TOKEN=sometoken && ./yourscript

This sets GITHUB_TOKEN and then runs your script if that assignment was successful (i.e. if GITHUB_TOKEN isn't a read-only variable). However, that variable is not seen inside the script. For that to happen, you either have to turn the variable into an environment variable using export GITHUB_TOKEN or start your script like so:

GITHUB_TOKEN=sometoken ./yourscript

... which sets the value of GITHUB_TOKEN in your script but not in the calling environment.

Since you did not have the correct value for GITHUB_TOKEN in your script, you were given the following JSON document in response from Github:

{
  "message": "Bad credentials",
  "documentation_url": "https://docs.github.com/rest"
}

Your jq expression then tried to access the tag_name key from the string Bad credentials, which obviously does not work.

Note that since the Github repository is public, no authentication is actually necessary to call the given REST API endpoint (if I understand things correctly).

Another comment about your jq expression: The jq utility has a way of allowing you to import data into its expressions that do not rely on injecting shell strings. This would be safer for you to use as it ensures that the value is encoded appropriately and not accidentally evaluated as an expression:

jq -r --arg tagname "checkstyle-$TARGET_RELEASE_NUM" '
    [.[].tag_name] | to_entries[] |
    select( .value == $tagname ).key' /var/tmp/cs-releases.json

Note that $tagname is an internal jq variable, not a shell variable.

Shorter variant (it's almost always unnecessary to extract things from an array into an array as in [.[].tag_name]):

jq -r --arg tagname "checkstyle-$TARGET_RELEASE_NUM" '
    to_entries[] | select(.value.tag_name == $tagname).key' /var/tmp/cs-releases.json

Thanks a lot, @Kusalananda for your time and answer but I still face an error. This short variant still gives me an error. ```jq: error (at /var/tmp/cs-releases.json:4): Cannot index string with string "tag_name"``` I have used this command: ```TARGET_RELEASE_INDEX=$(jq -r --arg tagname "checkstyle-$TARGET_RELEASE_NUM" 'to_entries[] | select(.value.tag_name == $tagname).key' /var/tmp/cs-releases.json)``` — Rahul Khinchi, Aug 21 '22 at 12:15
@RahulKhinchi Did you correctly pass the `GITHUB_TOKEN` variable into the environment of your script as I explained at the start of the answer, either by using `export` on the variable, or by calling your script like `GITHUB_TOKEN=tokenvalue ./yourscript` (without `&&` in-between)? — Kusalananda, Aug 21 '22 at 12:17
Sorry, I did some mistake in the script naming. Perfectly works now. Thanks a lot! — Rahul Khinchi, Aug 21 '22 at 12:19
@RahulKhinchi When I tested this, I tested without giving a token at all (removed the `-H` option and its argument from the `curl` call), and it seemed to work any way. The Github repository is obviously public, so authenticating is not necessary. — Kusalananda, Aug 21 '22 at 12:19
@RahulKhinchi When troubleshooting shell script problems, one of the first things you should do is put `set -x` before the problem section to turn on execution tracing -- in this case, it would've shown that `curl` was being passed `-H 'Authorization: token '`, which would immediately tell you there's a problem with `$GITHUB_TOKEN`. Also, checking /var/tmp/cs-releases.json to see what it contained would've been helpful. — Gordon Davisson, Aug 22 '22 at 04:06
Sure, thanks for the feedback. I will always troubleshoot first! — Rahul Khinchi, Aug 22 '22 at 15:09

Getting jq error when using variable in command line

1 Answers1