AWK to launch OS commands from items in a CSV file with a jump line

Question

I have a CSV file that looks like:

keyuat,carsim,logs-keyuat-carsim
lowuat,carsimserver,logs-lowuat-carsimserver
utils,dash,logs-utils-dash
utils,lifecycle,logs-utils-lifecycle
utils,lifecycle-nodejs,logs-utils-lifecycle-nodejs
workshop,cashier,logs-workshop-cashier
workshop,jfrog-dotnet,logs-workshop-jfrog-dotnet
workshop,labelsengine,logs-workshop-labelsengine

Based on this CSV file, I'm trying to run two commands that must go together:

oc project $1 
oc patch dc $2 -p '{"metadata":{"labels":{"logentries":"$3"}}}'

Using real examples from above, the commands would be:

oc project keyuat 
oc patch dc carsim -p '{"metadata":{"labels":{"logentries":"logs-keyuat-carsim"}}}'

I have been trying with awk, but I always find an issue with special characters or a /r invalid character that I don't see. If instead of system I use print, I see some of my characters overlapping at the beginning of the line instead of adding in the end:

 awk -F , '{ cmd="oc project " $1 "\;" "\n" "oc patch dc " $2 " \-p '\''\{\"metadata\"\:\{\"labels\"\:\{\"logentries\"\:\"" $3"\""; print(cmd) }' ./csv/labels.csv
 
"}}}oject keyuat; oc patch dc carsim -p '{"metadata":{"labels":{"logentries":"logs-keyuat-carsim
"}}}oject lowuat; oc patch dc carsimserver -p '{"metadata":{"labels":{"logentries":"logs-keyuat-carsimserver
 "}}}oject utils; oc patch dc dash -p '{"metadata":{"labels":{"logentries":"logs-utils-dash


awk -F , '{ cmd="oc project " $1 "\;" "oc patch dc " $2 " -p '\''\{\"metadata\"\:\{\"labels\"\:\{\"logentries\"\:\"" $3 "\"\}\}\}'\''"; system(cmd) }' ./csv/labels.csv
awk: cmd. line:1: warning: escape sequence `\;' treated as plain `;'
awk: cmd. line:1: warning: escape sequence `\{' treated as plain `{'
awk: cmd. line:1: warning: escape sequence `\:' treated as plain `:'
awk: cmd. line:1: warning: escape sequence `\}' treated as plain `}'
Already on project "keyuat" on server "https://test-ocp.exampleusage.eu:443".
Error from server (BadRequest): invalid character '\r' in string literal
Already on project "lowuat" on server "https://test-ocp.exampleusage.eu:443".
Error from server (BadRequest): invalid character '\r' in string literal
Now using project "utils" on server "https://test-ocp.exampleusage.eu:443".
Error from server (BadRequest): invalid character '\r' in string literal

How can I correct this script?

The cause for errors `invalid character '\r'` might be an input file with DOS (Windows) line endings. You can fix this with `dos2unix inputfile` — Bodo, Sep 14 '22 at 12:28

score 3 · Answer 1 · answered Sep 14 '22 at 12:30

I'd use

jq to generate the JSON -- this will the safest way to handle any quoting edge cases, and
bash to construct the commands -- due to bash FAQ #50

json=$(jq -n -c --arg le "$3" '{metadata: {labels: {logentries: $le}}}')

# no `$` here, I'm building arrays
cmd1=(oc project "$1")
cmd2=(oc patch dc "$2" -p "$json")

# now, execute the commands
"${cmd1[@]}"
"${cmd2[@]}"

To inspect the contents of the commands:

$ declare -p cmd1 cmd2
declare -a cmd1=([0]="oc" [1]="project" [2]="keyuat")
declare -a cmd2=([0]="oc" [1]="patch" [2]="dc" [3]="carsin" [4]="-p" [5]="{\"metadata\":{\"labels\":{\"logentries\":\"logs-keyuat-carsim\"}}}")

I like your solution as it is both clean and informative (on ways to use jq, and bash arrays) — Olivier Dulac, Sep 14 '22 at 12:35

Olivier Dulac · Accepted Answer · 2022-09-14T12:49:05.250

2

In awk, you can easily print a singlequote with: \047

You could try (I just type it here, I can't test it right now... I hope it is correct!)

# if needed: take out "^M" from the input file
tr -d '\015' <inputfile.csv >inputfile_sanitized.csv

# then parse it with:
awk -F',' '
  { cmd1="oc project " $1 
    cmd2="oc patch dc " $2 " -p \047{\"metadata\":{\"labels\":{\"logentries\":\"" $3 "\"}}}\047"
    ## once sure: you can here do: system(cmd1 " ; " cmd2); 
    print cmd1 " ; " cmd2
  }' inputfile_sanitized.csv

edited Sep 14 '22 at 12:49

answered Sep 14 '22 at 12:23

Olivier Dulac

5,924
1
23
35

It could be possible (I guess...) that a special locale have something different for singlequote? Then add : `LC_ALL=C ` before `awk` – Olivier Dulac Sep 14 '22 at 12:29
1

This worked, thanks!! – Vidanez Sep 14 '22 at 13:24

Kusalananda · Answer 3 · 2022-09-14T12:41:48.993

Assuming the input file is a Unix text file and that all the involved fields are "nice" (no embedded commas, quotes, or newlines, and nothing that needs any special JSON encoding or extra quoting in the shell), then you could do this with a simple shell loop:

while IFS=, read -r project dc logentries; do
        cat <<END_CMDS
oc project $project
oc patch dc $dc -p '{"metadata":{"labels":{"logentries":"$logentries"}}}'
END_CMDS
done <file | sh -s

This creates a stream of commands:

oc project keyuat
oc patch dc carsim -p '{"metadata":{"labels":{"logentries":"logs-keyuat-carsim"}}}'
oc project lowuat
oc patch dc carsimserver -p '{"metadata":{"labels":{"logentries":"logs-lowuat-carsimserver"}}}'

(etc.) ... which gets passed to sh -s for execution.

Again, but now using tools that actually knows how to parse CSV, how to quote strings for the shell, and how to create JSON:

mlr --icsv -N --ojson cat file | 
jq -r '
    @sh "oc project \(."1")",
    @sh "oc patch dc \(."2") \({metadata:{labels:{logentries:."3"}}}|tostring)"' |
sh -s

Here we use mlr (Miller) to convert the input from CSV to a rudimentary JSON document. The JSON document is then parsed by the JSON processor jq, which creates shell code containing quoted strings and correctly formatted JSON. The output is sent to sh -s for execution as before.

For an input line like

"The ""Project""","A study, in many parts",It'll be fine

this would generate and execute the two commands

oc project 'The "Project"'
oc patch dc 'A study, in many parts' '{"metadata":{"labels":{"logentries":"It'\''ll be fine"}}}'

AWK to launch OS commands from items in a CSV file with a jump line

3 Answers3