Replace string with output from command, based on value of that string

Question

I'm writing a script in sh to:

Read a config file
For each value between {{ value_0000 }} retrieve 0000
- Perform a command with value 0000 passed as argument
- Replace {{ value_0000 }} with the output of that command

Notes:

value_0000 always converts to numbers, e.g. 0000
The command is always the same, e.g. /bin/command <value>
I do not want to edit the current config file, it could just be run in memory

Example:

# this is a config file
key: {{ value_12345 }}:hello
something
another_key: return:{{ value_56789 }}/yeah

./run.sh

# this is a config file
key: returned-value:hello
something
another_key: return:another-returned-value2/yeah

I can retrieve the value from the config file, but requires a lot more code to make it work as desired.

#!/bin/sh
cat some.conf | while read line
do
   echo $line
   val="$(grep -o -P '(?<={{ value_).*(?= }})')"
   command $val
done

They're different keys, in the same file. Not really related. — Kevin C, Feb 19 '20 at 15:43
Not related to the question, but note that your shebang is wrong, it should be `#!/bin/sh` (but I guess that's a typo) and you don't need to cat the file, you can simply do `while read line; do ...; done < some.conf` — terdon, Feb 19 '20 at 15:43
" .. a command". What command? Is the command defined in the config file, or always the same command? What to do if the command fails, or returns nothing? — Paul_Pedant, Feb 19 '20 at 15:45
Are the values always numbers? Can you have `{{ value_foo }}`? And what exactly are you expecting as output? Is the objecting just to run the command on each value or do you then want to print out the new key with the new value? And how? To a new file? Do you want to edit the existing file? — terdon, Feb 19 '20 at 15:46
Easy for awk. It can use FS=":", match the value pattern, run an external command with cmd | getline, and substitute the result into the text. It can deal with multiple value patterns in one line too. Just need to know the command to run. You could pass that as an option too. — Paul_Pedant, Feb 19 '20 at 15:49

Stéphane Chazelas · Accepted Answer · 2020-02-20T13:32:00.050

2

With perl:

perl -pe 's(\{\{ value_(\d+) \}\})(`cmd $1`)ge' < some.conf

Note that it's safe here as $1 is guaranteed to be non-empty and not to contain characters that are special in the shell syntax (only digits) and perl even bypasses the shell here. But if you change the \d+ (sequence of one or more digits) with anything else, that would become dangerous (like for values such as foo;reboot) and you may want to take a safer approach like:

perl -pe '
  s(\{\{ value_(.*?) \}\})(
    open my $fh, "-|", "cmd", $1 or die "cannot start cmd: $!";
    local $/ = undef;
    <$fh>;
  )ge' < some.conf

If cmd accepts options, you'll probably want to replace "cmd" with "cmd", "--" or qw(cmd --) (assuming it supports -- as an end-of-option marker) to avoid problems with values that start with - (and possibly +).

In any case, while read shell loops are best avoided to process text.

edited Feb 20 '20 at 13:32

answered Feb 19 '20 at 15:47

Stéphane Chazelas

522,931
91
1,010
1,501

This... damn, what a sick answer. It works as expected. Thanks! – Kevin C Feb 19 '20 at 16:05
The safer approach doesn't seem to work, the output is exactly the same as the input. Do you know how I can add some error handling to the first perl command, or how do I make the safer approach work? – Kevin C Feb 20 '20 at 12:51
@KevinC, it works for me on your sample input and with `"cmd"` replaced with `"printf", "<%s>"`. What kind of error do you want to account for and what should be done in those cases? – Stéphane Chazelas Feb 20 '20 at 13:33
The error returned is "Error: Could not find specified account(s)." I'd like to provide a message with: "My custom message" and then stop the script. – Kevin C Feb 20 '20 at 16:35

score 0 · Answer 2 · answered Feb 19 '20 at 19:15

With GNU awk for the 3rd arg to match():

$ cat tst.awk
match($0,/(.*{{ value_)([0-9]+)( }}.*)/,a) {
    cmd = "echo \"<foo_" a[2] "_bar>\""
    if ( (cmd | getline output) > 0 ) {
        $0 = a[1] output a[3]
    }
    close(cmd)
}
{ print }

$ awk -f tst.awk some.conf
# this is a config file
key: {{ value_<foo_12345_bar> }}:hello
something
another_key: return:{{ value_<foo_56789_bar> }}/yeah

and with any awk:

$ cat tst.awk
match($0,/{{ value_[0-9]+ }}/) {
    cmd = "echo \"<foo_" substr($0,RSTART+9,RLENGTH-12) "_bar>\""
    if ( (cmd | getline output) > 0 ) {
        $0 = substr($0,1,RSTART+8) output substr($0,RSTART+RLENGTH-3)
    }
    close(cmd)
}
{ print }

$ awk -f tst.awk some.conf
# this is a config file
key: {{ value_<foo_12345_bar> }}:hello
something
another_key: return:{{ value_<foo_56789_bar> }}/yeah

Replace string with output from command, based on value of that string

2 Answers2