Grep inserting file names from working directory

Question

I have script to do a few jobs:

list config XMLs.
list which of those XMLs have a match.
show what that match is.

JENKINS_HOME=/var/lib/jenkins

_ALL_JOBS=$( find $JENKINS_HOME/jobs -name config.xml )

_CRON_JOBS=$( for _JOB in $( echo "$_ALL_JOBS" ); do grep -l 'triggers.TimerTrigger' $_JOB; done )

for _JOB in $( echo "$_CRON_JOBS" ); do
    _TIME_TRIGGER=$( grep -A 1 'hudson.triggers.TimerTrigger' $_JOB | head -2 | tail -1 )
    echo ${_JOB}: $_TIME_TRIGGER
done

$_ALL_JOBS something like:

/var/lib/jenkins/jobs/Testing/jobs/job1/config.xml
/var/lib/jenkins/jobs/Testing/jobs/job2/config.xml
/var/lib/jenkins/jobs/Testing/jobs/job3/config.xml
/var/lib/jenkins/jobs/Testing/jobs/job4/config.xml
/var/lib/jenkins/jobs/Testing/jobs/job5/config.xml

$_CRON_JOBS is a selection $_ALL_JOBS and looks something like:

/var/lib/jenkins/jobs/Testing/jobs/job2/config.xml
/var/lib/jenkins/jobs/Testing/jobs/job4/config.xml
/var/lib/jenkins/jobs/Testing/jobs/job5/config.xml

When I run GREP from the command line on one of these files, I get something like this back: <spec>H 1 * * *</spec>

When I run the same command on every file in a loop, I get this back plus the contents of the directory I'm currently working in:

<spec>H */6 file1 file2 file3 file4 file5*</spec>

If I change to a different working directory and run the script, I get the same thing, but the files that appear in between the XML tags are the those of the new working directory:

<spec>H */6 file6 file7 file8 file9 file10*</spec>

Is there something about the loop or the contents of the match that's causing the contents of my current working directory to be sandwiched inside the match?

Thanks for any thoughts.

Did you try to insert `set -x` before the suspicious point in the script and run it? Then you should immediately localise in the debug output which command `file1 file2 etc.` come from. As a note, you don't need to echo variables to run a loop, `for` can iterate over a variable containing a list of words like this `for _JOB in $_ALL_JOBS`. Also, `grep` can loop over files itself so it's sufficient to just do `_CRON_JOBS=$(grep -l 'triggers.TimerTrigger' $_ALL_JOBS)` — user3071170, May 04 '22 at 04:44
Also, use an array to hold multiple values. a plain string variable holding space (or whatever) separated items will inevitably break on some filenames, no matter what you use as the separator. The ONLY character that can't be in a pathname is a NUL, and bash variables can't contain NUL either. e.g. you can populate your _ALL_JOBS list with something like: `mapfile -d '' -t _ALL_JOBS < <(find "$JENKINS_HOME/jobs" -name config.xml -print0)`. BTW, double-quote your variables when you use them. Unquoted vars are a bug waiting to happen. — cas, May 04 '22 at 10:30
Finally, don't use `grep` to extract data from structured text like XML. Use an XML parser, like [xmlstarlet](http://xmlstar.sourceforge.net/), or at least pre-process the XML into a line-oriented format suitable for use with line-oriented tools like grep/sed/cut/awk/etc with a tool like [xml2](https://github.com/cryptorick/xml2). Both of these are packaged for most linux distros and are easily installed. — cas, May 04 '22 at 10:34
Related: [When is double-quoting necessary?](//unix.stackexchange.com/q/68694) and [Why is looping over find's output bad practice?](//unix.stackexchange.com/q/321697) and [Why is printf better than echo?](//unix.stackexchange.com/q/65803) — Stéphane Chazelas, May 05 '22 at 10:51

thanasisp · Answer 1 · 2022-05-05T10:34:15.190

Try this example:

$ mkdir -p test_dir && cd test_dir/
$ touch file1 file2
$ time_trigger='<spec>H 1 * * *</spec>'

$ echo "$time_trigger"
<spec>H 1 * * *</spec>

$ echo $time_trigger
<spec>H 1 file1 file2 file1 file2 *</spec>

Your output is the result of the unquoted variable, some stars expanded to the files of your current directory.

Even after you fix this, the remaining code could fail in some cases, for example for spaces into filenames and you should not use echo to print variables and assign this output to a variable again, it's vulnerable.

A possible version of your script could be:

find "$JENKINS_HOME"/jobs -name 'config.xml' -exec sh -c '
    for f do
        <apply your logic for every config file "$f">
    done' sh {} +

See this post for reference.

Grep inserting file names from working directory

1 Answers1