grep command in bash shell loop

Question

I have a log file, name out.log. I need to search a list of string (filelist.txt) from this log file.

I need each matching line from "out.log" for every file in filelist.txt. it can appear multiple times.

cat out.log | grep -i I466030.CXR910E >> search_result.txt

this works, it will search for I466030.CXR910E and output to search_result.txt

for i in `cat filelist.txt`; do  grep -i "$i" out.log >> result.txt; done


 for i in `cat filelist.txt`; 
   do  
     echo $i
     cat out.log | grep -i $i >> result2.txt
     grep $i out.log >> result3.txt 
  done

Above code the echo will output the search string but none of the grep not writing output to any result*.txt

What am I missing here ? Thanks for your advice.

In response to comments:

xxxxxm@cxxxrtrls01[DEV][rerun] $ grep -i -f filelist.txt out.log > result.txt
xxxxxm@cxxxrtrls01[DEV][rerun] $ ll
total 564
-rw-r----- 1 cnrlsadm cnrlsadm   4600 Aug 19 09:36 filelist.txt
-rw-r----- 1 cnrlsadm cnrlsadm    216 Aug 19 10:14 file.sh
-rw-r--r-- 1 cnrlsadm cnrlsadm 557403 Aug 19 10:10 out.log
-rw-r--r-- 1 cnrlsadm cnrlsadm      0 Aug 19 12:14 result.txt
-rw-r----- 1 cnrlsadm cnrlsadm    213 Aug 19 12:10 srch.sh
xxxxxm@cxxxrtrls01[DEV][rerun] $

size of result.txt is zero , it doesnt have anything.

this question comes up a lot, but i can't find an exact dupe at the moment. grep can do this (with the `-f` option) without needing a shell for loop. try `grep -i -f filelist.txt out.log > result.txt` — cas, Aug 19 '19 at 04:03
Is filelist.txt in DOS/Windows text format by any chance? If that's not the problem, try putting `set -x` before the loop so it'll print out what the shell thinks is happening and you can see what's different from what you expect. — Gordon Davisson, Aug 19 '19 at 04:35
without at least sample of the contents of both `filelist.txt` and `out.log` it's impossible to tell why neither the for loop nor the grep are working. @GordonDavisson's guess about dos/windows text files is a pretty good one and may be the source of the problem. — cas, Aug 19 '19 at 04:41
This is most likely a due to `filelist.txt` being a DOS text file. The patterns (filenames) in that file would then have an extra carriage-return character at the end, which would never match in the `out.log` file. OR _there are simply no matches in `out.log`_ (you haven't shown an example of a log entry that ought to be matched). — Kusalananda, Aug 19 '19 at 05:56
See https://stackoverflow.com/q/45772525/1745001, https://unix.stackexchange.com/q/169716/133219, http://porkmail.org/era/unix/award.html, http://porkmail.org/era/unix/award.html, http://mywiki.wooledge.org/BashFAQ/082, and http://mywiki.wooledge.org/BashFAQ/001 for descriptions of **most** of the issues in your script. — Ed Morton, Aug 19 '19 at 06:25

score 3 · Accepted Answer · answered Aug 19 '19 at 14:11

Don't read the lines of a file with for
use grep's -f option to supply a file containing patterns so you don't need to loop at all:
```
grep -i -f filelist.txt out.log > result.txt
```
You now have a single invocation of grep, instead of one per line. If out.log is large, your CPU will thank you.

score 0 · Answer 2 · answered Aug 20 '19 at 09:00

0

Well.. most of the reply here are correct, its my bad the "filelist.txt" was generated from excel and in windows. Once I convert the file into unix format. everything worked.

I learned a very time-consuming lesson. Thanks a million to every one answered the question :)

Have a great day ahead !

answered Aug 20 '19 at 09:00

Dwija

13
1
3

1

If that was your issue, then why did you accept an answer that had nothing to do with DOS text file formats? – Kusalananda Aug 20 '19 at 09:23
does that means I should not accept any answer ? My real answer were in comments and I cant make a comment as accepted answer. – Dwija Aug 20 '19 at 10:07
You may accept your own answer. – Kusalananda Aug 20 '19 at 10:09
Thanks for your reply. – Dwija Aug 21 '19 at 12:20

score -1 · Answer 3 · edited Aug 19 '19 at 08:30

-1

Tried with below script and it worked fine

for i in `cat filelist.txt`
do
for j in `cat out.log`
do
grep -i "$j" $i
done
done

filelist.txt.====> Consists of filenames

edited Aug 19 '19 at 08:30

Kevdog777

3,194
18
43
64

answered Aug 19 '19 at 08:06

Praveen Kumar BS

5,139
2
9
14

grep command in bash shell loop

3 Answers3

Linked