Delete lines from a file using awk or sed based on the variable from previous command

Question

I have a CSV file (say temp.csv) with strings and numbers. I need to delete the lines which contain a particular pattern of string at the beginning of the lines in the file. For example, here is my file -

req1,incl_patt1,excl_patt1,2,ind1
req1,incl_patt2,excl_patt2,1,ind1
req1,incl_patt3,excl_patt3,4,ind1
req2,inc_patt1,exc_patt1,1,ind2
req2,inc_patt2,exc_patt2,2,ind2
req2,inc_patt3,exc_patt3,3,ind2
req3,pattern3,expatt3,1,ind3
req4,pattern4,expatt4,1,ind4

I am writing a pattern to req_file_curr in a command ~ $ req_file_curr=req1

echo "${req_file_curr}" gives output as req1

However when using it in a sed command, like below:

sed '/\"${req_file_curr}\"/d' temp.csv

gives output as

req1,incl_patt1,excl_patt1,2,ind1
req1,incl_patt2,excl_patt2,1,ind1
req1,incl_patt3,excl_patt3,4,ind1
req2,inc_patt1,exc_patt1,1,ind2
req2,inc_patt2,exc_patt2,2,ind2
req2,inc_patt3,exc_patt3,3,ind2
req3,pattern3,expatt3,1,ind3
req4,pattern4,expatt4,1,ind4

I am expeceting the output as

req2,inc_patt1,exc_patt1,1,ind2
req2,inc_patt2,exc_patt2,2,ind2
req2,inc_patt3,exc_patt3,3,ind2
req3,pattern3,expatt3,1,ind3
req4,pattern4,expatt4,1,ind4

How can I use the variable from a shell as a pattern to look for in sed or awk?

score 6 · Accepted Answer · answered May 12 '14 at 19:29

6

You need to enclose the sed pattern in double quotes so that the shell can expand the variable before passing it to sed:

sed "/^${req_file}/d" temp.csv

You also need to use the ^ anchor to indicate to sed that this pattern needs to occur at the beginning of the line.

answered May 12 '14 at 19:29

Joseph R.

38,849
7
107
143

+1 and Thank you for explaining the use of `^` anchor. – Dhruuv May 12 '14 at 19:37

Stéphane Chazelas · Answer 2 · 2020-01-14T13:23:57.023

5

No need for awk or sed here:

grep -v "^$req_file_curr," file

Note that (like for sed or awk though to a lesser extent), that assumes that $req_file_curr does not contain regexp operators (like ., *...).

If it does, you'd have to escape those characters, or with awk, you could use an approach using its index() function:

S="$req_file_curr," awk 'index($0, ENVIRON["S"]) != 1' < file

edited Jan 14 '20 at 13:23

answered May 12 '14 at 19:53

Stéphane Chazelas

522,931
91
1,010
1,501

I tried to append `.req` to the contents of first columns to check if `sed` or `awk` works, it actually works fine with the code provided by @joseph. `req_file_curr=req1.req`. Using the sed command as provided above `sed "/^${req_file}/d" temp.csv` gives exactly what I need. Am I missing anything here? – Dhruuv May 13 '14 at 16:36
@Dhruuv, `.` matches any character, so `req1.req` would match `req1.req` but also `req12req` or `req1,req`. – Stéphane Chazelas May 13 '14 at 17:05
I do have `.` in the first column of my file. I am leaning more towards `sed` because I can use inplace edit option `sed -i` (which I learned recently) instead of streaming the output into a file and then renaming that file to the original file name. Is there a way to handle the regexp operators while using `sed`? Otherwise, I will have to go ahead with `grep` – Dhruuv May 13 '14 at 17:55
1

@Dhruuv See http://unix.stackexchange.com/a/129063/22565 (escape as per the LHS) – Stéphane Chazelas May 13 '14 at 19:14
grep also does regular expressions. awk has an index() function that compares actual strings, not using patterns. – Paul_Pedant Jan 14 '20 at 13:07
@Paul_Pedant, good point thanks. I've added an `index()`-based appraoch. – Stéphane Chazelas Jan 14 '20 at 13:24

cuonglm · Answer 3 · 2014-05-12T20:01:50.797

You have set req_file_curr but use req_file. And your sed command won't work, since shell variable can not be expanded in your sed command.

You can try this:

$ sed '/'"${req_file_curr}"'/d' file
req2,inc_patt1,exc_patt1,1,ind2
req2,inc_patt2,exc_patt2,2,ind2
req2,inc_patt3,exc_patt3,3,ind2
req3,pattern3,expatt3,1,ind3
req4,pattern4,expatt4,1,ind4

or using sed with double quote:

sed "/${req_file_curr}/d" file

and using awk:

$ awk -v req=$req_file_curr '$0 !~ req' file
req2,inc_patt1,exc_patt1,1,ind2
req2,inc_patt2,exc_patt2,2,ind2
req2,inc_patt3,exc_patt3,3,ind2
req3,pattern3,expatt3,1,ind3
req4,pattern4,expatt4,1,ind4

Thank you for pointing the mistake in my question, I edited my question. — Dhruuv, May 12 '14 at 19:32

score 0 · Answer 4 · answered Jan 14 '20 at 13:57

0

An ed solution would be something like.

  req_file_curr=req1

  printf '%s\n' "g/^$req_file_curr/d"  w | ed -s temp.csv

It edit's the file inplace so test first before running this against your files.

answered Jan 14 '20 at 13:57

Jetchisel

1,264
7
9

Delete lines from a file using awk or sed based on the variable from previous command

4 Answers4