Shell for loop through file and for each line use sed to delete lines from other file

Question

I'm trying to clean up long-cluttered-filename.txt by deleting the lines that match the strings in my delete-these-lines.txt...

$ cat delete-these-lines.txt
context
platform
server
civicrm
site_enabled
client
redirection
cron_key
^\s*[0-9]*\s
alias
profile
install
ssl_
address

...and with a simple for loop I can iterate through the lines of that file like so:

$ for l in delete-these-lines.txt; do cat $l; done
context
platform
server
civicrm
site_enabled
client
redirection
cron_key
^\s*[0-9]*\s
alias
profile
install
ssl_
address

but when I try to substitute cat for sed it doesn't work. Neither of these work:

for l in delete-these-lines.txt; do sed -i "/$l/d" long-cluttered-filename.txt; done
for l in delete-these-lines.txt; do sed -i '/"$l"/d' long-cluttered-filename.txt; done

Is this a problem with the $l variable being interpolated? Is this a limitation of for loops?

I see answers like loop sed to delete line with strings provided in a list of strings from a text file and Read line from file then delete that use while loops with read but I'd prefer to use a for loop if it's possible, and if it's not possible I'd like to know why.

`for l in delete-these-lines.txt ... ` loops over the file*name*, once. You presumably want `for l in $( — steeldriver, Feb 13 '20 at 18:10
OK perhaps I shouldn't have said "bad idea" - it's just hard to see why you are so determined to use a `for` loop in this context. — steeldriver, Feb 13 '20 at 18:44
Does `delete-these-lines.txt` contain _patterns_ or _strings_? That is, should `^\s*[0-9]*\s` match that string literally, or is that a PCRE regular expression? — Kusalananda, Feb 13 '20 at 23:00
@Kusalananda `^\s*[0-9]*\s` is a regex pattern (I'm not sure about PCRE though), not a literal string. — alec, Feb 14 '20 at 00:26
@steeldriver when you say "bad idea" are you thinking something along the lines of [using a shell loop to process text considered bad practice](https://unix.stackexchange.com/questions/169716)? I'm not as determined to "use" a `for` loop as I am to figure out if and how it's possible; the `grep -v -f....` option looks great in how it functions, but I'm trying learn about how these loops work... and your comment on the distinction between looping over the *filename* vs the *file contents* was helpful :) — alec, Feb 14 '20 at 00:41
I was thinking more specifically about the issue of *word splitting*, as described here [How to loop over the lines of a file?](https://unix.stackexchange.com/questions/7011/how-to-loop-over-the-lines-of-a-file) and explored more generically in [Why does my shell script choke on whitespace or other special characters?](https://unix.stackexchange.com/questions/131766/why-does-my-shell-script-choke-on-whitespace-or-other-special-characters). If you can guarantee that your lines contain no whitespace, you may not care. — steeldriver, Feb 14 '20 at 00:47

score 1 · Answer 1 · answered Feb 13 '20 at 19:14

If you have two files where one contains list of strings to delete from another one, you can do the entire thing with one simple grep:

grep -v -f delete-these-lines.txt long-cluttered-filename.txt

This will output all the strings from long-cluttered-filename.txt which do not match to patterns from delete-these-lines.txt.

Note, that in case some string in long-cluttered-filename.txt will contain client (e.g. 123superclient456) it still matches to client and will be excluded from the output.

score 1 · Answer 2 · answered Feb 15 '20 at 00:04

Never use the letter l as a variable name as it looks far too much like the number 1 and so obfuscates your code. What you want to do is a bad idea as you already know, but see https://mywiki.wooledge.org/DontReadLinesWithFor for how to read lines from a file in shell and then you'd write it as:

while IFS= read -r regexp; do
    sed -i "/$regexp/d" long-cluttered-filename.txt
done < delete-these-lines.txt

but again - don't do it for all the reasons you've already been warned of (plus you're just re-implementing grep -v -f delete-these-lines.txt long-cluttered-filename.txt > tmp && mv tmp long-cluttered-filename.txt less efficiently, less robustly, and less portably).

Shell *for* loop through file and for each line use sed to delete lines from other file

2 Answers2

Shell for loop through file and for each line use sed to delete lines from other file