Multi-line String Pattern Matching, Insertion and Deletion with sed or awk

Question

I'm trying to write a script to automatically update my .ssh/config. Basically it's an ssh management tool that will add an .ssh/config entry for an input computer, user, etc.

Question:

Are there any tools available (like sed, or awk) for automatically inserting around or removing all instances of multi-line strings?

For example with the sed command:

# inserts "stuff" in the line above "single-line-pattern" in file
sed -i.bkp "/single-line-pattern/i stuff" file

# inserts "stuff" in the line below "single-line-pattern" in file
sed -i.bkp "/single-line-pattern/a stuff" file

# removes all instances of "single-line-pattern" from file
sed -i.bkp "/single-line-pattern/d" file

Possible Work Around

A hacky workaround for deleting all pattern instances is to invert the problem and find all non-instances, print those to a new file, and then delete the old file. Ignoring appending above an entry (because I deleted them all) I just append to the end of the file in a fixed order.

That being said this does not answer the above question, or give examples of such tools or sed/awk functions that work with multi-line strings.

My Solution

The hacky script that works for me is given below. You'll note that I use awk and it handles the multi-line problem well. It's just that as far as I know, awk does not provide many tools for manipulating or working around this phrase like the sed examples given above.

# create backup of file
cp $FILE $FILE.bak

# delete previous .tmp file
if [ -f $FILE.tmp ]; then
    rm $FILE.tmp
fi

# if the expression is not in the paragraph, append it to $FILE.tmp
awk -v RS="" -v expr1="Host $COMPUTER" -v expr2="Match originalhost \
$COMPUTER" -v user="User $USER" \
'{ if (! (($0 ~ expr1 || $0 ~ expr2) && $0 ~ user)) print$0 "\n"}' \
$FILE >> $FILE.tmp

mv $FILE.tmp $FILE

My Problem Space

A matching set of paragraphs in $FILE for $COMPUTER="compy" and $USER="harpo" is given below. One could imagine entries for other computers or even for other users on the same computer (hence checking for both COMPUTER and USER matches).

Match originalhost compy exec /usr/local/bin/superscript.sh
   HostName 192.168.0.101
   User harpo
   Port 22
   IdentityFile /home/harpo/.ssh/id_rsa

Host compy
   HostName 123.123.123.123
   User harpo
   Port 33
   IdentityFile /home/harpo/.ssh/id_rsa

What is your end goal? if you are moving stanzas around so as to alter the "first match" for a particular user, you can avoid that altogether that by applying different parameters per-user as described here: [Specify Specific Identity file when ssh'ing as certain user in ~/.ssh/config](https://unix.stackexchange.com/questions/427587/specify-specific-identity-file-when-sshing-as-certain-user-in-ssh-config) — steeldriver, Mar 04 '18 at 17:32
@don_crissti: The output is contingent on the input, and I actually run out of comment characters if I spell out an example case here. So generally the output should be a paragraph, like given in the example `.ssh/config` except that any entry with matching COMPUTER and USER would be deleted, and the paragraph would be appended to the end of the file. Consider a partially completed paragraph like one without an IdentityFile. I want to delete an existing entry to handle that, or any other permutation of error. I would say that it might just be better to focus on those enumerated questions? — RACKGNOME, Mar 05 '18 at 01:17
@steeldriver: While I want to check for an existing, matching entry to handle the case where this COMPUTER/USER had already been added, primarily it would be for adding a new entry to the `.ssh/config` file. Think of sshing to many machines and wanting an easy way to add new machines to your list of aliases. The `Match originalhost` bit runs a script to check if I'm behind a local network and thus use a local IP. Hope this helps! — RACKGNOME, Mar 05 '18 at 01:19
I recommend writing the whole script in `perl` or `awk`, and not using `sh` code at all - both have good support for reading files a paragraph at a time, and doing regex matches and/or substitutions on each paragraph. BTW, you can split a paragraph into lines by splitting on `\n` newline characters if you need to. Both `awk` and `perl` have `split()` functions that can be used for this. — cas, Mar 05 '18 at 10:24

user1404316 · Answer 1 · 2018-03-05T02:56:49.877

1

How about placing a comment symbol between each paragraph, and ensuring that no other comment symbols exist in the file. Then you can use awk and set RS="#" to have awk read an entire paragraph as a single record, you can do your processing and checking, and then you can output (or not) what you want, how you want, either then or later in the file, or at the end of the file (using awk's END address specifier).

UPDATE: The awk documentation tells me that for this answer to work, there is no reason to require that comment symbols do not exist elsewhere in the file, because the value of RS can be a regex!

edited Mar 05 '18 at 02:56

answered Mar 05 '18 at 02:29

user1404316

3,028
12
23

So it's probably just me but I don't see how this will allow for deleting the paragraph. I guess also I already am using an awk command to search the existing file for a paragraph containing both "Host $COMPUTER" (or "Match originalhost $COMPUTER..." and "User $USER". I'll edit and give my GET_CONFIG_PARAGRAPH function. I guess also for stackoverflow's benefit it would be really useful to know how to 1.) find and delete a multi-line string stored in a variable and/or 2.) use `sed` or `awk` to find a multi-line string – RACKGNOME Mar 06 '18 at 02:20
@RACKGNOME - yep, it is just you (joke). To delete a "paragraph" - well, once a "paragraph" is just a record, you could just **not** print it upon a test of your decision or since it now **is** a variable, ie. `$0`, you could `$0=""` for a blank line, or do whatever else. Once you have changed the `RS` value, you can include newline characters in your regex searches, or remove them and have your searches traverse strings. – user1404316 Mar 06 '18 at 04:04
@RACKGNOME - it seems you didn't grok from the answer that once you modify the RS, awk automatically assigns all elements to its positional parameters `$1`... and the entire paragraph as `$0`, so you now can do anything multi-line you want within that paragraph using all of awk's powerful features. – user1404316 Mar 06 '18 at 04:08
is there anyway in awk to insert things around matching phrases like in sed? Also if you could kind of give more concrete examples to exemplify your description that be awesome (and upvotable!). I've also reworked the question and given my solution that uses awk as you suggest. Cheers – RACKGNOME Mar 22 '18 at 20:37
@RACKGNOME : The thing to do isn't to wish me cheers, but to mark the check mark next to the answer, as accepted. Also, question URL pages on this site are supposed to be static one-off events, for the benefit of others who use search engines; they aren't supposed to be running continuous support requests or collaboration forum, so your "reworking" undermines a major purpose of the site, which is to allow people (like me) to get answers by googling, without needing to ask. – user1404316 Mar 22 '18 at 21:59
The question was edited for clarification and generality. You have not given a sufficient solution for it to be considered. Thanks. – RACKGNOME Mar 26 '18 at 19:30

Multi-line String Pattern Matching, Insertion and Deletion with sed or awk

1 Answers1