Prefixing and appending blocks of text to a list of files

Question

I'm trying to prefix and append blocks of text to a list of files. Here is where I am so far. The sticking point is the sed -i "1i \$prefix" "$file" && line. sed won't substitute the value of prefix in. I tried to follow various threads about this on unix.sx and a couple of other places, but just got a headache.

Can someone tell me how to fix this? If it is too complicated I'm Ok with using something else instead of sed.

This is closely related to How do I append text to the beginning and end of multiple text files in Bash?, which is where I got the code below from. The difference is that the question does not cover expanding a variable, and also in my case the strings in question are multiline.

filelist=(foo.tex bar.tex)

prefix='\documentclass[12pt]{article}                                                                                                                                       
\usepackage{myarticle}                                                                                                                                                      
%\xpretocmd{\opening}{\insertname}{}{}                                                                                                                                      
\begin{document}                                                                                                                                                            
%\insertname                                                                                                                                                                
\begin{verbatim}'

suffix='\end{verbatim}                                                                                                                                                      
\end{document}'                                                                                                                                                                 

for file in "${filelist[@]}"; do
  sed -i "1i \$prefix" "$file" &&
  echo "$suffix" >> "$file"
done

See this: http://unix.stackexchange.com/a/45208/4873 and delete that backslash in front of `$prefix`. What is it for? — angus, Jan 24 '14 at 21:32
@angus I'm guessing you have in mind something like `for file in "${filelist[@]}"; do sed -i "1i $(echo -E "$prefix" | sed -e 's/\\/\\\\/g')" "$file" && echo "$suffix" >> "$file" done`, but this gives an error. — Faheem Mitha, Jan 24 '14 at 22:11
OK, I see why you had a backslash. It was part of the sed command; I missed it. You need to double it and add a new line after it: `1i\\ ` _NEWLINE_ `$(echo ...`. See the linked answer and write your `sed` line exactly as it is there, only add `-i` and change `5a` by `1i` and `$text` by `$prefix`. That's all. — angus, Jan 25 '14 at 01:04

score 2 · Answer 1 · edited Apr 13 '17 at 12:36

2

How about (assuming your shell has process substitution):

for file in "${filelist[@]}"; do
    cat <(printf "%s" "$prefix") "$file" <(printf "%s" "$suffix") > "$file"_$$\ 
     && mv "$file"_$$ "$file"
done

or, better, in Perl (untested):

perl -MTie::File -e '
    @ARGV//=<STDIN>;
    chomp(@ARGV);
    for(@ARGV){
        tie @lines,"Tie::File",$_;
        @lines=($ENV{prefix} @lines $ENV{suffix})
    }' "${filelist[@]}"

For an alternative approach, if the prefix and suffix are fixed, see the following question (which also happens to be about (La)TeX :) ):

extracting and copying parts of tex files into new tex files

edited Apr 13 '17 at 12:36

Community

1

answered Jan 24 '14 at 22:09

Joseph R.

38,849
7
107
143

Hi Joseph. Thanks for the suggestions. The last one does indeed seem like a nice simple way to approach the problem. – Faheem Mitha Jan 24 '14 at 22:25
@FaheemMitha Great. Let us know how things work out... – Joseph R. Jan 24 '14 at 22:30
I found you link to "extractig and copying parts..." more useful. – Jan 24 '14 at 22:43

Stéphane Chazelas · Answer 2 · 2014-01-25T13:33:09.010

2

With zsh:

zmodload zsh/mapfile
for i ($filelist) mapfile[$i]="$prefix
$mapfile[$i]$suffix
"

With ksh93 or bash (or zsh):

for file in "${filelist[@]}"; do
  {
    rm -- "$file" && {
      printf '%s\n' "$prefix"
      cat
      printf '%s\n' "$suffix"
    } > "$file"
  } < "$file"
done

edited Jan 25 '14 at 13:33

answered Jan 24 '14 at 22:50

Stéphane Chazelas

522,931
91
1,010
1,501

score 1 · Answer 3 · edited Apr 13 '17 at 12:36

1

The case is similar to this: Appending a string containing escape character with sed

I suggest the same answer (adapted):

for file in "${filelist[@]}"; do
  sed -i "1i\\
$(echo -E "$prefix" | sed -e 's/\\/\\\\/g' | sed -e '$! s/$/\\/')" "$file" &&
  echo "$suffix" >> "$file"
done

The sed command i requires each line except the last to be terminated with a backslash (this was missing from the referred answer). Also, backslashes need to be doubled, so sed won't try to interpret them. The echo outputs the contents of the variable $prefix. Next in the pipeline, the sed command doubles the backslashes. The second sed command adds backslashes at the end of every line except the last.

EDIT How the sed program $! s/$/\\/ works:

First, the address $ means to execute the command that follows on the last line. $! negates that, so the command will be executed on all lines except the last.

Then, the s/// command does substitution. It will substitute the empty string at the end of the line, denoted by $, with a backslash. But the backslash is a special character, so we need to escape it with another backslash for sed to read it correctly.

edited Apr 13 '17 at 12:36

Community

1

answered Jan 25 '14 at 01:25

angus

12,131
3
44
40

Hi Angus, thanks for the helpful reply. Unfortunately, the backslashes at the ends of lines appears in the final output. Is it possible to suppress this? Also, could you reference some sed documentation that (a) `i` requires each line except the last the be terminated with a backslash and (b) that backslashes need to be doubled? Finally if you go into a little more detail on how `sed -e '$! s/$/\\/'` works, that would be helpful. – Faheem Mitha Jan 25 '14 at 12:26
(a) -> http://www.gnu.org/software/sed/manual/html_node/Other-Commands.html#Other-Commands, (b) -> on the same page, see documentation for `a\ `. (c) see edited answer. – angus Jan 25 '14 at 19:10
Hi Angus. Thanks for the references. These could reasonably be in the question, I think. However, I don't see any change in the code (unless I'm missing something). How can one get rid of the extra backslashes? – Faheem Mitha Jan 25 '14 at 19:34
I can't reproduce the problem. There aren't extra backslashes in the output when I run the script exactly as written. Maybe you aren't using GNU sed? – angus Jan 25 '14 at 19:49
`sed --version GNU sed version 4.2.1`. What version are you using? In any case, odd. I'll check again in case I made a mistake. – Faheem Mitha Jan 25 '14 at 19:53
4.2.2. It shouldn't be any difference. – angus Jan 25 '14 at 20:03

score 1 · Answer 4 · answered Jan 25 '14 at 16:22

Here are a few choices, none of which use sed. I find that in general, trying to get sed to play nice with external variables is rarely worth the effort.

Shell

for file in "${filelist[@]}"; do 
  tmp=$(mktemp); 
  printf "%s\n%s\n%s\n" "$prefix" $(cat "$file") "$suffix" > "$tmp" && 
  mv "$tmp" "$file"; 
done

Perl/Shell

for file in "${filelist[@]}"; do 
  tmp=$(mktemp); 
  perl -lpe 'BEGIN{print $ENV{prefix}}END{print $ENV{suffix}}' $file 
  mv "$tmp" "$file"; 
done

Pure Perl

This one has the advantage of not relying on external modules:

perl -le 'foreach (@ARGV){
           open($f,"+<","$_"); @a=<$f>; 
           print $f "$ENV{suffix}\n@a$ENV{prefix}"
          }' "${filelist[@]}"

NOTE:

The variables need to be exported in order for the Perl solutions to work, otherwise, they won't be available in the %ENV hash.

Prefixing and appending blocks of text to a list of files

4 Answers4

Shell

Perl/Shell

Pure Perl

NOTE:

Linked