I have a bunch of files with the same extension (let's say .txt) and I would like to concatenate them. I am using cat *.txt > concat.txt, but I would like to add a new line between each file so to distinguish them in concat.txt.
Is it possible to do it with a single bash command rather than an implementation such as this?
Thank you
Not a single command, but a simple one-liner:
for f in *.txt; do cat -- "$f"; printf "\n"; done > newfile.txt
That will give this error:
cat: newfile.txt: input file is output file
But you can ignore it, at least on GNU/Linux systems. Stéphane Chazelas pointed out in the comments that apparently, on other systems this could result in an infinite loop instead, so to avoid it, try:
for f in *.txt; do
[[ "$f" = newfile.txt ]] || { cat -- "$f"; printf "\n"; }
done > newfile.txt
Or just don't add a .txt extension to the output file (it isn't needed and doesn't make any difference at all, anyway) so that it won't be included in the loop:
for f in *.txt; do cat -- "$f"; printf "\n"; done > newfile
Using GNU sed:
sed -s -e $'$a\\\n' ./*.txt >concat.out
This concatenates all data to concat.out while at the same time appending an empty line to the end of each file processed.
The -s option to GNU sed makes the $ address match the last line in each file instead of, as usual, the last line of all data. The a command appends one or several lines at the given location, and the data added is a newline. The newline is encoded as $'\n', i.e. as a "C-string", which means we're using a shell that understands these (like bash or zsh). This would otherwise have to be added as a literal newline:
sed -s -e '$a\
' ./*.txt >concat.out
Actually, '$a\\' and '$a\ ' seems to work too, but I'm not entirely sure why.
This also work, if one thinks the a command is too bothersome to get right:
sed -s -e '${p;g;}' ./*.txt >concat.out
Any of these variation would insert an empty line at the end of the output of the last file too. If this final newline is not wanted, deletede it by passing the overall result through sed '$d' before redirecting to your output file:
sed -s -e '${p;g;}' ./*.txt | sed -e '$d' >concat.out
zsh has a P glob qualifier to prefix each filename resulting from a glob with an arbitrary argument.
While it's typically used for things like cmd *.txt(P[-i]) to prefix each filename with a given option, you could use here to insert any given file before each file. A temporary file containing an empty line could be done with =(print), so you could do:
() { cat file*.txt(P[$1]); } =(print)
On Linux or Cygwin, you could also do:
cat file*.txt(P[/dev/stdin]) <<< ''
Using GNU awk:
gawk -v RS='^$' -v ORS= '{
print sep $0; sep="\n";
}' ./file*.txt >single.file
prefix dot-slash in files name ./ is used to avoid problems with files named like file=x.txt for instance as awk do reading these kind of strings as a variable when these come after awk codes;
Another GNU awk approach would be:
gawk 'BEGINFILE{if (ARGIND>1) print ""};1' ./file*.txt >single.txt
which is better as it would add an empty line even if the last line doesn't end in a newline character and would avoid loading the whole files in memory.
there is also a sed alternative, but to remove very last \newline, you should add another pipe sed ... | to remove that.
sed -s '$s/$/\n/' file*.txt >single.file
Perhaps not exactly what you were looking for, but like Quasímodo suggested in a comment, GNU's tail can add the empty line, in addition to a header with the filename:
$ echo 'this is foo' > foo.txt
$ echo 'this is bar' > bar.txt
$ tail -n+1 foo.txt bar.txt
==> foo.txt <==
this is foo
==> bar.txt <==
this is bar
The -n+1 causes it to print the whole file; it means "print the tail starting from line 1."
If you want the header to be added even when there is only one file for consistency, you can use -v.
$ tail -n+1 foo.txt
this is foo
$ tail -v -n+1 foo.txt
==> foo.txt <==
this is foo
This does not work in POSIX /bin/sh, but in bash:
cat file1 <(echo) file2 >concatenated
The <(echo) is replaced by a temporary named pipe that is connected to the output of the echo command, which generates a single newline.
An example using Perl.
$ perl -e 'while(<>){print}continue{print"\n" if eof}' *.txt > concat.txt
which can be simplified to
$ perl -ne 'print; print "\n" if eof' [abc].txt > concat.txt
