How to copy all the files in a specific folder and then concatenate them into one file?

Question

I would like to improve my shell script skills. What I want to do is that I will copy all the files in a specific folder into other destination folder and then save them into one file by concatenating them.

Below is my simple code.

path=/Users/sclee/Dropbox/project/java/projects/WaferPointDraw/src/main/resources/pattern
dest=/Users/sclee/Dropbox/journal/clustering_ieee_semi_2020/data/small_datasets/noise200000other2000
output=large_datast_3.txt

mkdir -p $dest;
echo "$dest directory is created"
for i in `find ${path} -iname '*.txt'`;
do
        cp $i $dest;
done
echo "all the files is copied"

cat ${dest}/* > ${dest}/${output}
echo "created the combined file"

I can achieve my goals with above codes, but I would like to process all the logic in the for loop. As you can see, the cat command is done apart from the for loop. I would like to process them in the for loop for the simple code.

Any help will be appreciated.

Thanks.

To improve your shell skills, read this: http://mywiki.wooledge.org/BashFAQ/001 -- don't read lines with `for` — glenn jackman, Sep 21 '20 at 12:53
More reading: [Security implications of forgetting to quote a variable in bash/POSIX shells](https://unix.stackexchange.com/questions/171346/security-implications-of-forgetting-to-quote-a-variable-in-bash-posix-shells) — glenn jackman, Sep 21 '20 at 12:59

score 0 · Answer 1 · answered Sep 21 '20 at 03:27

0

Oneliner using find :

find "$path" -iname '*.txt' -exec cp {} "$dest" \; -exec sh -c "cat {} >> $dest/$output" \;

answered Sep 21 '20 at 03:27

annahri

2,030
14
32

1

The last clause could be `-exec cat {} + > "$dest/$output"` I think – glenn jackman Sep 21 '20 at 12:51

score 0 · Answer 2 · answered Sep 21 '20 at 10:08

0

If I understand right you could replace

do
  cp $i $dest;
done

with

do
  cat $i>>${dest}/${output};
done

That appends the findings to your output file

answered Sep 21 '20 at 10:08

Dbrg

1

The Q says "copy and cat", so keep the cp command in the loop. – glenn jackman Sep 21 '20 at 12:52

score 0 · Answer 3 · answered Sep 21 '20 at 12:58

question: the txt files in $path, are there subdirectories into which you want to descend in order to find all the txt files? If not, you don't need find, for file in "$path"/*.txt will suffice.

In fact, you don't need a loop if all the files are in one directory:

cp -t "$dest" "$path"/*.txt
cat "$path"/*.txt > "$dest/$output"

Note that a variable in braces ${var} is not the same as a quoted variable "$var"

How to copy all the files in a specific folder and then concatenate them into one file?

3 Answers3