How to combine multiple text files and remove duplicates effectively?

Question

I tried many things but most of them just got stuck or didn't remove the duplicates properly. It doesnt matter if the output file is sorted or not

If file1.txt contains

a
b
c
d
e
c
d

If file2.txt contains

c
d
e
a
f
g
g
h

output.txt should contain

a
b
c
d
e
f
g
h

Answer 1

simply use sort -u file[12].txt >output.txt. that sorts the input while removed duplicates.

make sure your files are in Unix type newline format (LF \n) instead of Dos/Windows newline format (CRLF \r\n); if not convert them by tr -d $'\r' <filename or dos2unix filename command.