Not able to cat a string from a file under a string of another file

Question

I have encountered a strange issue in concatenating the files using the cat command. I have two files with one string in each of them:

file1:

ABC

file2:

DEF

Either I do cat file1 file2 or I do cat file1 >> file2. I expect an output as shown below:

ABC  
DEF

However, I have a funny output like this:

ABCDEF

I have checked the files and there is no extra space or characters. However, when I do manual deleting from the back of the string, I don't even see a single character. It works fine. I suppose there must be some kind of "hidden" character or line which I can't see.

It has been bugging me a lot because I have tons of files to concatenate. I can't do the same thing manually.

Any help is appreciated.

Answer 1

paste is probably the easiest (which is not to mention extremely efficient) means at your disposal to handle this problem.

printf abc >file1
printf def >file2
paste -sd\\n file[12]

---

abc
def

When paste is invoked -serially it will read each of its named input files in-turn and paste the output of each line within each file on either a <tab> or a specified -delimiter string. While paste will always end its output of each named infile with a \newline, here the -delimiter is also specified to be a \newline, and so it will just basically cat its input to output with the exception that each file will always end in a \newline.

---

As Peter points out below, an empty file can cause paste to emit an extra \newline though. If this is an issue, practically the same method might be applied with sed which will not do so:

: > file0
sed '' file[012]

---

abc
def

Now with this method, though, (with GNU sed at least) there may be a different problem. Any sed will always write out a \newline before pulling in another line, but if it is the very last line of the whole concatenated series of input files, then some seds (such as GNU) may not append a newline at the end. For example, with my input files, the def is not followed on by a newline.

And if that is a problem, well...

sed '' file[012] | paste -sd\\n

...the above pipeline should probably cover all your bases.

Answer 2

As Peter says, you first file does not have an end-of-line character. You can probably check it with ls -l --- if it's exactly three chars, this is it.

If you want to "cat" the files adding a newline only if the newline wasn't there, you can use the nice trick explained here.

If you have this three files:

[romano:~/tmp] % ls -l f1 f2 f3
-rw-rw-r-- 1 romano romano 3 Jul 12 14:58 f1
-rw-rw-r-- 1 romano romano 4 Jul 12 15:03 f2
-rw-rw-r-- 1 romano romano 4 Jul 12 15:03 f3
[romano:~/tmp] % cat f1 f2 f3
ABCDEF
GHI

where f1 has no end-of-line in the last line, while the others do, you can just do:

[romano:~/tmp] % sed -e '$a\' f1 f2 f3
ABC
DEF
GHI

... sed is a Stream EDitor, and you are instructing it to print all unchanged and the last line adding nothing --- but sed implicitly add a newline when operates, so it solves the problem.

Notice that just using the cat + echo thing will add a newline always. so you'll have two where you had one:

[romano:~/tmp] % for i in f?; do cat $i; echo; done;
ABC
DEF

GHI

[romano:~/tmp] %

Answer 3

It seems that your file1 has no trailing newline. If you want to concatenate a list of files. You can first check each one and cat a newline where needed, as follows:

# make some sample files
printf "%s\n" abc > file1  
printf "%s"   def > file2  # no trailing newline
printf "%s\n" ghi > file3
printf "%s"   jkl > file4  # no trailing newline

---

# find files to concatenate and build a sorted array `f[]`
unset f i; 
while IFS= read -r -d $'\0' path; do f[i++]="$path"
done < <(find . -type f -name 'file[0-9]' -print0 | sort -z)

# build the `cat` command
cmd=cat
tmp="$(mktemp)"; echo >$tmp   # a file which contains only `\n`
for file in "${f[@]}"; do
    lasthex=$(tail -c1 $file | hexdump -ve '1/1 "%02x"')
    [[ -z $lasthex ]] && continue     # skip enpty files
    [[ $lasthex == 0a ]] && nl= || nl=" $tmp"
    cmd="$cmd \"$file\"$nl"
done

# execute the `cat` command
eval "$cmd"

---

The concatenated result is:

abc
def
ghi
jkl

The generated command is:

cat "./file1" "./file2" /tmp/tmp.z7iKccY0T9 "./file3" "./file4" /tmp/tmp.z7iKccY0T9

Answer 4

You have missing newlines. You could do

cat file1; echo; cat file2;

or create a file which has only one linebrake and do

cat file1 NEWLINEFILE file2

or do this with a loop like this

for i in find[1-2]; do cat $i; echo; done;

to run through *, cat each file and add a newline afterwards.

With $(find OPTIONS) instead of find[1-2], you could refine your selection.

Answer 5

You can see special characters in a file with od -c filename. Newlines will look like \n. If Windows or source control changed the newlines, you will see \r\n using od -c, but you won't see anything different with cat. You can use dos2unix filename to repair those.

Sometimes you will see a sequence of control characters that cat would print out to look like a "-" or some other valid character.