comm command behaving strangely

Question

I have two files:

1. one generated using find command in a folder to list files, sorting them numerically and writing to a file,
2. and the other generated by a python script, which is not sorted, so I explicitly sort it numerically.

The problem is that my sort output only has two columns and is as follows:

500016
    500016
500174
    500174
500277
    500277

As you can see, even the common entries are shown separately in two columns and the third column is missing altogether, implying that there is nothing common between the two files, whereas these first three entries are indeed same. sort otherwise works as expected with some test files that I make.

I know that comm needs the two files to be lexically sorted, and here is a list of options I tried and failed:

comm <(sort file1.txt) <(sort file2.txt)

from https://unix.stackexchange.com/a/377689/187419 failed. I also tried giving the -d option to sort explicitly, and also tried explicitly rewriting the files with dictionary sort -- both didn't work

comm --check-order <(sort file1.txt) <(sort file2.txt)

from https://unix.stackexchange.com/a/186101/187419 did not return any order error; it ran as usual giving two output columns.

This solution for a problem very close to mine is also not working.

Thinking that it might be because of some additional characters in the file, I also the solution mentioned here to do :set list in vim.

Just to test if sort is causing issues, I deliberately sorted the test files I made (with which comm worked earlier) numerically and comm still worked.

I tried the solutions I could find, to no avail. Any other suggestions?

Answer 1

You are almost certainly right that additional characters on each line are causing corresponding lines to fail to match exactly. Those additional characters might have the form of carriage-return characters from Windows-style line terminators, space or tab characters, or possibly other non-printing characters. For example, maybe the Python script is right-justifying the numbers so that some or all of them have leading spaces.

The surest thing to do would be to filter out all such unwanted characters, and since the data are strictly numeric, that's pretty easy to do with, for example, sed:

sed 's/[^0-9]//g' < input > output

You could interpose that at various points in your process. Here's just one:

comm <(sed 's/[^0-9]//g' file1.txt | sort) <(sed 's/[^0-9]//g' file2.txt | sort)