29

What is reasonable scalability limit of sort -u? (in dimensions of "line length", "amount of lines", "total file size")

What is Unix alternative for files exceeding this in dimension of "amount of lines"?

Of course I can easily implement one, but I wondered if there is something that can be done with few standard Linux commands.

pevik
  • 1,397
  • 15
  • 27
Grzegorz Wierzowiecki
  • 13,865
  • 23
  • 89
  • 137
  • For those that might want to binary search in it or know how: http://unix.stackexchange.com/q/247508/9689 – Grzegorz Wierzowiecki Apr 26 '16 at 07:45
  • 3
    There are situations where a `uniq` before the `sort -u` helps. BTW, for ASCII data `LC_ALL=C sort` speeds up GNU `sort` an awful lot (see [this answer](http://unix.stackexchange.com/questions/169739/why-is-coreutils-sort-slower-than-python)) – Walter Tross Apr 28 '16 at 19:37

3 Answers3

48

The sort that you find on Linux comes from the coreutils package and implements an External R-Way merge. It splits up the data into chunks that it can handle in memory, stores them on disc and then merges them. The chunks are done in parallel, if the machine has the processors for that.

So if there was to be a limit, it is the free disc space that sort can use to store the temporary files it has to merge, combined with the result.

Anthon
  • 78,313
  • 42
  • 165
  • 222
  • 4
    Note that GNU sort can compress those temp files to pack even more (and increase performance with slow disks). – Stéphane Chazelas Apr 26 '16 at 12:11
  • 1
    @StéphaneChazelas Thanks for the update. I was wondering myself if sort was smart enough to remove chunk files when one is fully merged (which could easily happen if the source is already partly sorted) as a space optimization. I haven't got the time to dive into the source code these days :-( – Anthon Apr 26 '16 at 12:18
  • 4
    Apart from memory there's also another limit that applies to the merge phase: the number of files that can be simultaneously open. This is typically a limit imposed by the operating system. GNU sort copes with that as well, by recursively merging the number of files it can old open at one time! – Diomidis Spinellis Apr 26 '16 at 14:21
  • @StéphaneChazelas If I were designing a tool specifically for sorting very large files, I would store the lines as an index into the original file. Does GNU sort do this, or does it simply use a conventional compression algorithm? – Random832 Apr 27 '16 at 05:19
  • @Random832 sort cannot do that, at least not in the general case, because it must work on an STDIN stream. – Law29 Apr 27 '16 at 06:28
  • 4
    @Random832 and it's meant to be able to overwrite the file over itself (`sort -o file file`) – Stéphane Chazelas Apr 27 '16 at 06:46
4

I cannot speak for vendor specific implementations, but the UNIX sort implementation splits large files into smaller files, sorts these files and then combines the sorted smaller files into an aggregated sorted output.

The only limitation is the disk space for the smaller files created intermediately by sort, but the files can be redirected to an arbitrary directory by setting the environment variable TMPDIR.

schily
  • 18,806
  • 5
  • 38
  • 60
  • 3
    What exactly do you call the _UNIX sort implementation_? Is it the original one from Unix version 3? The man page there says it can't sort files bigger than 128KiB. – Stéphane Chazelas Apr 26 '16 at 12:15
  • 1
    What do you understand by Unix version 3? The version from 1973? The original UNIX sort implementation has been enhanced over the years and IIRC, the Solaris version is even much faster then the GNU version. Of course, 25 years ago sort was enhanced to understand multi-byte characters and what I remember from a USENET discussion, was that this has been done efficiently on Solaris. BTW: `man largefile` lists `sort` as large file aware. – schily Apr 26 '16 at 12:33
  • 2
    So are you actually talking of the Oracle vendor specific version of `sort`? Or any derivative of some version of AT&T Unix sort? Or any Unix certified version of `sort` (like GNU `sort` on OS/X)? – Stéphane Chazelas Apr 26 '16 at 13:03
  • The quality of modern `sort` implementations with respect to multi-byte chars may vary, the fact that `sort` uses split intermediate files is common to all UNIX implementations that are based on the original code. BTW: the Solaris version is OSS as "OpenSolaris", see https://sourceforge.net/p/schillix-on/schillix-on/ci/default/tree/usr/src/cmd/sort/common/ – schily Apr 26 '16 at 13:18
  • 25 years ago, UTF-8 was not invented yet? Support for UTF-8 locales was added in Solaris 7 ([1](http://czyborra.com/utf/), [2](https://web.archive.org/web/20060509010340/http://docs.sun.com/app/docs/doc/805-4123/6j3tmpc75?a=view)). Are you referring to some other multibyte character set? – Stéphane Chazelas Apr 26 '16 at 13:18
  • Aprox. 24 years ago, Solaris programs did already support multi-byte characters. If you read my text, you will not see "UTF-8". – schily Apr 26 '16 at 13:19
3

Based on https://blog.mafr.de/2010/05/23/sorting-large-files/ and https://unix.stackexchange.com/a/88704/9689 :

split -n l/20 input input-
for inpf in input-* ; do
    sort --parallel="$(nproc --all)" "${inpf}" > sorted-"{$inpf}"
done
sort -m sorted-input-* > sorted-input

Update:

From answers above we see that sort already does what mentioned snippet - i.e. External R-Way merge. So after all running just:

sort --parallel="$(nproc --all)" -u input > output

Should be sufficient.

My current assumptions (without checking code) about limits are:

  • maximum line length is limited by amount of physical memory. Sort need to fit at least two into memory
  • amount of lines - I am not aware of
  • file size - of course by filesystem
  • amount of opened files in parallel - depending on Operating System (Thanks Diomidis Spinellis for pointing this out!)

(This answer is marked as community wiki - feel encouraged to improve it! :) )

Community
  • 1
Grzegorz Wierzowiecki
  • 13,865
  • 23
  • 89
  • 137
  • 5
    GNU `sort` sorts in parallel by default (since 2010 after that page you're linking to), `--parallel` there is to reduce the number of concurrent threads instead of letting `sort` determine the optimum one. Sort already does a splitting and merging internally in a more efficient way. I doubt doing that extra splitting will help. – Stéphane Chazelas Apr 26 '16 at 12:20