7

Here is what I do right now,

sort -T /some_dir/ --parallel=4 -uo file_sort.csv -k 1,3 file_unsort.csv

the file is 90GB,I got this error message

sort: close failed: /some_dir/sortmdWWn4: Disk quota exceeded

Previously, I didn't use the -T option and apparently the tmp dir is not large enough to handle this. My current dir has free space of roughly 200GB. Is it still not enough for the sorting temp file?

I don't know if the parallel option affect things or not.

Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
ChubbyRuby
  • 123
  • 1
  • 2
  • 7
  • 5
    You seem to have a quota system in place. How much of that 200GB is actually available to you? Your error message suggests that your user has a limit to the space you're allowed to use. – terdon Mar 08 '17 at 17:39
  • Also keep in mind the quota (and available space) in `$HOME` might be different than in `/tmp` – derobert Mar 08 '17 at 18:41
  • @terdon That might be the reason. I checked my quota and I only got 100GB, can you believe that? I will have a serious talk with my admin. Thanks again. – ChubbyRuby Mar 08 '17 at 18:50
  • On a system that has 200GB free and is, obviously, a multi user system? Yes, of course I can believe that! Are you joking? You have a full 50% of available space just for you! Oh, and what OS are you using? If you have access to GNU `sort`, I might have a workaround for you. – terdon Mar 08 '17 at 18:50
  • LOL, ok, I'm not a system guy, but I don't believe the whole server has only 200GB free space. I think that's free space of my afs, yet I can only use 50% of my allocated space? That's strange. – ChubbyRuby Mar 08 '17 at 18:54
  • What operating system are you using? Do you have access to GNU `sort`? – terdon Mar 08 '17 at 18:55
  • @terdon, system being ubuntu 16.04. I think I can use GNU sort. What do you have in mind? – ChubbyRuby Mar 08 '17 at 18:55

1 Answers1

6

The problem is that you seem to have a disk quota set up and your user doesn't have the right to take up so much space in /some_dir. And no, the --parallel option shouldn't affect this.

As a workaround, you can split the file into smaller files, sort each of those separately and then merge them back into a single file again:

## split the file into 100M pieces named fileChunkNNNN
split -b100M file fileChunk
## Sort each of the pieces and delete the unsorted one
for f in fileChunk*; do sort "$f" > "$f".sorted && rm "$f"; done
## merge the sorted files    
sort -T /some_dir/ --parallel=4 -muo file_sort.csv -k 1,3 fileChunk*.sorted

The magic is GNU sort's -m option (from info sort):

‘-m’
‘--merge’
    Merge the given files by sorting them as a group.  Each input file
    must always be individually sorted.  It always works to sort
    instead of merge; merging is provided because it is faster, in the
    case where it works.

That will require you to have ~180G free for a 90G file in order to store all the pieces. However, the actual sorting won't take as much space since you're only going to be sorting in 100M chunks.

terdon
  • 234,489
  • 66
  • 447
  • 667
  • That seems to work. I'll give it a try. In fact, I have two 90GB files to sort, and then I need to diff them.... Any idea on the diff part? – ChubbyRuby Mar 08 '17 at 19:31
  • @XuezhouZhang not without more details, no. I suggest you ask a new question. Make sure you clarify whether you need to know what the differences are or if you simply need to know if the files are different, but don't care how or why. – terdon Mar 08 '17 at 21:27
  • Does `-b` honorus lines, i.e. to round chunk sizes to split on linebreaks? I can't see anything in `man split` , I am concerned if there can be portable expectation towards split regaring not breaking lines in the middle – Grzegorz Wierzowiecki May 25 '17 at 11:51
  • @GrzegorzWierzowiecki I assume so. `sort` works on lines by default, the entire concept of sorting is based on sorting *lines*, so it must respect line boundaries here. I haven't checked the source code, but it is the only thing that makes sense. – terdon May 25 '17 at 12:45