16

On Mac OS X Yosemite 10.10.5, when I try to run a calculation that needs to allocate and use 128 GB of memory (it is a command line program written in C), the kernel kills my process with extreme prejudice. This console log entry is an example of one instance:

9/25/15 7:08:40.000 PM kernel[0]: low swap: killing pid 6202 (huffgrp)

The calculation works fine and in a reasonable amount of time when it allocates and uses 64 GB of memory. My Mac has 32 GB of RAM and beaucoup space on the hard drive. I also tried this on another Mac with 8 GB of RAM, on which the 64 GB calculation runs fine as well, taking longer of course, but the 128 GB calculation gets killed by the kernel in the same way.

By the way, malloc() never returns an error, no matter how much space I ask for. The kernel will only kill the process once too much of that memory is actually being used by the process, resulting in lots of swapping to the hard drive.

So there appears to be a secret swap space limit somewhere between 64 GB and 128 GB.

My question is: how do I reconfigure the kernel to permit more swap space? I found a promising looking file, /System/Library/LaunchDaemons/com.apple.dynamic_pager.plist, but I don't see the secret number in there. The man page for dynamic_pager says that all it does is set the name and location of the swap files. There is an older version of the same man page that documents a -S option to set the size of the swapfiles created. I tried that, requesting 160 GB swapfiles, but it had no effect. The swapfiles were still 1 GB each, and the process was still killed by the kernel.

Mark Adler
  • 1,955
  • 4
  • 14
  • 15
  • I also posted this question on apple.stackexchange.com, but there has not been any response there. – Mark Adler Sep 27 '15 at 15:41
  • 1
    The reason that you can `malloc` more than you have is because the commit_limit is very high (probably infinite). Therefore the OS will allocate memory that it does not have (this is betting that the process will not use it, the os usually wins this bet). You may want to adjust the commit limit to be the memory limit, this way the process will fail early. – ctrl-alt-delor Sep 27 '15 at 20:38
  • I can not see why you thing that paragraph 5 is true, it does not follow from paragraph 4 (though you start it with “so”). Also have you tried adding lots of swap? (the question is not clear on this, though I suspect the answer is yes). – ctrl-alt-delor Sep 27 '15 at 20:41
  • @richard Yes, I know why `malloc()` does that. I was deflecting possible comments on someone thinking I'm not checking the return value of `malloc()`. By the way, my objective is not to fail earlier. My objective is to succeed. – Mark Adler Sep 27 '15 at 23:38
  • @richard Because the kernel message says "low swap". As to "adding lots of swap", no I haven't done that because I don't know how. That is precisely this question. _How_ do I add swap space? Of course, the kernel adds swap space automatically, but only up to a limit. Hence the root of question: _How do I increase the kernel's swap space limit?_ – Mark Adler Sep 27 '15 at 23:41
  • As I noted in my question, that man page currently does not provide that option. It only provides `-F` to change the base name and location of the swapfiles. – Mark Adler Sep 28 '15 at 15:41
  • `...calculation that needs to allocate` - or maybe rather, your implementation needs to? In general, if an app needs 10x more **RAM** than is contemporary available, it is a good idea to use some fast storage to offload from the memory. Perhaps a graph DB like Titan. – Ondra Žižka Feb 09 '21 at 10:22

2 Answers2

5

Not the answer you asked for, but if you create your own file of an appropriate size, mmap it into your process, and then run your calculation in this address space it should have the same effect as a swap file and you're guaranteed to have the space, as opposed to competing with other processes for available RAM/swap.

It might also be slower, depending on how often you overwrite the data, but should be much more portable.

dataless
  • 1,689
  • 1
  • 13
  • 13
  • 3
    Great idea! I had high hopes. Alas, they were dashed against the harsh shore of reality. The process was _still_ killed by the kernel for large amounts of swapping, just as before ("low swap" console message), even though it was using a file I had created to use as virtual memory by `mmap()`. – Mark Adler Oct 01 '15 at 05:48
  • Sorry, most of my experience is with Linux. But, maybe the problem is that the kernel isn't flushing back to the file and holding it in RAM instead. What if you periodically call ```msync()``` to force it to do so? – dataless Oct 01 '15 at 18:25
1

My mac info is quite dated there may no longer be the kernel magic to do this. As such I suggest using Linux for this program, where you can mount either a folder or partition as swap very simply.

Eliminate the uncertainties of using a dynamic swap folder. Create a physical swap partition (just an empty unformatted partition, with disk table type code for swap, in Linux it is hex code 0x82.) Then:

  1. Edit the /etc/rc file find the swap section comment out all of it. add the line mount -vat swap
    • means mount all swap partitions in /etc/fstab
  2. run pdisk /dev/disk? -dump
    • "?" is the disk number of your hd record the number next to the swap partition
  3. Edit or create /etc/fstab (it probably doesn't exist) add a line like /dev/disk?s?? none swap sw 0 0
    • ? is your disk.
    • ?? is the number of the swap partition.
  4. reboot.
Max Power
  • 229
  • 1
  • 9