2

For example, would it not be beneficial to have -jN > nrcpus during the configure phase, where CPU load is low? Then have -jN = nrcpus during compilation, where load should be evenly distributed to each CPU?

How difficult would this be to implement in a build environment? Would it require changing each configuration file for each step? Is it possible to automate for multiple projects, given they use the same "make" program?

This is assuming that the workload is never I/O bound.

avisitoritseems
  • 121
  • 2
  • Related: https://unix.stackexchange.com/q/208568/117549 – Jeff Schaller Dec 22 '19 at 03:34
  • 2
    Not clear to me what you are asking. The typical `configure` script is a single threaded shell script, so there is nothing to be gained from telling `make` that it can run multiple jobs. To answer the question in the title, look at the `-l` option to make. This limits jobs being started if the load average (which is a crude approximation to CPU load if the workload is not I/O bound) is too high. – icarus Dec 22 '19 at 04:31
  • @icarus it's not clear if the OP is using `configure`. The [tag:configure] tag was added by a moderator, and the original [tag:cmake] tag removed. That being said, I don't know if `cmake` supports that kind of parallelism, but I know `cmake` supports not only `make`, but also other build systems like `ninja`. –  Dec 22 '19 at 23:12
  • @mosvy cmake is a generator, which as you say can generate files such as build.ninja for ninja, Makefiles for various flavors of make, and even supports some IDEs. The typical use is a one time `mkdir build ; cd build ; cmake ..` which does the generation based on ../CMakeLists.txt, and then just use `make` as required. There is a lot more to it of course, support for cross compiling, caching build flags and rebuilding if they change etc. This invocation of cmake does a lot of the same things that a typical configure script does, e.g. finding installed optional libraries. – icarus Dec 23 '19 at 02:57
  • If this is what the OP was thinking of as the "configure" stage, then this is still single threaded. There is no `-j` flag to cmake. – icarus Dec 23 '19 at 03:00
  • @icarus many complex projects include a convenience `Makefile`, which will automatically call `configure`/`cmake`/`meson`/whatever into subprojects (sometimes even download them via `git`); _that_ can be run with `make -j`. I usually use just `nice -n19 make -j32` (on a machine with 4 cores) and let the system handle it ;-) –  Dec 23 '19 at 03:23
  • @mosvy yes, and the big advantage of that is (assuming the convenience Makefile is correctly written) is the "jobserver" of the top level make allows coordination of the 32 jobs between the subprojects. However we seem to be drifting away from answering the OP. Do we know what the OP's question is? – icarus Dec 23 '19 at 03:50
  • @icarus imho, just using __`nice -n19 make -j32`__ (with a `j` > the number of cores) will do for "evenly distributing the load", keeping the cores busy even when some make actions are blocked in i/o, and not let `make` hog the whole system ;-) –  Dec 23 '19 at 04:39

1 Answers1

2

While a specified N within make -jN is a fixed maximum, there is a way to limit parallel jobs based on system load. See make -l load or make --load-average[=load]

References:

https://www.gnu.org/software/make/manual/make.html#Options-Summary https://www.gnu.org/software/make/manual/make.html#Parallel

Note that option -l load is supported in newer gnu make, though an installed version could be older.

For selecting load, here's one example: first tune -jN for the best performance on your server (also try building sources with gcc -pipe); after observing load averages, tune -l load so the value is enough to support your build activity. Tuning -l load may also help avoid overloading the server.

forest
  • 129
  • 2
  • Thank you. Is this only limited to gnu make? A more pertinent question with example: Assuming I am in the top lvl directory of a project and run `make -j100 -l5`, will the nr. of jobs be able to dynamically for that one Makefile throughout? I have only experienced using load average with multiple projects, not one project, so I have some difficulty understanding how it works. Also then for a build host, would it then be ideal to set gnu make parameters to `-j(nr CPUs * 2) -l(avg. load)`? That way more jobs can be created when load is not exceeded for normal operation. Assuming RAM is no issue. – avisitoritseems Jan 02 '20 at 06:21