19

In Gentoo Linux it is possible to set the MAKEOPTS variable in /etc/portage/make.conf to tell make how many jobs it should run in parallel when building packages. Since I have a dual-core CPU, I naively chose to use the -j2 option: one job per core, so both have something to do. The "problem" is there are a lot of references that tell users having a dual-core CPU to set the -j3 option instead. Some of them are:

For example, the Gentoo handbook says:

A good choice is the number of CPUs (or CPU cores) in your system plus one, but this guideline isn't always perfect.

But what is the rationale for "CPUs + 1" rule? Why the extra job?

The make.conf(5) man page even says:

Suggested settings are between CPUs+1 and 2*CPUs+1.

I also read section 5.4 (Parallel Execution) in the make info page and make man page explanation for the -j option, but it seems there's no answers there.

4 Answers4

14

There isn't a simple rule that always works. People might recommend a particular figure because they experimented with a particular compilation on a particular machine and this was the best setting, or because they followed some reasoning that may or may not have some relation with reality.

If you're blessed with a lot of RAM, then the limiting factor in a long compilation will be CPU time. Then one task per CPU, plus one pending task for those occasional I/O blocks, is a good setting. That makes it -j3 for a dual-core CPU (or more precisely, for a dual-CPU machine — if each core is hyperthreaded, that would be 4 CPUs, so -j5).

If you have very little RAM, then a limiting factor may be that you can't have many concurrent jobs, or else they'll keep making each other swap out. For example, if you can't comfortably fit two compiler instances in memory, make -j2 may already be slower than make. Since this is dependent on how many compiler processes you can fit in RAM at once, there's no way to derive a general figure.

In between, it may be beneficial to have more jobs. If each compiler process is small, but the build as a whole touches a lot of data, then disk I/O may be the blocking factor. In this case, you'll want several jobs per CPU at once, so that there is always one job using each CPU while others are waiting for I/O. Again, this is very dependent on the build job and on the available RAM, here on what's available for data cache (there's an optimum after which having too many jobs pollutes the cache too much).

Gilles 'SO- stop being evil'
  • 829,060
  • I didn't know that if CPU cores are hyperthreaded, then each of them counts as two. Anyway, it seems that my CPU doesn't support Hyper Threading. –  Oct 15 '12 at 14:27
  • I accepted this answer. Anyway I chose to stick with -j2 on my system. This is because I tried emerging both gcc and firefox with settings from -j1 up to -j5 (for a total of 10 emerge commands), and it seems that while -j2 is definitely faster than -j1, the other three settings are on par with -j2. –  Oct 19 '12 at 12:28
7

I guess this is kinda heuristic — allowing make to launch CPUs + 1 processes is to make sure that:

  1. there wouldn't be a gap between a worker-process which just have finished and a worker yet-to-run — somewhat like pre-filling run queue.
  2. there wouldn't be too much competing processes to bring in noticeable overhead with that run-queue pre-filling.

But, again, that's heuristic and FreeBSD's handbook still recommends make -j4 for a single CPU.

poige
  • 6,231
5

Generally, there are reasons to start more jobs than number of cores. For C compiling using gcc, if -pipe isn't defined in gcc options, it does its actions (preprocessing, first run, optimizations and assembly) in sequence using temporary files; -pipe changes this to using pipes between subprocesses. (Adding -pipe is default e.g. for FreeBSD but isn't traditional on Linux.) So, if you have 2 cores and allow 2 jobs in parallel, they will spend some time in disk I/O. The recommendation to add 1 job seems to be related with this specifics. But to get final answer you should find who and when added this recommendation and ask him:) or ask in Gentoo devels' mailing list.

Netch
  • 2,529
2

Basically that number is what the authors call common sense. At best, it's a good guess. As far as I know the make process that is spawned when you type make is already counted so with -j3 you can end up with the main process waiting, while the two others are compiling.

However, back when I used Gentoo the rule of thumb was <#cpus>*2 + 1.

It all hinges on what your chicken trails, tea leaves or magic 8 ball tell you about the disk I/O that needs to take place and the scheduling of your current linux kernel. [begin this post's core] From my personal experience (-j is not Gentoo specific), everything between #cpus + 1 and #cpus *2 +1 yields fine results[end this post's core] and on average you will hardly note any differences. Processors and kernels are quite good these days.

BUT all this changes when: a) you actually use more than one box to compile (du'h) or b) are developing your own code

A higher -j attribute is more likely to show previously unknown dependencies.

And on a side note: dont go by the number of cores, but by the number of concurrent streams the CPUs take. (Hypertheading!)

Bananguin
  • 7,984