0
  • What is your experience? Can you confirm my experimental findings?
  • Can I generally use total RAM - 600 MB, or 0.4 * total RAM?
  • Or is it always trial and error, and hoping that it is low enough?

Context: I'm trying to set up jenkins on a T3 instance, experimenting with Ubuntu Server 16.04 and 18.04.

I started with a t3.micro instance (1 GB RAM), but found the OOM killer killing my java process, as soon as I use more than about -Xmx400m, which seems kind of low. I was expecting to be able to use more like -Xmx750m.

Does this mean Ubuntu Server requires about 600 MB to work?

The problem is that the java process starts, even if I set both -Xms and -Xmx to a very high value, like 700m. The process is killed only later when I make the first request to the website.

I now experiment with a t3.small instance (2 GB of RAM), but am again very unsure about what to configure.

On Windows it is kind of deterministic: I set both -Xms and -Xmx to the same value. If the service fails to start, the value was too high. If the service starts successfully, the value is fine, and the memory is reserved for my process.

Some background:

Reto Höhener
  • 431

1 Answers1

1

If you want to run any application on Linux in a limited resource environment you should make sure that you understand how memory on Linux actually works. Sounds harsh, but it really is the only way to understand what is happening behind the curtain.

If the OOM killer gets triggered at all it means you ran out of memory in the first place. If that happens with -Xmx400 then this is the limit for that specific application in that specific environment doing the specific thing it does.

Before you even think about setting vm.overcommit_memory to a non-default value, make sure that you really understand the difference between allocating and using memory (in terms of the linux virtual memory system). Otherwise you will just make sure that you'll never be able to use the system's memory efficiently.

Having said all that: the defaults are usually good in 80%-90% of the use-cases. For Jenkins the defaults in the kernel should actually work rather good. You will want the kernel to overcommit the available memory (which is the default) and make it trigger the OOM when it actually runs out (also the default).

If that doesn't work with the memory you have available, you'll need more memory. Also setting larger -Xmx values that are not required by the application is considered harmful, but you asked about Linux memory, not Java memory :)

Andreas Rogge
  • 2,918
  • Thank you for your answer. This is me trying to learn how memory on Linux works. I didn't change the OOM settings (overcommit_memory=0). The default -Xmx for OpenJDK 8 on Ubuntu 18.04 seems to be 1/4 * RAM, which would be just 256 MB on t3.micro. I know that a new jenkins can run with that memory (I tried it), but want it to be able to use all available memory on that server. I believe that is a pretty sensible requirement to have. – Reto Höhener May 09 '19 at 20:05
  • Also I believe your conclusion is not accurate: OOM killer is triggered with -Xmx500, but not with -Xmx256, so jenkins is fine with 256 MB, but the system lets it overcommit when started with -Xmx500m, and then panics and kills it. This behavior is very confusing to me. – Reto Höhener May 09 '19 at 20:08
  • Thanks for proving that you did not yet understand what the Linux virtual memory system does. :) – Andreas Rogge May 09 '19 at 21:47
  • When a program tries to allocate memory (calls malloc() or something like that) the linux kernel will decide whether or not it can allocate (reserve) that amount of memory for the given program. With the default settings the kernel usually decides that almost every allocation is ok and will allocate (reserve) way more memory than is present in the system. This behaviour is desirable, because most memory that is allocated is never used (or at least it is never written to) so the over-allocation is required to be able to use the whole system memory. – Andreas Rogge May 09 '19 at 21:52
  • Now when too many programs actually use the memory that has been allocated for them, the kernel will run out of memory. That is called an OOM and it is resolved by killing at least one process, preferably one that consumed a lot of memory. – Andreas Rogge May 09 '19 at 21:54
  • To stop the kernel from running into OOM you can do three things:
    1. add more memory to the system
    2. disable memory over-commit so things will fail really early on allocation
    3. make your applications use less memory (they can still allocate as much as before)
     – Andreas Rogge May 09 '19 at 21:56
  • Yes, I don't understand how it works, I just said so. I still don't understand why the OOM thinks it needs to kill my jenkins java process, when started with -Xmx500m. Aren't the remaining 500 MB more than enough for Ubuntu to work? – Reto Höhener May 09 '19 at 22:12
  • Let us continue this discussion in chat. – Andreas Rogge May 09 '19 at 22:59