non-mapped virtual memory & total number of connections

Question

We have two MongoDB data nodes (replica set) - Primary & Secondary. I noticed that the non-mapped virtual memory is relatively high and wondering if they are hurting our MongoDB performance (The server usually peaked at around 6-7K queries per sec).

In MMS, it was stated: "The most common case of usage of a high amount of memory for non-mapped is that there are very many connections to the database."

So we checked the memory usage with db.serverStatus().mem in our Secondary:

{
    "bits" : 64,
    "resident" : 6846,
    "virtual" : 416797,
    "supported" : true,
    "mapped" : 205549,
    "mappedWithJournal" : 411098,
    "note" : "virtual minus mapped is large. could indicate a memory leak"
}

Note: We are using 2.0.4 and now the default stack size should be 1MB per connection.

The current number of connections is around 1.1K, but the non-mapped virtual memory (virtual-mappedWithJournal) is around 5699 MB. The trend is quite stable so I can't say there is a leak here, but where is the memory gone?

Any idea?

Adam C · Accepted Answer · 2012-12-10T09:51:03.803

First, let me say that if it is relatively stable you don't have a memory leak, and that note in serverStatus() has been removed in 2.2 because of the frequency of false positives.

Non-mapped virtual memory is MongoDB's internal data structures and threads' stacks, essentially anything not backed by files on disk. This is generally driven, as you mention, by the connection stack at 1MB per connection. In your case it sounds like that should be around the 1.1-1.2 GB mark (unless you had a connection spike and the memory had not yet been reclaimed).

If you are doing a lot of inline map/reduce, in-memory sorts etc. they can also push up the usage of non-mapped memory. If you install MMS (which is free), non-mapped memory is one of the stats tracked over time and you can then correlate increases in the non-mapped stat with the activity on your database easily. Though, if you have not been running it you may need to restart the process to track the usage from scratch again.

This type of usage analysis is often easier than using pmap (or similar) to try to tie back the memory to specific pieces inside a process

Finally, if non-mapped usage is out of control, it can occasionally be caused by issues with libc malloc in 2.0 - this, among other reasons, was why the 2.2 version shipped with TCMalloc instead. So, 2.2 may well be worth checking out if this remains high without an obvious reason.

Hi Adam, we are not using MR or doing any kind of sorting in our production DB as it is almost impossible to keep our current QPS if we are doing so. The connections count is rather stable below 1.5K. We are already using MMS and one thing which is quite interesting is our SECONDADY's non-mapped virtual memory is almost 1.5 to 2 times more than our PRIMARY, and the SECONDADY's loading is higher than the PRIMARY if they serve the same amount of traffic, so currently we still serve quite a lot of read in our PRIMARY. — tszming, Sep 14 '12 at 12:08
if you give me the group name in MMS, I can take a look (I work for 10gen). Other things that should be looked at are readahead when you are dealing with memory allocation: http://serverfault.com/questions/408100/mongodb-and-datasets-that-dont-fit-in-ram-no-matter-how-hard-you-shove/408182#408182 and http://serverfault.com/questions/418352/readahead-settings-for-lvm-device-mapper-software-raid-and-block-devices-wha — Adam C, Sep 14 '12 at 13:00

non-mapped virtual memory & total number of connections

1 Answers1