Memory leak with NDSolve

Question

Bug solved in 11.2

Bug introduced in 10.3 or earlier and persists through 11.0

Edit The technical team confirms the memory leak in NDSolve and has forwarded an incident report to the developers.

This is not the first time the question in mentionned (cf in particular this recent question NDsolve memory leak), but there seems to be a serious issue of memory leak - or am I missing something?

First example (no leak) Let's loop 1000 times over the first example in the documentation of NDSolve:

$HistoryLength = 0;
s := NDSolve[{y'[x] == y[x] Cos[x + y[x]], y[0] == 1}, 
   y, {x, 0, 30}];
temp = MemoryInUse[];
Table[s, {i, 1000}];
MemoryInUse[] - temp

-1256

There is no significant increase in the memory usage.

Second example (memory leak?!) Now let's loop 1000 times with NDSolve using another example from the same documentation. I see no programming difference in terms of memory, however this time the used memory increases significantly (and for other systems of equations, it can increase much faster).

s2 := NDSolve[{y''[x] + Sin[y[x]] y[x] == 0, y[0] == 1, y'[0] == 0}, y, {x, 0, 30}]
temp = MemoryInUse[];
Table[s2, {i, 1000}];
MemoryInUse[] - temp

293 696

Additionally, Remove[s2] or Clear[s2] does not free any memory (I think that's normal, but what is the memory used for?).

How come does the memory usage increases, and how to overcome this serious problem?

Tested on MMA 10.3.0.0 with Ubuntu 15.10 64 bits

Update (09 Aug 2016): I just installed version 11.0 and the leak is still present.

With v9.0.1, win10 64bit, I observe similar memory "leakage" in the first example, too. — xzczd, Jun 28 '16 at 03:33
@xzczd Interesting... it would mean this might be a bug? I added info about my system. — anderstood, Jun 28 '16 at 03:41
With v10.4.1 on Win7 x64 I get 365528 as output for the first example and 721896 - for the second. If I put ClearSystemCache[] after Table I get -1760 and 294296 correspondingly. — Alexey Popkov, Jun 28 '16 at 03:51
@AlexeyPopkov How many times have you tested? As far as I can tell, ClearSystemCache will reduce the memory usage only once if you are not testing with a fresh kernel :) — xzczd, Jun 28 '16 at 04:02
@xzczd With fresh kernel and ClearSystemCache[] I get 2673480 after the first run and then -1760 for every further run. — Alexey Popkov, Jun 28 '16 at 04:05
@AlexeyPopkov That's interesting… I get 94352 except for the first run. @anderstood have you reported this to WRI? — xzczd, Jun 28 '16 at 04:09
@xzczd : No, as a beginner I was afraid I was missing something simple, so did not dare reporting this. Is this the right link to use? Does it mean there is no other way than clearing the kernel to avoid memory overflow? :'( — anderstood, Jun 28 '16 at 04:14
@xzczd I just reported this ([CASE:3648349]). Thanks for having tried the code on your computer. — anderstood, Jun 28 '16 at 04:22
I am curious to know what the application is where this comes up; surely you do not call s2 a thousand times? — user21, Jun 28 '16 at 17:16
@user21 I am exploring systematically the periodic orbits of a given nonlinear dynamic system. Without entering too much into details, I have to find curves in a space of dimension $p$, where $p$ is a chosen parameter (typically, 5). I know some points on the curves (typically, a few thousands) and I do continuation from each of these points. Each continuation uses NDSolve, so I need to use it a few thousand times, but because of the memory leak it overflows every 200 iterations. I hope this gives you enough insight on my need :-) — anderstood, Jun 28 '16 at 18:07
@user21 I don't have a set of parametric ODEs, but rather the ODEs (including their numbers, the number of terms in each ODE, etc.) is parametrised by $p$. Actually I should have avoided mentioning $p$; I did it to explain that I had more than a few thousand calls to NDSolve. However, I tried ParametricNDSolve with a dummy parameter (s2 := ParametricNDSolve[{y''[x] + Sin[y[x]] y[x] == 0, y[0] == 1, y'[0] == 0}, y, {x, 0, 30}, {a}]), but it also suffers memory leak... — anderstood, Jun 28 '16 at 20:30

anderstood · Accepted Answer · 2017-11-10T16:48:44.613

Update 2017 11 10 The technical support report this bug has been solved in the 11.2 release.

Partial answer

The technical support came back to me with the following workaround: write the ODE(s) in the first order form. Some symbols are apparently leaking when MMA performs the reduction.

For example,

NDSolve[{y'[x] == z[x], z'[x] + Sin[y[x]] y[x] == 0, y[0] == 1, z[0] == 0}, y, {x, 0, 30}]

does not lead to memory leak.

However, the following DAE still leaks, while I see no necessary first-order reduction:

DAE = ({{x1'[t] == x3[t]}, {x2[t] (1 - x2[t]) == 
      0}, {x1[t] x2[t] + x3[t] (1 - x2[t]) == t}});
Clear[old, x, y]
ClearSystemCache[];
MemoryInUse[]
Do[old = MemoryInUse[];
 NDSolve[{DAE, x1'[0] == 1}, {x1, x2, x3}, {t, 0, 1}]; 
 Print[MemoryInUse[] - old];, {10}]
MemoryInUse[]

Present status is that I am waiting for their answer. I'll update the post accordingly.

Edit I was told by the support team that they do not know when this bug will be fixed (but they confirmed that it's a bug).

"(...) they do not know when this bug will be fixed (...)", eh — mmal, Oct 02 '16 at 07:16

alan · Answer 2 · 2016-06-28T05:36:07.810

4

I have experienced serious memory leaks with NDSolve, NIntegrate, and FindRoot in every version of Mathematica I have owned (still on V9), perhaps over 20 years. They typically show up for me when the routine is called at a deep level in a complex program.

The only cure in many cases is to remove the offending routine and write your own. For NIntegrate, say a Simpson's rule, for FindRoot, say a Newton's method. If using NDSolve on an ODE, again a simple hand-coded solution. (I'm not saying that's the cure for this particular case, which is effectively cured by the HistoryLength trick. I'm saying that you will eventually run across one that seems 'unfixable' after trying $HistoryLength=0, Clearing[] every variable in sight, etc, etc.)

I know this sucks, but it's the only solution I have found in some cases. In my case, I am talking about a half-dozen cases over, say, several thousand programs. So, it's not the end of the world.

edited Jun 28 '16 at 05:36

answered Jun 28 '16 at 04:58

alan

181
1
4

2

What is surprising is that there are still memory leaks on widely used functions, after more than 20 years of existence. And it's not free software. Thank you for your answer anyway. – anderstood Jun 28 '16 at 14:23
2

The only true solution to memory leaks is the report them to WRI. Unreported leaks have a very low chance of getting fixed as developers don't know about them. – user21 Jun 28 '16 at 17:12
1

@anderstood, especially these widely used functions see a lot of change in the code, so the chance of introducing a leak is higher than in functions that do not see a lot of change. – user21 Jun 28 '16 at 17:15

Memory leak with NDSolve

2 Answers2

Linked