How to fix this memory leak

Question

I am using halirutan's code in this thread to unflatten a list. It works very well but the problem is that the Module's local function f is not deleted when the function returns the output and I consequently end up with hundreds of thousands of copies of the symbol of the form f$xxxx, which eventually crashes Mathematica. I tried other solutions from the same thread, but they don't seem to work. Is there any workaround?

This slight change to that code might suffice. unflatten[l_, o_] := Module[{f, i = 1, l1 = Flatten[l]}, Attributes[f] = {Listable}; f[_] := l1[[i++]]; ll = f[o]; Clear[f]; ll] — Daniel Lichtblau, Apr 03 '14 at 17:03
@DanielLichtblau You beat me to it. Why don't you write this as an answer? — Leonid Shifrin, Apr 03 '14 at 17:05

halirutan · Accepted Answer · 2014-04-04T07:02:00.547

8

The creation of an explicit function f can be completely avoided by using the 3rd argument to Function. There, you can specify what Attributes the anonymous function should have and therefore, you can enforce the same behavior as you did for f.

unflatten[l_, o_] := Module[{i = 1, l1 = Flatten[l]},
  Function[Null, l1[[i++]], {Listable}][o]
]

edited Apr 04 '14 at 07:02

answered Apr 04 '14 at 00:41

halirutan

112,764
7
263
474

That's very neat and works perfectly ! – Pisto Apr 04 '14 at 09:03

score 6 · Answer 2 · answered Apr 03 '14 at 17:29

Here is an alternative which is a bit shorter and closer to the original, and still completely free of this leak:

unflatten[l_, o_] :=
  Module[{res, f, i = 1, l1 = Flatten[l]},
    Block[{f},
       SetAttributes[f, Listable];
       f[_] := l1[[i++]];
       f[o]]]

rm -rf · Answer 3 · 2014-04-03T18:06:43.450

4

You can fix that with the following change:

unflatten[l_, o_] := 
    Module[{f, i = 1, l1 = Flatten[l]}, 
        SetAttributes[f, Listable]; 
        f[_] := l1[[i++]];
        With[{res = f[o]},
            Clear[f, l1];
            res
        ]
    ]

edited Apr 03 '14 at 18:06

answered Apr 03 '14 at 17:05

rm -rf

88,781
21
293
472

@LeonidShifrin Thanks, I've changed it, but why do you say it won't work (it worked in my test)? Isn't the Temporary attribute only necessary for automatic garbage collection? If I force erase all definitions, is it not the same? – rm -rf Apr 03 '14 at 17:08
It's not, because then this variable is not garbage -collected - Module only garbage - collects variables with attribute Temporary. You can make a simple experiment to verify that: Module[{f},f[1] = 1; Clear[f]], and then also try ClearAll, and monitor the existing symbols in Global` as you do this. – Leonid Shifrin Apr 03 '14 at 17:11
Actually, in this case, it seems to not work, no matter whether one uses Clear or ClearAll. I am a bit puzzled. – Leonid Shifrin Apr 03 '14 at 17:20
3

Ok, the statement Attributes[f]=Listable is the culprit. You also have to use instead SetAttributes[f,Listable], so that the Temporary is preserved. – Leonid Shifrin Apr 03 '14 at 17:24
This code reduced the number of these extra variables by a third in a test run but the number is still huge. SetAttributes[f,Listable], however, seem to do the trick. I feel lost and can anyone please explain what is going on? – Pisto Apr 03 '14 at 17:30
1

@Pisto Well, what happens is that the variable is only collected by Module if it has the attribute Temporary and it has no DownValues or SubValues (not sure about UpValues, and OwnValues are fine. There are also a couple of additional more subtle conditions, but they are not important for this case) at the moment when the execution leaves Module (and of course if the return value of Module is not some expression containing this local symbol). Now, originally, there were 2 problems: first, the code Attributes[f] = {Listable} removes the Temporary attribute ... – Leonid Shifrin Apr 03 '14 at 17:35
2

@Pisto ... This happens because it explicitly alters all set of attributes, which is one reason why this form is generally not advisable. SetAttributes fixes that, because it only adds Listable to existing ones, so Temporary is retained. Second, we need Clear at the end, to remove the DownValues which accumulated for f during the execution - then we fulfill both conditions and the symbol is garbage-collected. – Leonid Shifrin Apr 03 '14 at 17:37
@Pisto But have a look at my Block-based solution, I think it is a bit more elegant / higher level. It is also robust against exceptions / Aborts inside function's body, which is not an issue here but an issue in general. – Leonid Shifrin Apr 03 '14 at 17:38
@rm-rf Sorry for my stealing your check-mark with an aggressive marketing :) – Leonid Shifrin Apr 03 '14 at 17:47
@LeonidShifrin...Thanks. The Block-based solution seems to work perfectly. Even the original code is working well now when SetAttributes[f,Listable] is used (and even has two fewer variables) ! – Pisto Apr 03 '14 at 17:49
@LeonidShifrin No problem :) Also, thanks for noticing the issue with the attributes. I paid no attention to the original code! :D – rm -rf Apr 03 '14 at 18:07
@Pisto Actually, now that you mentioned this, it is indeed right. The subtle thing here is that such local symbols with DownValues are not collected only if they were referenced by some external symbols / expressions. In this case, they are not, so perhaps all this is an overkill and indeed, SetAttributes alone is sufficient. I have mentioned this also here – Leonid Shifrin Apr 03 '14 at 18:16

How to fix this memory leak

3 Answers3

Linked