5

I have a code that works over lists like

S0; (*initial list*)
S1=f[S0];
S2=g[S1];
S3=f[S2];
S4=f[S3];
.
.
.

where f and g are functions that produces larger lists that consume a large amount of RAM memory. I thought that running

S0; (*initial list*)
S1=f[S0]; Clear[S0];
S2=g[S1]; Clear[S1];
S3=f[S2]; Clear[S2];
S4=f[S3]; Clear[S3];
.
.
.

would free the memory used from the kernel, but it is not the case. The consumption of both codes are exactly the same. I stored (using Save) the result for S10, for example, and after Get it (with a clear kernel) the memory consumption is much less than running the code, even using Clear in each step as above.

Is there a stronger version of Clear which eliminates the information of a symbol from the kernel? Or does Clear do it, but the task manager keeps showing the same memory usage from before?

David G. Stork
  • 41,180
  • 3
  • 34
  • 96
Filburt
  • 578
  • 5
  • 13
  • One thing to try is $HistoryLength -- change it to something much less than the default. – bill s Oct 04 '17 at 22:36
  • @bills Can I redefine $HistoryLength later? Like, if I aim to get S10 and then make other operations using it and other things, can I redefine to infinity after the computation of S10? – Filburt Oct 04 '17 at 23:13
  • @bills And it removes lines from the beginning? Is is possible to keep some definitions before start the string of calculations in the OP? – Filburt Oct 04 '17 at 23:23

1 Answers1

7

As @bills says, you can just set $HistoryLength=0, but that's rather draconian. Instead, just make sure that Out doesn't store your results. To do this, you need to use something like:

S0; "";

instead of something like:

S0; Null;

Let's compare the two:

S0; "";
S1 = f[S0]; ""; Clear[S0];
S2 = g[S1]; ""; Clear[S1];

DownValues[Out][[-3;;]]

{HoldPattern[%56] :> "", HoldPattern[%57] :> "", HoldPattern[%58] :> ""}

versus your:

S0;
S1 = f[S0]; Clear[S0];
S2 = g[S1]; Clear[S1];

DownValues[Out][[-3;;]]

{HoldPattern[%60] :> S0, HoldPattern[%61] :> f[S0], HoldPattern[%62] :> g[f[S0]]}

Notice how the "" version doesn't store any outputs, while your version does. It is the storage of the outputs that prevents you from recovering memory.

Update

A more normal method to avoid memory bloat is to use a Module. For example:

f[x_] := x^2
g[x_] := x+1

func[s_] := Module[{res = s},
    res = f[res];
    res = g[res];
    res = f[res];
    res = f[res];
    Total[res]
]

Now, create a large variable:

MemoryInUse[]
S0 = RandomReal[1,10^8];"";
MemoryInUse[]

151355496

951357168

Note the use of ;""; above so that Out doesn't store the value of S0. Run your function:

MemoryInUse[]
func[S0]
MemoryInUse[]

951358872

4.21584*10^8

951359968

Notice that MemoryInUse has stayed basically the same.

Finally, clear S0 and recover memory:

MemoryInUse[]
Clear[S0]
MemoryInUse[]

951651720

151652648

If we hadn't used ;""; above, the memory wouldn't have gone down.

Carl Woll
  • 130,679
  • 6
  • 243
  • 355