9

I am running into an issue where HermiteDecomposition seems to leak memory, but only from certain matrices

RandomSeed[1];
M = RandomInteger[{0, 1}, {33, 40}];
Table[HermiteDecomposition[M]; MemoryAvailable[], {i, 0, 100}]

This gives a gradual decline in memory available to the operating system, by about 1 megabyte per call to HermiteDecomposition

{10629611520, 10628149248, 10627117056, ... 10524651520, 10523619328}

These seems high considering M is 10 kilobytes in size; intermediate steps probably involve large integers, and these do not get freed.

If the above random seed does not work to reproduce, the following matrix is the same as what I have:

M={{1,0,0,0,0,1,1,1,0,0,0,0,1,1,1,1,0,1,1,0,1,0,0,1,0,0,0,0,0,0,0,0,1,0,0,1,0,1,0,1},{0,0,1,0,1,0,1,0,0,1,1,1,1,0,1,0,1,1,0,0,0,1,0,1,0,0,1,1,0,0,1,0,0,1,1,0,1,0,0,0},{1,1,1,1,0,1,1,1,1,0,0,0,1,1,0,0,0,1,0,1,1,0,0,1,1,1,0,0,1,1,0,0,1,0,0,0,1,1,1,1},{1,0,0,0,0,1,1,1,1,1,0,1,0,0,0,0,1,1,0,0,1,0,1,0,1,1,0,1,0,1,0,1,0,1,0,0,1,0,0,0},{1,0,0,0,1,1,0,0,1,0,1,1,1,0,1,1,1,1,0,1,1,1,1,1,1,1,1,0,0,1,1,1,1,1,1,0,1,1,1,0},{0,1,0,1,1,0,1,1,1,0,0,1,1,0,1,0,0,0,1,0,0,0,1,1,0,1,1,1,1,1,0,0,1,0,1,1,0,1,1,0},{1,0,1,0,1,0,1,1,0,0,0,1,0,1,0,1,1,1,1,0,0,1,0,0,1,1,1,1,0,0,0,1,1,1,0,0,0,1,1,1},{1,1,1,0,0,1,0,1,0,1,1,1,0,0,1,0,0,1,0,1,1,1,0,0,1,1,1,0,1,1,1,1,1,0,1,1,0,1,1,1},{0,0,0,0,0,0,1,0,0,1,0,1,0,0,0,1,1,1,0,1,0,0,0,0,1,0,0,0,1,1,0,1,0,1,1,1,1,1,1,0},{0,1,0,0,0,1,0,1,1,1,1,1,0,0,0,1,1,1,1,1,0,1,1,1,1,1,1,0,0,0,0,0,0,0,1,0,0,0,0,0},{1,0,0,0,0,1,0,0,0,0,0,1,1,1,0,0,0,1,1,1,0,0,0,1,0,0,1,1,0,1,0,1,0,1,0,0,0,1,0,1},{1,1,0,1,1,1,0,0,1,0,0,1,0,1,0,1,0,0,1,1,1,1,1,0,0,0,1,1,0,0,1,0,0,0,1,1,0,0,0,0},{0,0,1,0,1,0,1,0,1,1,0,1,0,0,0,0,1,0,1,1,0,1,0,1,1,0,1,1,0,1,1,1,1,0,1,1,0,0,1,0},{1,0,1,0,1,1,0,1,0,0,0,1,0,1,1,1,1,0,1,1,0,0,1,0,0,0,1,1,1,1,1,0,0,0,1,0,1,0,1,0},{1,0,0,1,1,1,0,1,1,0,0,1,1,1,0,0,0,0,1,0,0,0,0,1,1,0,1,0,0,1,0,1,0,1,1,1,0,1,1,1},{0,0,1,0,0,0,1,0,0,1,0,0,0,0,1,1,0,0,0,0,1,1,1,1,1,1,1,0,0,0,0,1,0,1,1,0,1,0,1,0},{0,0,0,0,0,0,0,0,0,0,0,1,0,1,1,0,0,1,0,0,0,1,1,0,0,0,0,0,1,1,0,1,1,0,0,1,0,1,1,1},{0,1,0,1,1,0,0,1,1,1,1,1,1,1,0,1,1,0,1,1,1,0,0,1,1,1,0,0,1,1,1,1,0,1,0,1,1,1,1,1},{0,0,1,1,1,1,0,1,1,1,0,1,0,0,0,0,1,0,0,1,0,0,0,1,0,0,1,0,0,0,0,1,1,0,1,0,1,1,1,0},{0,0,1,1,1,1,0,0,0,1,1,0,1,1,1,0,1,0,0,0,1,1,1,1,0,0,0,1,1,0,0,0,1,1,1,1,1,1,0,1},{0,0,1,1,0,0,0,0,1,1,0,1,1,1,1,1,0,0,0,0,0,1,0,1,1,1,1,1,1,0,0,1,0,1,1,0,1,1,0,0},{0,0,0,0,1,1,1,0,1,0,0,0,0,0,1,1,1,1,0,1,0,0,1,1,1,0,1,0,0,0,0,0,0,1,1,0,0,0,0,0},{1,0,0,0,0,1,0,1,0,0,1,1,1,1,1,1,0,1,0,0,0,1,0,0,0,0,0,1,1,1,0,0,1,1,0,0,0,0,0,0},{0,1,1,0,0,0,1,1,0,1,0,1,0,0,0,1,0,1,0,1,0,1,0,0,0,0,1,1,0,0,1,0,1,0,1,1,0,0,0,0},{0,1,1,0,0,1,1,1,0,1,0,0,1,1,1,1,0,1,1,0,0,0,1,0,0,1,0,1,1,0,1,1,0,0,1,0,0,0,0,0},{0,0,1,1,0,1,0,1,0,1,1,0,0,1,1,0,0,0,1,0,1,1,1,1,1,0,1,0,0,1,1,1,1,0,1,0,1,0,0,1},{0,1,1,0,0,0,0,1,1,0,0,0,0,1,0,1,0,1,0,0,0,0,0,1,1,1,0,1,0,1,0,0,0,0,1,1,0,0,1,0},{0,0,1,0,1,1,0,1,0,0,1,1,0,1,0,1,1,1,1,0,0,1,0,1,0,1,1,0,1,1,1,1,1,0,1,0,0,1,1,0},{0,0,0,1,1,1,1,1,1,1,0,1,0,1,1,1,1,1,0,0,1,1,1,1,1,1,1,1,0,0,1,1,0,0,1,0,0,1,1,0},{0,1,0,1,1,1,1,1,1,0,0,0,1,0,0,0,0,0,1,1,0,0,0,1,1,0,1,1,1,0,0,0,0,1,0,1,0,1,1,0},{1,0,1,1,0,0,1,1,1,0,0,1,1,0,1,0,0,0,1,1,0,1,0,0,0,1,0,0,0,1,0,1,0,1,0,0,1,1,0,1},{0,0,0,0,1,0,0,1,0,0,0,0,0,0,0,0,1,0,1,1,1,0,0,1,0,1,1,1,1,1,0,1,0,1,0,0,1,1,0,0},{1,0,0,0,0,1,1,1,1,1,1,0,1,1,0,1,1,1,1,0,1,1,1,1,0,0,0,0,1,0,0,0,1,1,0,1,1,0,0,1}};

A few details:

  • Killing the kernel frees the memory
  • ClearSystemCache[] does not help
  • Putting it in a For loop instead of a Table did not help either.
  • The same leak also occurs with SmithDecomposition, which is similar to HermiteDecomposition, so they might share some code.

The leak also occurs with RowReduce on a different set of matrices, which led me to Memory Leak in RowReduce?, but that focuses more on identifying the bug, while I want to work around it.

The memory leak is much larger for bigger matrices, so it forces me to restart Mathematica about every 30 minutes for a set of calculations that use HermiteDecomposition on various trials.

Is there any way to avoid having to restart Mathematica by hand so I could do overnight trials? I could rewrite HermiteDecomposition by hand, but that seems prone to error. Maybe the kernel could be restarted and commands run automatically, but that seems unlikely since the kernel would be stopped at that point.

Alexey Popkov
  • 61,809
  • 7
  • 149
  • 368
fireflame241
  • 291
  • 1
  • 4

1 Answers1

10

Thanks to Mathematica Support, I have a workaround for the issue.

Running the following code avoids the leak entirely:

d = Max[Dimensions[M]];
SetSystemOptions[
  "LinearAlgebraOptions" -> {"IntegerBlasParallelizationThresholds" -> {2*d, 2*d, 8}}
];
fireflame241
  • 291
  • 1
  • 4