9

I have some code that process very large flat files in to separate HSQL(Standalone) databases using ParallelSubmit and JDBC for each file using the strategy in this post (19542). Lately I keep getting a Java error GC overhead limit exceeded. I found this post (2391189) with comments that suggest that the issue could be more than one program running in the Java VM.

I tried running a set of two files in series and they completed whereas the ParallelSubmit approach fails for them. Therefore I would like to run a separate Java VM on each sub-Kernel so that this issue is avoided.

I am already running ReinstallJava[JVMArguments -> "-Xms500m -Xmx6g"]; in the front end so I do believe it is the interaction and not memory.

How do I launch a separate Java VM on each subKernel?

In 19542 this would be creating a Java VM at the start of loadBigFile and then dropping it at the end of the function. Unless, of course, there is a better way to do this.

Edmund
  • 42,267
  • 3
  • 51
  • 143
  • While it is possible that there is some issue accessing JLink from multiple subkernels, I'm inclined to agree with the comments in the linked SO post which suggest there is at least one run-away process. JVisualVM could be used to view the heap and see what is clogging it up. If the only use of Java is JDBC, you might want to ensure that the subkernel tasks are releasing the JDBC resources in a timely fashion (e.g. CloseSQLConnection, SQLResultSetClose, etc). – WReach Jan 10 '18 at 00:38
  • @WReach It is only SQLInsert on separate HSQL(Standalone) databases. The function creates the database for its file and connections are opened and closed correctly. Each file takes between 15 to 30 mins to read into the database with 10's of millions of rows. It has been running fine until recently. – Edmund Jan 10 '18 at 01:16

1 Answers1

7

I opened a case with WRI ([CASE:3994742]) where they provided a supported solution.

A Java process can be launched on a sub-kernel by calling ReinstallJava on the sub-kernel. You need Jlink` loaded on the main kernel before loading on the sub-kernels.

Needs["JLink`"];

k = LaunchKernels[2];

ParallelEvaluate[
 Needs["JLink`"];
 ReinstallJava[JVMArguments -> "-Xms500m -Xmx2g"];
 , k
 ]

This starts two new Java processes; one for each of the sub-kernels. A warning was given that you need to keep an eye on your machine's resources. You can limit each process with JVMArguments as I have above; 500MB initial heap with a max of 2GB.

Closing the sub-kernels also closes the Java processes.

CloseKernels[k];

Hope this helps.

Edmund
  • 42,267
  • 3
  • 51
  • 143