2

I'm trying to launching my first jobs on LightWeight Grid using ParallelDo. I have 10000 jobs, each of which returns a 1M output, which I would like saved to a file on the head machine, one file per run. My basic challenge is that Mathematica saves by default to the slave node.

I have many solutions, none of which very good:

1-Save in a temp directory on slave, then collect the files at the end of the job. My best option so far, but not very elegant. I pass the file names as variables, save those to a file, and write a shell script to copy the files to the local machine.

2-Pass output as variables to the head node. The problem is that this fills up the memory of the master machine and saves nothing until the 10000 jobs are running. I can break the job into chunks by hand, but that's not efficient.

Is there a way to either:

-Pass the output as a stream-like object that can be saved on the fly, rather than at the end of the computation,

-Save efficiently from the slave to the master computer.

-easily collect all the output from the local temp directories?

Simon
  • 21
  • 1
  • Hi, welcome to Mathematica.SE, please consider taking the tour so you learn the basics of the site. Once you gain enough reputation by making good questions you will be able to vote up and down both questions and answers. When you see good ones, please vote them up by clicking the grey triangles, because the credibility of the system is based on the reputation gained by users sharing their knowledge. As you receive help, try to give it too, by answering questions in your area of expertise. – rhermans Dec 06 '14 at 10:24

1 Answers1

1

This isn't a perfect solution either, but it's an improvement on your solution 2:

Use ParallelSubmit and WaitNext for parallelization. Documentation is here and here. With this method it is not necessary to collect all results to the main kernel first, and export them only afterwards. You'll be able to collect one result, save it, collect the next, save that, etc.

Also, be aware of this bug which can cause the main kernel to run out of memory. See the workaround at that link.

Community
  • 1
Szabolcs
  • 234,956
  • 30
  • 623
  • 1,263