4

I am trying to run code on a Lightweight Grid using ParallelDo. I have ~300 jobs that each take ~20 minutes to run and produce ~8 MB of data, and I am using a fairly unstable grid, so I would like to save the data produced in each job to a file. For simplicity, I am trying to use built-in parallelization features --- I would like to avoid having to manage the parallelization manually if possible. When I try to do this naïvely by simply exporting a file and hoping that it gets transferred to the master kernel, I find (unsurprisingly) that only the files generated on the local kernels are saved to my machine. Here is the minimal working example I am using to test:

makeFile[i_] := Module[{file},
    file = StringTemplate["file_`i`.txt"][<|"i" -> i|>];
    Export[file, "hello"];
]

ParallelDo[makeFile[i], {i, 1, 14}]

Is there a way to transfer the exported file to my local machine on the completion of each job? I have been unable to find anything helpful in the Mathematica documentation. I came across a related thread here, but I would prefer not to have to manage the parallelization directly using ParallelSubmit and WaitNext if possible.

book_kees
  • 255
  • 1
  • 6
  • 2
    I think this is more a question about the setup of the cluster you run the grid on. If there is a shared filesystem for all nodes in the lightweight grid then write to that using a full path for the files and your problem is solved. If such a shared filesystem does not exist, you will need to collect the results from the remote machines. You could do this with Mathematica, but I doubt it would be very efficient... – Albert Retey Dec 09 '19 at 13:00

0 Answers0