0

I want to create a large number of variables, store data in them and then DumpSave each variable in a separate .mx file.

Here is a toy version of the problem:

Do[varname = StringJoin["var", ToString[i]];
 ToExpression[StringJoin[varname, "=RandomReal[1,{10}];"]];, {i, 10}]

This creates variables var1,var2,..., var10 and stores random data in each of them.

Problem #1: How do I parallelize the above, as ParallelDo doesn't appear to work?

Problem #2: How do I save each variable in a correspondingly-named .mx file. So the variable var1 would be saved in var1.mx, var2 would be saved in var2.mx, etc.? Again, the main issue appears to be parallelization...

Note: creating an initial, large table using ParallelTable is not an option, due the size and number of files.

MMAUser
  • 399
  • 1
  • 6
  • Maybe if you use built-in functionalities, you won't need to worry about parallelization problems: How can I make assignments persist across sessions? – Roman Nov 05 '22 at 19:30
  • Parallelization is most useful when tasks require lots of computation, but the results themselves are compact, so that the overhead due to communication between kernels is small. Is that the situation you are in? If all you do is create variables and write to files, then maybe parallelization will not help. – user293787 Nov 06 '22 at 14:38
  • Also consider that your storage media (esp. hard drives) are inherently serial at the "write bits to disk" stage, so there is a hard limit to how much parallelization can help a certain number of physical storage devices based on when their writing speed is maxed out. – eyorble Nov 06 '22 at 17:23

0 Answers0