15

I am executing a loop, and in each loop I have to store a parameter in a file. It is like

file = FileNameJoin[{NotebookDirectory[], "info.txt"}];
OpenWrite[file];
Do[out = Prime[i]; WriteString[file, i, "\t", out, "\n"], {i, 1, 10}]
Close[file]

It stores the values perfectly. But when I use Parallelize[], it produce only partial output.

file = FileNameJoin[{NotebookDirectory[], "infop.txt"}];
OpenWrite[file];
Do[out = Prime[i]; WriteString[file, i, "\t", out, "\n"], {i, 1, 10}]
Close[file]

It produces only 4-5 sets out of 10 (They are in random ordering, but that is not a problem). Although when I use Print[], it shows all the sets.

Parallelize[Do[out = Prime[i]; Print[i , "\t", out], {i, 1, 10}]]

The loop I am using for work is pretty big and I may have to check the info.txt file several times to know the status while the code is running. Is there any way to write the data (as the code is running) in a common file while using parallel processing.

Sumit
  • 15,912
  • 2
  • 31
  • 73

2 Answers2

10

Interaction between parallel kernels

This is a common parallelisation problem. Without proper blocking you'll produce such results, but with blocking you loose all your parallelisation. So you've to come up with a solution with at least interaction between the parallel sub-kernels as possible. The less interaction, the faster things will go.

A solution is to use a shared variable to communicate results found to the master kernel:

file = FileNameJoin[{$TemporaryDirectory, "info.txt"}];
OpenWrite[file];

SetSharedVariable[list]
list = {};
ParallelDo[out = Prime[i]; AppendTo[list, {i, out}], {i, 1, 10}]

list = Sort[list]; (* only if you care about order *)

WriteString[file, #[[1]], "\t", #[[2]], "\n"] & /@ list;

Close[file]

If you want to, you could aggregate this into a function, but then you've to take care about the distributed definition of your function to the parallel kernels. Not if you define your function in the current context, because those are distributed automatically.

Edit:

I do now parallelisation for years (not necessarily in Mma) and suffered a lot to get things real going, but what I learned from my experiences is:

  1. Explore all serial ways to speed things up, before you go parallel
  2. If you go parallel beware of synchronisation, because this is killing parallelism

Edit 2:

This is maybe not the most elegant solution to merge files in a directory, but it does the job. Imagine you have several files with data in it in a flat directory. For instance:

  • /tmp/test/test1.txt -> "Hello"
  • /tmp/test/test2.txt -> "World"

You can read all the files in the directory into a list:

strList = Read /@ OpenRead /@ (SetDirectory["/tmp/test"]; 
    FileNameJoin[{"/tmp/test", #}] & /@ FileNames[]);

And now you can OpenWrite and write to that channel mapping over strList and you're done.

This is all, if you don't want to leave your tool (like me; probably emacs affected ;) )

Stefan
  • 5,347
  • 25
  • 32
  • I guess you are suggesting to store the output in a variable and collect it after compiling the whole loop. like, file = FileNameJoin[{NotebookDirectory[], "info3.txt"}]; ParallelDo[out1[i] = {i, Prime[i]}, {i, 1, 10}] Export[file, Table[out1[i], {i, 10}], "Table"] But then I won't be able to know the status or access the evaluated data at some particular moment. If I can write it in a file while the code is running then I can use the partial data even before the complete execution of all the loop in a long run. – Sumit Jun 18 '13 at 13:39
  • 1
    You can do that, but then you have all the issues that I tried to list. Why just don't open several files and after processing is done merge them? If you really need a shared file (presumably because you write and another process reads the written result and does some further calculation), then you should use a blocking call like Print[]. – Stefan Jun 18 '13 at 14:01
  • Another possibility is to use WaitNext[] and/or WaitAll[] in conjunction with ParallelSubmit. So the writing process will be sugmitted and the reading process waits until finished. But as you might realize, this is hardly parallism. – Stefan Jun 18 '13 at 14:07
  • one solution for file backed list is this bright implementation of @Leonid http://mathematica.stackexchange.com/questions/36/file-backed-lists-variables-for-handling-large-data/209#209 Maybe this helps you, to divide your workload in parallel tasks and serial ones... – Stefan Jun 18 '13 at 14:12
  • Thanks @Stefan. Indeed I tried to create and merge individual file. fol = NotebookDirectory[]; ParallelDo[out = {{i, Prime[i]}}; Export[StringJoin[{fol, "/", ToString[i], ".txt"}], out, "Table"], {i, 1, 10}]; Do[dat[i] = Import[StringJoin[{fol, "/", ToString[i], ".txt"}]], {i,1, 10}]; and store all the values in dat[]. But then if a particular file is missed (say 7.txt is yet to be born) I get trouble . BTW is there any command to merge all the data files of a specified folder (like cat in unix)? – Sumit Jun 18 '13 at 15:30
  • @Sumit sorry for my late answer, but the sun and a beergarden are two good friends ;). What you could do, if you want to stay in Mma is that you Import the directory and Join all the files to one. Look here http://mathematica.stackexchange.com/questions/14902/how-to-batch-rename-a-directory-of-files and here http://mathematica.stackexchange.com/questions/17365/visualizing-directories – Stefan Jun 19 '13 at 07:46
5

I would just write to separate files:

ParallelEvaluate[file = OpenWrite[ToString[$KernelID] <> ".txt"]]
ParallelDo[out = Prime[i]; 
 WriteString[file, i, "\t", out, "\n"], {i, 1, 10}]
ParallelEvaluate[Close@file]
Ajasja
  • 13,634
  • 2
  • 46
  • 104
  • this can't work, since the returned OpenStream will have all the same serial number, which is nothing but a file handle. Opening several files with all the same file handle, can't work. – Stefan Jun 19 '13 at 09:55
  • 1
    @Stefan Sure it works. Why don't you try and see? The file variable is local to each kernel, so no problems there. – Ajasja Jun 19 '13 at 10:17
  • yes it does. i'm sorry for the inconvenient comment. had some symbols defined in my session, which resulted in weird messages...but as i wrote in my post, i wouldn't do it that way, since i/o is expensive and not needed in this particular case. – Stefan Jun 19 '13 at 11:03
  • @Stefan Well, this depends on many factors: Operating system file buffering, amount of data written, etc... This way one can monitor the progress of calculation or restart in case of crashes. – Ajasja Jun 19 '13 at 11:15