How to get Python multiprocessing module working on windows?

Question

I am trying to run a script within Blender that utilizes multiprocessing module. According to documentation it is possible to use it ( see note at the end of "Strange errors using ‘threading’ module" ).

Pythons threads only allow co-currency and won’t speed up your scripts on multi-processor systems, the subprocess and multiprocess modules can be used with Blender and make use of multiple CPU’s too..

So I tried running below script within Blender:

from multiprocessing import Pool

def test( v ):
    print(v)
    return v

p = Pool(2)

print( p.map(test, range(5) ) )

This however spawns as many new Blender instances (with separate windows) as there are workers in p (in my case 2). Also it spews in System Console :

Can't read file: "C:\Program Files\Blender Foundation\Blender\-c", Unable to open the file.
Can't read file: "C:\Program Files\Blender Foundation\Blender\from multiprocessing.forking import main; main()", Unable to open the file.
Can't read file: "C:\Program Files\Blender Foundation\Blender\--multiprocessing-fork", Unable to open the file.
Can't read file: "C:\Program Files\Blender Foundation\Blender\2888", Unable to open the file.

I also thought about using os.fork() to do the same thing, but sadly Windows does not support that.

I've been looking all over Internet, but so far I was not able to find even one example proving that documentation is correct.

The only thing I found was thread on blenderartists.org, but it is locked and with no responses.

---

Update: This is a windows spesific limitation in the multiprocessing module. ~ideasman42.

Answer 1

Edit, this workaround is no longer needed as of 2.91.

---

This looks like a kind of ms-windows specific problem... at least a conflict with having an embedded Python interpreter and the environment which multiprocessing expects. (its not exactly a bug, in that its not like Blender devs can fix some mistake to get this working).

Currently multiprocessing makes the assumption that its running in python and not running inside an application.

---

I managed to get multi-processing working on ms-windows, doing some workarounds.

• sys.executable needs to point to Python executable.
• The scripts __file__ needs to point to a file on-disk
  (not always the case - when executing a text block for example).

Heres an example of a workaround:

import sys
# are we running inside Blender?
bpy = sys.modules.get("bpy")
if bpy is not None:
    sys.executable = bpy.app.binary_path_python
    # get the text-block's filepath
    __file__ = bpy.data.texts[__file__[1:]].filepath
del bpy, sys
end workaround!
--------------------------------------------------
Example from
https://docs.python.org/3/library/multiprocessing.html
from multiprocessing import Process
import os
def info(title):
    print(title)
    print('module name:', name)
    if hasattr(os, 'getppid'):  # only available on Unix
        print('parent process:', os.getppid())
    print('process id:', os.getpid())
def f(name):
    info('function f')
    print('hello', name)
def main():
    info('main line')
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()
if name == 'main':
    main()

Answer 2

I think this statement in the same docs says a lot:

"So far, no work has gone into making Blender’s python integration thread safe, so until its properly supported, best not make use of this. "

The statement that subprocess and multiprocess can be used with Blender isn't false, just may be interpreted in reverse with regards to multiprocess module.

• subprocess can call external apps fine with .call or .Popen, non-blocking or blocking. Works as expected, threaded or non threaded.
• multiprocess probably doesn't work (right now) as you might expect, how I interpreted this statement originally was, "you can call several instances of Blender from a python program using multiprocessing, that way you have control over what ends when and can use multiple CPU's.

In the end the warning is reasonable clear (I think?) that we aren't to expect threaded execution of calls to Blender functions within the one Blender executable. (for instance function calls that take a lot of time, like baking.. it is possible to write scripts to do all that and finally when all blender processes have ended , merge the result in an additional script. The work (scripting wise) may be a little bit more, but that seems to be where it's at at the moment.

Ubuntu 14.04
On Ubuntu that code leaves headless processes behind, which only finish when the main Blender executable is terminated. If there's a problem it's not entirely localized to windows.

Answer 3

I entirely forgot I asked that question, but since then I've worked out a workaround for the issue.

I use Blender as video editor (even though it is not its primary purpose it works really well for my needs while still being free).

I ended up creating and add-on script that will launch multiple instances of Blender to render separate sections of my video thus utilising multiple cores in my CPU.

Full solution and actual add-on with that functionality can be seen here:

https://github.com/elmopl/ktba/blob/master/scripts/addons/parallel_render.py

It's not perfect in how it is coded, but it works fine. I simply save my current project's state into temporary .blend file and then launch multiple headless Blender instances (using subprocess module) that load that temporary project and launch into main() of my add-on. That main() is responsible for handling communication with main instance and performing render operation of headless instances.

It seems to work quite well.

Usage of addon documented here: https://github.com/elmopl/ktba/wiki/Addons#parallel-render

That solution could be split and wrapped up into separate module that does purely subprocess call + handles communication layer. Then it could be reused by others.

Answer 4

... I used python 3.9.4(64 bits) in Windows 10 the following code:

from multiprocessing import Pool
def test( v ):
    print(v)
    return v
if __name__ == '__main__':
    with Pool(2) as p:
        print(p.map(test, range(5)))

.... and at run-time obtain this output results:

0
1
2
3
4
[0, 1, 2, 3, 4]