On posix systems, multiprocessing
works by using the fork()
syscall, which clones the running process, and all of its state (including things like instances of complex classes, such as pycurl.Curl
).
Windows does not have fork()
, or anything like it, so multiprocessing
fires up a new python interpreter for each child process, with a special stub function that listens to the parent process and recreates the state so that it looks much like it would had fork()
been used. The key technique used by multiprocessing
for this is that it recreates each object in the child process, from the parent process, as they are used. This works by converting the objects to a bytecode representation (using the pickle
module), sending them over a pipe to the child, and converting them back to python objects once there.
For most kinds of python objects, this works perfectly, and is transparent to you. There are a number of obvious kinds of things where this cannot possibly work; open files cannot be passed around in this way; nor can the state of non-python library objects, which don't know anything about the pickle
system. Unfortunately, pycurl
is a bit of both of these things.