How do you share data between a parent and forked child process in Python?
Question
I'm pretty sure one would do this using the os.plock(op) function, but I have no idea how. Also, if there's a better way, I'd be grateful to find out. Code snippets are very welcome.
Solution
Subprocess replaces os.popen, os.system, os.spawn, popen2 and commands. A simple example for piping would be:
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
You could also use a memory mapped file with the flag=MAP_SHARED for shared memory between processes.
multiprocessing abstracts both pipes and shared memory and provides a higher level interface. Taken from the Processing documentation:
from multiprocessing import Process, Pipe
def f(conn):
conn.send([42, None, 'hello'])
conn.close()
if __name__ == '__main__':
parent_conn, child_conn = Pipe()
p = Process(target=f, args=(child_conn,))
p.start()
print parent_conn.recv() # prints "[42, None, 'hello']"
p.join()
OTHER TIPS
Take a look at the multiprocessing module new in python 2.6 (also available for earlier versions a pyprocessing
Here's an example from the docs illustrating passing information using a pipe for instance:
from multiprocessing import Process, Pipe
def f(conn):
conn.send([42, None, 'hello'])
conn.close()
if __name__ == '__main__':
parent_conn, child_conn = Pipe()
p = Process(target=f, args=(child_conn,))
p.start()
print parent_conn.recv() # prints "[42, None, 'hello']"
p.join()
This is pretty much Python-independent! It's a classic example of Unix interprocess communication. One good option is to use popen()
to open a pipe between the parent and child processes, and pass data/messages back and forth along the pipe.
Take a look at the subprocess
module, which can set up the necessary pipes automatically while spawning child processes.
You have two options: os.popen*
in the os
module, or you can use the subprocess
module to the same effect. The Python manual has pretty documentation and examples for popen and subprocess.
If you are doing low-level operating system forking and really want to avoid using pipes, it is possible to use shared memory-mapped files as well. This is not nearly as nice as using subprocess
or popen
pipes, but including the answer for completeness...
There's a full example here, but basically you can combine the os file handling and mmap modules:
import mmap, os, tempfile
fd, tmpfile = tempfile.mkstemp()
os.write(fd, '\x00' * mmap.PAGESIZE)
os.lseek(fd, 0, os.SEEK_SET)
child_pid = os.fork()
if child_pid:
buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_READ)
os.waitpid(child_pid, 0)
child_message = buf.readline()
print(child_message)
os.close(fd)
else:
buf = mmap.mmap(fd, mmap.PAGESIZE, mmap.MAP_SHARED, mmap.PROT_WRITE)
buf.write('testing\n')
os.close(fd)