Share Large, Read-Only Numpy Array Between Multiprocessing Processes

If you are on Linux (or any POSIX-compliant system), you can define this array as a global variable. multiprocessing is using fork() on Linux when it starts a new child process. A newly spawned child process automatically shares the memory with its parent as long as it does not change it (copy-on-write mechanism).

Since you are saying “I don’t need any kind of locks, since the array (actually a matrix) will be read-only” taking advantage of this behavior would be a very simple and yet extremely efficient approach: all child processes will access the same data in physical memory when reading this large numpy array.

Don’t hand your array to the Process() constructor, this will instruct multiprocessing to pickle the data to the child, which would be extremely inefficient or impossible in your case. On Linux, right after fork() the child is an exact copy of the parent using the same physical memory, so all you need to do is making sure that the Python variable ‘containing’ the matrix is accessible from within the target function that you hand over to Process(). This you can typically achieve with a ‘global’ variable.

Example code:

from multiprocessing import Process
from numpy import random


global_array = random.random(10**4)


def child():
    print sum(global_array)


def main():
    processes = [Process(target=child) for _ in xrange(10)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()


if __name__ == "__main__":
    main()

On Windows — which does not support fork()multiprocessing is using the win32 API call CreateProcess. It creates an entirely new process from any given executable. That’s why on Windows one is required to pickle data to the child if one needs data that has been created during runtime of the parent.

Leave a Comment