numpy를 가져온 후 멀티 프로세싱에서 단일 코어 만 사용하는 이유는 무엇입니까?

numpy를 가져온 후 멀티 프로세싱에서 단일 코어 만 사용하는 이유는 무엇입니까?

---

이것이 OS 문제로 간주되는지 확실하지 않지만 누군가 파이썬 끝에서 통찰력이있는 경우 여기에 물어볼 것이라고 생각했습니다.

을 for사용하여 CPU가 많은 루프 를 병렬화하려고 joblib했지만 각 작업자 프로세스가 다른 코어에 할당되는 대신 모든 코어가 동일한 코어에 할당되고 성능이 향상되지 않는 것으로 나타났습니다.

다음은 아주 간단한 예입니다.

from joblib import Parallel,delayed
import numpy as np

def testfunc(data):
    # some very boneheaded CPU work
    for nn in xrange(1000):
        for ii in data[0,:]:
            for jj in data[1,:]:
                ii*jj

def run(niter=10):
    data = (np.random.randn(2,100) for ii in xrange(niter))
    pool = Parallel(n_jobs=-1,verbose=1,pre_dispatch='all')
    results = pool(delayed(testfunc)(dd) for dd in data)

if __name__ == '__main__':
    run()

... htop이 스크립트가 실행 되는 동안 내가 본 내용은 다음과 같습니다.

4 개의 코어가있는 랩톱에서 Ubuntu 12.10 (3.5.0-26)을 실행 중입니다. 분명히 joblib.Parallel다른 작업자를 위해 별도의 프로세스를 생성하지만 이러한 프로세스를 다른 코어에서 실행할 수있는 방법이 있습니까?

---

After some more googling I found the answer here.

It turns out that certain Python modules (numpy, scipy, tables, pandas, skimage...) mess with core affinity on import. As far as I can tell, this problem seems to be specifically caused by them linking against multithreaded OpenBLAS libraries.

A workaround is to reset the task affinity using

os.system("taskset -p 0xff %d" % os.getpid())

With this line pasted in after the module imports, my example now runs on all cores:

My experience so far has been that this doesn't seem to have any negative effect on numpy's performance, although this is probably machine- and task-specific .

Update:

There are also two ways to disable the CPU affinity-resetting behaviour of OpenBLAS itself. At run-time you can use the environment variable OPENBLAS_MAIN_FREE (or GOTOBLAS_MAIN_FREE), for example

OPENBLAS_MAIN_FREE=1 python myscript.py

Or alternatively, if you're compiling OpenBLAS from source you can permanently disable it at build-time by editing the Makefile.rule to contain the line

NO_AFFINITY=1

---

Python 3 now exposes the methods to directly set the affinity

>>> import os
>>> os.sched_getaffinity(0)
{0, 1, 2, 3}
>>> os.sched_setaffinity(0, {1, 3})
>>> os.sched_getaffinity(0)
{1, 3}
>>> x = {i for i in range(10)}
>>> x
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
>>> os.sched_setaffinity(0, x)
>>> os.sched_getaffinity(0)
{0, 1, 2, 3}

---

This appears to be a common problem with Python on Ubuntu, and is not specific to joblib:

• Both multiprocessing.map and joblib use only 1 cpu after upgrade from Ubuntu 10.10 to 12.04
• Python multiprocessing utilizes only one core
• multiprocessing.Pool processes locked to a single core

I would suggest experimenting with CPU affinity (taskset).

참고 URL : https://stackoverflow.com/questions/15639779/why-does-multiprocessing-use-only-a-single-core-after-i-import-numpy