When I run Python's multiprocessing Pool with main environment, I get the expected output i.e. time is reduced due to parallel processing.
But when I run the same code without main enviroment, it just throws error
from multiprocessing import Pool
import os
import time
def get_acord_detected_json(page_no):
time.sleep(5)
return page_no*page_no
def main():
n_processes = 2
page_num_list = [1,2,3,4,5]
print("n_processes : ", n_processes)
print("page_num_list : ", page_num_list)
print("CPU count : ", os.cpu_count())
t1 = time.time()
with Pool(processes=n_processes) as pool:
acord_js_list = pool.map(get_acord_detected_json, page_num_list)
print("acord_js_list : ", acord_js_list)
t2 = time.time()
print("t2-t1 : ", t2-t1)
if __name__=="__main__":
main()
Output :
n_processes : 2
page_num_list : [1, 2, 3, 4, 5]
CPU count : 8
acord_js_list : [1, 4, 9, 16, 25]
t2-t1 : 15.423236846923828
But when I do
main()
instead of
if __name__=="__main__":
main()
I get non-stopping error logs(crash logs)
When Python launches a secondary multiprocessing.Process, it imports the script again in each Process's space. (I use Windows and this is always true, but according to the documentation it is not necessarily the case on other OS's.) This even applies to process Pools, which is what you are using.
During these extra imports, the __name__ local variable is something other than __main__. So you can separate code that you want to run in every child Process from code that you want to run only once. That is the purpose of the if __name__ == "__main__" statement.
When you run your script with this statement omitted, the module is loaded again and again, in every one of the child processes. Every process tries to run the function main(), which then tries to launch more child processes, which tries to launch more, and so on. The whole thing crashes, as you observe. With the line of code present, the main() function runs only once, in the main Process where it works properly, launching all the other Processes.
I'm afraid you are stuck writing that line of code. Life is full of unpleasant necessities. But it's probably less disruptive than switching everything to another operating system.
See also python multiprocessing on windows, if __name__ == "__main__"
Unfortunately, the standard python library docs present this issue as a commandment that must be obeyed rather than an explanation that can be understood (in my opinion).
Related
test1.py/myfunc1() does some work in parallel.
If I call myfunc1() from test2.py - it works fine (currently commented out).
If I create another pool in test2.py and call myfunc1() from those I get an unreported error in test1.py on the "pool = mp.Pool(5)" line .
result = {type} <class 'AssertionError'> args = {getset_descriptor}
<attribute 'args' of 'BaseException' objects>
How do I fix this issue?
test1.py
import time
import multiprocessing as mp
def worker(a):
print("Worker: "+str(a))
time.sleep(5)
return a
def mycallback(val ):
print("Callback: "+str(val))
def myfunc1(n=3):
print("start myfunc1")
slist = range(n)
pool = mp.Pool(5)
[pool.apply_async(worker,args=(s,), callback=mycallback) for s in slist]
pool.close()
pool.join()
if __name__ == "__main__":
myfunc1()
test2.py
from pythonProjectTEST.test1 import myfunc1
import multiprocessing as mp
def mycallback(val ):
print("CallbackMaster: "+str(val))
if __name__ == "__main__":
# This works
#myfunc1(5)
# This does not
slist = range(6)
pool = mp.Pool(3)
[pool.apply_async(myfunc1,args=(s,), callback=mycallback) for s in slist]
pool.close()
pool.join()
You are not allowed to spawn a daemon process from another daemon process. Note how the main of test2 spawns processes to call myfunc1 and then myfunc1 spawns processes to call worker. I suspect this restriction is to reduce the chances of fork bombs or deadlocks. If you really want to do this, there are workarounds: Python Process Pool non-daemonic?. However, I would avoid it if possible.
To debug an issue like this, it is often convenient to add an error callback. For example, the following code gives you a helpful error message "Error: daemonic processes are not allowed to have children":
def errorcallback(val):
print("Error: %s" % str(val))
...
[pool.apply_async(myfunc1,args=(s,), callback=mycallback, error_callback=errorcallback ) for s in slist]
The method apply_async will normally eat errors unless you specify an error_calblack (see the documentation here). The arguments in square brackets are optional, but you can add them one by one with the specified names.
apply_async(func[, args[, kwds[, callback[, error_callback]]]])
"If error_callback is specified then it should be a callable which accepts a single argument. If the target function fails, then the error_callback is called with the exception instance."
I have a list of jobs but due to certain condition not all of the jobs should run in parallel at the same time because sometimes it is important that a finishes before I start b or vice versa (actually its not important which one runs first just not that they run both at the same time) so i thought i keep a list of the currently running threads and when ever a new on starts it checks in this list of currently running threads if the thread can proceed or not. I wrote some sample code for that:
from time import sleep
from multiprocessing import Pool
def square_and_test(x):
print(running_list)
if not x in running_list:
running_list = running_list.append(x)
sleep(1)
result_list = result_list.append(x**2)
running_list = running_list.remove(x)
else:
print(f'{x} is currently worked on')
task_list = [1,2,3,4,1,1,4,4,2,2]
running_list = []
result_list = []
pool = Pool(2)
pool.map(square_and_test, task_list)
print(result_list)
this code fails with UnboundLocalError: local variable 'running_list' referenced before assignment so i guess my threads don't have access to global variables. Is there a way around this? If not is there another way to solve this problem?
I try to find a simple way to "speed up" simple functions for a big script so I googled for it and found 3 ways to do that.
but it seems the time they need is always the same.
so what I am doing wrong testing them?
file1:
from concurrent.futures import ThreadPoolExecutor as PoolExecutor
from threading import Thread
import time
import os
import math
#https://dev.to/rhymes/how-to-make-python-code-concurrent-with-3-lines-of-code-2fpe
def benchmark():
start = time.time()
for i in range (0, 40000000):
x = math.sqrt(i)
print(x)
end = time.time()
print('time', end - start)
with PoolExecutor(max_workers=3) as executor:
for _ in executor.map((benchmark())):
pass
file2:
#the basic way
from threading import Thread
import time
import os
import math
def calc():
start = time.time()
for i in range (0, 40000000):
x = math.sqrt(i)
print(x)
end = time.time()
print('time', end - start)
calc()
file3:
import asyncio
import uvloop
import time
import math
#https://github.com/magicstack/uvloop
async def main():
start = time.time()
for i in range (0, 40000000):
x = math.sqrt(i)
print(x)
end = time.time()
print('time', end - start)
uvloop.install()
asyncio.run(main())
every file needs about 180-200 sec
so i 'can't see' a difference.
I googled for it and found 3 ways to [speed up a function], but it seems the time they need is always the same. so what I am doing wrong testing them?
You seemed to have found strategies to speed up some code by parallelizing it, but you failed to implement them correctly. First, the speedup is supposed to come from running multiple instances of the function in parallel, and the code snippets make no attempt to do that. Then, there are other problems.
In the first example, you pass the result benchmark() to executor.map, which means all of benchmark() is immediately executed to completion, thus effectively disabling parallelization. (Also, executor.map is supposed to receive an iterable, not None, and this code must have printed a traceback not shown in the question.) The correct way would be something like:
# run the benchmark 5 times in parallel - if that takes less
# than 5x of a single benchmark, you've got a speedup
with ThreadPoolExecutor(max_workers=5) as executor:
for _ in range(5):
executor.submit(benchmark)
For this to actually produce a speedup, you should try to use ProcessPoolExecutor, which runs its tasks in separate processes and is therefore unaffected by the GIL.
The second code snippet never actually creates or runs a thread, it just executes the function in the main thread, so it's unclear how that's supposed to speed things up.
The last snippet doesn't await anything, so the async def works just like an ordinary function. Note that asyncio is an async framework based on switching between tasks blocked on IO, and as such can never speed CPU-bound calculations.
I am using multiprocessing in my code to do somethings parallel. Actually in a simple version of my goal, I want to change some global variables by two different processes in parallel.
But in the end of the code running, the result which is getting from mp.Queue is true but the variables are not changed.
here is a simple version of code:
import multiprocessing as mp
a = 3
b = 5
# define a example function
def f(length, output):
global a
global b
if length==5:
a = length + a
output.put(a)
if length==3:
b = length + b
output.put(b)
if __name__ == '__main__':
# Define an output queue
output = mp.Queue()
# Setup a list of processes that we want to run
processes = []
processes.append(mp.Process(target=f, args=(5, output)))
processes.append(mp.Process(target=f, args=(3, output)))
# Run processes
for p in processes:
p.start()
# Exit the completed processes
for p in processes:
p.join()
# Get process results from the output queue
results = [output.get() for p in processes]
print(results)
print ("a:",a)
print ("b:",b)
And the blow is the answers:
[8, 8]
a: 3
b: 5
How can I apply the results of processes to the global variables? or how can I run this code with multiprocessing and get answer like running a simple threat code ?
When you use Threading, the two (or more) threads are created within the same process and share their memory (globals).
When you use MultiProcessing, a whole new process is created and each one gets its own copy of the memory (globals).
You could look at mutiprocessing Value/Array or Manager to allow pseudo-globals, i.e. shared objects.
I am trying to write a Python function for fast calculation of the sum of a list, using parallel computing. Initially I tried to use the Python multithreading library, but then I noticed that all threads run on the same CPU, so there is no speed gain, so I switched to using multiprocessing. In the first version I made the list a global variable:
from multiprocessing import Pool
array = 100000000*[1]
def sumPart(fromTo:tuple):
return sum(array[fromTo[0]:fromTo[1]])
with Pool(2) as pool:
print(sum(pool.map(sumPart, [(0,len(array)//2), (len(array)//2,len(array))])))
This worked well and returned the correct sum after about half the time of a serial computation.
But then I wanted to make it a function that accepts the array as an argument:
def parallelSum(theArray):
def sumPartLocal(fromTo: tuple):
return sum(theArray[fromTo[0]:fromTo[1]])
with Pool(2) as pool:
return (sum(pool.map(sumPartLocal, [(0, len(theArray) // 2), (len(theArray) // 2, len(theArray))])))
Here I got an error:
AttributeError: Can't pickle local object 'parallelSum.<locals>.sumPartLocal'
What is the correct way to write this function?
When scheduling jobs to a Python Pool you need to ensure both the function and it's arguments can be serialized as they will be transferred over a pipe.
Python uses the pickle protocol to serialize its objects. You can see what can be pickled in the module documentation. In your case, you are facing this limitation.
functions defined at the top level of a module (using def, not lambda)
Under the hood, the Pool is sending a string with the function name and its parameters. The Python interpreter in the child process looks for that function name in the module and fails to find it as it's nested in the scope of another function parallelSum.
Move sumPartLocal outside parallelSum and everything will be fine.
I believe you are hitting this, or see the documentation
What you could do is leave def sumPartLocal at module level, and pass theArray as third component of your tuple so that would be fromTo[2] inside the sumPartLocal function.
Example:
from multiprocessing import Pool
def sumPartLocal(fromTo: tuple):
return sum(fromTo[2][fromTo[0]:fromTo[1]])
def parallelSum(theArray):
with Pool(2) as pool:
return (sum
(pool.map
(sumPartLocal, [
(0, len(theArray) // 2, theArray),
(len(theArray) // 2, len(theArray), theArray)
]
)
)
)
if __name__ == '__main__':
theArray = 100000000*[1]
s = parallelSum(theArray)
print(s)
[EDIT 15-Dec-2017 based on comments]
Anyone who is thinking of multi-threading in python, I strongly recommend reading up about the Global Interpreter Lock
Also, some good answers on this question here on SO