Unable to run pathos program from spyder IDE - python-3.x

Question

I have the following simple program:
from pathos.core import connect
tunnel = connect('192.168.1.5', port=50004)
print(tunnel)
print(type(tunnel._lport))
print(tunnel._rport)
def sleepy_squared(x):
from time import sleep
sleep(1.0)
return x**2
from pathos.pp import ParallelPythonPool as Pool
p = Pool(8, servers=('192.168.1.5:6260',))
print(p.servers)
x = [1, 2, 3, 4, 5, 6, 7, 8, 9]
y = p.map(sleepy_squared, x)
print(y)
When I try running this program from the Spyder 4 IDE I get the following error:
Tunnel('-q -N -L 4761:192.168.1.5:50004 192.168.1.5')
<class 'int'>
50004
('192.168.1.5:6260',)
Traceback (most recent call last):
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3319, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-1-e89974d31563>", line 20, in <module>
y = p.map(sleepy_squared, x)
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pathos/parallel.py", line 234, in map
return list(self.imap(f, *args))
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pathos/parallel.py", line 247, in imap
return (subproc() for subproc in list(builtins.map(submit, *args)))
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pathos/parallel.py", line 243, in submit
return _pool.submit(f, argz, globals=globals())
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pp/_pp.py", line 499, in submit
sfunc = self.__dumpsfunc((func, ) + depfuncs, modules)
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pp/_pp.py", line 683, in __dumpsfunc
sources = [self.__get_source(func) for func in funcs]
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pp/_pp.py", line 683, in <listcomp>
sources = [self.__get_source(func) for func in funcs]
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/pp/_pp.py", line 750, in __get_source
self.__sourcesHM[hashf] = importable(func)
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/dill/source.py", line 957, in importable
src = _closuredimport(obj, alias=alias, builtin=builtin)
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/dill/source.py", line 876, in _closuredimport
src = getimport(func, alias=alias, builtin=builtin)
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/dill/source.py", line 764, in getimport
return _getimport(head, tail, alias, verify, builtin)
File "/home/mahmoud/anaconda3/envs/trade_fxcm/lib/python3.6/site-packages/dill/source.py", line 713, in _getimport
try: exec(_str) #XXX: check if == obj? (name collision)
File "<string>", line 1
from __main__'> import sleepy_squared
^
SyntaxError: EOL while scanning string literal
When I run this program from the terminal using the following command python test_connect.py the program works fine. My question is why isn't the program running on the Spyder IDE 4 and how can I make the program run on Spyder IDE 4?

Answer 1

I'm the pathos author. Spyder, Jupyter, and other IDEs add an additional execution layer on top of the interpreter, and in some cases, even wrap the execution in a closure to add additional hooks into the rest of the IDE. You are using a ParallelPool, which uses ppft, which uses dill.source to "serialize" by extracting the source code of an object and it's dependencies. Since the IDE is adding a closure layer, dill.source has to try to serialize that as well, and it's not successful -- so in short it's a compatibility issue between dill.source and Spyder. If you pick one of the other pathos pools, it may succeed. The ProcessPool is essentially the same as the ParallelPool, but serializes by object instead of by source code -- it uses multiprocess, which uses dill. Then there's ThreadPool, which is probably the most likely to succeed, unless Spyder also messes with the main thread -- which most IDEs do. So, what can you do about it? Easy thing is to not run parallel code from the IDE. Essentially, write your code in the IDE, and then swap out the Pool and it should run in parallel. IDEs don't generally play well with parallel computing.