I have Python code that works on Windows, however when running on Linux it just hangs. I'm using JPype, so I suspect that there might be some issue with multiple shared processes trying to use the same pipe to access Java (the different processes are created but hang at the JPype line). Is there any way to enforce spawning in Pathos to copy the Windows implementation? (e.g. set_start_method, or get_context in the regular multiprocessing library?)
Thanks.
To answer my own question (and it's a bit nasty) but after digging through the code you can:
import multiprocess.context as ctx
ctx._force_start_method('spawn')
Which happily solves the issue I'm having with JPype hanging. The difference between Linux and Windows is that as Windows spawns a new process, a new JVM is started (jpype.startJVM()), whereas a forked process has to use the same one (so I'm guessing that there are multiple processes trying to use the same pipe to Java). set_start_method doesn't seem to have been implemented, as far as I can tell.
Related
I have a Python 3.6 script which is multi-threaded and runs on a windows 10 machine with 12 cores.
I want to run multiple instances of the script but I am worried about Pythons GIL issue in terms of the performance of each instance of the script.
Would doing any of the below work around this as I am under the impression that when I run an instance of the script, the Python process it runs within is running on just one CPU core and each thread called from within the script 'time slices' on that one core...
Therefore would:
A: Starting each instance of the script in its own CMD window allow the OS to automatically deal with starting each scripts parent Python processes on its own core and prevent any locking taking place...
or
B: Starting each CMD session via a shortcut that sets its affinity to a specific core and then run the Python script so that the python process and its threads run on the specific core the CMD process has been set to use...
or
C: My understanding of how threading and the Python GIL works is not correct and I need understand...
Any help will be much appreciated.
I want to run multiple instances of the script
Then go ahead and do it!
I am worried about Pythons GIL issue in terms of the performance of each instance of the script.
The "GIL issue" is only an issue within any single multi-threaded Python process: No more than one thread belonging to the same Python process will ever execute Python code at the same instant. But, each Python process has its own GIL, and nothing prevents a thread in one Python process from running at the same time as a thread in some other Python process.
would (A) Starting each instance of the script in its own CMD window [...run each] Python processes on its own core.
No, because it is not normal for any process on an SMP platform (i.e., most multi-processor systems) to have "its own core." Cores are a resource that the operating system uses to run threads/processes, and applications normally have no reason to know or care which core is running which thread/process at any given moment in time.
(B) Starting each CMD session via a shortcut that sets its affinity to a specific core.
Some operating systems let you do that, but usually there is no benefit unless your application has very special needs. Normally, you should just trust the operating system scheduler to schedule processes and threads as efficiently as possible.
I am in the process of testing an app that has been working on Windows 7 & 8 nicely on OSX, using FireMonkey 2 (XE3).
For some reason I am getting crashes with OSX and the debugger isn't really giving much of a clue as to what the problem is.
Until now I have been using a single TSQLConnection and using this throughout my application and in multiple threads. Since the problems on OSX have been happening, I have been looking at maybe the causes being the TSQLConnection, used in multiple threads.
So I am here hoping to find a definitive answer as to how to use TSQLConnection/SQLite on multiple threads, if there is anything I need to do in order to ensure there are no conflicts with threads or do I just use it as is without having to worry about possible conflicts?
I have a django application which relies heavily on threading and I'm noticing no performance increment no matter how much processes or threads I add to the WSGIDaemonProcess.
I can't find a YES/NO answer out there and I'm wondering. Could it be that mod_wsgi is using the same interpreter for each request so I'm running in a bottleneck due to a GIL limitation?
If so, would you recommend something else that would help me workaround this limitation?
For a typical configuration, yes, all requests would be handle in same sub interpreter.
If in different sub interpreters of same process, you are still affected by the GIL.
Post your actual mod_wsgi configuration to confirm you have set things up right.
Consider trying New Relic to find out where real bottlenecks are.
Watch my PyCon US 2012 talk on finding bottlenecks
Short answer:
No.
Long answer:
This ability to make good use of more than processor, even when using multithreading, is further enchanced by the fact that Apache uses multiple processes for handling requests and not just a single process. Thus, even when there is some contention for the GIL within a specific process, it doesn't stop other processes from being able to run as the GIL is only local to a process and does not extend across processes.
Citation: https://code.google.com/p/modwsgi/wiki/ProcessesAndThreading
You haven't given enough information for anybody to recommend how to improve performance, but if you've actually written a thread-heavy program in Python, that's your first mistake.
Instead of running your program on CPython, maybe you should try Jython or IronPython instead. But then it wouldn't work with mod_wsgi, so we really need more details to understand what you're trying to do...
I am trying to run a multiple threads program on a 4 core processor, and I want each thread to run on a different core.
How can I do that? Because right now I see that they all running on the same core.
(I'm using Linux OS and my code was wrriten on c.)
Process schedulers make processes have an affinity towards a specific CPU. You've already loaded a bunch of stuff into cache, you may as well keep using this 'hot' cache.
You may be getting all the threads on this same core, since you already have the program loaded here.
I did find this: pthread_setaffinity_np. It seems clumsy, but i hope it's of some use.
Now here's something interesting. When I have more than one thread in Tcl invoking package require Expect, I get a seg fault.
e.g.
package require Threads
package require Expect
set t [thread::create]
thread::send {package require Expect}
puts "blarg! Damned thing crashes before I get here"
This is not a good time. Any thoughts?
Expect and Threads don't go together too well. Its the complexity you get from fork() + threads that can bite a lot there and lead to deadlocks and all kinds of uglyness. Usually not a good idea to combine the two.
If you really need Expect and the added concurrency a multi process approach with on multi threaded driver program and one single threaded expect process might work better. If you used tcllibs comm package the api's for sending commands are not that much different either (you mostly miss the tsv and tpool stuff if you used comm).
But it shouldn't segfault for sure. Which Expect/Threads/Tcl core combination did you use (e.g. ActiveStates ActiveTcl bundle or some self compiled stuff on an unusual platform?)
It's all from the latest debian packages, Ubuntu 9.0.4, 64 bit.
One alternative is to organize the code such that one thread is dedicated to handling all expect calls...which isn't the most elegant, generic solution but it might have to do.
The C code of the expect library (loaded with package require Expect) is not thread-safe (it probably uses global variables or else). I tried a lot to work around this limitation because I wanted to have a load balancing algorithm based on the Thread library which would pilot some expect code launching builds on a pool of slave machines. Unless you are very good at C and want to enhance expect, I would rather suggest to launch expect interpreters (in their own OS process) each time you need to use it from your Thread-enabled program. But of course I don't know your problem to solve, and this would only work if the "expect works" are unrelated.. Good luck anyway..