I am using queues in the context of processes and threads and there is a constellation with a strange behavior which i do not understand.
the used queue is defined like this: multiprocessing.Queue(1)
First Constellation (works fine)
initiate queue
Process 1 starts Process 2 (queue in parameters)
Process 1 starts Thread
2 Loops (one inside Thread one in Process 2) are communcating via the queue: everything works fine
Second Constellation (does not work)
Process 1 starts Process 2
inside Process 2 queue is initiated
Process 2 starts Process 3 (queue in parameters)
Process 2 starts Thread
2 Loops (one inside Thread one in Process 3) are communcating via the queue:
--> Loop in Thread in Process2 says always queue is full (wants to put something in the queue)
--> Loop in Process 3 says always queue is empty
Third Constellation (works fine)
Process 1 starts Process 2
inside Process 2 queue is initiated
Process 2 starts Process 3 (queue in parameters)
2 Loops (one in Process 2 one in Process 3) are communcating via the queue: everything works fine
I do not understand what's the problem with the second constellation. I read that this could happen if one first starts the thread and then the child-process. But i start Process 3 first and then start the Thread in Process 2. And if the thread is the Problem why does constellation 1 work?
Related
q= queue.Queue()
for i in [3,2,1]:
def f():
time.sleep(i)
print(i)
q.put(i)
threading.Thread(target=f).start()
print(q.get())
For this piece of code, it returns 1. The reason for this is because the queue is FIFO and "1" is put first as it slept the least time.
extended question,
If I continue to run q.get() twice, it still outputs the same value "1" rather than "2" and "3". Can anyone tell me why that is? Is there anything to do with threading?
Another extended question,
When the code finishes running completely, but there are still threads that haven't finished, will they get shut down immediately as the whole program finishes?
q.get()
#this gives me 1, but I suppose it should give me 2
q.get()
#this gives me 1, but I suppose it should give me 3
Update:
It is a Python 3 code.
Assuming that the language is Python3.
The second and third calls to q.get() return 1 because each of the three threads puts a 1 into the queue. There is never a 2 or a 3 in the queue.
I don't fully understand what to expect in this case—I'm not a Python expert—but the function, f does not appear to capture the value of the loop variable, i. The i in the function f appears to be the same variable as the i in the loop, and the loop leaves i==1 before any of the three threads wakes up from sleeping. So, in all three threads, i==1 by the time q.put(i) is called.
When the code finishes running completely, but there are still threads that haven't finished, will they get shut down immediately?
No. The process won't exit until all of its threads (including the main thread) have terminated. If you want to create a thread that will be automatically, forcibly, abruptly terminated when all of the "normal" threads are finished, then you can make that thread a daemon thread.
See https://docs.python.org/3/library/threading.html, and search for "daemon".
We have an API endpoint that starts a thread, and another endpoint to check the status of the thread (based on a thread ID returned by the first API call).
We use the threading module.
The function that the thread is executing may or may not sleep for a duration of time.
When we create the thread, we override the default name provided by the module and add the thread ID that was generated by us (so we can keep track).
The status endpoint gets the thread ID from the client request and simply loops over the results from threading.enumerate(). When the thread is running and not sleeping, we see that the thread is returned by the threading.enumerate() function. When it is sleeping, it is not.
The function we use to see if a thread is alive:
def thread_is_running(thread_id):
all_threads = [ t.getName() for t in threading.enumerate() ]
return any(thread_id in item for item in all_threads)
When we run in debug and print the value of "all_threads", we only see the MainThread thread during our thread's sleep time.
As soon as the sleep is over, we see our thread in the value of "all_threads".
This is how we start the thread:
thread_id = random.randint(10000, 50000)
thread_name = f"{service_name}-{thread_id}"
threading.Thread(target=drain, args=(service_name, params,), name=thread_name).start()
Is there a way to get a list of all threads including idle threads? Is a sleeping thread marked as idle? Is there a better way to pause a thread?
We thought about making the thread update it's state in a database, but due to some internal issue we currently have, we cannot 100% count on writing to our database, so we prefer checking the system for the thread's status.
Turns out the reason we did not see the thread was our use of gunicorn and multi workers.
The thread was initiated on one of the 4 configured workers while the status api call could've been handled by any of the 4 workers. only when it was handled by the worker who is also responsible of running the thread - we were able to see it in the enumerate output
Folks how exactly are requests being sent from JMeter. Given this configuration does it mean that JMeter will create 200 threads, and send concurrent requests in a loop 20 times. I would assume that this means that each thread blocks until the previous request finishes, and sends the next request, and that this process is done 20 times for each thread.
Let's say I have a sample request A, num threads = 2, and loop count= 5 I imagine the workflow to be as follows
At time t0 Thread 1 -> send A to the target server
At time t0 Thread 2 -> send A to the taget server
At time t1 Thread 1 receives response, and sends A again to target server
At time t2 Thread 2 receives response and sends A again to target server
Is this the correct workflow that JMeter will follow. Please note I am using version 5.2.1
JMeter starts all the Threads defined in the Thread Group within the bounds of the Ramp-Up period
Each JMeter thread does not depend on the other threads
Once started each Thread starts executing Samplers upside down (or according to the Logic Controllers)
When there are no more samplers to execute the thread starts the next iteration
When there are no more loops to iterate - the thread is being shut down
The actual concurrency depends on test duration and application response time, it can be checked using i.e. Active Threads Over Time and Transactions Per Second listeners
You can easily track each virtual user/iteration using __threadNum() function and special ${__jm__Thread Group__idx} JMeter Variable:
I am new to Python threading and I have researched about it. I would like to implement the following functionality as written in pseudo code:
while True:
while(file size < 1 GB):
sleep for 1 minute
process(file)
file = next file
I want the process() function to be a daemon thread. The next time line 4 is reached in the code (next file is 1 GB), it should create a new thread if the previous thread is running. The maximum such threads are 3 and they loop one after the other. At any time there will be at least 2 threads free. So basically any one free thread should be called to do the job once the code reaches line 4.
Are daemon threads and thread queue the things I need to look at? Or is there some other way to solve this?
Am unable to understand (what I think) is a peculiar situation wrt process/thread scheduling on Linux.
[Env: Ubuntu 12.10 , kernel ver 3.5.0-... ]
A 'test' application (call it sched_pthread), will have a total of three threads - 'main' + two others; main() will spawn two new threads:
Thread 1 [main()]:
Runs as SCHED_NORMAL (or SCHED_OTHER). It:
Creates two threads (Thread 2 and Thread 3 below); they will automatically inherit the
scheduling policy and priority of main.
Prints the character “m” to the terminal in a loop.
Terminates.
Thread 2 [t2]:
Sleeps for 2 seconds.
Changes it's scheduling policy to SCHED_FIFO, setting it's realtime priority to the
value passed on the command line.
Prints the character “2” to the terminal in a loop.
Terminates.
Thread 3 [t3]:
Changes it's scheduling policy to SCHED_FIFO, setting it's realtime priority to the
value passed on the command line plus 10.
Sleeps for 4 seconds.
Prints the character “3” to the terminal in a loop.
Terminates.
We run it as root.
As per scheduling policy, we should first see main() print 'm' for about 2s, then it should
get preempted by t2 (as it awakens after 2s) and we should see '2' appearing on the terminal for about 2s, after which t3 wakes up (it was asleep for 4s); it should now preempt everyone else & emit '3' to the display; after it dies, we should see '2's until p2 dies, then 'm's until main() dies.
So okay, this works: when i test it in console mode (no X server).
Of course, i take care to run it as:
sudo taskset 02 ./sched_pthrd 8
so that in effect it runs on only 1 processor core.
When I run the same thing in graphical mode (with X), after the initial 'm's by main(), there is a long-ish pause (a few seconds) during which nothing appears on the screen; then all of a sudden we get the 2's and 3's and m's slapped onto the screen!
This can be explained : the X server (Xorg) was preempted by the SCHED_FIFO threads and hence could not 'paint' pixels on the screen.
However - here's the question at last - : how come the Xorg process was not scheduled / migrated onto some other core (so that it can continue updating the screen in parallel with the RT threads)??
taskset verifies that the cpu affinity mask of Xorg is 'f' (1111b) (i have 4 cores on my laptop).
Any ideas??
Here's the source code:
https://dl.dropboxusercontent.com/u/9301413/code_shared/so_sched_pthrd.c
-or-
http://goo.gl/PLHBrC
TIA!
-Kaiwan.