I have two cameras and this is important to read the frames with OpenCV exactly at the same time, I thought something like Lock but I cannot figure out the way I can implement this.
I need some trigger to push and enable the threads to read frames, and then wait for another trigger hit, something like below :
def get_frame(queue, cap):
while running:
if(read_frame):
queue.put(cap.read());
else:
# without this sleep this function just consumes unnecessary CPU time
time.sleep(some_time);
q = Queue.Queue()
# for every camera
for u in xrange(2):
t = threading.Thread(target=get_frame, args = (q, caps[u]))
t.daemon = True
t.start()
The problems with the above implementation are :
I need the sleep time to be defined since I don't know the delay in between every frame read (i.e. it might be long or short, depending on the calculation)
This does not enable me to read once for every trigger hit.
So this approach won't work, Any suggestions?
Consider getting FPS from VideoCapture. Also, note the difference between VideoCapture.grab and VideoCapture.retrieve frame. This is used for camera synchronization.
First call VideoCapture#grab for both cameras and then retrieve the frames. See docs.
Related
I have two threads concurrently running, speechRecognition and speakBack. Both of these threads are run in while loops (while True: #do something).
Speech recognition is constantly waiting for microphone input. Then, once it is received, it saves the text version of the verbal input to a file, which is loaded by my second thread, speakBack, and spoken through the speakers.
My issue is that when the phrase is spoken through the speakers, it is picked up by the microphone and then translated and once again saved to this file to be processed, resulting in an endless loop.
How can i make the speechRecognition thread suspend itself, wait for the speakBack thread to stop outputting sound through the speakers, and then continue listening for the next verbal input?
Im using the speechRecognition library and the pyttsx3 library for speech recognition and verbal ouput respectively.
The way to do this is to have shared state between the threads (either with global variables that the threads can store into and read from to indicate their progress, or with a mutable reference that is passed into each thread). The solution I’ll give below involves a global variable that stores a mutable reference, but you could just as easily pass the queue into both threads instead of storing it globally.
Using queues is a very standard way to pass messages between threads in python, because queues are already written in a thread-safe way that makes it so you don’t have to think about synchronization and locking. Furthermore, the blocking call to queue.get is implemented in a way that doesn’t involve repeatedly and wastefully checking a condition variable in a while loop.
Here’s how some code might look:
import queue
START_SPEAK_BACK = 0
START_SPEECH_RECOGNITION = 1
messageQueue = queue.Queue()
# thread 1
def speechRecognition():
while True:
# wait for input like you were doing before
# write to file as before
# put message on the queue for other thread to get
messageQueue.put(START_SPEAK_BACK)
# Calling `get` with no arguments makes the call be
# "blocking" in the sense that it won't return until
# there is an element on the queue to get.
messageFromOtherThread = messageQueue.get()
# logically, messageFromOtherThread can only ever be
# START_SPEECH_RECOGNITION, but you could still
# check that this is true and raise an exception if not.
# thread 2
def speakBack():
while True:
messageFromOtherThread = messageQueue.get()
# likewise, this message will only be START_SPEAK_BACK
# but you could still check.
# Here, fill in the code that speaks through the speakers.
# When that's done:
messageQueue.put(START_SPEECH_RECOGNITION)
Some comments:
This solution uses a single queue. It could just have easily used two queues, one for speakBack —> speechRecognition communication and the other for speechRecognition —> communication. This might make more sense if the two threads were generating messages concurrently.
This solution doesn’t actually involve inspecting the contents of the messages. However, if you need to pass additional information between threads, you could very easily pass objects or data as messages (instead of just constant values)
Finally, it’s not clear to me why you don’t just run all code in the same thread. It seems like there’s a very clear (serial) series of steps you want your program to follow: get audio input, write it to file, speak it back, start over. It might make more sense to write everything as a normal, serial, threadless python program.
I want to do a simple job. I have a list of n elements, and want to split the list into two smaller lists and use threading to perform a simple calculation and append them to a new list. I've written some testcode and it seems to work fine when I have a small amount of elements (say 3000). But when the element list is larger (30,000), over 12-20k tasks are being dropped and the append just doesn't go through.
I've read a lot about what constitutes threadsafe, and queueing. I believe it has something to do with that, but even after experimenting with Lock() I still seem to be unable to get a threadsafe Thread.
Can someone point me in the right direction? Cheers.
# Seperate thread workload
a_genes = genes[0:count_seperator]
b_genes = genes[count_seperator:genes_count]
class GeneThread (Thread):
def __init__(self, genelist):
Thread.__init__(self)
self.genelist = genelist
def run(self):
for gene in self.genelist:
total_reputation = 0
for local_snp in gene:
user_rsid = rsids[0]
if user_rsid is None:
continue
rep = "B"
# If multiplier is 0, don't waste time calculating
if not rep or rep == "G" or rep == "U":
continue
importance = 1
weighted_reputation = importance * mul[rep]
zygosity = "homozygous_minor"
if rep == "B":
weighted_reputation *= z_mul[zygosity]
# Now we apply the spread amplifier, we raise the score to the power of the spread number
rep_square = pow(spread, weighted_reputation)
total_reputation += rep_square
try:
with lock:
UserGeneReputation.append(total_reputation)
except:
pass
start_time = time.time()
# Create new threads
gene_thread1 = GeneThread(genelist=a_genes)
gene_thread2 = GeneThread(genelist=b_genes)
gene_thread1.daemon, gene_thread2.daemon = True, True
# Start new Threads
gene_thread1.start()
gene_thread2.start()
print(len(UserGeneReputation))
print("--- %s seconds ---" % (time.time() - start_time))
You have, broadly speaking, two choices with threads. You can have them be autonomous, do their work, and then terminate themselves quietly. Or you can have them be managed by some other thread that monitors their lifetime and knows when they're done. You have a design that absolutely requires the second option (how else will you know when you have all the results you need?), yet you've chosen the first (set them for self-termination and not monitored).
Don't make the threads daemon threads. Instead, wait for both threads to finish after you start them. That's not the most sophisticated or elegant solution, but it's the one everyone learns first.
The problem with this approach is that it forces your code to be dependent on how work is assigned to threads. This can cause performance problems as you wind up having to create and destroy a thread every time you want to know when work is done, and the only way you can know that work is done is by waiting for it. Ideally, you would treat threads as an abstraction that gets work done somehow and code that has to wait for work to be finished would wait for the work itself to be finished (through some synchronization associated with the work itself) rather than wait for the thread to be finished. That way, you can be flexible about what thread does what work and don't have to keep creating and destroying threads every time you need to assign work.
But everyone learns the create/join method. And sometimes it really is the best choice. Even when you use other methods, you likely still have an outer create/join to create the threads in the first place and, typically, ensure they cleanly finish to shut down your program in an orderly way.
I have a piece of code which has to get executed every 100ms and update the GUI. When I am updating the GUI - I am pressing a button, which calls a thread and in turn it calls a target function. The target function gives back the message to the GUI thread using pub sub as follows.
wx.CallAfter(pub.sendMessage, "READ EVENT", arg1=data, arg2=status_read) # This command line is in my target function
pub.subscribe(self.ReadEvent, "READ EVENT") # This is in my GUI file - whihc calls the following function
def ReadEvent(self, arg1, arg2):
if arg2 == 0:
self.MessageBox('The program did not properly read data from MCU \n Contact the Program Developer')
return
else:
self.data = arg1
self.firmware_version_text_control.Clear()
#fwversion = '0x' + ''.join('{:02X}'.format(j) for j in reversed(fwversion))
self.firmware_version_text_control.AppendText(str(SortAndDecode(self.data, 'FwVersion')))
# Pump Model
self.pump_model_text_control.Clear()
self.pump_model_text_control.AppendText(str(SortAndDecode(self.data, 'ModelName')))
# Pump Serial Number
self.pump_serial_number_text_control.Clear()
self.pump_serial_number_text_control.AppendText(str(SortAndDecode(self.data, 'SerialNum'))[:10]) # Personal Hack to not to display the AA , AB and A0
# Pressure GAIN
self.gain_text_control.Clear()
self.gain_text_control.AppendText(str(SortAndDecode(self.data, 'PresGain')))
# Pressure OFFSET Offset
self.offset_text_control.Clear()
self.offset_text_control.AppendText(str(SortAndDecode(self.data, 'PresOffset')))
#Wagner Message:
#self.status_text.SetLabel(str(SortAndDecode(self.data, 'WelcomeMsg')))
# PUMP RUNNING OR STOPPED
if PumpState(SortAndDecode(self.data, 'PumpState')) == 1:
self.led6.SetBackgroundColour('GREEN')
elif PumpState(SortAndDecode(self.data, 'PumpState')) == 0:
self.led6.SetBackgroundColour('RED')
else:
self.status_text.SetLabel(PumpState(SortAndDecode(self.data, 'PumpState')))
# PUMP RPM
self.pump_rpm_text_control.Clear()
if not self.new_model_value.GetValue():
self.pump_rpm_text_control.AppendText("000")
else:
self.pump_rpm_text_control.AppendText(str(self.sheet_num.cell_value(self.sel+1,10)*(SortAndDecode(self.data, 'FrqQ5'))/65536))
# PUMP PRESSURE
self.pressure_text_control.Clear()
self.pressure_text_control.AppendText(str(SortAndDecode(self.data, 'PresPsi')))
# ON TIME -- HOURS AND MINUTES --- EDITING IF YOU WANT
self.on_time_text_control.Clear()
self.on_time_text_control.AppendText(str(SortAndDecode(self.data, 'OnTime')))
# JOB ON TIME - HOURS AND MINUTES - EDITING IF YOU WANT
self.job_on_time_text_control.Clear()
self.job_on_time_text_control.AppendText(str(SortAndDecode(self.data, 'JobOnTime')))
# LAST ERROR ----- RECHECK THIS AGAIN
self.last_error_text_control.Clear()
self.last_error_text_control.AppendText(str(SortAndDecode(self.data, 'LastErr')))
# LAST ERROR COUNT --- RECHECK THIS AGAIN
self.error_count_text_control.Clear()
self.error_count_text_control.AppendText("CHECK THIS")
As you can see my READEVENT is very big and it takes a while for the GUI to take enough time to successfully do all these things. My problem here is, when my GUI is updating the values of TEXTCTRL it is going unresponsive - I cannot do anything else. I cant press a button or enter data or anything else. My question is if there is a better way for me to do this, so my GUI wont go unresponsive. I dont know how I can put this in a different thread as all widgets are in the main GUI. But that also requires keep creating threads every 100ms - which is horrible. Any suggestions would be greatly helpful.
Some suggestions:
How long does SortAndDecode take? What about the str() of the result? Those may be good candidates for keeping that processing in the worker thread instead of the UI thread, and passing the values to the UI thread pre-sorted-and-decoded.
You can save a little time in each iteration by calling ChangeValue instead of Clear and AppendText. Why do two function calls for each text widget instead of just one? Function calls are relatively expensive in Python compared to other Python code.
If it's possible that the same value will be sent that was sent on the last iteration then adding checks for the new value matching the old value and skipping the update of the widget could potentially save lots of time. Updating widget values is very expensive compared to leaving them alone.
Unless there is a hard requirement for 100ms updates you may want to try 150 or 200. Fewer updates per second may be fast enough for most people, especially since it's mostly textual. How much text can you read in 100ms?
If you are still having troubles with having more updates than the UI thread can keep up with, then you may want to use a different approach than pubsub and wx.CallAfter. For example you could have the worker thread receive and process the data and then add an object to a Queue.Queue and then call wx.WakeUpIdle(). In the UI thread you can have an EVT_IDLE event handler that checks the queue and pulls the first item out of the queue, if there are any, and then updates the widgets with that data. This will give the benefit of not flooding the pending events list with events from too many wx.CallAfter calls, and you can also do things like remove items from your data queue if there are too many items in it.
I am using MATLAB's PsychToolbox to run an experiment where I have to gather saccade information in real time, while also rendering a video frame by frame.
The problem that I have is that given the frame rate of the video and the display (~24fps), it means that I have about 40ms time window to render query and render every frame that I have previously stored in memory. This is fine, but since this process takes aditional time, it usually implies that I have about ~20ms to consistently poll for a saccade from beginning to end.
This is a problem, because when I poll for saccades, what I am usually doing (in say still images, that only have to be displayed once), is I wait for a start and end of a fixation, given consistent polling from the eye tracking machine, that detects that the observers gaze has shifted abruptly from one point to another with a
speed exceeding: 35 deg/s
and an
acceleration exceeding: 9500 deg/s^2
but if the beginning of a saccade or end of it takes places when a frame is being rendered (Which is most of the time), then it makes it impossible to get the data in real time without splitting the rendering and polling process into two separate MATLAB threads.
My code (relevant part) looks like this:
while GetSecs-t.stimstart(sess,tc)<fixation_time(stimshownorder(tc))
x =evt.gx(1);
y =evt.gy(1);
pa = evt.pa(1);
x_vec = [x_vec; x];
y_vec = [y_vec; y];
pa_vec = [pa_vec; pa];
evta=Eyelink('NewestFloatSample');
evtype=Eyelink('GetNextDataType');
#%% Ideally this block should detect saccades
#%% It works perfect in still images but it can't do anything here
#%% since it conflicts the main for loop ahead.
if evtype==el.ENDSACC
sacdata=Eyelink('GetFloatData',evtype);
sac.startx(sess,tc,sacc)=sacdata.gstx;
sac.starty(sess,tc,sacc)=sacdata.gsty;
sac.endx(sess,tc,sacc)=sacdata.genx;
sac.endy(sess,tc,sacc)=sacdata.geny;
sac.start(sess,tc,sacc)=sacdata.sttime;
sac.end(sess,tc,sacc)=sacdata.entime;
sacc=sacc+1;
end
#%Main loop where we render each frame:
if (GetSecs-t.space(sess,tc)>lag(tc))
z = floor((GetSecs-t.space(sess,tc)-lag(tc))/(1/24))+1;
if z > frame_number
z = frame_number;
end
Screen('DrawTexture',win,stimTex{z});
Screen('Flip',win);
#DEBUG:
#disp(z);
#%disp(frame_number);
end
end
Ideally, I'd want a MATLAB function that can render the video independently in one separate thread in the back end, while still polling for saccades in the main thread. Ideally like this:
#% Define New thread to render video
#% Some new function that renders video in parallel in another thread
StartParallelThread(1);
#%Play video:
Playmovie(stimTex);
#%Now start this main loop to poll for eye movements.
while GetSecs-t.stimstart(sess,tc)<fixation_time(stimshownorder(tc))
x =evt.gx(1);
y =evt.gy(1);
pa = evt.pa(1);
x_vec = [x_vec; x];
y_vec = [y_vec; y];
pa_vec = [pa_vec; pa];
evta=Eyelink('NewestFloatSample');
evtype=Eyelink('GetNextDataType');
if evtype==el.ENDSACC
sacdata=Eyelink('GetFloatData',evtype);
sac.startx(sess,tc,sacc)=sacdata.gstx;
sac.starty(sess,tc,sacc)=sacdata.gsty;
sac.endx(sess,tc,sacc)=sacdata.genx;
sac.endy(sess,tc,sacc)=sacdata.geny;
sac.start(sess,tc,sacc)=sacdata.sttime;
sac.end(sess,tc,sacc)=sacdata.entime;
sacc=sacc+1;
end
end
It also seems that the time it takes to run the Screen('Flip',win) command is about 16ms. This means that if any saccades happen in this interval, I would not be able to detect or poll them. Note that in the end I am having 42ms (for frame refresh rate) minus 16ms (for time it takes to query and display the frame), so a total of ~26ms of probing time per frame for getting eye movements and computing any real-time processing.
A possible solution might be to continually poll for gaze, instead of checking if an eye movement is a saccade or not. But I'd still have the problem of not capturing what goes on in about a third of each frame, just because it takes time to load it.
You need to reorganize your code. The only way to make this work is knowing how long the flip takes and knowing how long submission of the next video frame takes. Then you poll the eye tracker in a loop until you have just enough time left for the drawing commands to be executed before the next screen vertical blank.
You can't do any form of reliable multi-threading in matlab
I wonder if anyone of you know how to to use the function get_timer()
to measure the time for context switch
how to find the average?
when to display it?
Could someone help me out with this.
Is it any expert who knows this?
One fairly straightforward way would be to have two threads communicating through a pipe. One thread would do (pseudo-code):
for(n = 1000; n--;) {
now = clock_gettime(CLOCK_MONOTONIC_RAW);
write(pipe, now);
sleep(1msec); // to make sure that the other thread blocks again on pipe read
}
Another thread would do:
context_switch_times[1000];
while(n = 1000; n--;) {
time = read(pipe);
now = clock_gettime(CLOCK_MONOTONIC_RAW);
context_switch_times[n] = now - time;
}
That is, it would measure the time duration between when the data was written into the pipe by one thread and the time when the other thread woke up and read that data. A histogram of context_switch_times array would show the distribution of context switch times.
The times would include the overhead of pipe read and write and getting the time, however, it gives a good sense of big the context switch times are.
In the past I did a similar test using stock Fedora 13 kernel and real-time FIFO threads. The minimum context switch times I got were around 4-5 usec.
I dont think we can actually measure this time from User space, as in kernel you never know when your process is picked up after its time slice expires. So whatever you get in userspace includes scheduling delays as well. However, from user space you can get closer measurement but not exact always. Even a jiffy delay matters.
I believe LTTng can be used to capture detailed traces of context switch timings, among other things.