All of the GStreamer samples are initializing GLib main thread through some form of:
loop = g_main_loop_new(NULL, FALSE);
g_main_loop_run(loop);
As far as I understood this main loop is used for all the signals processing.
Also Bus messages are processed in it.
So I'm a little bit concerned what will happen if I run multiple pipelines simultaneously.
Or there is an issue/improper implementation in some of them.
Most probably for heavy loads the best solution is pipelines to be separated to multiple processes, thus mitigating all the possible problems with memory leaks, hangs, dead locks etc., without affecting the main application.
Anyway at least running them into separate threads will be beneficial.
Obviously it is possible to be started more than one GLib main thread, with creation of the GMainContext first. But I cannot understand (apparently I'm missing knowledge) how after that to "assign" them to pipelines or signaling to them, etc.
For example in "g_signal_connect" and "g_signal_emit" is not specified on which "main" thread to be executed.
Some posts here claim it is possible (GStreamer supports different main thread), but I wasn't able to find details.
Similar problem is discussed in this thread but to be honest I wasn't able to understand it.
In this StackOverflow post is discussed how timeouts could be attached to different GLib main threads.
I suppose that something similar could be made and for the GStreamer pipelines and objects, but I'm not sure.
Could someone enlighten me a little bit?
I'm posting here the answer of the same question in the GStreamer-devel forum:
http://gstreamer-devel.966125.n4.nabble.com/Running-every-GStremer-pipeline-into-a-separate-GLib-thread-td4694469.html
citate:
GMainLoop is optional with GStreamer (convenient but optional). You can
use the GstBus API directly. As for signals, these are synchronous, and
not using a messageé.
If you decide to use a GMainLoop, you will only need one to handle
asynchronous messages, as all message gets serialized into the loop
queue. Multiple pipeline is were that becomes convenient as you don't
have to deal with multiple GstBus object.
Streamer splits the streaming into seperate threads already. The
mainloop thread is always free for other task (like UI task).
:end of citate
Related
The SDL documentation for threading states:
NOTE: You should not expect to be able to create a window, render, or receive events on any thread other than the main one.
The glfw documentation for glfwCreateWindow states:
Thread safety: This function must only be called from the main thread.
I have read about issues regarding the glut library from people who have tried to run the windowing functions on a second thread.
I could go on with these examples, but I think you get the point I'm trying to make. A lot of cross-platform libraries don't allow you to create a window on a background thread.
Now, two of the libraries I mentioned are designed with OpenGL in mind, and I get that OpenGL is not designed for multithreading and you shouldn't do rendering on multiple threads. That's fine. The thing that I don't understand is why the rendering thread (the single thread that does all the rendering) has to be the main one of the application.
As far as I know, neither Windows nor Linux nor MacOS impose any restrictions on which threads can create windows. I do know that windows have affinity to the thread that creates them (only that thread can receive input for them, etc.); but still that thread does not need to be the main one.
So, I have three questions:
Why do these libraries impose such restrictions? Is it because there is some obscure operating system that mandates that all windows be created on the main thread, and so all operating systems have to pay the price? (Or did I get it wrong?)
Why do we have this imposition that you should not do UI on a background thread? What do threads have to do with windowing, anyways? Is it not a bad abstraction to tie your logic to a specific thread?
If this is what we have and can't get rid of it, how do I overcome this limitation? Do I make a ThreadManager class and yield the main thread to it so it can schedule what needs to be done in the main thread and what can be done in a background thread?
It would be amazing if someone could shed some light on this topic. All the advice I see thrown around is to just do input and UI both on the main thread. But that's just an arbitrary restriction if there isn't a technical reason why it isn't possible to do otherwise.
PS: Please note that I am looking for a cross platform solution. If it can't be found, I'll stick to doing UI on the main thread.
While I'm not quite up to date on the latest releases of MacOS/iOS, as of 2020 Apple UIKit and AppKit were not thread safe. Only one thread can safely change UI objects, and unless you go to a lot of trouble that's going to be the main thread. Even if you do go to all the trouble of closing the window manager connection etc etc you're still going to end up with one thread only doing UI. So the limitation still applies on at least one major system.
While it's possibly unsafe to directly modify the contents of a window from any other thread, you can do software rendering to an offscreen bitmap image from any thread you like, taking as long as you like. Then hand the finished image over to the main thread for rendering. (The possibly is why cross platform toolkits disallow/tell you not to. Sometimes it might work, but you can't say why, or even that it will keep working.)
With Vulkan and DirectX 12 (and I think but am not sure Metal) you can render from multiple threads. Woohoo! Of course now you have to figure out how to do all the coordination and locking and cross-synching without making the whole thing slower than single threaded, but at least you have the option to try.
Adding to the excellent answer by Matt, with Qt programs you can use invokeMethod and postEvent to have background threads update the UI safely.
It's highly unlikely that any of these frameworks actually care about which thread is the 'main thread', i.e., the one that called the entry point to your code. The real restriction is that you have to do all your UI work on the thread that initialized the framework, i.e., the one that called SDL_Init in your case. You will usually do this in your main thread. Why not?
Multithreaded code is difficult to write and difficult to understand, and in UI work, introducing multithreading makes it difficult to reason about when things happen. A UI is a very stateful thing, and when you're writing UI code, you usually need to have a very good idea about what has happened already and what will happen next -- those things are often undefined when multithreading is involved. Also, users are slow, so multithreading the UI is not really necessary for performance in normal cases. Because of all this, making a UI framework thread-safe isn't usually considered beneficial. (multithreading compute-intensive parts of your rendering pipeline is a different thing)
Single-threaded UI frameworks have a dispatcher of some sort that you can use to enqueue activities that should happen on the main thread when it next has time. In SDL, you use SDL_PushEvent for this. You can call that from any thread.
I'd like to be able to open a TDataSet asynchronously in its own thread so that the main VCL thread can continue until that's done, and then have the main VCL thread read from that TDataSet afterwards. I've done some experimenting and have gotten into some very weird situations, so I'm wondering if anyone has done this before.
I've seen some sample apps where a TDataSet is created in a separate thread, it's opened and then data is read from it, but that's all done in the separate thread. I'm wondering if it's safe to read from the TDataSet from the main VCL thread after the other thread opens the data source.
I'm doing Win32 programming in Delphi 7, using TmySQLQuery from DAC for MySQL as my TDataSet descendant.
Provided you only want to use the dataset in its own thread, you can just use synchronize to communicate with the main thread for any VCL/UI update, like with any other component.
Or, better, you can implement communication between the mainthread and worker threads with your own messaging system.
check Hallvard's solution for threading here:
http://hallvards.blogspot.com/2008/03/tdm6-knitting-your-own-threads.html
or this other one:
http://dn.codegear.com/article/22411
for some explanation on synchronize and its inefficiencies:
http://www.eonclash.com/Tutorials/Multithreading/MartinHarvey1.1/Ch3.html
I have seen it done with other implementations of TDataSet, namely in the Asta components. These would contact the server, return immediately, and then fire an event once the data had been loaded.
However, I believe it depends very much on the component. For example, those same Asta components could not be opened in a synchronous manner from anything other than the main VCL thread.
So in short, I don't believe it is a limitation of TDataSet per se, but rather something that is implementation specific, and I don't have access to the components you've mentioned.
One thing to keep in mind about using the same TDataSet between multiple threads is you can only read the current record at any given time. So if you are reading the record in one thread and then the other thread calls Next then you are in trouble.
Also remember the thread will most likely need its own database connection. I believe what is needed here is a multi-threaded "holding" object to load the data from the thread into (write only) which is then read only from the main VCL thread. Before reading use some sort of syncronization method to insure that your not reading the same moment your writing, or writing the same moment your reading, or load everything into a memory file and write a sync method to tell the main app where in the file to stop reading.
I have taken the last approach a few times, depdending on the number of expected records (and the size of the dataset) I have even taken this to a physical disk file on the local system. It works quite well.
I've done multithreaded data access, and it's not straightforward:
1) You need to create a session per thread.
2) Everything done to that TDataSet instance must be done in context of the thread where it was created. That's not easy if you wanted to place e.g. a db grid on top of it.
3) If you want to let e.g. main thread play with your data, the straight-forward solution is to move it into a separate container of some kind,e.g. a Memory dataset.
4) You need some kind of signaling mechanism to notify main thread once your data retrieval is complete.
...and exception handling isn't straightforward, either...
But: Once you've succeeded, the application will be really elegant !
Most TDatasets are not thread safe. One that I know is thread safe is kbmMemtable. It also has the ability to clone a dataset so that the problem of moving the record pointer (as explained by Jim McKeeth) does occur. They're one of the best datasets you can get (bought or free).
The Twisted documentation led me to believe that it was OK to combine techniques such as reactor.spawnProcess() and threads.deferToThread() in the same application, that the reactor would handle this elegantly under the covers. Upon actually trying it, I found that my application deadlocks. Using multiple threads by themselves, or child processes by themselves, everything is fine.
Looking into the reactor source, I find that the SelectReactor.spawnProcess() method simply calls os.fork() without any consideration for multiple threads that might be running. This explains the deadlocks, because starting with the call to os.fork() you will have two processes with multiple concurrent threads running and doing who knows what with the same file descriptors.
My question for SO is, what is the best strategy for solving this problem?
What I have in mind is to subclass SelectReactor, so that it is a singleton and calls os.fork() only once, immediately when instantiated. The child process will run in the background and act as a server for the parent (using object serialization over pipes to communicate back and forth). The parent continues to run the application and may use threads as desired. Calls to spawnProcess() in the parent will be delegated to the child process, which will be guaranteed to have only one thread running and can therefore call os.fork() safely.
Has anyone done this before? Is there a faster way?
What is the best strategy for solving this problem?
File a ticket (perhaps after registering) describing the issue, preferably with a reproducable test case (for maximum accuracy). Then there can be some discussion about what the best way (or ways - different platforms may demand different solution) to implement it might be.
The idea of immediately creating a child process to help with further child process creation has been raised before, to solve the performance issue surrounding child process reaping. If that approach now resolves two issues, it starts to look a little more attractive. One potential difficulty with this approach is that spawnProcess synchronously returns an object which supplies the child's PID and allows signals to be sent to it. This is a little more work to implement if there is an intermediate process in the way, since the PID will need to be communicated back to the main process before spawnProcess returns. A similar challenge will be supporting the childFDs argument, since it will no longer be possible to merely inherit the file descriptors in the child process.
An alternate solution (which may be somewhat more hackish, but which may also have fewer implementation challenges) might be to call sys.setcheckinterval with a very large number before calling os.fork, and then restore the original check interval in the parent process only. This should suffice to avoid any thread switching in the process until the os.execvpe takes place, destroying all the extra threads. This isn't entirely correct, since it will leave certain resources (such as mutexes and conditions) in a bad state, but you use of these with deferToThread isn't very common so maybe that doesn't affect your case.
The advice Jean-Paul gives in his answer is good, but this should work (and does in most cases).
First, Twisted uses threads for hostname resolution as well, and I've definitely used subprocesses in Twisted processes that also make client connections. So this can work in practice.
Second, fork() does not create multiple threads in the child process. According to the standard describing fork(),
A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread ...
Now, that's not to say that there are no potential multithreading issues with spawnProcess; the standard also says:
... to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called ...
and I don't think there's anything to ensure that only async-signal-safe operations are used.
So, please be more specific as to your exact problem, since it isn't a subprocess with threads being cloned.
Returning to this issue after some time, I found that if I do this:
reactor.callFromThread(reactor.spawnProcess, *spawnargs)
instead of this:
reactor.spawnProcess(*spawnargs)
then the problem goes away in my small test case. There is a remark in the Twisted documentation "Using Processes" that led me to try this: "Most code in Twisted is not thread-safe. For example, writing data to a transport from a protocol is not thread-safe."
I suspect that the other people Jean-Paul mentioned were having this problem may be making a similar mistake. The responsibility is on the application to enforce that reactor and other API calls are being made within the correct thread. And apparently, with very narrow exceptions, the "correct thread" is nearly always the main reactor thread.
fork() on Linux definitely leaves the child process with only one thread.
I assume you are aware that, when using threads in Twisted, the ONLY Twisted API that threads are permitted to call is callFromThread? All other Twisted APIs must only be called from the main, reactor thread.
I've heard that mixing forking and threading in a program could be very problematic, often resulting with mysterious behavior, especially when dealing with shared resources, such as locks, pipes, file descriptors. But I never fully understand what exactly the dangers are and when those could happen. It would be great if someone with expertise in this area could explain a bit more in detail what pitfalls are and what needs to be care when programming in a such environment.
For example, if I want to write a server that collects data from various different resources, one solution I've thought is to have the server spawns a set of threads, each popen to call out another program to do the actual work, open pipes to get the data back from the child. Each of these threads responses for its own work, no data interexchange in b/w them, and when the data is collected, the main thread has a queue and these worker threads will just put the result in the queue. What could go wrong with this solution?
Please do not narrow your answer by just "answering" my example scenario. Any suggestions, alternative solutions, or experiences that are not related to the example but helpful to provide a clean design would be great! Thanks!
The problem with forking when you do have some threads running is that the fork only copies the CPU state of the one thread that called it. It's as if all of the other threads just died, instantly, wherever they may be.
The result of this is locks aren't released, and shared data (such as the malloc heap) may be corrupted.
pthread does offer a pthread_atfork function - in theory, you could take every lock in the program before forking, release them after, and maybe make it out alive - but it's risky, because you could always miss one. And, of course, the stacks of the other threads won't be freed.
It is really quite simple. The problems with multiple threads and processes always arise from shared data. If there is not shared data then there can be no possible issues arising.
In your example the shared data is the queue owned by the main thread - any potential contention or race conditions will arise here. Typical methods for "solving" these issues involve locking schemes - a worker thread will lock the queue before inserting any data, and the main thread will lock the queue before removing it.
I'm developing a DirectShow application. I encounter a deadlock problem, the problem seems caused by acquire lock in a callback function called from a thread. This is the quest I asked in MSDN forum:
http://social.msdn.microsoft.com/Forums/en-US/windowsdirectshowdevelopment/thread/f9430f17-6274-45fc-abd1-11ef14ef4c6a
Now I have to avoid to acquire lock in that thread. But the problem is, I have to output the audio to another thread, how can I put data to another thread without lock?
There's someone tell me that I can use PostMessage of win32 sdk to post data to another thread. But however, to get the message, I have to run a windows program. My program is a Python C++ extension module. That might be very difficult to add a loop to pull message. So I am think another way to pass data among threads without locking.
(Actually... the producer thread can't be locked, but the consumer thread can do that. )
To lock or not to lock, that's the question.
So the question is how to do?
Thanks.
------EDIT------
I think I know why I got a deadlock, that might not be the problem of DirectShow.
The main thread is own by Python, it call stop, namely, it hold GIL. And the stop wait for callback of DirectShow in thread return. But callback acquire the GIL.
It looks like this
Main(Hold GIL) -> Stop(Wait callback) -> Callback(Wait GIL) -> GIL(Hold by Main thread)
Damn it! That's why I don't like multi-thread so much.
No matter what, thanks your help.
If you were doing this in pure Python, I'd use a Queue object; these buffer up data which is written but block on read until something is available, and do any necessary locking under the hood.
This is an extremely common datatype, and some equivalent should always be available, whatever your current language or toolchain; there's a STL Queue available in C++, for instance, but the standard doesn't specify thread-safety characteristics (so see your local implementation docs).
Well, theoretically locks can be avoided if both of your threads can work on duplicate copies of the same data. After reading your question in the MSDN forum...
"So to avoid deadlock, I should not acquire any lock in the graber callback function? How can I do if I want to output audio to another thread?"
I think that you should be able to deposit your audio data in a dequeue (an STL class) and then fetch this data from another thread. This other thread can then process your audio data.
I am glad that your problem has been resolved the reason I asked about your Os was that the documentation you referred to said that you should not wait on other threads because of some problem with win16Mutexes. There are no win16mutexes on windows XP (except when programs are running on ntvdm/wow16) so you should be able to use locks to synchronize these threads.