In RR scheduling policy what will happen if a low priority thread locks a mutex and is removed by the scheduler because another high priority thread is waiting?
Will it also releases the lock held by low priority thread?
For example consider 3 threads running in a process with priorities 10,20 and 30 in RR scheduling policy.
Now at a given point of time low priority thread 1 locks mutex and is still doing execution mean while the high priority thread pops in and also waits on mutex held by thread 1. Now thread 2 comes in to picture which also needs the same mutex locked by thread 1.
As far as I know as per scheduling algorithm the threads sleeping or waiting for mutex,semaphore etc are removed and the other ones, even having low priority are allowed to execute. Is this correct? If so, in the above example ultimately high priority threads wait for the completion of low priority thread which doesn't make any sense.
Is this how the system works if at all threads are designed like I said above?
or
the thread priority should be set in such a way that high priority one's will not depend on low priority one's mutexe's ?
Also can anyone please explain me how scheduling works at process level? How do we set priority for a process?
Typically, scheduling and locks are unrelated in any other aspect than a "waiting thread is not scheduled until it's finished waiting". It would be rather daft to have a MUTEX that "stops other thread from accessing my data" but it only works if the other thread has the same or lower priority than the current thread.
The phenomena of "a low priority holds a lock that a high priority thread 'needs'" is called priority inversion, and it's a well known scenario in computer theory.
There are some schemes that "temporarily increase priority of a lock-holding thread until it releases the lock to the highest priority of the waiting threads" (or the priority of the first waiting thread if it's higher than the current thread, or some other variation on that theme). This is done to combat priority inversion - but it has other drawbacks too, so it's not implemented in all OS's/schedulers (after all, it affects other threads than the one that is waiting too).
Edit:
The point of a mutex (or other similar locks) is that it prevents two threads from accessing the same resources at once. For example, say we want to update five different variables with some pretty lengthy processing (complicated math, fetching data from a serial port or a network drive, or some such), but if we only do two of the variables, some other process using these would get an invalid result, then we clearly can't "let go" of the lock.
The high priority thread simply has to wait for all five variables to be updated and the low priority lock.
There is no simple workaround that the application can do to "fix" this problem - don't hold the locks more than necessary of course [and it may be that we can actually fix the problem described above, by performing the lengthy processing OUTSIDE of the lock, and only do the final "store it in the 5 variables" with the lock on. This would reduce the potential time that the high priority thread has to wait, but if the system is really busy, it won't really fix the problem.
There are whole PhD thesis written on this subject, and I'm not a specialist on "how to write a scheduler" - I have a fair idea how one works, just like I know how the engine in my car works - but if someone gave me a bunch of suitable basic shapes of steel and aluminium, and the required tools/workspace and told me to build an engine, I doubt it would work great...Same with a scheduler - I know what the parts are called, but not how to build one.
If a high priority thread needs to wait on a low priority thread (mutex semaphore etc), the low priority thread is temporarily elevated to the same priority as the high priority thread.
The high priority thread is not going to have the lock for which it is requesting until the low priority thread will unlock it.
To avoid this we can use semaphore where any other thread can initiate to unlock but in mutex it is not possible.
Related
Apologies, if it is not the right place for this question.
I am working on an RTOS (RTX), my question is if a High Priority Thread is waiting for a Thread Flag (says the flag is X_High) (currently is in a blocked state) and a Low Priority Thread is running, and this low priority thread sets "X_High" thread flag.
Would context switching from low priority to High Thread take place immediately or low priority thread will keep on running till a blocking statement (Delay or waiting for thread flag)?
The guarantee of a pre-emptive priority bases real-time scheduler is that the highest priority thread that is ready to run, will run (outside if interrupt processing).
The scheduling decision is made whenever a scheduling call is made in a normal thread and in exit from the interrupt context. osEventFlagsSet is a scheduling call. If any event set by it causes a higher priority thread to become ready, the calling thread is pre-empted immediately (before the function returns).
It depends on the specific RTOS you're using, and how it is configured. For example, FreeRTOS can be configured to use either cooperative scheduling or preemptive scheduling. When using preemptive scheduling, task switching takes place as soon as the flag is set (semaphore is given). But when using cooperative scheduling, task switching takes place when the currently running task is blocked, or yields voluntarily.
Can someone explain to me how message queues handle waking multiple
threads blocked on a single message queue?
My situation is I have multiple writers blocking on a full message
queue, each posting messages with priority equal to the thread
priority. I want to make sure they wake and post in priority order,
however my application is behaving as if they are waking in FIFO order
(i.e. the order in which they blocked). Each blocking thread is
scheduled with the SCHED_FIFO policy with a different priority with
system level scope.
I've searched the Internet high and low for something describing how
this should work and all I can find is POSIX man pages describing that
multiple blockers wake in priority order if Priority Scheduling is
supported. Since the kernel scheduler is a priority scheduler I
would think that the threads would wake in priority order and post to
the queue, however that doesn't appear to be the case. I'm sure I'm
just missing some subtle detail and was hoping the experts here on
this list can help shine some light on what I'm seeing, since its at
the kernel level that these threads are made ready to run.
I have a small test application that I can post here if necessary. It simply fills a queue, then has a few threads all try and write to it, all with different thread priorities and posting with a message priority equal to the thread priority. I then remove a message from the queue and would expect the highest priority thread wake up and post its message. However, the first thread to wait posts its message first.
Any help or documentation anyone can point me to in order to get to the bottom of this?
Thanks in advance!
Turns out the Linux kernel looks at the tasks' priority value if the queue is full and adds them to a wait queue in task niceness order (which is priority order for non-RT tasks). The wait queue does not honor real-time priorities which my application uses. Non-RT priorities (nice values) are being handled properly and wake up in niceness order.
The root cause of my issue down in how the kernel was handling priorities when adding tasks to the internal kernel wait queues. I submitted a patch to the linux-kernel list which was accepted and will be rolled into future releases which changes the priority check when adding tasks to the wait queue - it now honors both non-RT priority as well as RT priority. It does NOT handle priorities of deadline scheduled tasks.
I have an interesting problem related to Java thread live lock. Here it goes.
There are four global locks - L1,L2,L3,L4
There are four threads - T1, T2, T3, T4
T1 requires locks L1,L2,L3
T2 requires locks L2
T3 required locks L3,L4
T4 requires locks L1,L2
So, the pattern of the problem is - Any of the threads can run and acquire the locks in any order. If any of the thread detects that a lock which it needs is not available, it release all other locks it had previously acquired waits for a fixed time before retrying again. The cycle repeats giving rise to a live lock condition.
So, to solve this problem, I have two solutions in mind
1) Let each thread wait for a random period of time before retrying.
OR,
2) Let each thread acquire all the locks in a particular order ( even if a thread does not require all the
locks)
I am not convinced that these are the only two options available to me. Please advise.
Have all the threads enter a single mutex-protected state-machine whenever they require and release their set of locks. The threads should expose methods that return the set of locks they require to continue and also to signal/wait for a private semaphore signal. The SM should contain a bool for each lock and a 'Waiting' queue/array/vector/list/whatever container to store waiting threads.
If a thread enters the SM mutex to get locks and can immediately get its lock set, it can reset its bool set, exit the mutex and continue on.
If a thread enters the SM mutex and cannot immediately get its lock set, it should add itself to 'Waiting', exit the mutex and wait on its private semaphore.
If a thread enters the SM mutex to release its locks, it sets the lock bools to 'return' its locks and iterates 'Waiting' in an attempt to find a thread that can now run with the set of locks available. If it finds one, it resets the bools appropriately, removes the thread it found from 'Waiting' and signals the 'found' thread semaphore. It then exits the mutex.
You can twiddle with the algorithm that you use to match up the available set lock bools with waiting threads as you wish. Maybe you should release the thread that requires the largest set of matches, or perhaps you would like to 'rotate' the 'Waiting' container elements to reduce starvation. Up to you.
A solution like this requires no polling, (with its performance-sapping CPU use and latency), and no continual aquire/release of multiple locks.
It's much easier to develop such a scheme with an OO design. The methods/member functions to signal/wait the semaphore and return the set of locks needed can usually be stuffed somewhere in the thread class inheritance chain.
Unless there is a good reason (performance wise) not to do so,
I would unify all locks to one lock object.
This is similar to solution 2 you suggested, only more simple in my opinion.
And by the way, not only is this solution more simple and less bug proned,
The performance might be better than solution 1 you suggested.
Personally, I have never heard of Option 1, but I am by no means an expert on multithreading. After thinking about it, it sounds like it will work fine.
However, the standard way to deal with threads and resource locking is somewhat related to Option 2. To prevent deadlocks, resources need to always be acquired in the same order. For example, if you always lock the resources in the same order, you won't have any issues.
Go with 2a) Let each thread acquire all of the locks that it needs (NOT all of the locks) in a particular order; if a thread encounters a lock that isn't available then it releases all of its locks
As long as threads acquire their locks in the same order you can't have deadlock; however, you can still have starvation (a thread might run into a situation where it keeps releasing all of its locks without making forward progress). To ensure that progress is made you can assign priorities to threads (0 = lowest priority, MAX_INT = highest priority) - increase a thread's priority when it has to release its locks, and reduce it to 0 when it acquires all of its locks. Put your waiting threads in a queue, and don't start a lower-priority thread if it needs the same resources as a higher-priority thread - this way you guarantee that the higher-priority threads will eventually acquire all of their locks. Don't implement this thread queue unless you're actually having problems with thread starvation, though, because it's probably less efficient than just letting all of your threads run at once.
You can also simplify things by implementing omer schleifer's condense-all-locks-to-one solution; however, unless threads other than the four you've mentioned are contending for these resources (in which case you'll still need to lock the resources from the external threads), you can more efficiently implement this by removing all locks and putting your threads in a circular queue (so your threads just keep running in the same order).
I have a static list of objects.
During the program, many thread are created.
Immediately after each thread is created, it creates a new object, and add it to the static list.
There is another thread in the program, that responsible to iterating over the static list.
Suppose that a thread with a low priority 'A' is access to the list, and another thread with a higher priority 'C' asks also access to it, May (in a rare case indeed),that thread with medium priority 'B' that exists in the system too, will get the CPU time of 'A'.
So, 'C' will wait for 'B', contrary to common sense.
How do I can get a lock to the List, without getting involved with this Priority inversion problem?
The function 'Lock()' can help?
Thank you!
That is, at worst, a short term priority inversion problem. Unless, of course, the low priority thread A holds the lock for a very long time. Thread C can't make progress because it's waiting on the lock. As Hans Passant said in his answer, the thread scheduler detects this problem and boosts the priority of the lower-priority thread so that it can release the lock. The first MSDN link he posted explains it quite well.
If your low priority thread A holds the lock for a very long time (i.e. it's doing complex calculations on the list) and that's causing problems in your application, then you can do one of the following:
Increase the priority of thread A
Have thread A lock the list, get an item, unlock the list, and then process the item
Lock the list, make a copy, unlock the list, and process the copy.
some variation on or combination of the above
In any case, the problem isn't the lock. The problem is coding the program so that a high-priorty thread can be left waiting for a long time on a data structure that a lower priority thread needs exclusive access to.
Priority inversion is a very generic problem on an operating system that uses thread priority to pick the next thread to schedule. Like Windows. The operating system thread scheduler has specific countermeasures against it, artificially bumping the thread priority when it detects an inversion problem so the low-priority thread is allowed to run and given an opportunity to release the lock. The MSDN page that describes the feature is here. And old KB article with more details is here.
Do not help.
How do you tell the thread scheduler in linux to not interrupt your thread for any reason? I am programming in user mode. Does simply locking a mutex acomplish this? I want to prevent other threads in my process from being scheduled when a certain function is executing. They would block and I would be wasting cpu cycles with context switches. I want any thread executing the function to be able to finish executing without interruption even if the threads' timeslice is exceeded.
How do you tell the thread scheduler in linux to not interrupt your thread for any reason?
Can't really be done, you need a real time system for that. The closes thing you'll get with linux is to
set the scheduling policy to a realtime scheduler, e.g. SCHED_FIFO, and also set the PTHREAD_EXPLICIT_SCHED attribute. See e.g. here , even now though, e.g. irq handlers and other other stuff will interrupt your thread and run.
However, if you only care about the threads in your own process not being able to do anything, then yes, having them block on a mutex your running thread holds is sufficient.
The hard part is to coordinate all the other threads to grab that mutex whenever your thread needs to do its thing.
You should architect your sw so you're not dependent on the scheduler doing the "right" thing from your app's point of view. The scheduler is complicated. It will do what it thinks is best.
Context switches are cheap. You say
I would be wasting cpu cycles with context switches.
but you should not look at it that way. Use the multi-threaded machinery of mutexes and blocked / waiting processes. The machinery is there for you to use...
You can't. If you could what would prevent your thread from never releasing the request and starving other threads.
The best you can do is set your threads priority so that the scheduler will prefer it over lower priority threads.
Why not simply let the competing threads block, then the scheduler will have nothing left to schedule but your living thread? Why complicate the design second guessing the scheduler?
Look into real time scheduling under Linux. I've never done it, but if you indeed do NEED this this is as close as you can get in user application code.
What you seem to be scared of isn't really that big of a deal though. You can't stop the kernel from interrupting your programs for real interrupts or of a higher priority task wants to run, but with regular scheduling the kernel does uses it's own computed priority value which pretty much handles most of what you are worried about. If thread A is holding resource X exclusively (X could be a lock) and thread B is waiting on resource X to become available then A's effective priority will be at least as high as B's priority. It also takes into account if a process is using up lots of cpu or if it is spending lots of time sleeping to compute the priority. Of course, the nice value goes in there too.