I'm running vxWorks 6.3 and have run into a problem. I have a series of tasks running as in an RTP. I create a task, do stuff then destroy the task. Then create two tasks, very close together, do some stuff and destroy them. These tasks have to do crazy things like, malloc and free memory. Unfortunately, if I do this enough times, one of the tasks will get stuck in the memory (both malloc and free) routines on a semaphore. It's always the second task that gets "lost" at the very start of the task in either free or malloc. After the failure, I can still create tasks and I can still malloc memory. The failing task sits forever, waiting for the semaphore... A semaphore that other tasks MUST be using.
Does anyone have any idea how a task can get stuck in the memory routines?
0x08265e58 malloc +0x2c : 0x082416f4 ()
0x08267e50 memPartAlloc +0x28 : 0x08241734 ()
0x08267e0c memPartAlignedAlloc+0x70 : 0x08267c04 ()
0x08267c7c memPartFree +0xfc : 0x08240654 ()
0x082753c0 semTake +0x90 : 0x08242534 ()
0x082752ec semUMTake +0xd8 : 0x08242514 ()
---- system call boundary ----
-> tw 0x69d21b0
NAME ENTRY TID STATUS DELAY OBJ_TYPE OBJ_ID OBJ_NAME
---------- ---------- ---------- ---------- ----- ---------- ---------- --------
tHttp631-2 0x827dbfc 0x69d21b0 PEND 0 SEM_M 0x6859650 N/A
Semaphore Id : 0x6859650
Semaphore Type : MUTEX
Task Queuing : PRIORITY
Pended Tasks : 1
Owner : 0x69d1a08 Deleted!
Options : 0xd SEM_Q_PRIORITY
SEM_DELETE_SAFE
SEM_INVERSION_SAFE
VxWorks Events
--------------
Registered Task : NONE
Event(s) to Send : N/A
Options : N/A
Pended Tasks
------------
NAME TID PRI TIMEOUT
---------- -------- --- -------
tHttp631-25502 69d21b0 120 0
value = 0 = 0x0
->
It is recommended that you allocate enough memory for the worst case at init time, and then just re-use that memory throughout the duration of your program. Especially if you actually have real time requirements as malloc/free are non-deterministic operations, I also recommend re-using the tasks rather that recreating new tasks at runtime, then use a semaphore or msgQueue to kick off the appropriate tasks at the appropriate times. So your program flow might look something like this:
initTime()
{
t1mem = malloc(t1memSize);
t2mem = malloc(t2memSize);
t3mem = malloc(t3memSize);
t1q = msgQCreate(qlen, msglen, MSG_Q_FIFO);
t2q = msgQCreate(qlen, msglen, MSG_Q_FIFO);
t3q = msgQCreate(qlen, msglen, MSG_Q_FIFO);
rspq = msgQCreate(qlen, msglen, MSG_Q_FIFO);
taskSpawn("t1", t1pri, ..., t1Entry, t1mem, t1q, rspq, ...);
taskSpawn("t2", t2pri, ..., t2Entry, t2mem, t2q, rspq, ...);
taskSpawn("t3", t3pri, ..., t3Entry, t3mem, t3q, rspq, ...);
runTime(t1sem, t2sem, t3sem, rspq);
msgQDelete(t1q);
msgQDelete(t2q);
msgQDelete(t3q);
msgQDelete(rspq);
free(t1mem);
free(t2mem);
free(t3mem);
}
runTime(MSG_Q_ID t1q, MSG_Q_ID t2q, MSG_Q_ID t3q, MSG_Q_ID rspq)
{
while (programRun)
{
tasksDone = 0;
msgQSend(t1q, t1start, msglen, 100, MSG_PRI_NORMAL);
if (msgQReceive(rspq, buf, msglen, errorCaseTimeout) == OK)
{
// check to make sure the msg is t1done...
// report error if it isn't...
msgQSend(t2q, t2start, msglen, 100, MSG_PRI_NORMAL);
msgQSend(t3q, t3start, msglen, 100, MSG_PRI_NORMAL);
for (int x = 0; x < 2; x++)
{
if (msgQReceive(rspq, buf, msglen, errorCaseTimeout) == OK)
{
// check to make sure the msg is t2done/t3done...
// report error if it isn't...
tasksDone++;
}
}
}
if (tasksDone == 2)
{
// everything is good... keep on running...
}
else
{
// a task didnt finish within the errorCaseTimeout time...
// report error or something, maybe set programRun to false...
}
}
}
t1Entry(void* mem, MSG_Q_ID q, MSG_Q_ID rspq)
{
while (programRun)
{
if (msgQReceive(q, buf, msglen, 100) == OK)
{
doTask1(mem);
msgQSend(rspq, t1done, msglen, 100, MSG_PRI_NORMAL);
}
}
}
t2Entry(void* mem, MSG_Q_ID q, MSG_Q_ID rspq)
{
while (programRun)
{
if (msgQReceive(q, buf, msglen, 100) == OK)
{
doTask2(mem);
msgQSend(rspq, t2done, msglen, 100, MSG_PRI_NORMAL);
}
}
}
t3Entry(void* mem, MSG_Q_ID q, MSG_Q_ID rspq)
{
while (programRun)
{
if (msgQReceive(q, buf, msglen, 100) == OK)
{
doTask3(mem);
msgQSend(rspq, t3done, msglen, 100, MSG_PRI_NORMAL);
}
}
}
Obviously the above code is not very DRY, and not all error cases are fully handled, but it is a start and has a good chance of working deterministically.
A few questions:
Are all the tasks created/deleted in an RTP?
How are you "destroying the task"?
When then task blocks, are new malloc/tasks creation in the same RTP or different RTP?
Are you deleting entire RTPs?
It sounds like you are using a taskDelete from one task to destroy those other tasks. If that's the case, then it is possible that a task is being deleted in the middle of a memory operation.
Since this is a malloc operation in an RTP, each RTP created contains it's own heap (malloc) semaphore. I would think this would be the semaphore being held.
I would suggest contacting Wind River support. This might be an issue they are familiar with.
This may be related to a problem I am having, although I am seeing a different symptom.
In both cases the owner of a semaphore is being deleted. In my case, I am having the tWebTask hang and tracing that to a missing semaphore owner on a web socket.
Here's the link to my SO question.
Related
Using shared memory with the shmget() system call, the aim of my C++ program, is to fetch a bid price from the Internet through a server written in Rust so that each times the value changes, I m performing a financial transaction.
Server pseudocode
Shared_struct.price = new_price
Client pseudocode
Infinite_loop_label:
Wait until memory address pointed by Shared_struct.price changes.
Launch_transaction(Shared_struct.price*1.13)
Goto Infinite_loop
Since launching a transaction involve paying transaction fees, I want to create a transaction only once per buy price change.
Using a semaphore or a futex, I can do the reverse, I m meaning waiting for a variable to reachs a specific value, but how to wait until a variable is no longer equal to current value?
Whereas on Windows I can do something like this on the address of the shared segment:
ULONG g_TargetValue; // global, accessible to all process
ULONG CapturedValue;
ULONG UndesiredValue;
UndesiredValue = 0;
CapturedValue = g_TargetValue;
while (CapturedValue == UndesiredValue) {
WaitOnAddress(&g_TargetValue, &UndesiredValue, sizeof(ULONG), INFINITE);
CapturedValue = g_TargetValue;
}
Is there a way to do this on Linux? Or a straight equivalent?
You can use futex. (I assumed "var" is in shm mem)
/* Client */
int prv;
while (1) {
int prv = var;
int ret = futex(&var, FUTEX_WAIT, prv, NULL, NULL, 0);
/* Spurious wake-up */
if (!ret && var == prv) continue;
doTransaction();
}
/* Server */
int prv = NOT_CACHED;
while(1) {
var = updateVar();
if (var != prv || prv = NOT_CACHED)
futex(&var, FUTEX_WAKE, 1, NULL, NULL, 0);
prv = var;
}
It requires the server side to call futex as well to notify client(s).
Note that the same holds true for WaitOnAddress.
According to MSDN:
Any thread within the same process that changes the value at the address on which threads are waiting should call WakeByAddressSingle to wake a single waiting thread or WakeByAddressAll to wake all waiting threads.
(Added)
More high level synchronization method for this problem is to use condition variable.
It is also implemented based on futex.
See link
I'm currently trying to make my device (STM32F105) which is usually running 12 threads on CMSIS RTOS go to low power mode. In order to simplify the algorythm I think (definitely not sure) that it's a good idea to terminate all the threads using osThreadTerminate and after a wake up recreate them using osThreadCreate
void os_idle_demon (void) {
/* The idle demon is a system thread, running when no other thread is */
/* ready to run. */
for (;;) {
/* HERE: include optional user code to be executed when no thread runs.*/
if (Sleep.SleepEnabled == 1)
{
if (Sleep.IsSleeping == 1)
{
// __wfi();
// PWR_EnterSTOPMode(PWR_Regulator_ON, PWR_STOPEntry_WFI); //PWR_Regulator_LowPower
__nop();
// osDelay(5000);
if (Sleep.WakeUp)
{
Sleep.IsSleeping = 0;
WakeUp();
// SetSysClock();
Sleep.WakeUp = 0;
Sleep.SleepEnabled = 0;
Sleep.TimeTillSleep = 60;
}
}
else
{
if (Sleep.TimeTillSleep == 0 )
{
TerminateTasks();
ResetPeripherals();
Sleep.IsSleeping = 1;
// PWR_EnterSTANDBYMode();
// __wfi();
// PWR_EnterSTOPMode(PWR_Regulator_ON, PWR_STOPEntry_WFI);
__nop();
// osDelay(5000);
}
}
}
}
}
As you can see I use some global variables to determinte when to sleep. TerminateTasks(); is used to terminate all of my running threads using osThreadTerminate function which doesn't seem to cause any trouble, but after I call WakeUp(); which uses osThreadCreate function to recreate terminated threads I run into an os stack overflow. So there are a few questions I struggle to find answers to. Does osThreadTerminate command in CMSIS-RTOS release stack after execution? Is there a better way to go into a low power mode ? I hope I made my point clear, if there's a need to be more specific let me know. Would be grateful if you shared your experience with similar problems.
Do you use dynamic allocation in your other thread ? Because if so, killing your thread when there are running could result in memory leak.
I have N threads performing various task and these threads must be regularly synchronized with a thread barrier as illustrated below with 3 thread and 8 tasks. The || indicates the temporal barrier, all threads have to wait until the completion of 8 tasks before starting again.
Thread#1 |----task1--|---task6---|---wait-----||-taskB--| ...
Thread#2 |--task2--|---task5--|-------taskE---||----taskA--| ...
Thread#3 |-task3-|---task4--|-taskG--|--wait--||-taskC-|---taskD ...
I couldn’t find a workable solution, thought the little book of Semaphores http://greenteapress.com/semaphores/index.html was inspiring. I came up with a solution using std::atomic shown below which “seems” to be working using three std::atomic.
I am worried about my code breaking down on corner cases hence the quoted verb. So can you share advise on verification of such code? Do you have a simpler fool proof code available?
std::atomic<int> barrier1(0);
std::atomic<int> barrier2(0);
std::atomic<int> barrier3(0);
void my_thread()
{
while(1) {
// pop task from queue
...
// and execute task
switch(task.id()) {
case TaskID::Barrier:
barrier2.store(0);
barrier1++;
while (barrier1.load() != NUM_THREAD) {
std::this_thread::yield();
}
barrier3.store(0);
barrier2++;
while (barrier2.load() != NUM_THREAD) {
std::this_thread::yield();
}
barrier1.store(0);
barrier3++;
while (barrier3.load() != NUM_THREAD) {
std::this_thread::yield();
}
break;
case TaskID::Task1:
...
}
}
}
Boost offers a barrier implementation as an extension to the C++11 standard thread library. If using Boost is an option, you should look no further than that.
If you have to rely on standard library facilities, you can roll your own implementation based on std::mutex and std::condition_variable without too much of a hassle.
class Barrier {
int wait_count;
int const target_wait_count;
std::mutex mtx;
std::condition_variable cond_var;
Barrier(int threads_to_wait_for)
: wait_count(0), target_wait_count(threads_to_wait_for) {}
void wait() {
std::unique_lock<std::mutex> lk(mtx);
++wait_count;
if(wait_count != target_wait_count) {
// not all threads have arrived yet; go to sleep until they do
cond_var.wait(lk,
[this]() { return wait_count == target_wait_count; });
} else {
// we are the last thread to arrive; wake the others and go on
cond_var.notify_all();
}
// note that if you want to reuse the barrier, you will have to
// reset wait_count to 0 now before calling wait again
// if you do this, be aware that the reset must be synchronized with
// threads that are still stuck in the wait
}
};
This implementation has the advantage over your atomics-based solution that threads waiting in condition_variable::wait should get send to sleep by your operating system's scheduler, so you don't block CPU cores by having waiting threads spin on the barrier.
A few words on resetting the barrier: The simplest solution is to just have a separate reset() method and have the user ensure that reset and wait are never invoked concurrently. But in many use cases, this is not easy to achieve for the user.
For a self-resetting barrier, you have to consider races on the wait count: If the wait count is reset before the last thread returned from wait, some threads might get stuck in the barrier. A clever solution here is to not have the terminating condition depend on the wait count variable itself. Instead you introduce a second counter, that is only increased by the thread calling the notify. The other threads then observe that counter for changes to determine whether to exit the wait:
void wait() {
std::unique_lock<std::mutex> lk(mtx);
unsigned int const current_wait_cycle = m_inter_wait_count;
++wait_count;
if(wait_count != target_wait_count) {
// wait condition must not depend on wait_count
cond_var.wait(lk,
[this, current_wait_cycle]() {
return m_inter_wait_count != current_wait_cycle;
});
} else {
// increasing the second counter allows waiting threads to exit
++m_inter_wait_count;
cond_var.notify_all();
}
}
This solution is correct under the (very reasonable) assumption that all threads leave the wait before the inter_wait_count overflows.
With atomic variables, using three of them for a barrier is simply overkill that only serves to complicate the issue. You know the number of threads, so you can simply atomically increment a single counter every time a thread enters the barrier, and then spin until the counter becomes greater or equal to N. Something like this:
void barrier(int N) {
static std::atomic<unsigned int> gCounter = 0;
gCounter++;
while((int)(gCounter - N) < 0) std::this_thread::yield();
}
If you don't have more threads than CPU cores and a short expected waiting time, you might want to remove the call to std::this_thread::yield(). This call is likely to be really expensive (more than a microsecond, I'd wager, but I haven't measured it). Depending on the size of your tasks, this may be significant.
If you want to do repeated barriers, just increment the N as you go:
unsigned int lastBarrier = 0;
while(1) {
switch(task.id()) {
case TaskID::Barrier:
barrier(lastBarrier += processCount);
break;
}
}
I would like to point out that in the solution given by #ComicSansMS ,
wait_count should be reset to 0 before executing cond_var.notify_all();
This is because when the barrier is called a second time the if condition will always fail, if wait_count is not reset to 0.
I have a main thread which creates another thread to perform some job.
main thread has a reference to that thread. How do I kill that thread forcefully some time later, even if thread is still operating. I cant find a proper function call that does that.
any help would be appreciable.
The original problem that I want to solve is I created a thread a thread to perform a CPU bound operation that may take 1 second to complete or may be 10 hours. I cant predict how much time it is going to take. If it is taking too much time, I want it to gracefully abandon the job when/ if I want. can I somehow communicate this message to that thread??
Assuming you're talking about a GLib.Thread, you can't. Even if you could, you probably wouldn't want to, since you would likely end up leaking a significant amount of memory.
What you're supposed to do is request that the thread kill itself. Generally this is done by using a variable to indicate whether or not it has been requested that the operation stop at the earliest opportunity. GLib.Cancellable is designed for this purpose, and it integrates with the I/O operations in GIO.
Example:
private static int main (string[] args) {
GLib.Cancellable cancellable = new GLib.Cancellable ();
new GLib.Thread<int> (null, () => {
try {
for ( int i = 0 ; i < 16 ; i++ ) {
cancellable.set_error_if_cancelled ();
GLib.debug ("%d", i);
GLib.Thread.usleep ((ulong) GLib.TimeSpan.MILLISECOND * 100);
}
return 0;
} catch ( GLib.Error e ) {
GLib.warning (e.message);
return -1;
}
});
GLib.Thread.usleep ((ulong) GLib.TimeSpan.SECOND);
cancellable.cancel ();
/* Make sure the thread has some time to cancel. In an application
* with a UI you probably wouldn't need to do this artificially,
* since the entire application probably wouldn't exit immediately
* after cancelling the thread (otherwise why bother cancelling the
* thread? Just exit the program) */
GLib.Thread.usleep ((ulong) GLib.TimeSpan.MILLISECOND * 150);
return 0;
}
Is it possible to pause a different task than the one on behalf of which the kernel is currently executing? To stop the current task, one can just set it to inactive and call schedule, but what about a different one?
What I have currently:
void disable_thread(struct task_struct *tsk) {
if (tsk->state == TASK_RUNNING) {
/*
* A running task - mark it stopped and wait for it to be descheduled
*/
tsk->state = TASK_INTERRUPTIBLE;
wait_task_inactive(tsk, TASK_INTERRUPTIBLE);
} else if (tsk->state == TASK_INTERRUPTIBLE || tsk->state == TASK_UNINTERRUPTIBLE) {
/*
* // TODO: what to do with tasks already waiting for something else?
*/
} else {
/*
* This task's state seems to indicate that it's dead, no need to disable it anymore.
*/
}
}
Stopping a running thread seems to work this way, but what can we do if the thread is already waiting for something else (e.g. waiting to acquire a lock), to prevent it from restarting even if it would get the lock?
I'm implementing a security feature, and can give more context if needed.
Thanks in advance.