I use RtlCaptureStackBackTrace to debug exceptions raised in the main thread of my app when it works on a client's PC. It gives only adresses but its quite enough to locale most problems except when the main thread is frozen.
For this kind of bug I made a watchdog thread like this
procedure TWatchDog.Execute;
var
res: NativeInt;
begin
repeat
If not Terminated then sleep(1000);
res:=SendMessageTimeout(MainFormHandle,WM_NULL,0,0,SMTO_NOTIMEOUTIFNOTHUNG,10000,nil);
If (res=0) and (HangsReported<4) then begin
WasHanged:=true;
APIs.SendMessage('WatchDog *HANG DETECTED* (v.'+Version_Num+') codes: '+ GenCRushRep);
inc(HangsReported);
end;
until Terminated;
end;
The function GenCRushRep returns last 16 stack frames given by RtlCaptureStackBackTrace but it contains nothing helpful about the situation caused the main thread to hang.
Is there any way to collect some more useful info from the main thread in a watchdog thread?
Related
Background
My Flagship App seems to leak memory. Well, not really but just during runtime. Investigating this showed the memory 'leak' is resolved when the app closes, but not in between.
I am using a background thread , that iself starts another thread. Both threads (parent and child) are 'Unobserved' which should result in the TOmniTaskControl object to be released and freed when the tread is finished.
Experimenting code
procedure TfrmMain.MyTestProc(const aTask: IOmniTask);
begin
sleep(100);
SGOutputDebugStringFmt('%s.MyTestProc for %s',[ClassName,aTask.Name]);
sleep(100);
end;
procedure TfrmMain.MyNestedTestProc(const aTask: IOmniTask);
var lTask:IOmniTaskControl;
begin
sleep(100);
SGOutputDebugStringFmt('%s.MyTestProc for %s',[ClassName,aTask.Name]);
lTask:=CreateTask(MyTestProc,'NestedTask');
lTask.Unobserved.Run;
sleep(100);
end;
procedure TfrmMain.btSimpleThreadClick(Sender: TObject);
var lTask:IOmniTaskControl;
begin
lTask:=CreateTask(MyTestProc,'SimpleThread');
lTask.Unobserved.Run;
end;
procedure TfrmMain.btNestedThreadClick(Sender: TObject);
var lTask:IOmniTaskControl;
begin
lTask:=CreateTask(MyNestedTestProc,'NestedThread');
lTask.Unobserved.Run;
end;
When debugging, and setting breakpoint on TOmniTaskControl.Destroy, and having a watch TOmniTaskControl.Name on I see the following:
btSimpleThreadCLick:
TOmniTaskControl for 'SimpleThread' gets to be created
TOmniTaskControl for 'SimpleThread' gets to be destroyed
btNestedThreadCLick:
TOmniTaskControl for 'NestedThread' is Created
TOmniTaskControl for 'NestedTask' is Created
TOmniTaskControl for 'NestedThread' is Destroyed
Problem: TOmniTaskControl for 'NestedTask' is NOT destroyed. Another issue is that the OnTerminate isn't called either.
Then, when closing the app, the TOmniTaskCOntrol for 'SimpleThread' is destroyed. (And also, the OnTherminate is fired)
Workaround
I came up with this solution which seems to do the trick. The thing is however, I do usually NOT run my subthreads from a form where a TOmniEventMonitor is at hand. So I'd have to create a global TOmniEventMonitor object for this.
But isn't this the whole point of the UnObserved method?
procedure TfrmMain.MyNestedTestProc(const aTask: IOmniTask);
var lTask:IOmniTaskControl;
begin
sleep(100);
SGOutputDebugStringFmt('%s.MyTestProc for %s',[ClassName,aTask.Name]);
lTask:=CreateTask(MyTestProc,'NestedTask');
lTask.MonitorWith(OmniEventMonitor).Run; // OmniEventMonitor is a component on my form
sleep(100);
end;
Well, secondary workarounbd... kind-of. It does not allow for my thread to be freed unattended.
if I change the NestedTestProc to the code below, then the NestedTaskgets to be destroyed at the expected moment. Unfortunately, this solution is clearly not 'Unobserved'
procedure TfrmMain.MyNestedTestProc(const aTask: IOmniTask);
var lTask:IOmniTaskControl;
begin
sleep(100);
SGOutputDebugStringFmt('%s.MyTestProc for %s',[ClassName,aTask.Name]);
lTask:=CreateTask(MyTestProc,'NestedTask');
try
lTask.Run;
lTask.WaitFor(2000);
finally
lTask:=nil;
end;
sleep(100);
end;
Update 20201029 >>
An invalid handle (1400) error typically occurs when the master task was completed and its associated Monitor was already destroyed.
So - The 'Master' task thread should not die in case there is an "owned" monitor that is monitoring other threads.
So to check this, I changed the timing (using sleep()) to ensure the child task was completed before the master task is completed.
The Invalid handle error is gone now, and the COmniTaskMsg_Terminated message gets to be posted successfully.
But still the ComniTaskMsg_Terminated from the child task is not processed. (I expected the thread of the MasterTask to handle this.)
IMO there are 2 problems:
life time management of the Unobserved monitor
shutdown management of the Unobserved monitor, which should keep the "owning" thread alive and keep processing messages until all
monitored threads/tasks are gone.
Also I wonder whether these shutdownmessages hould be handled/processed by the Application main thread (It seems this is the case now) or otherwise through a separate thread that checks all the monitors in GTaskControlEventMonitorPool . AIA, pretty complicated stuff :s...
Giving this some thought, monitors that were created by the application main thread (thus Monitor.ThreadID=MainThreadID) should be handle their messages in the main thread message loop, and all others probably need to be handled by a separate thread... Its just too confusing! I will see if I write a unit test for this, just to demonstrate what I expect to happen.
<< Update 20201029
The question
With OmniThreadLibrary, How can I use unobserved threads inside threads, and avoid the described memory leak?
Our program creates a background thread at the beginning of the program. The background thread does some database integrity checks and checks for stuff in the Internet using Indy. After 10 seconds, the background thread should be finished and since FreeOnTerminate is true, it will also clean itself up.
We have noticed that in some cases, if the user closes the program too quickly, the process will still be alive until the background thread is finished.
Since we couldn't exactly reproduce the issue, I have created a demo project to try a few things:
type
TBackgroundThread = class(TThread)
protected
procedure Execute; override;
end;
{ TForm1 }
var
bt: TBackgroundThread;
procedure TForm1.FormCreate(Sender: TObject);
var
i: integer;
begin
// Create a background thread which runs X seconds and then terminates itself.
bt := TBackgroundThread.Create(false);
bt.FreeOnTerminate := true;
end;
procedure TForm1.FormShow(Sender: TObject);
begin
// The user closes the app while the background thread is still active
Sleep(2000);
Close;
end;
{ TBackgroundThread }
procedure TBackgroundThread.Execute;
var
i: integer;
x: cardinal;
begin
inherited;
// Simulate some work that the background thread does
x := MaxInt;
for i := 0 to MaxInt do
begin
x := Random(x);
end;
end;
The result is a bit surprising to me: After I close the MainForm, the process will be immediately terminated and the background thread will get hard-killed.
Now I have a few questions about this:
After the closing of the MainForm (= exit of the main thread), should I manually terminate all created threads via .Terminate or will that be done automatically?
Shall my threads only check for Self.Terminated or should they also check for Application.Terminated ?
Why does my busy thread as shown above gets immediately killed when I close the application? I expected that the process Project1.exe will run until all threads have finished by themselfes. (And as described above, we had seen an application where the main form is closed, but a thread is preventing the process of being closed).
How is it possible then, that our real application's process does not terminate because of a running background thread? Might it have something to do with the Internet stuff, which might cause the app to wait until a connection timeout is reached?
Closing the main form is not synonymous with exiting the main thread. Code continues to run after the form is closed. In particular, units are finalized.
If you handle your test thread's OnTerminate event, or put a breakpoint in the Terminate method, you'll see that it's not called automatically when your program exits. You'll have to call it yourself. But note also that a thread doesn't stop running just because Terminate is called. It continues running until it stops itself or it's terminated forcefully. Call WaitFor to wait for it to terminate.
Don't bother checking Application.Terminated; the thread's property should be sufficient.
Your thread gets terminated forcefully as your program exits because eventually your program calls ExitProcess, and one of the things the OS does there is to terminate all other threads. It doesn't call Terminate on them because the OS doesn't know about Delphi classes and methods.
You'll have to do some more debugging to determine why your program doesn't terminate promptly for your customers. You say you can't reproduce the problem in house, and you've written a test program that doesn't exhibit the problem, either. You'll have to find a customer who will cooperate with your further debugging efforts. Do you really know it's the thread that's holding things up, or is that just a guess so far?
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I want to run my Thread then run application instruction. why run all after Sleep?
I have a TQuery(it has many record and slow fetch) instead Sleep and when Open, Thread no run before Open TQuery.
ShowMessage and Sleep are for test.
What's solution?
TCustomThread = class(TThread)
public
procedure Execute; override;
procedure doProc;
end;
.
.
.
procedure TCustomThread.Execute;
begin
Synchronize(doProc);
end;
procedure TCustomThread.doProc;
begin
ShowMessage('Thread');
end;
.
.
.
procedure TForm1.Button1Click(Sender: TObject);
var
thrd : TCustomThread;
begin
thrd := TCustomThread.Create(True);
thrd.Resume;
Sleep(3000);
ShowMessage('Form');
end;
I understand your question to be
Why does the form message appear before the thread message?
Simply put, your use of Synchonize means that all the code executes on the main thread. This means that the first message box must wait for the second message box to complete.
So, why does the form message box show first? The Synchronize method is used from threads to invoke code on the main thread. This Synchronize method signals the main thread that there is work to be done and then blocks until the main thread can do it.
The main thread won't start this work whilst it is busy in an event handler as your program is. So the call to Sleep actually blocks both threads because both threads are waiting on the main thread.
So to re-cap, the Synchronize method blocks until the main thread completes the work described in the procedure passed to Synchronize. If the main thread is busy then Synchronize blocks until the main thread completes its busy tasks.
Looking more broadly at your question you describe long running database tasks happening on the main thread. There is the fundamental problem. You must not have those tasks on the main thread. The main thread is dedicated to serving the GUI. That requires it to be responsive at all times and be able to service its message queue. Putting long running tasks on the main thread will destroy your GUI. Put those tasks on worker threads away from the main thread.
I suspect that you are using Synchronize because you've realised that the GUI is unresponsive because of your database code. And you reason that you can use a thread to keep the GUI responsive. But that can never work. The main thread must process messages in a timely manner. If the database code does not do so then you cannot service the message queue from the outside. There has to be co-operation.
The bottom line here is that long-running tasks must execute away from the main thread.
Button1Click is stopping the execution of the main thread for 3 seconds.
During the sleep(3000) execution, no GDI message is processed, so the dialog boxes are NOT displayed immediately.
In fact, TCustomThread.doProc works as expected, but the dialog box is not displayed until the GDI messages are processed.
You have to change this method e.g. into:
procedure TForm1.Button1Click(Sender: TObject);
var
thrd : TCustomThread;
expectedEnd: TDateTime;
begin
thrd := TCustomThread.Create(True);
thrd.Resume;
expectedEnd := Now+(1/24/60/60*3);
repeat
Sleep(50); // or any long process
Application.ProcessMessages; // to be called to process the GDI messages
until Now>=expectedEnd;
ShowMessage('Form');
end;
In short: never use Sleep() in the main GDI thread, for extended period of time (more than some ms), otherwise your application won't be responsive any more. And Windows will complain for it and ask you to kill the application!
So in your case, you have to call Application.ProcessMessages in your main thread when a long process is taken place (e.g. a TQuery), or run the query in a background thread (this is IMHO the preferred method), as we do e.g. for our Open Source mORMot framework on client side.
Hello
At my form I create TFrame at runtime. At this frame I create background thread with executes commands in endless loop. But when I destroy this frame I should destroy this thread.
I try
thread.Suspend;
thread.Terminate;
FreeAndNil(thread);
but get AV and ThreadError.
How should i destroy thread?
You must make sure that the thread exits its Execute method to terminate it properly.
Code could be something like this:
procedure TThread.Execute;
begin
while not Self.Terminated do
begin
//do something
end;
end;
Call this when You want to destroy thread:
thread.Terminate;
thread.WaitFor;
FreeAndNil(thread);
It is sufficient to do thread.Terminate. But you will probably want to set thread.FreeOnTerminate := true when it is created.
Of course, in the tight loop of your thread (that is, in Execute), you need to check if the thread has been requested to terminate (check the Terminated property). If you find that the thread has been requested to terminate, simply break from the loop and exit from Execute.
You should never call suspend on a tthread its not safe to do so and resume should only be used to start a thread that was created suspended.
In Delphi 2010 the suspend and resume where depreciated and the method start was introduced to reinforce this.
For a more complete explanation see this thread at Codegears forums.
having said that there are 2 ways I will terminate and free a tthread.
1: I Set FreeOnTerminate when the thread is created so I just call.
Thread.Terminate;
2: Free the thread explicitly, as I need to read a result from a public property before the thread is freed and after it has terminated.
Thread.Terminate;
Thread.WaitFor;
//Do somthing like read a public property from thread object
if Thread <> nil then FreeAndNil(Thread);
In the main execute loop it may be a good idea to put some exception handling in.
Or you may be left wondering why the thread appears to terminate its self. This may be causing the AV if the thread is set to FreeOnTerminate and it has already been freed when you try to free it.
procedure TThread.Execute;
begin
while not Terminated do
begin
try
//do something
except
on E:Exception do
//handle the exception
end;
end;
end;
I am having a problem with some code that was written by a developer that has now left our company, the code implements a tcpserver that responds to an XML based protocol. This appears to be working absolutely fine in our test environment but one or two customers are having problems with application shutdown.
I have traced this to what appears to be a deadlock when tidtcpserver.active = false is called. I am already aware that a deadlock can be caused by one of the connection treads making a synchronised call to the main thread, whilst the main thread is waiting for the connection threads to terminate.
I am already using a tidthreadsafestringlist to pass the data to the main thread for processing, and where I need to call a procedure from the main thread I have created a tidnotify descendant to do this. can anyone think of anything else to look for.
I had already been checking the exception handling,
this is what i have in the onexecute event
try
// code to handle connection including tidnotify etc....
except
on E:Exception do
begin
if (e.InheritsFrom(EIdSilentException) = False) then
TXMLTraceNotify.XMLTrace('TCPServerExecute: ' + E.Message,ttProblem, FTraceProc);
raise; //we must raise all exceptions for indy to handle them.
end;
end;
Here is how i'm using the TS-stringlist
Declaration.
public
TransactionStrings: TIdThreadSafeStringList;
its created in the constructor and destroyed in the destructor.
this is how i'm adding to it in the context of the tcpserver.
TransactionStrings.Add(newTrans.AsString);
And this is how i'm reading from it in the context of the main application thread
slXMLTrans := TStringList.Create;
try
slTemp := FCustomXMLServer.TransactionStrings.Lock;
try
slXMLTrans.Assign(slTemp);
slTemp.Clear;
finally
FCustomXMLServer.TransactionStrings.Unlock;
end;
if slXMLTrans.Count > 0 then
begin
for i := 0 to Pred(slXMLTrans.Count) do
TAbstractTerminal.ProcessXMLTrans(slXMLTrans[i]);
slXMLTrans.Clear;
end;
finally
slXMLTrans.Free;
end;
I think this is the correct way to use it but I await your comments.
Deadlocks when setting the Active property to False are a classic, and often discussed, issue when using TIdTCPServer incorrectly. The only way the server deadlocks is when the connection threads do not terminate correctly. That MAY be caused by synchronizing operaton to the main thread while the main thread is busy deactivating the server, in which case the use of TIdNotify would does eliminate any such deadlock conditions.
However, that is not the only way to prevent a connection thread from terminating. Another possibility is if your server event handlers have exception handling in them that is blocking Indy's internal notifications from being processed. That can cause runaway threads that keep running but never know that they need to stop. If you are catching exceptions in your code, make sure you re-throw any EIdException-derived exceptions and let the server handle them internally.