Question: I'm interested to know the best practice for killing a long standing operation that is running as a background thread (lets call this thread WorkerThread) in Java 1.2.
Scenario
Specifically, I'm developing an application for Blackberry devices whereby I make a HTTP connection. Big picture: a URL request if forwarded to a background thread (WorkerThread), the thread makes the request and returns the result using a call back.
Scenario Details
Now there exists a situation where at connection time, a data connection exists but then for whatever reason (drives through a tunnel) that connection no longer exists. Due to a limitation in Blackberry's design architecture, that actual connection will hang as the time out is fixed to be 2 minutes. As a result, there's a crucial need to kill a connection that has been hanging for a relatively (15 seconds) long period of time.
My Current Solution - 2 Theads?
Right now my current solution is to run WorkerThread inside another thread (lets call this new thread MonitorThead). MonitorThread starts WorkerThread, sleeps for 1000ms and then routinely checks if WorkerThread is still alive. If after 15 seconds WorkerThread is still alive, MonitorThread puts WorkerThread to sleep and exits. Is this really the best possible approach?
Summary of Question & Key points
In summary, below is the core question and key restraints associated with the question. Cheers!
How do I successful kill a java background thread that is stuck in a specific operation?
Scenario Restraints:
No control of having operation pause
and check the threads requested state
Specific to Blackberry's
implementation of Java ME 1.2 and its
Thread API so no explicit
kill() method
Most concerned about the best practice and how to
most safely kill a holding thread.
Follow Up/Edit
Neil Coffey recommended that I simply hold a reference to the connection object and instead call close() on that object. I am currently looking into this...
How to kill a Thread is a difficult question. There is no guaranteed way to be able to stop or interrupt a Thread. However, if you take your current architecture and upon timeout, just close the stream (not the Connection), that should cause an I/O Exception to occur on the thread that is stuck in I/O. If it doesn't cause an IOException, then it should at least cause the read or write to return with EOF.
Note that closing the Connection doesn't help, as the JavaDoc says:
Any open streams will cause the connection to be held open until
they themselves are closed.
You have to close the stream that was derived from the Connection.
Well, the best practice would normally be to get the connection to close, and then let the consequences of that ripple through to the thread, allowing it to exit cleanly.
How are you making the connection? Rather than waiting for it to time out, what are your chances of forcing it to close? Can you get hold of some connection object? Does the Blackberry have some other command that can be executed to kill a given connection?
I always believed that passing a flag into background thread in an atomic transaction has been the best way to ask a thread to stop. If it doesn't stop for a while, kill it.
Well, to add to that, if you believe 2 minutes is a long time, good for you. I'd let the user decide what's a long time with a cancel button.
Related
Is there anyway in F# that I can detect if a current waiting thread is waiting for too long without being contacted?
I have a case where threads must be actively contacting other waiting threads to pass their work to once they're finished. My solution is having a bug somewhere that sometimes one or more threads just wait for too long and eventually the program got deadlocked because other threads don't contact them.
I think by detecting if a waiting thread is simply waiting for too long, it will just actively go looking for available work, rather than keeping waiting for other threads to pass to it.
It's probably better to try and understand why your threads are getting stuck than just terminating them. If you can reproduce this with the Visual Studio debugger attached, you can click the Pause button and use the Threads window to see what code all threads are in.
That said; if you still have the need to do this, the solution will depend on how you're managing your threads. To monitor them from the outside, you'll need some process that has a list of threads and the ability to tell whether they're dead.
The Thread class doesn't appear have any built-in mechanism for sharing state between the thread and its control except for Name. You could possibly abuse name, but I would probably have a thread-safe collection (eg. a ConcurrentDictionary<Thread, DateTime>) to store all of the threads and the timestamp of their last communication, and pass an Action into each thread when it's started that allows it to "Ping" by calling the action periodically. The action would simply update the DateTime stored against that thread.
The controlling process then simply scans through the dictionary periodically for anything with a timestamp that is too old, declares that thread dead and Aborts() it.
It's hard to give a code sample without knowing exactly how you're spawning your threads and describe what a thread "being contacted" means in more detail.
I am writing a server application in which there is a thread deployed to read/write many sockets connecting to clients. My manager tells me that it is not a good design, because if the thread aborts due to unknown reason then all the read/write work will stop forever.
So I wonder in what conditions will a thread abort, except the case we return from the Run() function of a thread. Do we need consider the case that the thread stops running abnormally?
It depends. One thread per client can be a bad thing scalability wise, especially if the thread doesn't do that much work per client. In that circumstance it can be better to have a thread that handles a number of clients, the idea to achieve a good balance between the number of threads and having them do a decent amount of work.
If on the other hand each thread is doing a lot of work per client then one thread isn't such a bad idea, the overhead of the thread not being significant in comparison to the work load.
So setting that aside, a thread will abort if your code is written so that the thread returns or self-terminates. If another thread in your program knows the thread's handle/id then the library you're using may have a function with a name like thread_kill(). That would allow that other thread to kill this thread, though that's almost always a bad idea.
So as far as I'm concerned your thread will only abort and disappear if you've written your code to make that happen deliberately.
Handling exceptions is probably best done in its entirety within the thread where the exception arose. I've never tried to do otherwise (still writing in pure C), but the word is that it's difficult to handle them outside the thread. Irrespective of whether each thread handles one or many clients you still have to handle all errors and events within thread.
It may be simpler to get that correct if you write I so that a thread handles handles a single client. Getting it wrong could lead to a thread getting into a stalled state (eg waiting for the client that is listening too) and accumulating those as time goes past will eventually kill your whole system.
I am writing a server application in which there is a thread deployed to read/write many sockets connecting to clients.
Not a good design. There should be at least one thread per client, in some circumstances two: one to read and one to write. If you're dealing in blocking I/O, servicing one client could block out all the others. (If you're dealing in non-blocking I/O you don't need threads at all.)
My manager tells me that it is not a good design, because if the thread aborts due to unknown reason then all the read/write work will stop forever.
He's right, for more reasons than he is advancing.
Using epoll_wait to manage multiple connections using multiple threads, there is a risk trying to release custom data associated with a closed socket.
Consider the following scenario, where T is the custom data :
Data is received,
Because of 1, thread A deblocks from epoll_wait and processes the event (access T)
At same time, another thread B, wants to close the connection
Thread B can't assume that T can be safely deleted, eventhough the call to close will immediatly remove the socket from the epoll.
I had the following standard idea :
Maintain a variable within T that gets incremented each time a call to write/read returns EAGAIN, and gets decremented each time the socket is ready.
When close is called, wait for that variable to go down to zero before deleting T.
The issue I experienced is that if close is called, epoll_wait does not return an indication of a cancellation of previous calls to arm the socket.
Anybody had this same problem ? How did you managed to overcome it ?
At least three possible ways here:
Do not use threads, simple and clean, and usually works.
Have a dedicated thread do all file descriptor polling and publish events to a pool of worker threads that do actual I/O and processing.
Have one epoll(7) instance per thread, so threads manage non-intersecting sets of descriptors, with the exception of maybe the listening socket(s) to get these sets populated, and some control mechanism like eventfd(2), or self-pipe(2) to be able to shutdown the whole rig cleanly.
Hope this helps.
After many research, I found this recent and remarkable article :
http://lwn.net/Articles/520012/
Basically it acknowledge the issue I am describing and speaks about a possible future patch to Linux kernel that allows to extend the epoll API in a way that solves the issue.
The extension bring a new command called : EPOLL_CTL_DISABLE.
When it is issued, and by means of return value, the calling thread will know if some other thread has just been deblocked from epoll_wait upon same socket.
This can help know the safe moment of closure and release of custom data.
I'm writing an application that must do some things in background: check emails and parse them to inject some data in a database, and connect to a web service to check status for some asynchronous operations.
Right now, my solution is a simple timer that performs these operations on a predefined schedule: email every five minutes, and web service checks every minute (but these are only performed if there is pending activity, so most of the time this does nothing.)
Right now I'm not using a thread for this (I'm early in the development stage.) But my plan is to create a background thread and let it do the work offline.
Couple of questions:
I plan to control everything in the timer(s). Set up a global variable (rudimentary "locking",) start the thread. If the "lock" is already set, ignore it. The thread cleans it up on termination. Should I use a more robust locking / queue mechanism for my threads? (I already have OmniThread installed)
How can I run a thread with low priority? I don't want the application to feel sluggish when the background thread is performing data insertion or networking.
Is there a clean way to verify for user activity and start this thread only when the user is not busy at the keyboard / mouse?
Please have in mind that I'm not experienced with threads. I wrote an FTP sync application once so I'm not a complete newbie, but that was long time ago.
For part 3 of your question, the Windows API has a GetLastInputInfo function which should return information about the last time the user did something. It even says it's
"This function is useful for input idle detection". I did plan to use this for something myself, but haven't had a chance to test it.
Edit: Delphi implementation link
I plan to control everything in the timer(s). Set up a global variable (rudimentary "locking",) start the thread. If the "lock" is already set, ignore it. The thread cleans it up on termination. Should I use a more robust locking / queue mechanism for my threads? (I already have OmniThread installed)
I wouldn't bother with the Timer at all. Make your thread's loop look like this, and you'll have your delays. You will NOT need a lock because there's only one thread, it will not sleep until the previous job is over.
procedure YourThread;
var N: Integer;
begin
while not Terminated do
begin
// Figure out if there's a job to do
// Do the job
// Sleep for a while, but give the thread a chance to notice
// it needs to terminate.
for N := 1 to 500 do
if not Terminated then
Sleep(100);
end;
end;
How can I run a thread with low priority? I don't want the application to feel sluggish when the background thread is performing data insertion or networking.
Don't bother. You can easily use SetThreadPriority but it's not worth the trouble. If your background thread is waiting for I/O (networking), then it will not consume any CPU resource. Even if your background thread works full-speed, your GUI will not feel sluggish because Windows does a good job of splitting available CPU time among all available threads.
Is there a clean way to verify for user activity and start this thread only when the user is not busy at the keyboard / mouse?
Again, why bother checking for user activity? Checking for email is network (ie: I/O) bound, the thread checking for email will mostly be idle.
Can you not just do all this in the background thread, getting rid of all the thread micro-management? Seems to me that you could just loop around a sleep(60000) call in the background thread. Check the web service every time round the loop, check the email every 5 times round. You can set the priority to tpLower, if you want, but this thread is going to be sleeping or blocked on I/O nearly all the time, so I don't think it's even worth the typing.
I would be surprised if such a thread is noticeable at all to the user at the keyboard/mouse, no matter when it runs.
'Set up a global variable (rudimentary "locking",) start the thread' - what is this global variable intended to do? What is there to lock?
I've got a service that I need to shut down and update. I'm having difficulties with this in two different cases:
I have some threads that sleep for large amounts of time. Obviously I can't wait for them to wake up to finish shutting down the service. I had a thought to use an AutoResetEvent that gets set by some controller thread when the sleep interval is up (by just checking every two seconds or something), and triggering it immediately at OnClose time. Is there a better way to facilitate that?
I have one thread that makes a call to a blocking method call (one which I cannot modify). How do you signal such a thread to stop?
I'm not sure if I understood your first question correctly, but have you looked at using WaitForSingleObject as an alternative to Sleep? You can specify a timeout as well as an object to wait on, so if you want it to wake up earlier, just signal the object.
What exactly do you mean by "call to a blocking thread"? Or did you just mean a blocking call? In general, there isn't a way to interrupt a thread without forcefully terminating it. However, if the call is a system call, there might be ways to return control by making the call fail, eg. cancelling I/O or closing an associated handle.
For 1. you can get your threads into an interruptable Sleep by using SleepEx rather than Sleep. Once they get this shutdown kick (initiated from your termination logic using QueueUserApc), you can detect it happened using the return code from SleepEx and terminate those threads accordingly. This is similar to the suggestion to use WaitForSingleObject, but you don't need another per-thread handle that's just used to terminate the associated thread.
The return value is zero if the
specified time interval expired.
The return value is WAIT_IO_COMPLETION
if the function returned due to one or
more I/O completion callback
functions. This can happen only if
bAlertable is TRUE, and if the thread
that called the SleepEx function is
the same thread that called the
extended I/O function.
For 2., that's a tough one unless you have access to some resource used in that thread that can cause the blocking call to abort in such a way that the calling thread can handle it cleanly. You may just have to implement code to kill that thread with extreme prejudice using TerminateThread (probably this should be the last thing you do before exiting the process) and see what happens under test.
An easy and reliable solution is to kill the service process. A process is the memory-safe abstraction of the OS, after all, so you can safely terminate one without regard for process-internal state - of course, if your process is communicating or fiddling with external state, all bets are off...
Additionally, you could implement the solution which OS's themselves commonly do: one warning signal asking the process to clean up as best possible (which sets a flag and gracefully exits what can be gracefully stopped), and then forceful termination if the process doesn't exit by itself (which ends pesky things like blocking I/O).
All services should be built such that forceful termination isn't harmful, since these processes are system managed and may be terminated by things such as a reboot - i.e., your service ideally should permit this without corrupting storage anyhow.
Oh, and one final warning; windows services may share a process (I presume for efficiency, though it strikes me as an avoidable optimization), so if you go this route, you want to make sure your service is not sharing a process with other services. You can ensure this by passing the option SERVICE_WIN32_OWN_PROCESS to ChangeServiceConfig.