I have an unusual issue with TaskCompletionSource that has me baffled. I have a TaskCompletionSource waiting for the task to complete once i call the TrySetResult. I call this in three places in the code: from a WCF thread immediately to return a value to an APM WCF BeginXXX EndXXX; from another WCF thread to return immediately to the APM; lastly from an NServiceBus handler thread.
I started with the ubiquitous ToAPM provided by MS-PL. http://blogs.msdn.com/b/pfxteam/archive/2011/06/27/using-tasks-to-implement-the-apm-pattern.aspx
I noticed that the two WCF based threads worked 100% of the time. in 100 hours of hard testing, additionally extensive unit tests, I have never experienced a single failure to return a completed task to the AsyncCallback.
From the MS provided ToAPM code, the code uses a ContinueWith on the completed task to call the AsyncCallback in a schedule enabled task.
The problem I have not solved is the NServiceBus threads calling the TrySetResult on the TaskCompletionSource object. I find times of outages, where for undefined periods of time, the call simply fails. I set break points in the code for both the call and inside the ContinueWith code. I get the break point on the TrySetResult always, but only sometimes on the code inside the ContinueWith code.
The following information hopefully will shed some light on the matter.
I use a CancellationTokenSource with a timeout and setting a result to call the TrySetResult on TaskCompletionSource obj. When the above call does not work to move the task to completed, the timeout code fires. This timeout code has never not worked. it succeeds 100% of the time.
What is interesting is this, in the same code that calls the TrySetResult from the NServiceBus thread, when it works, it works as easily calling the cancellation object's Cancel as it does the TrySetResult on the TaskCompletionSource obj.
When one fails they both fail.
Then after an indiscriminate period of time it works again.
This is a WCF server in a production and QA environment and each displays identical results.
What is most weird is the following, for one WCF connection, the NServiceBus thread succeeds and another fails at the same time. Then at times both work, and then both fail. Again, all at the same time.
I have tried a number of things to work around the issue to no avail:
I wrapped the call to TrySetResult in a TaskCompletionSource + ContinueWith -- fail
I wrapped the call in a Task.Factory.StartNew -- fail
I call it directly -- fail
I really do not know what else to try.
I put in checks to ensure that the TaskCompletionSource obj is not completed, and during the outage it is not.
I put in checks to ensure the CancellationTokenSource object is not cancelled or has a cancellation pending during the outage, it does not.
I examined the objects in the debugger and they seem good.
They just do not work sometimes.
Could there be an inconsistency in the NserviceBus threads that sometimes prevent the calls from working?
Is there some thread marshaling I can try?
I searched everywhere and I have not see one mention of this problem. Is it unique?
I am totally baffled and need some ideas.
Remove the call from the NServiceBus thread execution. Isolate the call to TrySetResult using a thread such as QueueUserWorkItem or spinning your own thread. Since, the executing resumes using the thread, you may need some additional threads to handle the throughput. Ether spin multiple dedicated threads or use the thread pool. I tested calling TrySetResult in a dedicate threads and they work.
Here is code to demonstrate a single dedicated thread:
public static void Spin()
{
ClientThread = new Thread(new ThreadStart(() =>
{
while (true)
{
try
{
if (!HasSomething.WaitOne(1000, false))
continue;
while (true)
{
WaitingAsyncData entry = null;
lock (qlocker)
{
if (!Trigger.Any())
break;
entry = Trigger.Dequeue();
}
if (entry == null)
break;
entry.TrySetResult("string");
}
}
catch
{
}
}
}));
ClientThread.IsBackground = true;
ClientThread.Start();
}
Here is the ThreadPool example code:
ThreadPool.QueueUserWorkItem(delegate
{
entry.TrySetResult("string");
});
Using the ThreadPool rather than static thread provides greater flexibility and scaleability.
Related
Context:
I have a cmd application in java which is written to work in peer-to-peer mode in different servers. Once a server starts, all other instances must stop. So I have written a piece of code that runs in a low priority thread and monitors an AtomicBoolean value autoClose, and whenever autoClose is set to true, thread will close application. (P.S.: I don't want to manually add close because the application has 2 main high priority threads and many temporary normal priority threads).
Here is the code:
/**
* Watches autoClose boolean value and closes the connector once it is true
* <p>
* This is a very low priority thread which continuously monitors autoClose
*/
protected void watchAndClose() {
Thread watchAutoClose = new Thread(() -> {
while (true) {
if (autoClose.get()) {
close();
// wait till closing is successful
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException ignored) {
// I want instance of thread watchAutoClose so I can call this
// watchAutoClose.interrupt();
}
if (!component.getStatus()) setAutoClose(false);
}
}
});
watchAutoClose.setPriority(Thread.MIN_PRIORITY);
watchAutoClose.start();
}
Question:
SonarLint says I can't leave InterruptedException part empty. I have to either throw it again or call thatThread.interrupt().
So how can I do this? I want an instance of thread watchAutoClose inside that thread so I can call watchAutoClose.interrupt(). I tried Thread.currentThread() but I fear with that many threads, the currently executing thread wouldn't be this thread. (i.e, there is a possibility of JVM can choose to switch to another thread by the time it is inside the catch clause and calls Thread.currentThread() so at that time current thread would be the other one and I would interrupt that other thread... correct me if I am too worrying or my concept is totally wrong.)
Or should I ignore the warning altogether and leave catch block?
First of all, it’s not clear why you think that waiting for a second was necessary at all. By the time, the close() method returns, the close() method has been completed. On the other hand, if close() truly triggers some asynchronous action, there is no guaranty that waiting one second will be sufficient for its completion.
Further, addressing your literal question, Thread.currentThread() always return the calling thread’s instance. It’s impossible for a thread to execute that method without being in the running state. When a task switch happens, the thread can’t read the reference at all, until it gets CPU time again. Besides that, since the specification says that this method returns the Thread instance representing the caller, the environment has to ensure this property, regardless of how it implements it. It works even when multiple threads call this method truly at the same time, on different CPU cores.
So, regardless of how questionable the approach of waiting a second is, handling interruption like
try {
TimeUnit.SECONDS.sleep(1);
} catch (InterruptedException ignored) {
Thread.currentThread().interrupt();
}
is a valid approach.
But you may also replace this code with
LockSupport.parkNanos(TimeUnit.SECONDS.toNanos(1));
The parkNanos method will return silently on interruption, leaving the calling thread in the interrupted state. So it has the same effect as catching the InterruptedException and restoring the interrupted state, but is simpler and potentially more efficient as no exception needs to be constructed, thrown, and caught.
Another point is that you are creating a polling loop on the atomic variable consuming CPU cycles when the variable is false, which is discouraged, even when you give the thread a low priority.
I'm using the Azure Service Bus SubscriptionClient.OnMessage method; configured to process up to 5 messages concurrently.
Within the code I need to wait for all messages to finish processing before I can continue (to properly shutdown an Azure Worker Role). How do I do this?
Will SubscriptionClient.Close() block until all messages have finished processing?
Calling Close on SubscriptionClient or QueueClient will not block. Calling Close closes off the entity immediately as far as I can tell. I tested quickly just using the Worker Role With Service Bus Queue project template that shipped with Windows Azure SDK 2.0. I added a thread sleep for many seconds in the message process action and then shut down the role while it was running. I saw the Close method get called while the messages were processing in their thread sleep but it certainly did not wait for the for message processing to complete, the role simple closed down.
To handle this gracefully you'll need to do the same thing we did when dealing with any worker role that was processing messages (Service Bus, Azure Storage queue or anything else): keep track of what is being worked on and shut down when it is complete. There are several ways to deal with that but all of them are manual and made messy in this case because of the multiple threads involved.
Given the way that OnMessage works you'll need to add something in the action that looks to see if the role has been told to shutdown, and if so, to not do any processing. The problem is, when the OnMessage action is executed it HAS a message already. You'd probably need to abandon the message but not exit the OnMessage action, otherwise it will keep getting a message if there are ones in the queue. You can't simply abandon the message and let the execution leave the action because then the system will be handed another message (possibly the same one) and several threads doing this may cause messages to get too many dequeue counts and get dead lettered. Also, you can't call Close on the SubscriptionClient or QueueClient, which would stop the receive loop internally, because once you call close any of the outstanding message processing will throw an exception when .Complete, .Abandon, etc. is called on the message because the message entity is now closed. This means you can't stop the incoming messages easily.
The main issue here is because you are using the OnMessage and setting up the concurrent message handling by setting the MaxConcurrentCalls on the OnMessageOptions, which means the code that starts and manages the threads is buried in the QueueClient and SubscriptionClient and you don't have control over that. You don't have a way to reduce the count of threads, or stop the threads individually, etc. You'll need to create a way to put the OnMessage action threads into a state where they are aware that the system is being told to shut down and then complete their message and not exit the action in order for them to not continuously be assigned new messages. This means you'll likely need to also set the MessageOptions to not use autocomplete and manually call complete in your OnMessage action.
Having to do all of this may severely reduce the actual benefit of using the OnMessage helper. Behind the scenes OnMessage is simply setting up a loop calling receive with the default timeout and handing of messages to another thread to do the action (loose description). So what you get by using the OnMessage approach is away from having to write that handler on your own, but then the problem you are having is because you didn't write that handler on your own you don't have control over those threads. Catch-22. If you really need to stop gracefully you may want to step away from the OnMessage approach, write your own Receive loop with threading and within the main loop stop receiving new messages and wait for all the workers to end.
One option, especially if the messages are idempotent (which means processing them more than once yields the same results... which you should be mindful of anyway) then if they are stopped in mid processing they will simply reappear on the queue to be processed by another instance later. If the work itself isn't resource intensive and the operations are idempotent then this really can be an option. No different than when an instance might fail due to hardware failure or other issues. Sure, it's not graceful or elegant, but it certainly removes all the complexity I've mentioned and is still something that can happen anyway due to other failures.
Note that the OnStop is called when an instance is told to shut down. You've got 5 minutes you can delay this until the fabric just shuts it off, so if your messages take longer than five minutes to process it won't really matter if you attempt to shut down gracefully or not, some will be cut off during processing.
You can tweak OnMessageAsync to wait for processing of messages to complete, and block new messages from beginning to be processed:
Here is the implementation:
_subscriptionClient.OnMessageAsync(async message =>
{
if (_stopRequested)
{
// Block processing of new messages. We want to wait for old messages to complete and exit.
await Task.Delay(_waitForExecutionCompletionTimeout);
}
else
{
try
{
// Track executing messages
_activeTaskCollection[message.MessageId] = message;
await messageHandler(message);
await message.CompleteAsync();
}
catch (Exception e)
{
// handle error by disposing or doing nothing to force a retry
}
finally
{
BrokeredMessage savedMessage;
if (!_activeTaskCollection.TryRemove(message.MessageId, out savedMessage))
{
_logger.LogWarning("Attempt to remove message id {0} failed.", savedMessage.MessageId);
}
}
}
}, onMessageOptions);
And an implementation of Stop that waits for completion:
public async Task Stop()
{
_stopRequested = true;
DateTime startWaitTime = DateTime.UtcNow;
while (DateTime.UtcNow - startWaitTime < _waitForExecutionCompletionTimeout && _activeTaskCollection.Count > 0)
{
await Task.Delay(_waitForExecutionCompletionSleepBetweenIterations);
}
await _subscriptionClient.CloseAsync();
}
Note that _activeTaskCollection is a ConcurrentDictionary (we can also use a counter with interlock to count the number of in progress messages, but using a dictionary allows you to investigate what happend easily in case of errors.
I already have a windows service running with a System.Timers.Timer that do a specific work. But, I want some works to run at the same time, but in different threads.
I've been told to create a different System.Timers.Timer instance. Is this correct? Is this way works running in parallel?
for instance:
System.Timers.Timer tmr1 = new System.Timers.Timer();
tmr1.Elapsed += new ElapsedEventHandler(DoWork1);
tmr1.Interval = 5000;
System.Timers.Timer tmr2 = new System.Timers.Timer();
tmr2.Elapsed += new ElapsedEventHandler(DoWork2);
tmr2.Interval = 5000;
Will tmr1 and tmr2 run on different threads so that DoWork1 and DoWork2 can run at the same time, i.e., concurrently?
Thanks!
It is not incorrect.
Be careful. System.Timers.Timer will start a new thread for every Elapsed event. You'll get in trouble when your Elapsed event handler takes too long. Your handler will be called again on another thread, even though the previous call wasn't completed yet. This tends to produce hard to diagnose bugs. Something you can avoid by setting the AutoReset property to false. Also be sure to use try/catch in your event handler, exceptions are swallowed without diagnostic.
Multiple timers might mean multiple threads. If two timer ticks occur at the same time (i.e. one is running and another fires), those two timer callbacks will execute on separate threads, neither of which will be the main thread.
It's important to note, though, that the timers themselves don't "run" on a thread at all. The only time a thread is involved is when the timer's tick or elapsed event fires.
On another note, I strongly discourage you from using System.Timers.Timer. The timer's elapsed event squashes exceptions, meaning that if an exception escapes your event handler, you'll never know it. It's a bug hider. You should use System.Threading.Timer instead. System.Timers.Timer is just a wrapper around System.Threading.Timer, so you get the same timer functionality without the bug hiding.
See Swallowing exceptions is hiding bugs for more info.
Will tmr1 and tmr2 run on different threads so that DoWork1 and DoWork2 can run at the same time, i.e., concurrently?
At the start, yes. However, what is the guarantee both DoWork1 and DoWork2 would finish within 5 seconds? Perhaps you know the code inside DoWorkX and assume that they will finish within 5 second interval, but it may happen that system is under load one of the items takes more than 5 seconds. This will break your assumption that both DoWorkX would start at the same time in the subsequent ticks. In that case even though your subsequent start times would be in sync, there is a danger of overlapping current work execution with work execution which is still running from the last tick.
If you disable/enable respective timers inside DoWorkX, however, your start times will go out of sync from each other - ultimately possible they could get scheduled over the same thread one after other. So, if you are OK with - subsequent start times may not be in sync - then my answer ends here.
If not, this is something you can attempt:
static void Main(string[] args)
{
var t = new System.Timers.Timer();
t.Interval = TimeSpan.FromSeconds(5).TotalMilliseconds;
t.Elapsed += (sender, evtArgs) =>
{
var timer = (System.Timers.Timer)sender;
timer.Enabled = false; //disable till work done
// attempt concurrent execution
Task work1 = Task.Factory.StartNew(() => DoWork1());
Task work2 = Task.Factory.StartNew(() => DoWork2());
Task.Factory.ContinueWhenAll(new[]{work1, work2},
_ => timer.Enabled = true); // re-enable the timer for next iteration
};
t.Enabled = true;
Console.ReadLine();
}
Kind of. First, check out the MSDN page for System.Timers.Timer: http://msdn.microsoft.com/en-us/library/system.timers.timer.aspx
The section you need to be concerned with is quoted below:
If the SynchronizingObject property is null, the Elapsed event is
raised on a ThreadPool thread. If processing of the Elapsed event
lasts longer than Interval, the event might be raised again on another
ThreadPool thread. In this situation, the event handler should be
reentrant.
Basically, this means that where the Timer's action gets run is not such that each Timer has its own thread, but rather that by default, it uses the system ThreadPool to run the actions.
If you want things to run at the same time (kick off all at the same time) but run concurrently, you can not just put multiple events on the elapsed event. For example, I tried this in VS2012:
static void testMethod(string[] args)
{
System.Timers.Timer mytimer = new System.Timers.Timer();
mytimer.AutoReset = false;
mytimer.Interval = 3000;
mytimer.Elapsed += (x, y) => {
Console.WriteLine("First lambda. Sleeping 3 seconds");
System.Threading.Thread.Sleep(3000);
Console.WriteLine("After sleep");
};
mytimer.Elapsed += (x, y) => { Console.WriteLine("second lambda"); };
mytimer.Start();
Console.WriteLine("Press any key to go to end of method");
Console.ReadKey();
}
The output was this:
Press any key to go to end of method
First lambda.
Sleeping 3 seconds
After sleep
second lambda
So it executes them consecutively not concurrently. So if you want "a bunch of things to happen" upon each timer execution, you have to launch a bunch of tasks (or queue up the ThreadPool with Actions) in your Elapsed handler. It may multi-thread them, or it may not, but in my simple example, it did not.
Try my code yourself, it's quite simple to illustrate what's happening.
We recently adopted the TPL as the toolkit for running some heavy background tasks.
These tasks typically produce a single object that implements IDisposable. This is because it has some OS handles internally.
What I want to happen is that the object produced by the background thread will be properly disposed at all times, also when the handover coincides with application shutdown.
After some thinking, I wrote this:
private void RunOnUiThread(Object data, Action<Object> action)
{
var t = Task.Factory.StartNew(action, data, CancellationToken.None, TaskCreationOptions.None, _uiThreadScheduler);
t.ContinueWith(delegate(Task task)
{
if (!task.IsCompleted)
{
DisposableObject.DisposeObject(task.AsyncState);
}
});
}
The background Task calls RunOnUiThread to pass its result to the UI thread. The task t is scheduled on the UI thread, and takes ownership of the data passed in. I was expecting that if t could not be executed because the ui thread's message pump was shut down, the continuation would run, and I could see that that the task had failed, and dispose the object myself. DisposeObject() is a helper that checks if the object is actually IDisposable, and non-null, prior to disposing it.
Sadly, it does not work. If I close the application after the background task t is created, the continuation is not executed.
I solved this problem before. At that time I was using the Threadpool and the WPF Dispatcher to post messages on the UI thread. It wasn't very pretty, but in the end it worked. I was hoping that the TPL was better at this scenario. It would even be better if I could somehow teach the TPL that it should Dispose all leftover AsyncState objects if they implement IDisposable.
So, the code is mainly to illustrate the problem. I want to learn about any solution that allows me to safely handover Disposable objects to the UI thread from background Tasks, and preferably one with as little code as possible.
When a process closes, all of it's kernel handles are automatically closed. You shouldn't need to worry about this:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms686722(v=vs.85).aspx
Have a look at the RX library. This may allow you to do what you want.
From MSDN:
IsCompleted will return true when the Task is in one of the three
final states: RanToCompletion, Faulted, or Canceled
In other words, your DisposableObject.DisposeObject will never be called, because the continuation will always be scheduled after one of the above conditions has taken place. I believe what you meant to do was :
t.ContinueWith(t => DisposableObject.DisposeObject(task.AsyncState),
TaskContinuationOptions.NotOnRanToCompletion)
(BTW you could have simply captured the data variable rather than using the AsyncState property)
However I wouldn't use a continuation for something that you want to ensure happens at all times. I believe a try-finally block will be more fitting here:
private void RunOnUiThread2(Object data, Action<Object> action)
{
var t = Task.Factory.StartNew(() =>
{
try
{
action(data);
}
finally
{
DisposableObject.DisposeObject(task.AsyncState);
//Or use a new *foreground* thread if the disposing is heavy
}
}, CancellationToken.None, TaskCreationOptions.None, _uiThreadScheduler);
}
I’m trying to issue web requests asynchronously. I have my code working fine except for one thing: There doesn’t seem to be a built-in way to specify a timeout on BeginGetResponse. The MSDN example clearly show a working example but the downside to it is they all end up with a
SomeObject.WaitOne()
Which again clearly states it blocks the thread. I will be in a high load environment and can’t have blocking but I also need to timeout a request if it takes more than 2 seconds. Short of creating and managing a separate thread pool, is there something already present in the framework that can help me?
Starting examples:
http://msdn.microsoft.com/en-us/library/ms227433(VS.100).aspx
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.begingetresponse.aspx
What I would like is a way for the async callback on BeginGetResponse() to be invoked after my timeout parameter expires, with some indication that a timeout occurred.
The seemingly obvious TimeOut parameter is not honored on async calls.
The ReadWriteTimeout parameter doesn't come into play until the response returns.
A non-proprietary solution would be preferable.
EDIT:
Here's what I came up with: after calling BeginGetResponse, I create a Timer with my duration and that's the end of the "begin" phase of processing. Now either the request will complete and my "end" phase will be called OR the timeout period will expire.
To detect the race and have a single winner I call increment a "completed" counter in a thread-safe manner. If "timeout" is the 1st event to come back, I abort the request and stop the timer. In this situation, when "end" is called the EndGetResponse throws an error. If the "end" phase happens first, it increments the counter and the "timeout" foregoes aborting the request.
This seems to work like I want while also providing a configurable timeout. The downside is the extra timer object and the callbacks which I make no effort to avoid. I see 1-3 threads processing various portions (begin, timed out, end) so it seems like this working. And I don't have any "wait" calls.
Have I missed too much sleep or have I found a way to service my requests without blocking?
int completed = 0;
this.Request.BeginGetResponse(GotResponse, this.Request);
this.timer = new Timer(Timedout, this, TimeOutDuration, Timeout.Infinite);
private void Timedout(object state)
{
if (Interlocked.Increment(ref completed) == 1)
{
this.Request.Abort();
}
this.timer.Change(Timeout.Infinite, Timeout.Infinite);
this.timer.Dispose();
}
private void GotRecentSearches(IAsyncResult result)
{
Interlocked.Increment(ref completed);
}
You can to use a BackgroundWorker to run your HttpWebRequest into a separated thread, so your main thread still alive. So, this background thread will be blocked, but first one don't.
In this context, you can to use a ManualResetEvent.WaitOne() just like in that sample: HttpWebRequest.BeginGetResponse() method.
What kind of an application is this? Is this a service proces/ web application/console app?
How are you creating your work load (i.e requests)? If you have a queue of work that needs to be done, you can start off 'N' number of async requests (with the framework for timeouts that you have built) and then, once each request completes (either with timeout or success) you can grab the next request from the queue.
This will thus become a Producer/consumer pattern.
So, if you configure your application to have a maximum of "N' requests outstanding, you can maintain a pool of 'N' timers that you reuse (without disposing) between the requests.
Or, alternately, you can use ThreadPool.SetTimerQueueTimer() to manage your timers. The threadpool will manage the timers for you and reuse the timer between requests.
Hope this helps.
Seems like my original approach is the best thing available.
If you can user async/await then
private async Task<WebResponse> getResponseAsync(HttpWebRequest request)
{
var responseTask = Task.Factory.FromAsync(request.BeginGetResponse, ar => (HttpWebResponse)request.EndGetResponse(ar), null);
var winner = await (Task.WhenAny(responseTask, Task.Delay(new TimeSpan(0, 0, 20))));
if (winner != responseTask)
{
throw new TimeoutException();
}
return await responseTask;
}