singleton in azure queuetrigger not working as expected - azure

My understanding of this is obviously wrong, any clarification would be helpful.
I thought that adding [Singleton] to a web job would force it to run one after another.
This does not seem to be the case.
This is my very basic test code (against a queue with about 149 messages)
[Singleton] //just run one at a time
public static void ProcessQueueMessage([QueueTrigger("datatrac-stops-to-update")] string message, TextWriter log)
{
monitorEntities mDb = new monitorEntities();
//go get the record
int recordToGet = Convert.ToInt32(message);
var record = (from r in mDb.To_Process where r.Id == recordToGet select r).FirstOrDefault();
record.status = 5;
mDb.SaveChanges();
Console.WriteLine($"Finished record {message}");
}
When it runs I get this on the console:
and as I step though it I am getting conflict errors.
What am I not understanding?
RESOLVED - MORE INFO
Here is what I did to address this, like Haitham said in his answer [Singleton] refers to how many instances of the webjob itself is running -- not how many items are processed per instance.
That was addressed by modifying my Main like:
static void Main(string[] args)
{
var config = new JobHostConfiguration();
config.Queues.BatchSize = 2;
Which when set to 1 only ran 1 at a time.
When set to 2 like above then modifying the below code:
public static void ProcessQueueMessage([QueueTrigger("datatrac-stops-to-update")] string message, TextWriter log)
{
var threadID = Thread.CurrentThread.ManagedThreadId;
Console.WriteLine($"{threadID} : started record {message}");
Produces this behavior (which is what was expected):
Link where I found documentation on above:
https://github.com/Azure/azure-webjobs-sdk/wiki/Queues#config

Singleton does not mean it will run it one after another but mainly about instantiation the instance for the web job class.
If you need to run just one at a time, you can use locks on a static variable to prevent the code to execute more than one time.
But I would not recommend that anyway and you have to see why there are conflict errors

Related

Azure TimerTrigger not firing when trying to output multiple events to event grid topic

I have an azure timer function.
It works when I run it locally.
It works if I remove this code when running in Azure:
[EventGrid(TopicEndpointUri = "EventGridUri", TopicKeySetting = "EventGridKey")] IAsyncCollector outputEvents
Otherwise it does not work. The timer never seems to go off and the function is not triggered.
The code is below. I have tried removing the for loop and everything else with the idea that I might be getting a silent exception, but the function still does not execute. Any idea how to troubleshoot or what might be causing the issue?
public static class TimeTest
{
[FunctionName("TimeTest")]
public static async Task Run(
[TimerTrigger("0 */1 * * * *")] TimerInfo myTimer,
[EventGrid(TopicEndpointUri = "EventGridUri", TopicKeySetting = "EventGridKey")] IAsyncCollector<EventGridEvent> outputEvents)
{
for (var i = 0; i < 3; i++)
{
var myEvent = new EventGridEvent("message-id-" + i, "subject-name", "event-data", "event-type", DateTime.UtcNow, "1.0");
await outputEvents.AddAsync(myEvent);
}
}
}
I had a lot of trouble finding the exception, but was finally able to see it when I went to application insights for the entire functions app (as opposed to trying to see logs at the function level).
It said that my eventuri was incorrect. After taking a closer look I realized that I had not given my functions permission to the keyvault.

TrackTrace is not logging In to Application Insight

I have the below code. It's Not logging to Trace.I am not sure why. If possible can you help me on this?
public static void SAPLogger(string Message)
{
TelemetryConfiguration.Active.InstrumentationKey = "XXX-XXX-XXX";
TelemetryClient TelePositive = new TelemetryClient
{
InstrumentationKey = "XXX-XXX" (Optional Value)
};
//TelePositive.TrackRequest(Req);
TelePositive.TrackTrace(Message, SeverityLevel.Verbose, new Dictionary<string, string> { { "Information", "SAP" } });
}
I am calling this method in the Main() method.
static void Main(string[] args)
{
try
{
int a = 5;
int c = a / 2;
SAPLogger("The value is Success" + c);
}
}
I am totally not sure why this is not logging. Please help
Your example app is probably exiting before your telemetry gets sent.
DeveloperMode should cause it to send immediately, however, if your process exists immediately like your test app appears to, the process might still end before the web request gets created and sent.
For short lived applications like that test app, you'll probably need a flush and a sleep call of some kind at the end to ensure telemetry has a chance to send before the app quits.
For a real application that lives for a long time, telemetry will be batched and sent after an amount of time, or number of events is met, then that batch will be sent. you app probably still would want to flush/wait at the end just to make sure any batched up telemetry gets sent.
but in either case, the flush/wait should only occur once, at the end. not with every call to track telemetry.

How to read Azure Service Bus messages from Multiple Queues with one worker

I have three queues and one worker that I want monitoring the three queues (or only two of them)
One queue is qPirate
One queue is qShips
One queue is qPassengers
The idea is that workers will either be looking at all 3 of them, 2 of them, or one of them, and doing different things depending on what the message says.
The key though is that say a message is failing because ship1 is offline, all queues in qships will refresh, workers that are looking at that and other queues will get hung up slightly from it as they will try to process the messages for that queue while only looking at the other queues a little bit, while the other workers that are looking at the other 2 queues and skipping qships will continue to process through messages without holdup or delays.
public static void GotMessage([ServiceBusTrigger("%LookAtAllQueuesintheservicebus%")] BrokeredMessage message)
{
var handler = new MessageHandler();
var manager = new MessageManager(
handler,
"PirateShips"
);
manager.ProcessMessageViaHandler(message);
}
Looking around online I'm guessing this isn't something that's possible, but it seems like it would be? Thanks in advance either way!
Edit1: I'll add the Job Host as well to attempt to clarify things a bit
JobHostConfiguration config = new JobHostConfiguration()
{
DashboardConnectionString = "DefaultEndpointsProtocol=https;AccountName=PiratesAreUs;AccountKey=Yarr",
StorageConnectionString = "DefaultEndpointsProtocol=https;AccountName=PiratesAreUs;AccountKey=Yarr",
NameResolver = new QueueNameResolver()
};
ServiceBusConfiguration serviceBusConfig = new ServiceBusConfiguration()
{
ConnectionString = "Endpoint=AllPirateQueuesLocatedHere;SharedAccessKeyName=PiratesAreUs;SharedAccessKey=Yarr"
};
serviceBusConfig.MessageOptions.AutoComplete = false;
serviceBusConfig.MessageOptions.AutoRenewTimeout = TimeSpan.FromMinutes(1);
serviceBusConfig.MessageOptions.MaxConcurrentCalls = 1;
config.UseServiceBus(serviceBusConfig);
JobHost host = new JobHost(config);
host.RunAndBlock();
Also the QueueNameResolverClass is simply
public class QueueNameResolver : INameResolver
{
public string Resolve(string name)
{
return name;
}
}
I don't appear to have anyway to have the NameResolver be multiple queues, while I can say that I want the jobhost to look at a certain ServiceBus, I don't know how to tell it to look at all the queues within the ServiceBus.
In other words, I want multiple servicebustriggers on this worker so that if a message gets sent to qpirate1 and qships1 which are both located in service bus AllPirateQueuesHere, the worker can pick up the message in qpirate1, process it, then pick up the message in qships1 and process it.
Figured out the answer... This is possible and its simpler than I thought I'm not sure why I didn't connect the dots but I'm still curious why there isn't more documentation about this. Apparently it's simply make a function per queue you want a worker to look at multiple queues. So if you had three queues you'd want something like the below (you can handle each message differently).
public static void GotMessage1([ServiceBusTrigger("%qPirate1%")] BrokeredMessage message)
{
var handler = new MessageHandler();
var manager = new MessageManager(
handler,
"Pirates"
);
manager.ProcessMessageViaHandler(message);
}
public static void GotMessage2([ServiceBusTrigger("%qShip1%")] BrokeredMessage message)
{
var handler = new MessageHandler();
var manager = new MessageManager(
handler,
"Ships"
);
manager.ProcessMessageViaHandler(message);
}
public static void GotBooty([ServiceBusTrigger("%qBooty%")] BrokeredMessage message)
{
var handler = new MessageHandler();
var manager = new MessageManager(
handler,
"Booty"
);
manager.ProcessMessageViaHandler(message);
}

How to optimize tests validating asynchronous code?

We are developing a WPF application using TDD. As we're already working on this solution for almost two years, we've written a huge bunch of tests (almost 2000 Unittests right now).
There are some classes, that need to implement functionality multithreaded and asynchronously. For example a communication-component that can both send and receive messages and parse them. The dependencies are always mocked using RhinoMocks.
Our Test-Methods targeting these classes look very similar, as following:
[TestMethod]
public void Method_Description_ExpectedResult(){
// Arrange
var myStub = MockRepository.GenerateStub<IMyStub>();
var target = new MyAsynchronousClass(myStub);
// Act
var target.Send("Foo");
Thread.Sleep(200);
//Assert
myStub.AssertWasCalled(x => x.Bar("Foo"));
}
As you can see, this test runs at least for 200 ms due to the Thread.Sleep(). We optimized the test replacing the AssertWasCalled with a active polling method, s.th. like this:
public static bool True(Func<bool> condition, int times, int waitTime)
{
for (var i = 0; i < times; i++)
{
if (condition())
return true;
Thread.Sleep(waitTime);
}
return condition();
}
We can now use this WaitFor.True(...) Method by changing the AssertWasCalled to:
var fooTriggered = false;
myStub.Stub(x => x.Bar("Foo")).Do((Action)(() => fooTriggered = true)));
WaitFor.True(() => fooTriggered, 20, 20);
Assert.IsTrue(fooTriggered);
This construct will terminate earlier if the condition matches, but anyway - this takes too long for us. Running all of our 2000 Tests need about 5 Minutes (building and running them).
Is there any smart trick how we could optimize code like this?
You can use a monitor. I'm making this up so please excuse me if it isn't quite compiling, but it'll look something like:
[TestMethod]
public void Method_Description_ExpectedResult(){
// Arrange
var waitingRoom = new object();
var myStub = MockRepository.GenerateStub<IMyStub>();
myStub.Setup(x => x.Bar("Foo")).Callback(x =>
{
Monitor.Enter(waitingRoom);
Monitor.Pulse(waitingRoom);
Monitor.Exit(waitingRoom);
}
var target = new MyAsynchronousClass(myStub);
// Act
Monitor.Enter(waitingRoom);
target.Send("Foo");
Monitor.Wait(waitingRoom);
Monitor.Exit(waitingRoom);
//Assert
myStub.AssertWasCalled(x => x.Bar("Foo"));
}
Code written within the Monitor can't run until it's free. The test will cause the acting thread to wait until Monitor.Wait has been called. Then the callback can enter and pulse the Monitor. The test then "wakes up", and once the callback has exited the monitor, it gets control back and exits too, allowing you to Assert.
The only thing I haven't covered is that if Bar("Foo") doesn't get called it will hang, so you might want to have a timer pulse the thread too.
You can create a class which does the complex monitoring bits for you if you use it a lot. This is one I wrote to deal with asynchronous checks in UI automation; adapting it for what you're doing might help you.

How to ensure unique local state among threads for email queue processing

THE SCENARIO:
I want to ask this question regarding Parallel.For(or any other multithreading approach in C#.Net). I have to build a MultiThreaded Mailer Windows service that will send mails to all the recipients as fast as it can. I get the serialized rows from the database that contains the email message and SmtpDetails and then deSerialize them in code.
An emails may have 1000 reciepients and so on a Dual Core machine ( development machine) at least 2 threads can run simultaneously. So i use parallel.For in order ro do this. I have read about the LocalInit delegate that runs once for every thread.
THE CODE:
int itemCount = serMailObj.ReceipientList.Count;
Parallel.For(0, itemCount, () =>
{
return new ThreadLocalStateCache()
{
Receipient = serMailObj.ReceipientList.Dequeue(),
mail = serMailObj.Email,
SerializableSmtpDetails = serSmtpObj
};
}
, doWork, (x) => { });
private static ThreadLocalStateCache doWork(int instance, ParallelLoopState state, ThreadLocalStateCache threadInstance)
{
KeyValuePair<string, string> kvp = threadInstance.Receipient;
SerializableSmtpDetails serSmtpObj = threadInstance.SerializableSmtpDetails;
MailMessage email = threadInstance.mail;
email.To.Add(new MailAddress(kvp.Key, kvp.Value));
SmtpClient client = new SmtpClient();
client.Credentials = new System.Net.NetworkCredential(serSmtpObj.UserName, serSmtpObj.Password);
client.Host = serSmtpObj.Host;
client.Port = serSmtpObj.Port;
client.EnableSsl = serSmtpObj.EnableSSL;
try
{
client.Send(email);
Console.WriteLine("sending mail....");
}
catch (Exception)
{
throw;
}
return null;
}
public class ThreadLocalStateCache
{
public KeyValuePair<string, string> Receipient { get; set; }
public MailMessage mail { get; set; }
public SerializableSmtpDetails SerializableSmtpDetails { get; set; }
}
The above code is pretty straight forward. The localInit delegate constructs a local object foreach thread. and then the doWork tries to process the queue.
THE PROBLEMS:
I am getting multiple mails for each recipient. seems as if the email object is being shared among threads.
getting failure sending mail sometimes.
Kindly explain as to how i can isolate the mail and smtpclient objects in each thread. and process the queue.
EDIT 1: If the multithreading gurus would help me please tell that is there any way for every thread to have a unique copy of its local variables and not shared ones. Since the MailMessage object is not immutable i cannot create a clone of it also. apart from deseralizing it in each thread(which would ensure a new object is created) is there any magic way to achieve this?
There might be problems due to doWork() returning null. As I learned when answering your recent comment here, the thread local object should be passed between subsequent invocations of the Parallel.For body at the same thread, because it is supposed to work as an accumulator; see the usage example at MSDN. It's unclear what happens when you return null, but I would fix that and see whether it makes difference.
below might be the issue:
serMailObj.ReceipientList.Dequeue()
Try using ConcurrentQueue (.NET 4) or put locks around so that one thread at a time can Dequeue it.
also make to sure Concurrent classes or locks anywhere yo uhave access to shared resource from threads.
A Queue <(Of <(T >)>) can support
multiple readers concurrently, as long
as the collection is not modified.
Even so, enumerating through a
collection is intrinsically not a
thread-safe procedure. To guarantee
thread safety during enumeration, you
can lock the collection during the
entire enumeration. To allow the
collection to be accessed by multiple
threads for reading and writing, you
must implement your own
synchronization.

Resources