CloudQueueClient.ResponseReceived Event broken? - azure

I'm trying to build an event driven Azure Queue where a event is to fired every time a message is put in the Azure Queue. With AzureXplorer I see that the messages are put in the Azure Queue properly but the CloudQueueClient.ResponseReceived Event never fires. I'm using Azure V1.4. This is the code from my Worker role:
public class WorkerRole : RoleEntryPoint
{
public override void Run()
{
while (true)
{
Thread.Sleep(10000);
}
}
public override bool OnStart()
{
// Set the maximum number of concurrent connections
ServicePointManager.DefaultConnectionLimit = 12;
var queuDataSource = new AzureQueueDataSource();
queuDataSource.GetCloudQueueClient().ResponseReceived +=new EventHandler<ResponseReceivedEventArgs>(WorkerRole_ResponseReceived);
// For information on handling configuration changes
// see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
return base.OnStart();
}
void WorkerRole_ResponseReceived(object sender, ResponseReceivedEventArgs e)
{
var i = 1; // Breakpoint here never happends
}
}

Windows Azure Queues need to be polled for new messages. See SDK samples or code here for examples on how to query queues for new messages.
Quick list of things to take into account:
Because polling is counted as a
transaction in Windows Azure, you
will be paying for those.
It is usually better to implement some kind of retry mechanism if no messages are found (e.g. exponential back-off, etc)
It is usually good to retrieve messages in batches (less round trips, less transactions, etc)
Remember that messages can be delivered more than once (plan for duplicate messages)
Use the "dequeuecount" property to deal with "poison messages".
There's plenty of coverage on all these. See the documentation/samples in the link above. This article is pretty good too: http://blogs.msdn.com/b/appfabriccat/archive/2010/12/20/best-practices-for-maximizing-scalability-and-cost-effectiveness-of-queue-based-messaging-solutions-on-windows-azure.aspx

Related

Pipe Events from Azure Event Hub to Azure Service Bus

I am listening to Event hub for various events.
Each event is high value and cannot be missed.
Events are partitioned based on device id.
Events from one device id are sparse and not very frequent (couple of events per few dasy) . It only occurs in response to a user action which is infrequent.
The number of devices are huge , so I will have a lot of events for a variety of device Ids.
For each event , I need to make 3-4 API calls to Systems which are not super reliable. And since some of these are cross Geo Calls it might take some time.
I am planning to take the events from Event hub and put them into Service Bus. My reasons are as follows.
Event hub can be scaled to only 32 partitions and if one event takes time , the entire partition gets blocked.
Service bus on the other hand is more horizontally scalable. If the throughput drops I can just add more subscribers to the Service Bus.
I have been looking for patterns like this but I have not seen patterns where we get data from a log based messaging system and push them to a queue based one.
Are there better approach to handle such scenarios ?
I think you can use Event hub trigger and service bus output binding to achieve what you want.
For example, I want to monitor Event hub 'test' and I am using C# library:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.EventHubs;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
namespace FunctionApp68
{
public static class Function1
{
[FunctionName("Function1")]
[return: ServiceBus("test1", Connection = "ServiceBusConnection")]
public static string Run([EventHubTrigger("test", Connection = "str")] EventData[] events, ILogger log)
{
var exceptions = new List<Exception>();
string messageBodyt = "";
foreach (EventData eventData in events)
{
try
{
string messageBody = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
messageBodyt = messageBodyt + messageBody;
// Replace these two lines with your processing logic.
log.LogInformation($"C# Event Hub trigger function processed a message: {messageBody}");
//await Task.Yield();
}
catch (Exception e)
{
// We need to keep processing the rest of the batch - capture this exception and continue.
// Also, consider capturing details of the message that failed processing so it can be processed again later.
exceptions.Add(e);
}
}
// Once processing of the batch is complete, if any messages in the batch failed processing throw an exception so that there is a record of the failure.
if (exceptions.Count > 1)
throw new AggregateException(exceptions);
if (exceptions.Count == 1)
throw exceptions.Single();
return messageBodyt;
}
}
}
The above code will collect from event hub 'test' and save to service bus queue 'test1'.
Have a look of these doc:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs-trigger?tabs=csharp
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus-output?tabs=csharp#example
What you need is actually a private queue per device-Id. As soon as event comes to event hub, Pull events from it and put that into device-Id's private queue, and then process it serially.
How to build queue per device-Id:
Simple way to build queue is to use SQL database(mostly it works if request per second are not very very high, for sql-db 100 req/second are normal.)
another horizontally scalable way is to use azure append blobs(if your event processors are stateless).
You can also use advanced methodology like using Azure Service Fabric Reliable Queue.

Sequence processing with Azure Function & Service Bus

I have an issue with Azure Function Service Bus trigger.
The issue is Azure function cannot wait a message done before process a new message. It process Parallel, it not wait 5s before get next message. But i need it process sequencecy (as image bellow).
How can i do that?
[FunctionName("HttpStartSingle")]
public static void Run(
[ServiceBusTrigger("MyServiceBusQueue", Connection = "Connection")]string myQueueItem,
[OrchestrationClient] DurableOrchestrationClient starter,
ILogger log)
{
Console.WriteLine($"MessageId={myQueueItem}");
Thread.Sleep(5000);
}
I resolved my problem by using this config in my host.json
{
"version": "2.0",
"extensions": {
"serviceBus": {
"messageHandlerOptions": {
"maxConcurrentCalls": 1
}
}
}}
There are two approaches you can accomplish this,
(1) You are looking for Durable Function with function chaining
For background jobs you often need to ensure that only one instance of
a particular orchestrator runs at a time. This can be done in Durable
Functions by assigning a specific instance ID to an orchestrator when
creating it.
(2) Based on the messages that you are writing to Queue, you need to partition the data, that will automatically handle the order of messages which you do not need to handle manually by azure function
In general, ordered messaging is not something I'd be striving to implement since the order can and at some point will be distorted. Saying that, in some scenarios, it's required. For that, you should either use Durable Function to orchestrate your messages or use Service Bus message Sessions.
Azure Functions has recently added support for ordered message delivery (accent on the delivery part as processing can still fail). It's almost the same as the normal Function, with a slight change that you need to instruct the SDK to utilize sessions.
public async Task Run(
[ServiceBusTrigger("queue",
Connection = "ServiceBusConnectionString",
IsSessionsEnabled = true)] Message message, // Enable Sessions
ILogger log)
{
log.LogInformation($"C# ServiceBus queue trigger function processed message: {Encoding.UTF8.GetString(message.MessageId)}");
await _cosmosDbClient.Save(...);
}
Here's a post for more detials.
Warning: using sessions will require messages to be sent with a session ID, potentially requiring a change on the sending side.

Polling storage queue for messages using console app webjob

I wanted to create a console app as a WebJob using .NET Core but the WebJobs SDK is not yet available in .NET Core.
I've been advised to handle the scenario of reading messages from Azure Storage Queue manually. Looks like all the WebJobs SDK does is to keep polling the queue anyway.
Is the following code the basic idea in doing this? It doesn't look very sophisticated but not sure how it can be more sophisticated.
static void Main(string[] args)
{
var runContinuously = true;
while (runContinuously)
{
ReadAndProcessMessage();
System.Threading.Thread.Sleep(1000);
};
}
private static void ReadAndProcessMessage()
{
// Read message
ReadMessage();
// Process message and handle the work
HandleWork();
}
That will work. And I like simplicity.
The QueueTriggerAttribute makes use of a random exponential back-off algorithm to help minimize your transaction costs. If you'd like to trace through the logic of how this is accomplished, starting with the QueueListener class is a good way to go. Clone the project and then hop over to the RandomizedExponentialBackoffStrategy class.

Setup webjob ServiceBusTriggers or queue names at runtime (without hard-coded attributes)?

Is there any way to configure triggers without attributes? I cannot know the queue names ahead of time.
Let me explain my scenario here.. I have one service bus queue, and for various reasons (complicated duplicate-suppression business logic), the queue messages have to be processed one at a time, so I have ServiceBusConfiguration.OnMessageOptions.MaxConcurrentCalls set to 1. So processing a message holds up the whole queue until it is finished. Needless to say, this is suboptimal.
This 'one at a time' policy isn't so simple. The messages could be processed in parallel, they just have to be divided into groups (based on a field in message), say A and B. Group A can process its messages one at a time, and group B can process its own one at a time, etc. A and B are processed in parallel, all is good.
So I can create a queue for each group, A, B, C, ... etc. There are about 50 groups, so 50 queues.
I can create a queue for each, but how to make this work with the Azure Webjobs SDK? I don't want to copy-paste a method for each queue with a different ServiceBusTrigger for the SDK to discover, just to enforce one-at-a-time per queue/group, then update the code with another copy-paste whenever another group is needed. Fetching a list of queues at startup and tying to the function is preferable.
I have looked around and I don't see any way to do what I want. The ITypeLocator interface is pretty hard-set to look for attributes. I could probably abuse the INameResolver, but it seems like I'd still have to have a bunch of near-duplicate methods around. Could I somehow create what the SDK is looking for at startup/runtime?
(To be clear, I know how to use INameResolver to get queue name as at How to set Azure WebJob queue name at runtime? but though similar this isn't my problem. I want to setup triggers for multiple queues at startup for the same function to get the one-at-a-time per queue processing, without using the trigger attribute 50 times repeatedly. I figured I'd ask again since the SDK repo is fairly active and it's been a year..).
Or am I going about this all wrong? Being dumb? Missing something? Any advice on this dilemma would be welcome.
The Azure Webjob Host discovers and indexes the functions with the ServiceBusTrigger attribute when it starts. So there is no way to set up the queues to trigger at the runtime.
The simpler solution for you is to create a long time running job and implement it manually:
public class Program
{
private static void Main()
{
var host = new JobHost();
host.CallAsync(typeof(Program).GetMethod("Process"));
host.RunAndBlock();
}
[NoAutomaticTriggerAttribute]
public static async Task Process(TextWriter log, CancellationToken token)
{
var connectionString = "myconnectionstring";
// You can also get the queue name from app settings or azure table ??
var queueNames = new[] {"queueA", "queueA" };
var messagingFactory = MessagingFactory.CreateFromConnectionString(connectionString);
foreach (var queueName in queueNames)
{
var receiver = messagingFactory.CreateMessageReceiver(queueName);
receiver.OnMessage(message =>
{
try
{
// do something
....
// Complete the message
message.Complete();
}
catch (Exception ex)
{
// Log the error
log.WriteLine(ex.ToString());
// Abandon the message so that it can be retry.
message.Abandon();
}
}, new OnMessageOptions() { MaxConcurrentCalls = 1});
}
// await until the job stop or restart
await Task.Delay(Timeout.InfiniteTimeSpan, token);
}
}
Otherwise, if you don't want to deal with multiple queues, you can have a look at azure servicebus topic/subscription and create SqlFilter to send your message to the right subscription.
Another option could be to create your own trigger: The azure webjob SDK provides extensibility points to create your own trigger binding :
Binding Extensions Overview
Good Luck !
Based on my understanding, your needs seems to be building a message batch system in parallel. The #Thomas solution is good, but I think Azure Batch service with Table storage may be better and could be instead of the complex solution of ServiceBus queue + WebJobs with a trigger.
Using Azure Batch with Table storage, you can control the task creation and execute the task in parallel and at scale, even monitor these tasks, please refer to the tutorial to know how to.

How to get notification from about newly added topics or queue?

I'm new Azure developer. My scenario is something like manager will publish new topic/queue by website/wp8 and worker should get notification (by push notification) in wp8 about newly added topic/queue. At this moment I have all the pieces ready such as topic/queue creation, sending receiving. But it works on pulling basis. Meaning manager can crate topic and publish message. Then worker has to subscribe to the topic for receiving message and pull every time to check is there anything new?.
So I want to make this system based on notification. I meant whenever anything newly added in topic user should get a notification (by push notification). So can you suggest me how can I achieve this goal? Is there any event generates from service bus if topic added or removed, etc? Thanks in advance!
Unfortunately there is no "notification hook" for when a queue/topic is created. The two options I'd recommend are to either use the service bus management API to periodically scan for new queues/topics, or better yet, set up a "notification topic" which your worker role instances can then listen to. Dropping a message into this topic can be another action performed by the "manager" process when it provisions a new topic/queue.
However, if you could explain the larger scenario of what you're trying to accomplish, I can't help but suspect that there may be a better way to accomplish what you're after. As after a period of time, all those topics/queues could present some management challenges.
off course we do have events which notifies the client when a new message is added to Topic .With message pump mechanism you can hook a client to Topic messages with a valid subscription .
Essentially the code bellow shows how to subscribe to the topic .
static void Main(string[] args)
{
SubscriptionClient Client = null;
OnMessageOptions options;
string connectionString = "your topic Endpoint";
Client =
SubscriptionClient.CreateFromConnectionString
(connectionString, "YourTopicName", "YoursubscriberName");
// Configure the callback options.
options = new OnMessageOptions();
options.AutoComplete = false;
options.AutoRenewTimeout = TimeSpan.FromMinutes(1);
Client.OnMessage((message) =>
{
try
{
Console.WriteLine("Topic Message : ID :" + message.MessageId + " , " + message.Label);
message.Complete();
}
catch (Exception exp)
{
message.Abandon();
Console.WriteLine("**Error Reciving Message**");
}
}, options);
Console.ReadLine();
}

Resources