I've got service A communicating with service B thru a Queue.
The message contains information that requires service B to reach out to a DB and run a query, before storing it in another DB.
Service B won't be returning anything, and is async to Service A.
The thing, is that service A is sending over the messages in a loop, which is triggering 429 errors from service B.
What's the best way to manage this kind of situation?
If it helps, both services are Azure functions and I'm using Azure's standard Storage Queue.
TIA!
429 is usually related to Rate Limit. You're either adding too many messages at once or the SDK to add a message to a queue is failing somehow.
You can fix in a few ways:
1-Implement a retry policy
2-Add a 'sleep' every x seconds
3-Add less messages in a chunk
Related
We have a usecase where we need to schedule jobs which will be sent as a message from Azure web api service to Azure Service Bus Queue. As we need to schedule it at later point in time one solution is to use Scheduled Delivery ScheduledEnqueueTimeUtc.
What i understand is message gets engqueued only after the time specified expires . My concern is what happens if Web API crashes or undergoes upgrade meanwhile.
1.Will the messages be lost as its not enqueued yet?
2.Where does this messages are stored in the intermediate time ?
Second solution is to use visibilityTimeOut of storage queue where messages are enqueued and will not be impacted by Web API.
From stability and scalability perspective which would be a better option ?
The message is sent to Service Bus, which is enqueued (available to receive) according to the schedule. So, to answer your queries
Nope
In the queue, just not available to receive
visibilityTimeOut is for storage queues. Refer the comparison doc for making the decision.
Note that while you cannot receive scheduled messages, you can peek them.
Need a bit of architectural guidance. I have a set of stateless services that do various functions. My architecture allows for multiple copies of each service to run at the same time (as they are stateless), allowing me to:
scale up as needed for handling larger workloads
have fault-tolerance (if one instance of a service fails, no problem as there will be others to take on that work).
However, I don't want duplication of work.
If Service A, Instance 1 has already taken Job ABC, I don't want Service A, Instance 2, to take on that same job. So, I could avoid this problem by using Azure Service Bus Queues. Only a single worker would get a particular item from the queue and would only be reassigned to another worker, if the worker didn't mark it as complete in a set time.
So what's an appropriate use-case for Topics (Pub/Sub)? It seems like if I ever have multiple copies of the same service, I must rely on Queues. Is that right?
Asked another way, is there a way to use Topics in Azure Service Bus or similar products/services but avoid duplication of work? Also, if there is a way to lock a message (for a short period of time) when using Topics, is it possible to lock that message to just one instance of Service A (so no other instances of Service A will have access to it) but the message will be broadcast to Service B, Service, C, etc.?
is there a way to use Topics in Azure Service Bus or similar
products/services but avoid duplication of work?
Yes, there is. Basically with that you would need to use each subscription as a queue. What you will need to do is define proper filters so that one kind of message is sent to a single subscription (that way it acts as a queue) and have multiple listeners (service instances in your case) listen to a specific subscription only.
Also, if there is a way to lock a message (for a short period of time)
when using Topics, is it possible to lock that message to just one
instance of Service A (so no other instances of Service A will have
access to it) but the message will be broadcast to Service B, Service,
C, etc.?
It is certainly possible to lock a message. For that you will need to fetch messages in Peek-Lock mode. However if multiple subscribers (services) are involved, then only one subscriber will be able to lock the message and access it. For other subscribers, the message will be invisible. You can't have a scenario where one service acquires the lock and other services still receive the message.
Azure function triggers would provide all what you are looking for out of the box.
If you are not leveraging any advanced queuing features of service bus then I would recommend you look at storage queues to save some money.
If you need service bus then you can use service bus triggers.
Hope that helps.
My scenario: I have an Azure Storage Queue where messages can come in at any time. If I have 10 items in that queue, it's imperative that they be processed in order. I'm using c# and the windows azure storage SDK.
If the first item fails after, say, 2 seconds it remains invisible on the queue for another 28 seconds (30 second invisibility by default).
Now, my worker will just continue to check a queue for messages and process them as and when. If a queue message fails, it remains invisible and so the next queue item will be processed before the first message is retried.
This seems like really basic functionality for anyone needing a queue where the items are processed in order.
No, I can't set the timeout to a smaller amount because tasks can take varying lengths of time.
George, if you are looking for a messaging queue solution that processes items in order, you should consider using Azure Service Bus Queues:
As a solution architect/developer, you should consider using Service Bus queues when:
Your solution must be able to receive messages without having to poll the queue. With Service Bus, this can be achieved through the use of the long-polling receive operation using the TCP-based protocols that Service Bus supports.
Your solution requires the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
You want a symmetric experience in Azure and on Windows Server (private cloud).
For more information, see Service Bus for Windows Server.
Your solution must be able to support automatic duplicate detection.
There is a good article comparing both Storage Queues and Service Bus: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted , you may find the latter better suitable for your case.
You can subscribe to asynchronous updates from Azure topics and queues by using SubscriptionClient/QueueClient's .OnMessage call which will presumably create a separate thread polling the topic/queue with default settings and calling a defined callback if it receives anything.
Azure website says that receiving a message is a billable action, which is understandable. However, it isn't clear enough if each those poll requests are considered billable even when they do not return anything, i.e. the queue in question has no pending messages.
Based on the Azure Service Bus Pricing FAQ - the answer to your question is yes
In general, management operations and “control messages,” such as
completes and deferrals, are not counted as billable messages. There
are two exceptions:
Null messages delivered by the Service Bus in
response to requests against an empty queue, subscription, or message
buffer, are also billable. Thus, applications that poll against
Service Bus entities will effectively be charged one message per poll.
Setting and getting state on a MessageSession will also result in
billable messages, using the same message size-based calculation
described above.
Given the price is $0.01 per 10,000 messages, I don't think you should worry too much about that.
I need to keep track of how many failed attempts have been made to process a message in an azure storage queue and delete the message after N unsuccesful attempts.
I have searched, but have not found any particular property that does this automaticaly and was wondering if there was a way other than using a counter in a storage table.
Each cloud queue message has a DequeueCount property. Does this help?
REST API reference here.
As for how to delete messages automatically after n attempts: There's nothing that automatically does this. You'll need to implement your own poison-message handling in Windows Azure queues, based on DequeueCount.
Alternatively, Azure Service Bus queues have a dead-letter queue for undeliverable messages (or ones that can't be processed). More info here.