Is it possible to implement the following pseudo scenario with an Azure Service Bus?
I have a function that can scale out to 50 instances, it uses a service bus trigger. I would like to guarantee that related messages are only processed if an existing related message is NOT currently being processed.
Let's say I have a message (Message A) being processed by a function instance that's associated with UserID 1234. Another message (Message B) appears on the queue which is also associated with UserID 1234, the service bus should "ignore" it because a related message is already being processed. Another message (Message C) with UserID 9876 appears on the queue, this gets handled straight away because there is no in action message with UserID 9876.
Message A finishes processing and Message B is now picked up.
Currently I have a routing function which consumes the initial service bus trigger and then routes it to one of 10 functions each of which is responsible for messages where the last digit of the UserID is 0-9.
This means that if function "4" is busy with a request it won't be able to process any other requests where the UserID ends with 4, thus guaranteeing the system cannot process a related message at the same time. It does it's job but doesn't scale.
There's no conditional locking. From the description, sounds like you want to process messages associated with the same user ID, one at a time. For that, Azure Service Bus has a feature called Message Sessions.
As far as I know, azure service bus has a locking mechanism built into it. So no matter the message, if you have a single queue you are reading from, when a function picks that message up the rest of your functions would not pick up that same message. To solve the issue of not processing a duplicate message by userID I would recommend using something table storage to validate if that userID has been processed already (So in your function when you pick up a message you insert that into azure table storage before doing any processing and also have a check to see if it exists before processing).
service bus - https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement
table storage - https://learn.microsoft.com/en-us/azure/cosmos-db/table/quickstart-dotnet?toc=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fazure%2Fstorage%2Ftables%2Ftoc.json&bc=https%3A%2F%2Flearn.microsoft.com%2Fen-us%2Fazure%2Fbread%2Ftoc.json&tabs=azure-cli%2Cwindows
In summary I think the solution here would be using multiple technologies and use a central point your scaled out functions could validate against whats processed by other functions, whatever that central database is
Related
I have an Azure Service Bus topic. Messages send to the topic contain application properties.
I want to set permissions on the topic based on values of one or more application properties, i.e. only when identity A sends a message with property e.g. Destination = 'service' the message is allowed to be added to the topic.
The reason behind this: assume a topic where there are multiple publishers and multiple subscribers to consume the messages from the publishers. But each message shall only go to specific consumers using filters. An actor with bad intentions or just by accident could choose a filter value and send the message therefore to wrong consumers. And I don't want to create multiple topics. I want to limit the publishers to send messages to certain consumers by limiting their permissions by message attributes with certain values on a certain topic.
Is this possible? If yes, how?
You can start a separate app which will peek the message in the queue
check for the properties and then abandon the messages.
To peek create a receiver and use receiver.ReceiveMessageAsync();
to peek the message. After this you can apply your logic to filter
the messages using the properties.
After that use AbandonAsync to abandon the message with wrong properties.
// create a reciever and peek the messages.
await using var client = new ServiceBusClient(connectionString);
//reciever
ServiceBusReceiver receiver = client.CreateReceiver(queueName) ;
// get the messages
ServiceBusReceivedMessage receivedMessage = await receiver.ReceiveMessageAsync();
// write you logic for filtering the message
//now we can abandon the message. Here "reason":" Wrong message" is optional
await receivedMessage.AbandonAsync(new Dictionary<string, object> { { "Reason", "wrong properties"} });
Refernce:
Refer the article by sfeldman
Peek messages
You can't use ServiceBus (or any message broker) the way you describe.
My advice is to consider messages only as signals to do something. But the service is entirely responsible to check if the message is legitimate or not (by calling the source service or another source of truth synchronously for example).
Imagine this scenario:
SourceApp send a DeletedEvent{CustomerId="42"}.
DestApp receives this message, and before deleting data related to this customer, makes an HTTP call to the customer service GET CustomerService/42.
If the customer is deleted => perfect
If not... something went wrong, what you described by an actor with bad intentions or just by accident
Also, if you use the same topic for a lot of applications consider using AAD authentication, with managed identity per app. Or at least forbid sharing SAS keys. Every app should have dedicated connection string, scoped to the subscription or the topic with only the access needed.
I am using Azure Service Bus with .NET core
In our application, we are sending Session Messages to Service Bus. Whenever we receive a cancellation request with the session-Id, we need to remove/delete/complete messages with that specific sessionId without any further processing
I tried to access the MessageSession from another receiver to cancel messages from the queue
But I got error - The requested session 'session-name' cannot be accepted. It may be locked by another receiver.
Is there any other possible way to delete messages from service bus queue with a specific sessionId (session may be locked in few cases)
The only workaround, I'm able to get is
Update in database stating sessionId is cancelled & checking this for each message.
But this is not efficient for a large number of messages, as we need to extra DB hit for each message.
Is there any other possible way to delete messages from service bus queue with a specific sessionId?
It is possible to delete the message with SessionId using a search feature in Serverless360 where you can specify the query as SessionId = "1" and retrieve the message and delete it. But, it is obvious that you cannot retrieve session messages when it is locked and hence your second test case is not possible anyhow unless you could manage to use some action like renew lock in your orchestration with some delay in processing the messages.
Below is the screenshot of sample orchestration where each time the lock gets renewed and hence you can retrieve the message once you have the Session-Id
Scenario
When business transactions are performed, we're supposed to make that data available to end clients.
Current Design
Our web app publishes transaction messages are added to a topic on the Azure Service Bus.
We expose APIs to clients through which they can consume the data from those transactions.
Upon calling these APIs, we read the messages from the Subscription and return it to the client.
Problem
We want a guaranteed delivery - we want to make sure the client acknowledges the delivery of the data. So we don't want to remove the message from the subscription immediately. We want to keep it until the client acknowledges it.
So we only want to do a "Peek" instead of "Receive".
So the client calls the first API, to get the data, where we do a Peek.
And once the client has received the packets, the client would call a second API, to acknowledge.
At this point, we want to remove the message from the Subscription, making it Complete.
The current design of the Service Bus Message Receiver is that, a Complete can be performed only using the same Receiver instance that performed the Peek, as per the documentation, and we also observed the same when we tried it out.
Both the APIs, are two separate APIs and we cannot do the Peek and Complete using the same instance of the Receiver.
Thinking about options to somehow make the Receiver as a Singleton, across APIs within that App Service.
However this will be a problem when the App Service scales out.
Is there a different way to achieve what we're trying to do here ?
There is an option available in Azure Service Bus to defer messages. Once a message is deferred, it can be received with the help of it's sequence number.
The first client should receive the message and instead of completing it, it should defer it and return it.
The second client (which has sequence number) can receive the message from the Subscription. Refer here for more details.
Another option would be to not use a Service Bus Client on your backend and instead your clients could directly work with Service Bus using its Service REST API (assuming they can't use the AMQP client if I am understanding your scenario correctly).
There are APIs to
Peek-Lock
Renew Lock
Unlock
Delete (Complete)
You could also proxy these requests if you'd like using your backend itself or a service like APIM if you are already using it.
PS: Cross posting the answer for the same query on the MSDN forum
Is there a way to have multiple listening clients on one Azure Topic Subscription, and they all recieve ALL messages?
My understanding is that the only implementation of a Subscription is that the Published message is only delivered to ONE client on that subscription, as it is like a queue.
Can these messages be copied to multiple clients using the same Subscription?
EDIT: Potential use case example
A server notifies all of its clients (web clients via browser, or application), that are subscribed to the topic, of an object that has changed its value
More simply, multiple PCs are able to see a data value change
EDIT 2: My setup/what I'm looking for
The issue that I am running into is that a message is marked as consumed by one client, and not delivered to the other client. I have 3 PCs in a test environment:(1 PC publishing messages (we'll call this the Publisher) to the topic, and 2 other PCs subscribed to the topic using the same SubscriptionName (We'll call these Client 1 and Client 2)).
So we have this setup:
Publisher - Publishes to topic
Client 1 - Subscibed using SubscriptionName = Test1
Client 2 - Subscribed using SubscriptionName = Test1
The Publisher publishes 10 messages to the topic.
Client 1 gets Message 0
Client 2 gets Message 1
Client 1 gets Message 2
... And so on (Not all 10 messages are recieved by both Client 1 and Client 2)
I want the Clients to recieve ALL messages, like this:
Client 1 AND Client 2 get Message 0
Client 1 AND Client 2 get Message 1
Client 1 AND Client 2 get Message 2
... And so on.
Service Bus is a one-to-one or end-to-end messaging system.
What you need is Azure Event Hub or Event Grid.
It is not possible for both the client1 and client2 to get the same messsage.
To put it straight, when a message is received by client1 from a subscription and processed successfully, the message is removed from the subscription, so the client2 will not be able to receive the same message again.
Hope this clarifies.
Yes, its a one-to-one implementation, but, if you have real concern about message processing completing in sequential order then it depends on the Receive mode.
You can specify two different modes in which Service Bus receives messages.
Receive and delete.
In this mode, when Service Bus receives the request from the consumer, it marks the message as being consumed and returns it to the consumer application. This mode is the simplest model. It works best for scenarios in which the application can tolerate not processing a message if a failure occurs. To understand this scenario, consider a scenario in which the consumer issues the receive request and then crashes before processing it. As Service Bus marks the message as being consumed, the application begins consuming messages upon restart. It will miss the message that it consumed before the crash.
Peek lock.
In this mode, the receive operation becomes two-stage, which makes it possible to support applications that can't tolerate missing messages.
Finds the next message to be consumed, locks it to prevent other consumers from receiving it, and then, return the message to the application.
After the application finishes processing the message, it requests the Service Bus service to complete the second stage of the receive process. Then, the service marks the message as being consumed.
If the application is unable to process the message for some reason, it can request the Service Bus service to abandon the message. Service Bus unlocks the message and makes it available to be received again, either by the same consumer or by another competing consumer. Secondly, there's a timeout associated with the lock. If the application fails to process the message before the lock timeout expires, Service Bus unlocks the message and makes it available to be received again.
If the application crashes after it processes the message, but before it requests the Service Bus service to complete the message, Service Bus redelivers the message to the application when it restarts. This process is often called at-least once processing. That is, each message is processed at least once. However, in certain situations the same message may be redelivered. If your scenario can't tolerate duplicate processing, add additional logic in your application to detect duplicates. For more information, see Duplicate detection. This feature is known as exactly once processing.
Check this link for more details.
I have several subscriptions that listens to different Topics for messages, and some of this messages are dependent on each other. So one message for one subscription "needs" to arrive before another messages in another subscriptions.
I could solve this by storing the messages temporary in a database, but I thought that if I get a message on one subscription and it's correlated messages on a another subscription hasn't arrived yet, I would just wait 1 second and put the first messages back on it's subscription so the correlated messages get's some more time to arrive first.
It's easy if it would have been a Queue, but now it's a subscription and that client don't have any "Send" methods on it.
I don't want to put the messages back on the Topics, since other subscription might not want that messages again.
Since subscriptions basically is a Queue it should be possible, so is there some "base object" that could be used to put messages directly to a subscription queue.
Best Regards
Magnus Gladh
While subscription is a queue behind the scenes, you cannot send messages directly to that queue. Instead, you should target a topic.
If you wish to abort the receive operation, you can when receiving in PeekLock mode.