I work with azure Topics/Subscriptions for a project.
And i want to find the fastest solution to know if a BrokeredMessage will be received by 1 subscription at least.
I found a basic solution :
Before to send the message, i call GetRules method to iterate throught sql filters :
var rules = NamespaceMgr.GetRules("topict1", s.Name);
foreach (var ruleDescription in rules)
{
Console.Write(ruleDescription.Name);
var filter = ruleDescription.Filter as SqlFilter;
if(filter != null)
{
expressions.Add(filter.SqlExpression);
}
//...examine exisitngs expressions to know if the message will be handled by a subscription receiver
}
Is there a faster way ?
For example, is there a way to send the message instataneously the message in another queue if he didn't find a receiver to go ?
What you are doing is wrong. Topics are created to decouple publishers and subscribers. Your publisher should not be concerned about the fact if subscribers exist or not. Events are broadcasted, so if no listeners are found, those messages go nowhere.
There's a setting topic setting, TopicDescription.EnableFilteringMessagesBeforePublishing that when is set to true will throw NoMatchingSubscriptionException exception where there are no subscriptions that would be able to process a message. It is not intended for production.
Related
I have the logic below that I am using to peek at the messages on a subscription
var path = EntityNameHelper.FormatSubscriptionPath(TopicName, subscriptionName);
var receiver = new MessageReceiver(connection string, path);
var messages = await receiver.PeekAsync(1000);
When I look at Service Bus Explorer it shows that there are 800 messages on the subscription
However the logic only returns 23
Does anyone know why this happens, is there some kind of caching or something?
Paul
That's by design. Peek and receive operations will return as much as broker can at that specific moment. If you want to retrieve all the messages, you'd have to write some code to iterate over the request one or more time until the number of items you need is reached.
If you want to raise a broker request to clarify this, there's a service issue tracker here.
C# .NetCore 2.2 -
Azure Service Bus 3.4.0
I have 3 queues in Azure Service Bus with same properties. While sending messages to these queues, the messages in one of the queues always get delivered to Dead letter queues, while other 2 queues receive active messages.
I have tried playing with the properties - increase TTL, maximum delivery count etc. The properties of all 3 queues are same, the only difference is the name of the queues.
I have used this tutorial - https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues
queue properties image
static async Task SendMessagesAsync(int numberOfMessagesToSend)
{
try
{
for (var i = 0; i < numberOfMessagesToSend; i++)
{
// Create a new message to send to the queue.
string messageBody = $"Message {i}";
var message = new Message(Encoding.UTF8.GetBytes(messageBody));
Console.WriteLine($"Sending message: {messageBody}");
// Send the message to the queue.
await queueClient.SendAsync(message);
}
}
catch (Exception exception)
{
Console.WriteLine($"{DateTime.Now} :: Exception: {exception.Message}");
}
}
How do I prevent messages from going to Dead Letter Queue? Why does it happen with only 1 queue, not the other 2?
When messages are dead-lettered, there is a reason user property gets added. Check that property to see the reason and troubleshoot accordingly. Specifically, check for DeadLetterReason and DeadLetterErrorDescription custom properties.
The common reasons for a message to be dead-lettered are
Maximum transfer hop count exceeded
Session Id Is Null
TTLExpiredException
HeaderSizeExceeded
The messages might also have got dead-lettered due to some errors while receiving the message from the Queue. As Sean Feldman mentioned, looking into the DeadLetterReason and DeadLetterDescription property will help you diagnose the error reason clearly.
Also try to increase or set the time to live of the message sent if the DeadLetterReason is TTLExpiredException. Because if you have set the time to live of the message to a lower value then it will override the time to live property of the Queue.
Check whether the Queue whether the queue where the messages are getting dead-lettered is a Session enabled queue and the message sent has the Session Id value set.
Without seeing your app / messages it's hard to help. But probably there's an error with the application that is trying to consume the message. As it could not be completed, the message goes to the dead letter queue.
Log the messages from this particular queue and see if there's any missing required properties. Sometimes you're trying to deserialize to an uncompatible type.
The purpose of the dead-letter queue is to hold messages that cannot
be delivered to any receiver, or messages that could not be processed.
Background
I have several clients sending messages to an azure service bus queue. To match it, I need several machines reading from that queue and consuming the messages as they arrive, using Node.js.
Research
I have read the azure service bus queues tutorial and I am aware I can use receiveQueueMessage to read a message from the queue.
However, the tutorial does not mention how one can listen to a queue and read messages as soon as they arrive.
I know I can simply poll the queue for messages, but this spams the servers with requests for no real benefit.
After searching in SO, I found a discussion where someone had a similar issue:
Listen to Queue (Event Driven no polling) Service-Bus / Storage Queue
And I know they ended up using the C# async method ReceiveAsync, but it is not clear to me if:
That method is available for Node.js
If that method reads messages from the queue as soon as they arrive, like I need.
Problem
The documentation for Node.js is close to non-existant, with that one tutorial being the only major document I found.
Question
How can my workers be notified of an incoming message in azure bus service queues ?
Answer
According to Azure support, it is not possible to be notified when a queue receives a message. This is valid for every language.
Work arounds
There are 2 main work arounds for this issue:
Use Azure topics and subscriptions. This way you can have all clients subscribed to an event new-message and have them check the queue once they receive the notification. This has several problems though: first you have to pay yet another Azure service and second you can have multiple clients trying to read the same message.
Continuous Polling. Have the clients check the queue every X seconds. This solution is horrible, as you end up paying the network traffic you generate and you spam the service with useless requests. To help minimize this there is a concept called long polling which is so poorly documented it might as well not exist. I did find this NPM module though: https://www.npmjs.com/package/azure-awesome-queue
Alternatives
Honestly, at this point, you may be wondering why you should be using this service. I agree...
As an alternative there is RabbitMQ which is free, has a community, good documentation and a ton more features.
The downside here is that maintaining a RabbitMQ fault tolerant cluster is not exactly trivial.
Another alternative is Apache Kafka which is also very reliable.
You can receive messages from the service bus queue via subscribe method which listens to a stream of values. Example from Azure documentation below
const { delay, ServiceBusClient, ServiceBusMessage } = require("#azure/service-bus");
// connection string to your Service Bus namespace
const connectionString = "<CONNECTION STRING TO SERVICE BUS NAMESPACE>"
// name of the queue
const queueName = "<QUEUE NAME>"
async function main() {
// create a Service Bus client using the connection string to the Service Bus namespace
const sbClient = new ServiceBusClient(connectionString);
// createReceiver() can also be used to create a receiver for a subscription.
const receiver = sbClient.createReceiver(queueName);
// function to handle messages
const myMessageHandler = async (messageReceived) => {
console.log(`Received message: ${messageReceived.body}`);
};
// function to handle any errors
const myErrorHandler = async (error) => {
console.log(error);
};
// subscribe and specify the message and error handlers
receiver.subscribe({
processMessage: myMessageHandler,
processError: myErrorHandler
});
// Waiting long enough before closing the sender to send messages
await delay(20000);
await receiver.close();
await sbClient.close();
}
// call the main function
main().catch((err) => {
console.log("Error occurred: ", err);
process.exit(1);
});
source :
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-nodejs-how-to-use-queues
I asked myslef the same question, here is what I found.
Use Google PubSub, it does exactly what you are looking for.
If you want to stay with Azure, the following ist possible:
cloud functions can be triggered from SBS messages
trigger an event-hub event with that cloud function
receive the event and fetch the message from SBS
You can make use of serverless functions which are "ServiceBusQueueTrigger",
they are invoked as soon as message arrives in queue,
Its pretty straight forward doing in nodejs, you need bindings defined in function.json which have type as
"type": "serviceBusTrigger",
This article (https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus#trigger---javascript-example) probably would help in more detail.
I'm new Azure developer. My scenario is something like manager will publish new topic/queue by website/wp8 and worker should get notification (by push notification) in wp8 about newly added topic/queue. At this moment I have all the pieces ready such as topic/queue creation, sending receiving. But it works on pulling basis. Meaning manager can crate topic and publish message. Then worker has to subscribe to the topic for receiving message and pull every time to check is there anything new?.
So I want to make this system based on notification. I meant whenever anything newly added in topic user should get a notification (by push notification). So can you suggest me how can I achieve this goal? Is there any event generates from service bus if topic added or removed, etc? Thanks in advance!
Unfortunately there is no "notification hook" for when a queue/topic is created. The two options I'd recommend are to either use the service bus management API to periodically scan for new queues/topics, or better yet, set up a "notification topic" which your worker role instances can then listen to. Dropping a message into this topic can be another action performed by the "manager" process when it provisions a new topic/queue.
However, if you could explain the larger scenario of what you're trying to accomplish, I can't help but suspect that there may be a better way to accomplish what you're after. As after a period of time, all those topics/queues could present some management challenges.
off course we do have events which notifies the client when a new message is added to Topic .With message pump mechanism you can hook a client to Topic messages with a valid subscription .
Essentially the code bellow shows how to subscribe to the topic .
static void Main(string[] args)
{
SubscriptionClient Client = null;
OnMessageOptions options;
string connectionString = "your topic Endpoint";
Client =
SubscriptionClient.CreateFromConnectionString
(connectionString, "YourTopicName", "YoursubscriberName");
// Configure the callback options.
options = new OnMessageOptions();
options.AutoComplete = false;
options.AutoRenewTimeout = TimeSpan.FromMinutes(1);
Client.OnMessage((message) =>
{
try
{
Console.WriteLine("Topic Message : ID :" + message.MessageId + " , " + message.Label);
message.Complete();
}
catch (Exception exp)
{
message.Abandon();
Console.WriteLine("**Error Reciving Message**");
}
}, options);
Console.ReadLine();
}
I'm trying to figure out the correct behavior when setting AutoDeleteOnIdle. I have a topic called MyGameMessages (not disclosing the game name since it might be considered advertisement).
What I do is that I create a subscription on each node in my server farm.
var manager = GetNameSpaceManager();
_subscriptionId = Guid.NewGuid().ToString();
var description = new SubscriptionDescription(topic, _subscriptionId);
description.AutoDeleteOnIdle = TimeSpan.FromHours(1);
manager.CreateSubscription(description);
Then I start up a thread that pretty much loops for eternity (or at least until signaled to quit)
while(_running)
{
if (_subscriptionId == null)
break;
var message = client.Receive(TimeSpan.FromMinutes(1)); // MARK A
if (message != null)
{
var body = message.GetBody<T>();
// Do stuff with message
message.Complete();
}
}
Question A:
The first implementation had no timeout at MARK A. If no message is sent to this topic within one hour the subscription was autodeleted. Is this the behavior to expect? The client isn't really dead but I guess it just sits around waiting for a message. Is there no keep alive?
Question B:
Would it help to add the timeout as in MARK A or is it a better solution to create a new subscription every 50th minute (to create a small overlap just in case) and abandon the old one?
Thanks
Johan
Johan, the scenario you describe above should work per your expectations. A pending receive call will keep the subscription alive even if no messages are flowing. Using longer timeouts for the Receive are better so you do not have chatty traffic when message volume is low. One thing to confirm is if your are setting the AutoDeleteOnIdle value for the Topic, in that case a receive on a subscription will NOT keep the Topic alive and if no messages are sent to the Topic for one hour then it will get deleted. Deleting a Topic results in all the Subscriptions being deleted too.
Are you still seeing this behavior of Subscriptions being deleted? If so then please create a ticket with Azure live site support and the product team an investigate the specifics.