sorry if this is a dumb question, but I cannot seem to find the answer:
I am using an external source to read in Audit Log events in Azure Eventhub. I am able to get the data flowing and working, but I see that there are messages with the records field that have 2 messages, but some records only have 1 message. For those records that have 2 json events in them, why is this the case? I see that they might be related.
What I mean is that some logs I will see for some:
category:NoninteractiveSignin:
records:[{..},{..}]
Event Hub messages are binary, and opaque to Event Hubs. It’s entirely up to the sender what’s in each one.
So you’ll need to ask whatever application creates the messages about that.
Related
I have a request to implement a dashboard with the information about which message in Azure Service Bus queue was completed when (with some info about message parameters). Unfortunately we do not have an access to the reciever's code and cannot change the code to log the time of the message delivery. So, we need to subscribe somehow to a moment when reciever takes away the message.
I have already investigated Azure portal API in order to find something, but there is no such a possibility, I have tried to find something on stackoverflow and in Google, but no results.
There is 1 idea: use 2 queues and azure function between them. Put all messages to the first queue, azure function recieves a message, logs the info about the message and puts it to the second queue and waits until other services takes the message away from the second queue. Second queue will always have only 1 message and this way we will be able to understand what message was for sure delivered and when.
However what I do not like is the second message queue executes not the role of the real queue (it means something is wrong here and I need to use something else), performance of such a system can be not high enough...
Any help is appreciated (articles, videos, ideas). Thank you.
I'm creating a consumer of an Azure Service Bus topic (subscription) that does nothing but store some statistics. The messages sent to the topic contains a rather large body, that is handled by another consumer on a second subscription on the same topic.
Since the statistics consumer can handle a large number of messages in one go, I was wondering if it is possible to receive a lot of messages but leave out the body to improve performance when communicating with Service Bus and to receive even more messages in one go.
I'm currently doing this:
this.messageReceiver = new MessageReceiver(conn, path);
...
await messageReceiver.ReceiveAsync(10, TimeSpan.FromSeconds(5));
It works pretty sweet but it would be nice to be able to receive 100 or more messages, without having to worry about moving large messages over the network.
Before anyone suggests it, I already know that I can ask for a count, etc. on a topic subscription. I still need the Message object since that contains an entry in the UserProperties dictionary that is used to calculate the stats.
Not possible. You can peek, but that brings the whole payload and headers w/o incrementing the DeliveryCount of the message. You could request it as a broker feature here.
I have 65k records in Azure Service Bus Topic, while testing, whenever my test application is started, it reads all the 65k records. Can you please help me how can we avoid reading messages that have already read or How can we read only the messages that are send after executing test application?
From the question, it's unclear what exactly you're after. Here are a few things for consideration.
Queues/subscriptions are intended to be read by the consumers, not to store messages and access conditionally. To avoid consuming messages, you should consume those either by using ReceiveAndDelete receiving more, or PeekLock and completing the received messages.
If these messages are test messages and are not intended for the production, do not mix the environments and use different namespaces.
Alternatively, set a short TimeToLive on your test messages to get rid of those. You could also drop the entity and recreate it, but I try to avoid this if your performing testing quite often.
I might be confused how EventHubs supposed to be used or need guidance on how to reliably process events posted into Eventhub. I export Azure ActivityLog to Eventhub and currently just using console application to read those messages. What I don't understand is what I'm supposed to do with events which I already read and processed. Say I want to write content of all messages into Storage account AppendLog. For this I need to delete messages which I already processed (like it would be done if it will be message queue), how do I do that with eventhub?
You cannot delete them. From the docs:
Event Hubs retains data for a configured retention time that applies across all partitions in the event hub. Events expire on a time basis; you cannot explicitly delete them.
Back to your question:
Say I want to write content of all messages into Storage account AppendLog. For this I need to delete messages which I already processed
I am not sure why you need this though. You can keep a pointer to the last read message so you are able to process only new messages. Why should you need to delete the older ones? You can read about offsets and ceckpointing here.
What technique are you using for reading the messages?
If you need a pattern of popping messages, you need the Queue or Topic from the Azure Service Bus.
When you ack that message, it is popped from the queue.
I have an Azure queue (ServiceBus Topic Subscription DeadLetter queue) with ~900 messages. I would like to list them for debug purposes. I call PeekBatch method with parameter messageCount set to 100. I expect to get 100 messages but I only receive 69. Why is that???
I ran into a similar issue recently where the reported Active Message Count number was higher than the actual number of messages that could be read. After some investigation, I contacted Microsoft's Azure support team. Here's what they told me:
After researching I see that there is bug about this where the count is not showing correct value. The bug is active so it’s not resolved. Product group is tracking it but I don’t have any ETA on it.
Although in my case it's the Active Message Count that was off, this could very well be the same issue.
I found a way (that only works some of the times - I don't know why) to correct the message count. Use Service Bus Explorer, select the topic that is displaying the issue, hit Update (without having changed any of the settings) and then hit Refresh.
I also had the same issue, when I tried to peek messages from queue which is enabled partitioning. if I use PeekBatch method with passing message count as parameter, I can read only some of the messages even the actual message count was higher than that.
Partition queue - Partitioned queue consists of multiple fragments handled by different message brokers. when a message is sent to the partitioned queue, Service Bus assigns the message to one of the fragments. When a client wants to receive a message from a queue, Service Bus checks all fragments for messages. if it finds any, it picks one and passes to the receiver.
So, When we try to get messages using PeekBatch() method, it checks the fragments for messages and picks the messages from the fragment which it found at first, even if it is not matched with the given message count.