We have a job hosted in an azure website, the job reads entries from a topic subscription. Everything works fine when we only have one instance to host the website. Once we scale out to more than one instance we observe the message is processed as many times as instances we have. Each instance points to the same subscription. From what we read, once the item is read, it won't be available for any other process. The duplicated processing is happening inside the same instance, meaning that if we have two instances, the item is processed twice in one of the instances, it is not splitted.
What can be possible be wrong in the way we are doing things?
This is how we proceed to configure the connection to the queue, if the subscription does not exists, it is created:
var serviceBusConfig = new ServiceBusConfiguration
{
ConnectionString = transactionsBusConnectionString
};
config.UseServiceBus(serviceBusConfig);
var allRule1 = new RuleDescription
{
Name = "All",
Filter = new TrueFilter()
};
SetupSubscription(transactionsBusConnectionString,"topic1", "subscription1", allRule1);
private static void SetupSubscription(string busConnectionString, string topicNameKey, string subscriptionNameKey, RuleDescription newRule)
{
var namespaceManager =
NamespaceManager.CreateFromConnectionString(busConnectionString);
var topicName = ConfigurationManager.AppSettings[topicNameKey];
var subscriptionName = ConfigurationManager.AppSettings[subscriptionNameKey];
if (!namespaceManager.SubscriptionExists(topicName, subscriptionName))
{
namespaceManager.CreateSubscription(topicName, subscriptionName);
}
var subscriptionClient = SubscriptionClient.CreateFromConnectionString(busConnectionString, topicName, subscriptionName);
var rules = namespaceManager.GetRules(topicName, subscriptionName);
foreach (var rule in rules)
{
subscriptionClient.RemoveRule(rule.Name);
}
subscriptionClient.AddRule(newRule);
rules = namespaceManager.GetRules(topicName, subscriptionName);
rules.ToString();
}
Example of the code that process the topic item:
public void SendInAppNotification(
[ServiceBusTrigger("%eventsTopicName%", "%SubsInAppNotifications%"), ServiceBusAccount("OutputServiceBus")] Notification message)
{
this.valueCalculator.AddInAppNotification(message);
}
This method is inside a Function static class, I'm using azure web job sdk.
Whenever the azure web site is scaled to more than one instance, all the instances share the same configuration.
It sounds like you're creating a new subscription each time your new instance runs, rather than hooking into an existing one. Topics are designed to allow multiple subscribers to attach in that way as well - usually though each subscriber has a different purpose, so they each see a copy of the message.
I cant verify this from your code snippet but that's my guess - are the config files identical? You should add some trace output to see if your processes are calling CreateSubscription() each time they run.
I think I can access the message id, I'm using azure web job sdk but I think I can find a way to get it. Let me check it and will let you know.
Related
I want to stream the events from different subscription to a single eventhub on azure.
At present I have configured eventhub to a single subscription and events are being streamed. I have a java client which consumes these events and stores it on my persistence layer. My java client looks like..
private void processUsingProcessorClient(){
List<Disposable> subscriptions = null;
try {
EventHubConsumerAsyncClient eventHubConsumerAsyncClient = new EventHubClientBuilder()
.consumerGroup(EventHubClientBuilder.DEFAULT_CONSUMER_GROUP_NAME)
.connectionString(CONNECTION_STRING, EVENT_HUB_NAME)
.credential("*******.servicebus.windows.net","maney-event-hub",createClientSecretCredential())
.buildAsyncConsumerClient();
ReceiveOptions receiveOptions = new ReceiveOptions().setTrackLastEnqueuedEventProperties(true);
List<String> block = eventHubConsumerAsyncClient.getPartitionIds().collectList().block();
Iterator<String> iterator = block.stream().iterator();
String partitionID = null;
subscriptions = new ArrayList<>(block.size());
while(iterator.hasNext()){
partitionID = iterator.next();
Disposable subscription = eventHubConsumerAsyncClient.receiveFromPartition(
partitionID,
EventPosition.fromOffset(0),receiveOptions).subscribe(PARTITION_PROCESSOR,ERROR_HANDLER);
subscriptions.add(subscription);
}
System.in.read();
}catch (Exception ex){
ex.printStackTrace();
} finally {
if(subscriptions != null){
subscriptions.forEach( subscrip -> {
subscrip.dispose();
});
}
}
}
private final Consumer<PartitionEvent> PARTITION_PROCESSOR = partitionEvent -> {
EventData event = partitionEvent.getData();
PartitionContext partitionContext = partitionEvent.getPartitionContext();
String contents = new String(event.getBody(), UTF_8);
LastEnqueuedEventProperties properties = partitionEvent.getLastEnqueuedEventProperties();
System.out.printf("Information received at %s. Last enqueued sequence number: %s%n",properties.getRetrievalTime(), properties.getSequenceNumber());
System.out.printf("Partition[%s] with Offset-[%s] and Sequence Number[%s] has contents: '%s'%n",
partitionContext.getPartitionId(),
event.getOffset(),
event.getSequenceNumber(),
contents);
};
private final Consumer<Throwable> ERROR_HANDLER = errorContext -> {
System.out.printf("Error occurred in partition processor");
errorContext.printStackTrace();
};
public ClientSecretCredential createClientSecretCredential() {
ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
.clientId("****************")
.clientSecret("******************")
.tenantId("**********************")
.build();
return clientSecretCredential;
}
I'm able to read all the events from a single subscription. However I need to do data analytics on these events from different subscriptions too. How do I configre Azure Eventhub to listen to multiple subscriptions ?
I read on Stackoverflow suggestions about creating consumer groups to solve this issue, however I'm not able to figure out how? I did create the consumer group, but how do i connect the newly created consumer group to different subscriptions in my azure aaccount and get the events streamed to the eventhub that i just created?
[Note : I have followed - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create to create an evenhub on azure]
Just in case if i need to clarify on what subscription I'm talking about, below is the screenshot
How do i achieve this?
Thank you in advance
Maney
So i figured out a way to solve my problem (stated above). After going through Microsoft documentation and some trial and error methods, Here's how i solved it;
I have SUBSCRIPTION-1 and SUBSCRIPTION-2. I have created an eventhub in SUBSCRIPTION-2.
I go to SUBSCRIPTION-1 one and create a Resource-Group. After creating an Resource-Group, I create a EVENT-GRID. Within the eventgrid, I create a EVENT-SUBSCRIPTION that givens an option to point it to an endpoint. I chose the endpoint and selected the eventhub that was created in SUBSCRIPTION-1.
Now, i able to stream all the events from SUBSCRIPTION-1 to SUBSCRIPTION-2.
-Maney
We are migrating a transaction-processing service which was processing messages from MSMQ and storing transacitons in a SQLServer Database to use the Azure Storage Queue (to store the id's of the messages and placing the actual messages in the Azure Storage Blob).
We should at least be able to process 200.000 messages per hour, but at the moment we barely reach 50.000 messages per hour.
Our application requests batches of 250 messages from the Queue (which now takes about 2 seconds to get the id's from the azure queue and about 5 seconds to get the actual data from the azure blob storage) and we're storing this data in one time into the database using a stored procedure accepting a datatable.
Our service also resides in Azure on a virtual machine, and we use the nuget-libraries Azure.Storage.Queues and Azure.Storage.Blobs suggested by Microsoft to access the Azure Storage queue and blob storage.
Does anyone have suggestions how to improve the speed of reading messages from the Azure Queue and then retrieving the data from the Azure Blob?
var managedIdentity = new ManagedIdentityCredential();
UriBuilder fullUri = new UriBuilder()
{
Scheme = "https",
Host = string.Format("{0}.queue.core.windows.net",appSettings.StorageAccount),
Path = string.Format("{0}", appSettings.QueueName),
};
queue = new QueueClient(fullUri.Uri, managedIdentity);
queue.CreateIfNotExists();
...
var result = await queue.ReceiveMessagesAsync(1);
...
UriBuilder fullUri = new UriBuilder()
{
Scheme = "https",
Host = string.Format("{0}.blob.core.windows.net", storageAccount),
Path = string.Format("{0}", containerName),
};
_blobContainerClient = new BlobContainerClient(fullUri.Uri, managedIdentity);
_blobContainerClient.CreateIfNotExists();
...
public async Task<BlobMessage> GetBlobByNameAsync(string blobName)
{
Ensure.That(blobName).IsNotNullOrEmpty();
var blobClient = _blobContainerClient.GetBlobClient(blobName);
if (!blobClient.Exists())
{
_log.Error($"Blob {blobName} not found.");
throw new InfrastructureException($"Blob {blobName} not found.");
}
BlobDownloadInfo download = await blobClient.DownloadAsync();
return new BlobMessage
{
BlobName = blobClient.Name,
BaseStream = download.Content,
Content = await GetBlobContentAsync(download)
};
}
Thanks,
Vincent.
Based on the code you posted, I can suggest two improvements:
Receive 32 messages at a time instead of 1: Currently you're getting just one message at a time (var result = await queue.ReceiveMessagesAsync(1);). You can receive a maximum of 32 messages from the top of the queue. Just change the code to var result = await queue.ReceiveMessagesAsync(32); to get 32 messages. This will save you 31 trips to storage service and that should lead to some performance improvements.
Don't try to create blob container every time: Currently you're trying to create a blob container every time you process a message (_blobContainerClient.CreateIfNotExists();). It is really unnecessary. With fetching 32 messages, you're reducing this method call by 31 times however you can just move this code to your application startup so that you only call it once during your application lifecycle.
I'm new to azure service bus and masstransit. I'm looking for a solution to a specific situation.
I have a azure service bus topic with multiple subscribers. Subscriber will receive message based on filters. I've created the topic and subscriber with code below
class Program
{
static void Main(string[] args)
{
string connectionString = "Endpoint connection string";
// the names of topics and subscriptions we'll be working with
const string topicName = "MyTestTopic";
const string allMessagesSubName = "AllMessages";
const string filteredSubName1 = "Filtered1";
const string filteredSubName2 = "Filtered2";
// let's create the topic if it doesn't already exist...
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
if (!namespaceManager.TopicExists(topicName))
{
var td = new TopicDescription(topicName);
namespaceManager.CreateTopic(td.Path);
}
if (!namespaceManager.SubscriptionExists(topicName, allMessagesSubName))
{
namespaceManager.CreateSubscription(topicName, allMessagesSubName);
}
if (!namespaceManager.SubscriptionExists(topicName, filteredSubName1))
{
namespaceManager.CreateSubscription(
new SubscriptionDescription(topicName, filteredSubName1),
new Microsoft.ServiceBus.Messaging.SqlFilter("From LIKE '%Smith'"));
}
if (!namespaceManager.SubscriptionExists(topicName, filteredSubName2))
{
namespaceManager.CreateSubscription(
new SubscriptionDescription(topicName, filteredSubName2),
new Microsoft.ServiceBus.Messaging.SqlFilter("sys.Label='important'"));
}
var message1 = new BrokeredMessage("Hello World");
var message2 = new BrokeredMessage("Second message");
message2.Label = "important";
var message3 = new BrokeredMessage("Third message");
message3.Properties["From"] = "Kelly Smith";
message3.Label = "information";
var client = TopicClient.CreateFromConnectionString(connectionString, topicName);
client.Send(message1);
client.Send(message2);
client.Send(message3);
client.Close();
}
}
Here in the code we're adding Message custom properties like From.
Now I want to send such message using masstransit. In masstransit I cannot find any option of adding Message custom properties using the Publish() method. Is there any way that I can send these messages using masstransit where these filters can be used?
NB: I've read the answer of this question But the anwer here tells us to filter the messages in subscriber side. What I want is that this filtering will occur before reaching the subscriber.
When using Azure Service Bus with MassTransit, you can add subscription endpoints in additional to regular endpoints. When configuring a subscription endpoint, you should be able to specify rules and/or filters as part of the subscription. Which is exactly what you're doing above, so that is handled.
The other part, adding properties to the message, can be done by adding text headers to the SendContext. Those headers are copied to the message Properties collection, which I believe can be used to filter messages using a "SQL" filter (which is configured on the subscription endpoint, or the topic subscription on a receive endpoint).
Trying to convert an implementation using the .net library from using QueueClient.Create to the MessagingFactory.CreateQueueClient to be able to better control the BatchFlushInterval as well as to allowing the use of multiple factories over multiple connections to increase send throughput but running into roadblocks.
Right now we are creating QueueClients (they are maintained throughout the app) like this:
QueueClient.CreateFromConnectionString(address, queueName, ReceiveMode.PeekLock); // address is the connection string from the azure portal in the form of Endpoint=sb....
Trying to change it to creating a MessagingFactory in the class construtor that will be used to create the QueueClients:
messagingFactory = MessagingFactory.Create(address.Replace("Endpoint=",""),mfs);
// later on in another part of the class
messagingFactory.CreateQueueClient(queueName, ReceiveMode.PeekLock);
// error Endpoint not found.,
This throws the error Endpoint not found. If I don't replace the Endpoint= it won't even create the MessagingFactory. What is the proper way to handle this?
Notes:
address = Endpoint=sb://pmg-bus-mybus.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=somekey
As an aside, we have a process that is trying to push as many messages as possible to a queue and others reading it. The readers seem to easily keep up with the sender and I'm trying to maximize the send rate.
The address is the base address of namespace(sb://yournamespace.servicebus.windows.net/) you are connecting to. For more information, please refer to MessagingFactory. The following is the demo code :
var Address = "sb://yournamespace.servicebus.windows.net/"; //base address of namespace you are connecting to.
MessagingFactorySettings MsgFactorySettings = new MessagingFactorySettings
{
NetMessagingTransportSettings = new NetMessagingTransportSettings
{
BatchFlushInterval = TimeSpan.FromSeconds(2)
},
TokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider("RootManageSharedAccessKey", "balabala..."),
OperationTimeout = TimeSpan.FromSeconds(30)
}; //specify operating timeout (optional)
MessagingFactory messagingFactory = MessagingFactory.Create(Address, MsgFactorySettings);
var queue = messagingFactory.CreateQueueClient("queueName",ReceiveMode.PeekLock);
var message = queue.Receive(TimeSpan.Zero);
I know there is a way to determine the number of messages (or approximate number) in the Azure Queue (Store Account); however is there a way to query for the number of pending messages on an Azure Service Bus queue?
var nsmgr = Microsoft.ServiceBus.NamespaceManager.CreateFromConnectionString(connectionString);
long count = nsmgr.GetQueue(queueName).MessageCount;
It is called MessagesCountDetails.ActiveMessageCount.
It returns the number of the Active Messages in the Queue. You probably have some Dead letter messages:
var msg = Microsoft.ServiceBus.NamespaceManager.CreateFromConnectionString(Settings.Default.ConnectionString);
numofmessages.Text = msg.GetQueue(QueueName).MessageCountDetails.ActiveMessageCount.ToString();
Correct answer as of 2020+
Use of new packages as follows:
<PackageReference Include="Azure.Messaging.ServiceBus" Version="x.x.x" />
also two namespaces in the same package
using Azure.Messaging.ServiceBus;
using Azure.Messaging.ServiceBus.Administration;
and then you can use the new class ServiceBusAdministrationClient
var administrationClient = new ServiceBusAdministrationClient("connectionString");
var props = await administrationClient.GetQueueRuntimePropertiesAsync("queue");
var messageCount = props.Value.ActiveMessageCount;
have you looked at the Queue Description API? There's a property called MessageCount.
Here's the .NET SDK reference documentation page as well.
Based off what Joseph had as an answer I came up with, but for Topics and Subscriptions.
public async Task<long> GetCounterMessages()
{
var client = new ManagementClient(ServiceBusConnectionString);
var subs = await client.GetSubscriptionRuntimeInfoAsync(TopicName, SubscriptionName);
var countForThisSubscription = subs.MessageCount; //// (Comes back as a Long.)
return countForThisSubscription;
}
I ran into this same problem trying to get the count from the dead letter queue. It looks like the deadletterqueue doesn't allow you to get a count directly, you get it from the MessageCountDetails of the normal Queue.
string connectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.Connstr"].ToString();
NamespaceManager nsmgr = Microsoft.ServiceBus.NamespaceManager.CreateFromConnectionString(connectionString);
return nsmgr.GetQueue(QueueName).MessageCountDetails.DeadLetterMessageCount;
Here's a PowerShell example to continually eyeball the queue length as used in Azure Portal Cloud Shell
cd "Azure:\<MySubscription>\"
while (1) {(Get-AzureRmServiceBusQueue -ResourceGroup <myRG> -NamespaceName <myNS> -QueueName <myQueueName>).CountDetails | Select -expand ActiveMessageCount}
I've spent good 2 hours digging through docs to get that and for people using .net core and Microsoft.Azure.ServiceBus nuget package, code looks like that:
var managementClient = new ManagementClient("queue connection string"));
var runtimeInfo = await managementClient.GetQueueRuntimeInfoAsync("queueName");
var messagesInQueueCount = runtimeInfo.MessageCountDetails.ActiveMessageCount;
Aparently you get the information about all Counts(including deadletter, active, etc.) from QueueRuntimeInfo object instead of old QueueDescription object.
As per the recommendation by Microsoft, it is recommended to use Microsoft.Azure.ServiceBus in which you can easily fetch the message count by
var managementClient = new ManagementClient("connection string for queue");
var queue = await managementClient.GetQueueRuntimeInfoAsync("queue name");
var messages = queue.MessageCount;
Also..you can check the pending messages on Azure Management Portal...on the dashboard for service bus queue...under quick glance...you can see the queue length...this is the number of current/pending messages in length at the time of dashboard page load.