Sending message to a specific group of subscriber in azure service bus topic with masstransit - azure

I'm new to azure service bus and masstransit. I'm looking for a solution to a specific situation.
I have a azure service bus topic with multiple subscribers. Subscriber will receive message based on filters. I've created the topic and subscriber with code below
class Program
{
static void Main(string[] args)
{
string connectionString = "Endpoint connection string";
// the names of topics and subscriptions we'll be working with
const string topicName = "MyTestTopic";
const string allMessagesSubName = "AllMessages";
const string filteredSubName1 = "Filtered1";
const string filteredSubName2 = "Filtered2";
// let's create the topic if it doesn't already exist...
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
if (!namespaceManager.TopicExists(topicName))
{
var td = new TopicDescription(topicName);
namespaceManager.CreateTopic(td.Path);
}
if (!namespaceManager.SubscriptionExists(topicName, allMessagesSubName))
{
namespaceManager.CreateSubscription(topicName, allMessagesSubName);
}
if (!namespaceManager.SubscriptionExists(topicName, filteredSubName1))
{
namespaceManager.CreateSubscription(
new SubscriptionDescription(topicName, filteredSubName1),
new Microsoft.ServiceBus.Messaging.SqlFilter("From LIKE '%Smith'"));
}
if (!namespaceManager.SubscriptionExists(topicName, filteredSubName2))
{
namespaceManager.CreateSubscription(
new SubscriptionDescription(topicName, filteredSubName2),
new Microsoft.ServiceBus.Messaging.SqlFilter("sys.Label='important'"));
}
var message1 = new BrokeredMessage("Hello World");
var message2 = new BrokeredMessage("Second message");
message2.Label = "important";
var message3 = new BrokeredMessage("Third message");
message3.Properties["From"] = "Kelly Smith";
message3.Label = "information";
var client = TopicClient.CreateFromConnectionString(connectionString, topicName);
client.Send(message1);
client.Send(message2);
client.Send(message3);
client.Close();
}
}
Here in the code we're adding Message custom properties like From.
Now I want to send such message using masstransit. In masstransit I cannot find any option of adding Message custom properties using the Publish() method. Is there any way that I can send these messages using masstransit where these filters can be used?
NB: I've read the answer of this question But the anwer here tells us to filter the messages in subscriber side. What I want is that this filtering will occur before reaching the subscriber.

When using Azure Service Bus with MassTransit, you can add subscription endpoints in additional to regular endpoints. When configuring a subscription endpoint, you should be able to specify rules and/or filters as part of the subscription. Which is exactly what you're doing above, so that is handled.
The other part, adding properties to the message, can be done by adding text headers to the SendContext. Those headers are copied to the message Properties collection, which I believe can be used to filter messages using a "SQL" filter (which is configured on the subscription endpoint, or the topic subscription on a receive endpoint).

Related

Multiple Subscriptions in Azure Account Streaming to a single EventHub

I want to stream the events from different subscription to a single eventhub on azure.
At present I have configured eventhub to a single subscription and events are being streamed. I have a java client which consumes these events and stores it on my persistence layer. My java client looks like..
private void processUsingProcessorClient(){
List<Disposable> subscriptions = null;
try {
EventHubConsumerAsyncClient eventHubConsumerAsyncClient = new EventHubClientBuilder()
.consumerGroup(EventHubClientBuilder.DEFAULT_CONSUMER_GROUP_NAME)
.connectionString(CONNECTION_STRING, EVENT_HUB_NAME)
.credential("*******.servicebus.windows.net","maney-event-hub",createClientSecretCredential())
.buildAsyncConsumerClient();
ReceiveOptions receiveOptions = new ReceiveOptions().setTrackLastEnqueuedEventProperties(true);
List<String> block = eventHubConsumerAsyncClient.getPartitionIds().collectList().block();
Iterator<String> iterator = block.stream().iterator();
String partitionID = null;
subscriptions = new ArrayList<>(block.size());
while(iterator.hasNext()){
partitionID = iterator.next();
Disposable subscription = eventHubConsumerAsyncClient.receiveFromPartition(
partitionID,
EventPosition.fromOffset(0),receiveOptions).subscribe(PARTITION_PROCESSOR,ERROR_HANDLER);
subscriptions.add(subscription);
}
System.in.read();
}catch (Exception ex){
ex.printStackTrace();
} finally {
if(subscriptions != null){
subscriptions.forEach( subscrip -> {
subscrip.dispose();
});
}
}
}
private final Consumer<PartitionEvent> PARTITION_PROCESSOR = partitionEvent -> {
EventData event = partitionEvent.getData();
PartitionContext partitionContext = partitionEvent.getPartitionContext();
String contents = new String(event.getBody(), UTF_8);
LastEnqueuedEventProperties properties = partitionEvent.getLastEnqueuedEventProperties();
System.out.printf("Information received at %s. Last enqueued sequence number: %s%n",properties.getRetrievalTime(), properties.getSequenceNumber());
System.out.printf("Partition[%s] with Offset-[%s] and Sequence Number[%s] has contents: '%s'%n",
partitionContext.getPartitionId(),
event.getOffset(),
event.getSequenceNumber(),
contents);
};
private final Consumer<Throwable> ERROR_HANDLER = errorContext -> {
System.out.printf("Error occurred in partition processor");
errorContext.printStackTrace();
};
public ClientSecretCredential createClientSecretCredential() {
ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
.clientId("****************")
.clientSecret("******************")
.tenantId("**********************")
.build();
return clientSecretCredential;
}
I'm able to read all the events from a single subscription. However I need to do data analytics on these events from different subscriptions too. How do I configre Azure Eventhub to listen to multiple subscriptions ?
I read on Stackoverflow suggestions about creating consumer groups to solve this issue, however I'm not able to figure out how? I did create the consumer group, but how do i connect the newly created consumer group to different subscriptions in my azure aaccount and get the events streamed to the eventhub that i just created?
[Note : I have followed - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create to create an evenhub on azure]
Just in case if i need to clarify on what subscription I'm talking about, below is the screenshot
How do i achieve this?
Thank you in advance
Maney
So i figured out a way to solve my problem (stated above). After going through Microsoft documentation and some trial and error methods, Here's how i solved it;
I have SUBSCRIPTION-1 and SUBSCRIPTION-2. I have created an eventhub in SUBSCRIPTION-2.
I go to SUBSCRIPTION-1 one and create a Resource-Group. After creating an Resource-Group, I create a EVENT-GRID. Within the eventgrid, I create a EVENT-SUBSCRIPTION that givens an option to point it to an endpoint. I chose the endpoint and selected the eventhub that was created in SUBSCRIPTION-1.
Now, i able to stream all the events from SUBSCRIPTION-1 to SUBSCRIPTION-2.
-Maney

azure service bus message Id us email ID

I am creating a batch job to read email from an inbox using Microsoft Graph API and publish the content of the email using service bus.
I put a duplicate detection in my azure service bus based on the assigned Message ID. To prevent publishing the same email, I put the Email message Id to the Azure Service Bus Message ID.
However it seems that the length of the email message ID is too long and I am unable to fit it as the azure service bus message ID. Any solution on how to put the email message ID as the Azure Service Bus message ID? (I am not allowed to store the email message ID in any persistence storage, I can only publish it via Azure Service Bus)
Thanks
The length of the service bus message ID can't exceed 128 characters. Generally, the value of
email ids that you retrieved from Microsoft Graph API look like:
AAMkADU1NzNkY2I2LWIxY2QtNGFhOC1iYmE5LWEzODA2ZjJkMjQxNwBGAAAAAABRRlt89urpT62rDl3o5R1RBwCSQ84srYEXTY2slhDtcrhrAAAAAAEMAACSQ84srYEXTY2slhDtcrhrAABlQp2FAAA=
Generally, its length is 188 characters which caused this issue.
To solve this issue, you can try to use MD5 to create a unified identifier for the email ID, by default, its length will be 32 characters, though this value has the possibility to collision with a hash that from a different string, but its possibility is low(about 2^-128).
If you are using C#, just try the code below:
using Azure.Messaging.ServiceBus;
using System;
using System.Security.Cryptography;
using System.Text;
namespace serviceBusTest
{
class Program
{
static void Main(string[] args)
{
string connectionString = "";
string queueName = "";
var client = new ServiceBusClient(connectionString);
ServiceBusSender sender = client.CreateSender(queueName);
ServiceBusMessage message = new ServiceBusMessage();
var emailID = "AAMkADU1NzNkY2I2LWIxY2QtNGFhOC1iYmE5LWEzODA2ZjJkMjQxNwBGAAAAAABRRlt89urpT62rDl3o5R1RBwCSQ84srYEXTY2slhDtcrhrAAAAAAEMAACSQ84srYEXTY2slhDtcrhrAABlQp2FAAA=";
message.MessageId = GetMd5Hash(MD5.Create(), emailID);
sender.SendMessageAsync(message).GetAwaiter().GetResult();
ServiceBusReceiver receiver = client.CreateReceiver(queueName);
ServiceBusReceivedMessage receivedMessage = receiver.ReceiveMessageAsync().GetAwaiter().GetResult();
Console.WriteLine(receivedMessage.MessageId);
}
static string GetMd5Hash(MD5 md5Hash, string input)
{
byte[] data = md5Hash.ComputeHash(Encoding.UTF8.GetBytes(input));
StringBuilder sBuilder = new StringBuilder();
for (int i = 0; i < data.Length; i++)
{
sBuilder.Append(data[i].ToString("x2"));
}
return sBuilder.ToString();
}
}
}
Result:

Azure service bus - how to add metadata to the message

I am using .net core web app as the publisher and .net core console app as subscriber.
I am able to successfully pass messages between these two systems using Managed Identities - set up in Azure portal.
My question is I need to add metadata to the the message that is being sent. How do I do that ?
Below is my publisher code :
string data = JsonConvert.SerializeObject(payloadEvents);
Message message = new Message(Encoding.UTF8.GetBytes(data));
var tokenProvider = TokenProvider.CreateManagedIdentityTokenProvider();
TopicClient sendClient = new TopicClient(_serviceBusNamespace, _topicName, tokenProvider, retryPolicy: null);
await sendClient.SendAsync(message);
Message object has a property called UserProperties that can be used to set custom metadata for that message.
Something like:
string data = JsonConvert.SerializeObject(payloadEvents);
Message message = new Message(Encoding.UTF8.GetBytes(data));
message.UserProperties.Add("key1", "value1");
message.UserProperties.Add("key2", "value2");
var tokenProvider = TokenProvider.CreateManagedIdentityTokenProvider();
TopicClient sendClient = new TopicClient(_serviceBusNamespace, _topicName, tokenProvider, retryPolicy: null);
await sendClient.SendAsync(message);

Azure Servicebus, MassTransit and DLQ's. Moving from DLQ to original queue

It really annoys me that we're unable to move messages from a Dead Letter Queue over to the Original Queue for processing when using Azure Servicebus. So, I figured out that I will try to implement this feature myself. We are using Masstransit to publish events. The queuename in ASB will be an events full assembly name.
I've created an REST Endpoint in my application to move messages from the DLQ to the original queue for reprocessing. This is where I'm stuck at the moment.
To get all messages in a DLQ, the user gives me the queuename, and I will format it to contain the DeadLetterQueue. Like this:
myproject.events.usercreatedevent -> myproject.events.usercreatedevent/$DeadLetterQueue
I get all the messages from this queue by using classes from the Nuget package Microsoft.Azure.Servicebus
public async Task RequeueMessagesAsync(string queueName)
{
var msg = new MessageReceiver(BuildConnectionString(), queueName);
var messages = await msg.PeekAsync(50);
foreach (var message in messages)
{
var content = Encoding.UTF8.GetString(message.Body);
var jsonObject = JsonConvert.DeserializeObject<JObject>(content);
var destinationAddress = jsonObject["destinationAddress"].ToString();
var messageContent = jsonObject["message"].ToString();
var messageType = destinationAddress.Split("/").Last();
await _bus.SendAsync(jsonObject, messageType);
}
}
The when calling _bus.SendAsync(object, address) the message ends up in a _skipped queue. I think the reason for this is that the messageHeaders is set to JObject, and not the actual message type. I cannot use reflection to recreated the event either, as we have a lot of microservices and source code of the event it not necessarily available. The code behind the _bus.SendAsync(object, address) looks like this:
public async Task SendAsync(object message, string queueName, CancellationToken cancellationToken = default)
{
ISendEndpoint sender = await GetSenderAsync(queueName);
sender.ConnectSendObserver(new ErrorQueueConfiguration(_addressProvider.GetAddress("error")));
await sender.Send(message, cancellationToken);
}
Can I trick Masstransit to forward this "unknown" type to my Consumer by changing the MessageHeaders somehow? Have anyone successfully moved messages from a DLQ to its original queue?

Azure Service Bus - Message Peek on Subscription

I have a listen access on Topic/Subscription for Azure Service Bus. Is there a way to verify if a subscription has a message and how many messages are there. I don't have manage connection string and I think I cannot run Service Bus Explorer. I don't intent to read the message out of the subscription.
The count of the messages in a Topic Subscription can be retrieved only using Manage Connection string. With the Listen Connection string, you cannot read the properties or count of messages.
But, you can peek the messages in the Subscription using Listen Connection string. Peeking the messages will not lock or remove the messages from the Subscription. You can peek the messages as many times as you need.
Thus, with the help of Listen Connection string, you can find the number of messages in a Subscription by peeking the messages one by one inside a while loop and add a counter inside the loop. The counter value is the number of messages inside the Subscription.
I don't intent to read the message out of the subscription.
The short answer is no. Manage access is required to get the topic or subscription message information.
If you want to get the subscription message count, you could have a try following 2 ways:
1.You have manage access, you could use the following demo code to do that.
var topicName = "topic1";
var subscriptionName = "subscription";
var address = "sb://xxx.servicebus.windows.net/"; //base address of namespace you are connecting to.
MessagingFactorySettings MsgFactorySettings = new MessagingFactorySettings
{
TokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider("SharedAccessKeyName", "xxxxxx"),
};
MessagingFactory msgFactory = MessagingFactory.Create(address, MsgFactorySettings);
NamespaceManager manager = new NamespaceManager(msgFactory.Address,msgFactory.GetSettings().TokenProvider);
var subscriptioncDescription = manager.GetSubscription(topicName, subscriptionName);
var count = subscriptioncDescription.MessageCountDetails.ActiveMessageCount;
2.Use Azure Microsoft.Azure.Management.ServiceBus.Fluent sdk. You need to registry Azure AD application and assign corresponding role it. For more information please refer to another SO thread.
var subscriptionId = "Azure subscriptionId";
var resourceGroupName = "rgName";
var nameSpace = "tomtestsb";
var topicName = "topic1";
var subscriptionName = "subscription";
var credentials = SdkContext.AzureCredentialsFactory.FromFile(#"D:\Tom\Documents\azureCred.txt");
var client = new ServiceBusManagementClient(credentials);
client.SubscriptionId = subscriptionId;
var topic = client.Topics.GetAsync(resourceGroupName, nameSpace, topicName).Result;
var topicMessagecount = topic.CountDetails.ActiveMessageCount;
var subscription = client.Subscriptions.GetAsync(resourceGroupName, nameSpace, topicName, subscriptionName).Result;
var subscriptionMessagecount = subscription.CountDetails.ActiveMessageCount;

Resources