I have a Microsoft Azure Service Bus connection string that I want to validate if it is a valid one. I want to validate the topics and subscriptions as well if they exist. If the connection string, topics, and subscriptions are invalid then we can take appropriate actions.
The sample connection string can be like - TransportType=AmqpWebSockets;Endpoint=sb://myservicebus.servicebus.windows.net/;SharedAccessKeyName=sendkey;SharedAccessKey=MQHnx3voLhH/xVgoamX3KijzkZ0qb7U6oHTolj7LM9H=;
I tried ServiceBusClient but it is not having such support. Is there a way we can validate the connection string and topics/subscriptions?
The client will validate that the connection string and entity names are well-formed, but there is no dedicated operation for testing its ability to communicate with Service Bus.
Connections and links are established lazily when the first network operation that requires them is invoked. There are two approaches that occur to me which would trigger network resource creation without being intrusive.
Create a message batch
In order to ensure that the batch size does not exceed the limits for a given resource, the ServiceBusSender must query the queue/topic. In the case where the client is misconfigured, it will throw.
// In a real scenario, you would want to create these as
// singletons and reuse them for the lifetime of the application.
await using var client = new ServiceBusClient("<< Connection String>>");
await using var sender = client.CreateSender("<< Queue/Topic >>");
var valid = true;
try
{
using var batch = await sender.CreateMessageBatchAsync().ConfigureAwait(false);
}
catch (Exception ex)
when (ex is ServiceBusException
|| ex is IOException
|| ex is SocketException)
{
valid = false;
}
Peek a message
Peeking a message will not cause the message to be locked or otherwise impact receiving it.
await using var client = new ServiceBusClient("<< Connection String>>");
await using var receiver = client.CreateReceiver("<< Queue/Subscription >>");
var valid = true;
try
{
_ = await receiver.PeekMessageAsync().ConfigureAwait(false);
}
catch (Exception ex)
when (ex is ServiceBusException
|| ex is IOException
|| ex is SocketException)
{
valid = false;
}
Interactions with each queue, topic, and subscription make use of a dedicated AMQP link, so each would need to be tested individually if you'd like to ensure that the name matches a known Service Bus entity.
Related
I created a simple Azure Service bus (Queue) and a client that is sending message to service bus. Using below code to send message:
using Microsoft.Azure.ServiceBus;
using Microsoft.Extensions.Configuration;
public async Task SendMessageAsync<T>(T message, string queueName)
{
try
{
var queueClient = new QueueClient(_config.GetConnectionString("AzureServiceBus"), queueName);
string messageBody = JsonSerializer.Serialize(message);
var byteMessage = new Message(Encoding.UTF8.GetBytes(messageBody));
queueClient.SendAsync(byteMessage);
Console.WriteLine((message as Employee).FirstName);
}
catch (Exception ex)
{
var c = ex;
}
}
Sending message using:
using SenderApp;
Console.WriteLine("Hello, World!");
QueueService service = new QueueService();
for (int i = 0; i < 100; i++)
{
Employee e = new Employee();
e.FirstName = "1 " + i.ToString();
e.LastName = "2 " + i.ToString();
service.SendMessageAsync<Employee>(e, "employeequeue");
}
When I try to see active messages, There is nothing in the queue:
However I do see some traffic. But the number of message I sent (over 100) is not equal to number of incoming request show (62) at the bottom of the image. I am not sure what is happening to my messages? This defeats the purpose of the queue.
Please guide me why I am not seeing any messages. What is the best way to handle this ?
I am using following nuget packages:
<PackageReference Include="Microsoft.Azure.ServiceBus" Version="5.2.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.1" />
A message sent to an Azure Service Bus queue will be delivered to the queue unless operation is failing. In that case, an exception will be thrown. Check the following:
Exception handling doesn't swollow exceptions
Await asynchronous send operations to ensure messages are dispatched
Namespace/queue used for sending is what you use to receive
There are no competing consumers, actively receiving messages from the queue.
Validate TCP ports needed for AMQP are not blocked. If those ports are blocked, you could configure your client to use WebSockets.
So I still dont know what caused this issue. But I realized Microsoft.Azure.ServiceBus package was deprecated and later I started using Azure.Messaging.ServiceBus package to connect to service bus and things started to work.
I used following code to send message to queue:
string connectionString = "Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=f3f+qMYTyVwE18YNl5J6ygJFi30v6J/Smph5HZvyQyE=";
string queueName = "employeequeue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create the sender
ServiceBusSender sender = client.CreateSender(queueName);
// create a message that we can send. UTF-8 encoding is used when providing a string.
ServiceBusMessage message = new ServiceBusMessage("Hello world! " + id);
// send the message
await sender.SendMessageAsync(message);
return "Sent";
Used following code to receive message:
string queueName = "employeequeue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create a receiver that we can use to receive and settle the message
ServiceBusReceiver receiver = client.CreateReceiver(queueName);
// the received message is a different type as it contains some service set properties
ServiceBusReceivedMessage receivedMessage = await receiver.ReceiveMessageAsync();
string body = receivedMessage.Body.ToString();
// complete the message, thereby deleting it from the service
await receiver.CompleteMessageAsync(receivedMessage);
More info is available # https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.7.0/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
I want to stream the events from different subscription to a single eventhub on azure.
At present I have configured eventhub to a single subscription and events are being streamed. I have a java client which consumes these events and stores it on my persistence layer. My java client looks like..
private void processUsingProcessorClient(){
List<Disposable> subscriptions = null;
try {
EventHubConsumerAsyncClient eventHubConsumerAsyncClient = new EventHubClientBuilder()
.consumerGroup(EventHubClientBuilder.DEFAULT_CONSUMER_GROUP_NAME)
.connectionString(CONNECTION_STRING, EVENT_HUB_NAME)
.credential("*******.servicebus.windows.net","maney-event-hub",createClientSecretCredential())
.buildAsyncConsumerClient();
ReceiveOptions receiveOptions = new ReceiveOptions().setTrackLastEnqueuedEventProperties(true);
List<String> block = eventHubConsumerAsyncClient.getPartitionIds().collectList().block();
Iterator<String> iterator = block.stream().iterator();
String partitionID = null;
subscriptions = new ArrayList<>(block.size());
while(iterator.hasNext()){
partitionID = iterator.next();
Disposable subscription = eventHubConsumerAsyncClient.receiveFromPartition(
partitionID,
EventPosition.fromOffset(0),receiveOptions).subscribe(PARTITION_PROCESSOR,ERROR_HANDLER);
subscriptions.add(subscription);
}
System.in.read();
}catch (Exception ex){
ex.printStackTrace();
} finally {
if(subscriptions != null){
subscriptions.forEach( subscrip -> {
subscrip.dispose();
});
}
}
}
private final Consumer<PartitionEvent> PARTITION_PROCESSOR = partitionEvent -> {
EventData event = partitionEvent.getData();
PartitionContext partitionContext = partitionEvent.getPartitionContext();
String contents = new String(event.getBody(), UTF_8);
LastEnqueuedEventProperties properties = partitionEvent.getLastEnqueuedEventProperties();
System.out.printf("Information received at %s. Last enqueued sequence number: %s%n",properties.getRetrievalTime(), properties.getSequenceNumber());
System.out.printf("Partition[%s] with Offset-[%s] and Sequence Number[%s] has contents: '%s'%n",
partitionContext.getPartitionId(),
event.getOffset(),
event.getSequenceNumber(),
contents);
};
private final Consumer<Throwable> ERROR_HANDLER = errorContext -> {
System.out.printf("Error occurred in partition processor");
errorContext.printStackTrace();
};
public ClientSecretCredential createClientSecretCredential() {
ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()
.clientId("****************")
.clientSecret("******************")
.tenantId("**********************")
.build();
return clientSecretCredential;
}
I'm able to read all the events from a single subscription. However I need to do data analytics on these events from different subscriptions too. How do I configre Azure Eventhub to listen to multiple subscriptions ?
I read on Stackoverflow suggestions about creating consumer groups to solve this issue, however I'm not able to figure out how? I did create the consumer group, but how do i connect the newly created consumer group to different subscriptions in my azure aaccount and get the events streamed to the eventhub that i just created?
[Note : I have followed - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-create to create an evenhub on azure]
Just in case if i need to clarify on what subscription I'm talking about, below is the screenshot
How do i achieve this?
Thank you in advance
Maney
So i figured out a way to solve my problem (stated above). After going through Microsoft documentation and some trial and error methods, Here's how i solved it;
I have SUBSCRIPTION-1 and SUBSCRIPTION-2. I have created an eventhub in SUBSCRIPTION-2.
I go to SUBSCRIPTION-1 one and create a Resource-Group. After creating an Resource-Group, I create a EVENT-GRID. Within the eventgrid, I create a EVENT-SUBSCRIPTION that givens an option to point it to an endpoint. I chose the endpoint and selected the eventhub that was created in SUBSCRIPTION-1.
Now, i able to stream all the events from SUBSCRIPTION-1 to SUBSCRIPTION-2.
-Maney
I have a scenario in which I am calling RegisterMessageHandler of SubscriptionClient class of Azure Service Bus library.
Basically I am using trigger based approach while receiving the messages from Service Bus in one of my services in Service Fabric Environment as a stateless service.
So I am not closing the subscriptionClient object immediately, rather I am keeping it open for the lifetime of the Service so that it keeps on receiving the message from azure service bus topics.
And when the service needs to shut down(due to some reasons), I want to handle the cancellation token being passed into the service of Service Fabric.
My question is how can I handle the cancellation token in the RegisterMessageHandler method which gets called whenever a new message is received?
Also I want to handle the closing of the Subscription client "Gracefully", i.e I want that if a message is already being processed, then I want that message to get processed completely and then I want to close the connection.
Below is the code I am using.
Currently We are following the below approach:
1. Locking the process of the message using semaphore lock and releasing the lock in finally block.
2. Calling the cancellationToken.Register method to handle cancellation token whenever cancellation is done. Releasing the lock in the Register Method.
public class AzureServiceBusReceiver
{
private SubscriptionClient subscriptionClient;
private static Semaphore semaphoreLock;
public AzureServiceBusReceiver(ServiceBusReceiverSettings settings)
{
semaphoreLock = new Semaphore(1, 1);
subscriptionClient = new SubscriptionClient(
settings.ConnectionString, settings.TopicName, settings.SubscriptionName, ReceiveMode.PeekLock);
}
public void Receive(
CancellationToken cancellationToken)
{
var options = new MessageHandlerOptions(e =>
{
return Task.CompletedTask;
})
{
AutoComplete = false,
};
subscriptionClient.RegisterMessageHandler(
async (message, token) =>
{
semaphoreLock.WaitOne();
if (subscriptionClient.IsClosedOrClosing)
return;
CancellationToken combinedToken = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, token).Token;
try
{
// message processing logic
}
catch (Exception ex)
{
await subscriptionClient.DeadLetterAsync(message.SystemProperties.LockToken);
}
finally
{
semaphoreLock.Release();
}
}, options);
cancellationToken.Register(() =>
{
semaphoreLock.WaitOne();
if (!subscriptionClient.IsClosedOrClosing)
subscriptionClient.CloseAsync().GetAwaiter().GetResult();
semaphoreLock.Release();
return;
});
}
}
Implement the message client as ICommunicationListener, so when the service is closed, you can block the call until message processing is complete.
Don't use a static Semaphore, so you can safely reuse the code within your projects.
Here is an example of how you can do this.
And here's the Nuget package created by that code.
And feel free to contribute!
I'm new to azure service bus and masstransit. I'm looking for a solution to a specific situation.
I have a azure service bus topic with multiple subscribers. Subscriber will receive message based on filters. I've created the topic and subscriber with code below
class Program
{
static void Main(string[] args)
{
string connectionString = "Endpoint connection string";
// the names of topics and subscriptions we'll be working with
const string topicName = "MyTestTopic";
const string allMessagesSubName = "AllMessages";
const string filteredSubName1 = "Filtered1";
const string filteredSubName2 = "Filtered2";
// let's create the topic if it doesn't already exist...
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
if (!namespaceManager.TopicExists(topicName))
{
var td = new TopicDescription(topicName);
namespaceManager.CreateTopic(td.Path);
}
if (!namespaceManager.SubscriptionExists(topicName, allMessagesSubName))
{
namespaceManager.CreateSubscription(topicName, allMessagesSubName);
}
if (!namespaceManager.SubscriptionExists(topicName, filteredSubName1))
{
namespaceManager.CreateSubscription(
new SubscriptionDescription(topicName, filteredSubName1),
new Microsoft.ServiceBus.Messaging.SqlFilter("From LIKE '%Smith'"));
}
if (!namespaceManager.SubscriptionExists(topicName, filteredSubName2))
{
namespaceManager.CreateSubscription(
new SubscriptionDescription(topicName, filteredSubName2),
new Microsoft.ServiceBus.Messaging.SqlFilter("sys.Label='important'"));
}
var message1 = new BrokeredMessage("Hello World");
var message2 = new BrokeredMessage("Second message");
message2.Label = "important";
var message3 = new BrokeredMessage("Third message");
message3.Properties["From"] = "Kelly Smith";
message3.Label = "information";
var client = TopicClient.CreateFromConnectionString(connectionString, topicName);
client.Send(message1);
client.Send(message2);
client.Send(message3);
client.Close();
}
}
Here in the code we're adding Message custom properties like From.
Now I want to send such message using masstransit. In masstransit I cannot find any option of adding Message custom properties using the Publish() method. Is there any way that I can send these messages using masstransit where these filters can be used?
NB: I've read the answer of this question But the anwer here tells us to filter the messages in subscriber side. What I want is that this filtering will occur before reaching the subscriber.
When using Azure Service Bus with MassTransit, you can add subscription endpoints in additional to regular endpoints. When configuring a subscription endpoint, you should be able to specify rules and/or filters as part of the subscription. Which is exactly what you're doing above, so that is handled.
The other part, adding properties to the message, can be done by adding text headers to the SendContext. Those headers are copied to the message Properties collection, which I believe can be used to filter messages using a "SQL" filter (which is configured on the subscription endpoint, or the topic subscription on a receive endpoint).
One can receive messages in azure service bus using either of the the two methods..
queueClient.BeginReceiveBatch OR messageReceiver.ReceiveBatchAsync
Is there any difference between these two methods speedwise or in any other way.
Thanks
If you don't need to the batch receive functionalilty, I prefer the method of wiring up a callback on the OnMessage event of the queue client. We have some fairly high throughput services relying on this pattern of message processing without any issues (1M+ messages/day)
I like that you end up with less, and simpler code, and can easily control the options of how many messages to process in parallel, which receive mode (peek and lock, vs receive and delete), etc
There's a sample of it in this documentation:
string connectionString =
CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
QueueClient Client =
QueueClient.CreateFromConnectionString(connectionString, "TestQueue");
// Configure the callback options
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = false;
options.AutoRenewTimeout = TimeSpan.FromMinutes(1);
// Callback to handle received messages
Client.OnMessage((message) =>
{
try
{
// Process message from queue
Console.WriteLine("Body: " + message.GetBody<string>());
Console.WriteLine("MessageID: " + message.MessageId);
Console.WriteLine("Test Property: " +
message.Properties["TestProperty"]);
// Remove message from queue
message.Complete();
}
catch (Exception)
{
// Indicates a problem, unlock message in queue
message.Abandon();
}
};