in my case AMOP protocol is blocked by firewall. I just can work with Azure eventhub with https protocol.
I just find:
.NET EventHubConnectionOptions.TransportType property with EventHubsTransportType.AmqpTcp or EventHubsTransportType.AmqpWebSockets
I do not know, how to change the protocol in receiver of my application.
this is my sample code:
static async Task Main()
{
// Read from the default consumer group: $Default
string consumerGroup = EventHubConsumerClient.DefaultConsumerGroupName;
// Create a blob container client that the event processor will use
storageClient = new BlobContainerClient(blobStorageConnectionString, blobContainerName);
// Create an event processor client to process events in the event hub
processor = new EventProcessorClient(storageClient, consumerGroup, ehubNamespaceConnectionString, eventHubName);
// Register handlers for processing events and handling errors
processor.ProcessEventAsync += ProcessEventHandler;
processor.ProcessErrorAsync += ProcessErrorHandler;
// Start the processing
await processor.StartProcessingAsync();
// Wait for 30 seconds for the events to be processed
await Task.Delay(TimeSpan.FromSeconds(30));
// Stop the processing
await processor.StopProcessingAsync();
}
static async Task ProcessEventHandler(ProcessEventArgs eventArgs)
{
// Write the body of the event to the console window
Console.WriteLine("\tReceived event: {0}", Encoding.UTF8.GetString(eventArgs.Data.Body.ToArray()));
// Update checkpoint in the blob storage so that the app receives only new events the next time it's run
await eventArgs.UpdateCheckpointAsync(eventArgs.CancellationToken);
}
It's a bit convoluted process.
You would want to use EventProcessorClient(BlobContainerClient, String, String, EventProcessorClientOptions) constructor and then specify ConnectionOptions in your client options.
There's a property called TransportType in ConnectionOptions which is of type EventHubsTransportType.
You can specify TransportType as EventHubsTransportType.AmqpWebSockets and then you should be able to use AMQP over WebSockets.
Related
with servicebus_client:
receiver = servicebus_client.get_subscription_receiver(topic_name=TOPIC_NAME, subscription_name=SUBSCRIPTION_NAME, max_wait_time=5)
with receiver:
for msg in receiver:
print("Received: " + str(msg))
receiver.complete_message(msg)
This code is taken from the Azure service bus code snippet in Microsoft portal. The code works fine till the it sends the messages, but when it reaches this part of the code, it is not getting in the for loop. The print statement is not getting execute. But I can see the Incoming and outcoming messages on the particular metrics page of that topic. Can I get some help here as I am new to Azure Service Bus?
Code for receive message from topic. its working as expected. you can try again with all library reference and follow all steps.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Azure.Messaging.ServiceBus;
namespace SubscriptionReceiver
{
internal class Program
{
// connection string to your Service Bus namespace
static string connectionString = "Endpoint=sb://xxx/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxx";
// name of the Service Bus topic
static string topicName = "mytopic";
// name of the subscription to the topic
static string subscriptionName = "S1";
// the client that owns the connection and can be used to create senders and receivers
static ServiceBusClient client;
// the processor that reads and processes messages from the subscription
static ServiceBusProcessor processor;
static async Task Main()
{
// The Service Bus client types are safe to cache and use as a singleton for the lifetime
// of the application, which is best practice when messages are being published or read
// regularly.
//
// Create the clients that we'll use for sending and processing messages.
client = new ServiceBusClient(connectionString);
// create a processor that we can use to process the messages
processor = client.CreateProcessor(topicName, subscriptionName, new ServiceBusProcessorOptions());
try
{
// add handler to process messages
processor.ProcessMessageAsync += MessageHandler;
// add handler to process any errors
processor.ProcessErrorAsync += ErrorHandler;
// start processing
await processor.StartProcessingAsync();
Console.WriteLine("Wait for a minute and then press any key to end the processing");
Console.ReadKey();
// stop processing
Console.WriteLine("\nStopping the receiver...");
await processor.StopProcessingAsync();
Console.WriteLine("Stopped receiving messages");
}
finally
{
// Calling DisposeAsync on client types is required to ensure that network
// resources and other unmanaged objects are properly cleaned up.
await processor.DisposeAsync();
await client.DisposeAsync();
}
}
// handle received messages
static async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
Console.WriteLine($"Received: {body} from subscription: {subscriptionName}");
// complete the message. messages is deleted from the subscription.
await args.CompleteMessageAsync(args.Message);
}
// handle any errors when receiving messages
static Task ErrorHandler(ProcessErrorEventArgs args)
{
Console.WriteLine(args.Exception.ToString());
return Task.CompletedTask;
}
}
}
Output :- Receive Message from Topic Subscription
I am trying a sample code of Azure Event Hub Producer and trying to send some message to Azure Event Hub.
The eventhub and its policy is correctly configured for sending and listening messages. I am using Dotnet core 3.1 console application. However, the code doesn't move beyond CreateBatchAsync() call. I tried debugging and the breakpoint doesn't go to next line. Tried Try-catch-finally and still no progress. Please guide what I am doing wrong here. The Event hub on Azure is shows some number of successful incoming requests.
class Program
{
private const string connectionString = "<event_hub_connection_string>";
private const string eventHubName = "<event_hub_name>";
static async Task Main()
{
// Create a producer client that you can use to send events to an event hub
await using (var producerClient = new EventHubProducerClient(connectionString, eventHubName))
{
// Create a batch of events
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();
// Add events to the batch. An event is a represented by a collection of bytes and metadata.
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("First event")));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Second event")));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Third event")));
// Use the producer client to send the batch of events to the event hub
await producerClient.SendAsync(eventBatch);
Console.WriteLine("A batch of 3 events has been published.");
}
}
}
The call to CreateBatchAsync would be the first need to create a connection to Event Hubs. This indicates that you're likely experiencing a connectivity or authorization issue.
In the default configuration you're using, the default network timeout is 60 seconds and up to 3 retries are possible, with some back-off between them.
Because of this, a failure to connect or authorize may take up to roughly 5 minutes before it manifests. That said, the majority of connection errors are not eligible for retries, so the failure would normally surface after roughly 1 minute.
To aid in your debugging, I'd suggest tweaking the default retry policy to speed things up and surface an exception more quickly so that you have the information needed to troubleshoot and make adjustments. The options to do so are discussed in this sample and would look something like:
var connectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";
var options = new EventHubProducerClientOptions
{
RetryOptions = new EventHubsRetryOptions
{
// Allow the network operation only 15 seconds to complete.
TryTimeout = TimeSpan.FromSeconds(15),
// Turn off retries
MaximumRetries = 0,
Mode = EventHubsRetryMode.Fixed,
Delay = TimeSpan.FromMilliseconds(10),
MaximumDelay = TimeSpan.FromSeconds(1)
}
};
await using var producer = new EventHubProducerClient(
connectionString,
eventHubName,
options);
I'm testing retry options for Azure Service Bus publisher/subscriber client because after a sudden connection failure the client will not retry to send or receive messages.
Following is the code for publisher client sendMessage() method and I have set maximum delivery count to 1000 for the subscription. Still the client uses default retryPolicy values and I cannot see it retries as I have given in amqpRetryOptions.
static void sendMessage() {
// create Retry Options for the Service Bus client
AmqpRetryOptions amqpRetryOptions = new AmqpRetryOptions();
amqpRetryOptions.setDelay(Duration.ofSeconds(1));
amqpRetryOptions.setMaxRetries(120);
amqpRetryOptions.setMaxDelay(Duration.ofMinutes(5));
amqpRetryOptions.setMode(AmqpRetryMode.EXPONENTIAL);
amqpRetryOptions.setTryTimeout(Duration.ofSeconds(5));
// create a Service Bus Sender client for the queue
ServiceBusSenderClient senderClient = new ServiceBusClientBuilder()
.connectionString(connectionString)
.retryOptions(amqpRetryOptions)
.sender()
.topicName(topicName)
.buildClient();
// send one message to the topic
senderClient.sendMessage(new ServiceBusMessage("Hello, World! "));
System.out.println("Sent a single message to the topic");
}
Is my approach wrong?
If so, what is the standard way?
If not how can approach retry mechanism?
If not how to
I was able to get retrying mechanism work using ServiceBusSenderAsyncClient. Also, I could catch exceptions to check whether the cause is transient or not.
static void sendMessage() {
// create Retry Options for the Service Bus client
AmqpRetryOptions amqpRetryOptions = new AmqpRetryOptions();
amqpRetryOptions.setDelay(Duration.ofSeconds(1));
amqpRetryOptions.setMaxRetries(5);
amqpRetryOptions.setMaxDelay(Duration.ofSeconds(15));
amqpRetryOptions.setMode(AmqpRetryMode.EXPONENTIAL);
amqpRetryOptions.setTryTimeout(Duration.ofSeconds(5));
// instantiate a client that will be used to call the service
ServiceBusSenderAsyncClient serviceBusSenderAsyncClient = new ServiceBusClientBuilder()
.connectionString(connectionString)
.retryOptions(amqpRetryOptions)
.sender()
.topicName(topicName)
.buildAsyncClient();
// create a message
ServiceBusMessage serviceBusMessage = new ServiceBusMessage("Hello, World!\n")
// send the message to the topic
serviceBusSenderAsyncClient.sendMessage(serviceBusMessage).subscribe(
unused -> System.out.println("Message sent successfully"),
error -> {
ServiceBusException serviceBusException = (ServiceBusException) error;
System.out.println(serviceBusException.isTransient());
},
() -> {
System.out.println("Message sent successfully");
}
);
}
I hope someone can clarify this for me:
I have 2 consumers in the same ConsumerGroup, it is my understanding that they should coordinate between them, but I am having the issue that both consumers are getting all the messages. My code is pretty simple:
const connectionString =...";
const eventHubName = "my-hub-dev";
const consumerGroup = "processor";
async function main() {
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log(`Received event...`, event)
}
},
}
);
If I run two instances of this consumer code and publish an event, both instances will receive the event.
So my questions are:
Am I correct in my understanding that only 1 consumer should receive the message?
Is there anything I am missing here?
The EventHubConsumerClient requires a CheckpointStore that facilitates coordination between multiple clients. You can pass this to the EventHubConsumerClient constructor when you instantiate it.
The #azure/eventhubs-checkpointstore-blob uses Azure Storage Blob to store the metadata and required to coordinate multiple consumers using the same consumer group. It also stores checkpoint data: you can call context.updateCheckpoint with an event and if you stop and start a new receiver, it will continue from the last checkpointed event in the partition that event was associated with.
There's a full sample using the #azure/eventhubs-checkpointstore-blob here: https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/eventhub/eventhubs-checkpointstore-blob/samples/javascript/receiveEventsUsingCheckpointStore.js
Clarification: The Event Hubs service doesn't enforce a single owner for a partition when reading from a consumer group unless the client has specified an ownerLevel. The highest ownerLevel "wins". You can set this in the options bag you pass to subscribe, but if you want the CheckpointStore to handle coordination for you it's best not to set it.
I have created an Event Hub Namespace and 2 event hubs. I defined a Shared Access Policy (SAP) on the Event Hub Namespace. However, when I use the connection string defined on the namespace, I am able to send events to only one of the hubs even though I create the client using the correct event hub name
function void SendEvent(connectionString, eventHubName){
await using(var producerClient = new EventHubProducerClient(connectionString, eventHubName)) {
// Create a batch of events
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();
var payload = GetEventModel(entity, entityName);
// Add events to the batch. An event is a represented by a collection of bytes and metadata.
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes(payload.ToString())));
// Use the producer client to send the batch of events to the event hub
await producerClient.SendAsync(eventBatch);
System.Diagnostics.Debug.WriteLine($"Event for {entity} sent to Hub {eventHubName}");
}
}
The above code is called for sending events to Hub1 and Hub2. When I use the connection string from the SAP defined on the Namespace, I can only send events to Hub1 or Hub2 whichever happens to be called first. I am specifying the eventHubName as Hub1 or Hub2 as appropriate.
I call the function SendEvent in my calling code.
The only way I can send to both hubs is to define SAP on each hub and use that connection string when creating the EventHubProducer
Am I missing something or is this by design?
I did a quick test at my side, and it can work well at my side.
Please try the code below, and let me know if it does not meet your need:
class Program
{
//the namespace level sas
private const string connectionString = "Endpoint=sb://yyeventhubns.servicebus.windows.net/;SharedAccessKeyName=mysas;SharedAccessKey=xxxx";
//I try to send data to the following 2 eventhub instances.
private const string hub1 = "yyeventhub1";
private const string hub2 = "yyeventhub2";
static async Task Main()
{
SendEvent(connectionString, hub1);
SendEvent(connectionString, hub2);
Console.WriteLine("**completed**");
Console.ReadLine();
}
private static async void SendEvent(string connectionString, string eventHubName)
{
// Create a producer client that you can use to send events to an event hub
await using (var producerClient = new EventHubProducerClient(connectionString, eventHubName))
{
// Create a batch of events
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();
// Add events to the batch. An event is a represented by a collection of bytes and metadata.
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("First event: "+eventHubName)));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Second event: "+eventHubName)));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Third event: "+eventHubName)));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Fourth event: " + eventHubName)));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Fifth event: " + eventHubName)));
// Use the producer client to send the batch of events to the event hub
await producerClient.SendAsync(eventBatch);
Console.WriteLine("A batch of 3 events has been published to: "+ eventHubName);
}
}
}
After running the code, I can see the data are sent to both of the 2 eventhub instances. Here is the screenshot: