What is the correct way to add a correlation-id to azure events ?
Right now, I send the events as follows:
const { EventHubProducerClient } = require('#azure/event-hubs');
const producer = new EventHubProducerClient(connectionString, eventHubName);
const batch = await producer.createBatch();
batch.tryAdd({
body: {
foo: "bar"
}
});
await producer.sendBatch(batch);
Of course as a workaround I could just add my own field to the body. However, I suspect that there is a built-in mechanism or default approach to do this.
The latest release exposes a correlationId property on EventData, which corresponds to the correlation-id field of the message properties section of the underlying AMQP message.
One important call-out is that the correlationId is intended to enable tracing of data within an application, such as an event's path from producer to consumer. It has no meaning to the Event Hubs service or within a distributed tracing/AppInsights/OpenTelemetry context.
Related
I am trying a sample code of Azure Event Hub Producer and trying to send some message to Azure Event Hub.
The eventhub and its policy is correctly configured for sending and listening messages. I am using Dotnet core 3.1 console application. However, the code doesn't move beyond CreateBatchAsync() call. I tried debugging and the breakpoint doesn't go to next line. Tried Try-catch-finally and still no progress. Please guide what I am doing wrong here. The Event hub on Azure is shows some number of successful incoming requests.
class Program
{
private const string connectionString = "<event_hub_connection_string>";
private const string eventHubName = "<event_hub_name>";
static async Task Main()
{
// Create a producer client that you can use to send events to an event hub
await using (var producerClient = new EventHubProducerClient(connectionString, eventHubName))
{
// Create a batch of events
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();
// Add events to the batch. An event is a represented by a collection of bytes and metadata.
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("First event")));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Second event")));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Third event")));
// Use the producer client to send the batch of events to the event hub
await producerClient.SendAsync(eventBatch);
Console.WriteLine("A batch of 3 events has been published.");
}
}
}
The call to CreateBatchAsync would be the first need to create a connection to Event Hubs. This indicates that you're likely experiencing a connectivity or authorization issue.
In the default configuration you're using, the default network timeout is 60 seconds and up to 3 retries are possible, with some back-off between them.
Because of this, a failure to connect or authorize may take up to roughly 5 minutes before it manifests. That said, the majority of connection errors are not eligible for retries, so the failure would normally surface after roughly 1 minute.
To aid in your debugging, I'd suggest tweaking the default retry policy to speed things up and surface an exception more quickly so that you have the information needed to troubleshoot and make adjustments. The options to do so are discussed in this sample and would look something like:
var connectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";
var options = new EventHubProducerClientOptions
{
RetryOptions = new EventHubsRetryOptions
{
// Allow the network operation only 15 seconds to complete.
TryTimeout = TimeSpan.FromSeconds(15),
// Turn off retries
MaximumRetries = 0,
Mode = EventHubsRetryMode.Fixed,
Delay = TimeSpan.FromMilliseconds(10),
MaximumDelay = TimeSpan.FromSeconds(1)
}
};
await using var producer = new EventHubProducerClient(
connectionString,
eventHubName,
options);
I have an event hub trigger that I've configured to listen to an event hub for message. On the sending side javascript script I'm initializing a client as follows
const client = new EventHubProducerClient(hubConnectionString, hubName);
and initializing a batch as follows:
const batchOptions = {
partitionKey: sessionId,
};
const batch = await client.createBatch(batchOptions)
and submitting the batch using the same options:
await client.sendBatch(batch, batchOptions);
In my host.json I've conifgured the eventHub maxBatchSize as follows:
"eventHub": {
"maxBatchSize": 1
},
I'm finding that I can't get the messages in order in the EventHubTrigger implemented in Python. While processing each of the List[func.EventHubEvent] and while logging the partition key as follows:
def main(events: List[func.EventHubEvent]):
for event in events:
logging.info(f"PartitionId: ${event.partition_key}")
I always seem to get PartitionId: $None which seems to indicate that the partition is not being set and potentially why it is out of order.
It looks like I wasn't passing the value down to the method so once I fixed that it appeared to be working.
I hope someone can clarify this for me:
I have 2 consumers in the same ConsumerGroup, it is my understanding that they should coordinate between them, but I am having the issue that both consumers are getting all the messages. My code is pretty simple:
const connectionString =...";
const eventHubName = "my-hub-dev";
const consumerGroup = "processor";
async function main() {
const consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
const subscription = consumerClient.subscribe({
processEvents: async (events, context) => {
for (const event of events) {
console.log(`Received event...`, event)
}
},
}
);
If I run two instances of this consumer code and publish an event, both instances will receive the event.
So my questions are:
Am I correct in my understanding that only 1 consumer should receive the message?
Is there anything I am missing here?
The EventHubConsumerClient requires a CheckpointStore that facilitates coordination between multiple clients. You can pass this to the EventHubConsumerClient constructor when you instantiate it.
The #azure/eventhubs-checkpointstore-blob uses Azure Storage Blob to store the metadata and required to coordinate multiple consumers using the same consumer group. It also stores checkpoint data: you can call context.updateCheckpoint with an event and if you stop and start a new receiver, it will continue from the last checkpointed event in the partition that event was associated with.
There's a full sample using the #azure/eventhubs-checkpointstore-blob here: https://github.com/Azure/azure-sdk-for-js/blob/master/sdk/eventhub/eventhubs-checkpointstore-blob/samples/javascript/receiveEventsUsingCheckpointStore.js
Clarification: The Event Hubs service doesn't enforce a single owner for a partition when reading from a consumer group unless the client has specified an ownerLevel. The highest ownerLevel "wins". You can set this in the options bag you pass to subscribe, but if you want the CheckpointStore to handle coordination for you it's best not to set it.
Given this snippet of code:
//DirectApi
mqServer.RegisterHandler<LeadInformationInfo>(m =>
{
repository.SaveMessage(m as Message);
LeadInformationInfoResponse response = new LeadInformationInfoResponse();
try
{
var client = new JsonServiceClient(settingsFactory.GetMasterSetting("ProcessorApi:baseUri"));
response = client.Post(m.GetBody());
}
catch (WebServiceException webServiceException)
{
_log.Error("RegisterHandler<LeadInformationInfo>", webServiceException);
response = ((LeadInformationInfoResponse) webServiceException.ResponseDto);
response.CorrelationId = m.Id;
}
// Log response message here
return response;
}, 1);
I've gone to great lengths to make sure that a correlationId based off the original message Id property is propagated through the life of this message and any child messages spawned from this action. How do I get a handle on the response message so that I may log it in the handler? I only have access to the ResponseDto and not the message.
One of the reasons for this request is that the message queue client does not have access to the database, only the process that has the handler registered does. Hope that explains the situation better.
Just to clarify, this question is about persisting a MQ Response Message in the handler, the correlation Id is something that all messages in 1 request/response workflow will share. I'm also using ServiceStack ORMlite to persist the Message object, so querying this table by ID for troubleshooting is paramount.
Thank you,
Stephen
You're calling a WebService from within your MQ Handler:
var client = new JsonServiceClient(...);
response = client.Post(m.GetBody());
So there is no MQ Response which is only available in MQ Services. Although the WebService will return the response for the request that's sent so you can either use the CorrelationId on the MQ Request, otherwise you can have your Response DTO implement an interface like IHasCorrelationId and get it that way, e.g:
var correlationResponse = response as IHasCorrelationId;
if (correlationResponse != null)
{
var correlationId = correlationResponse.CorrelationId;
}
Create your own Instance of Message
As the Message<T> class is just a POCO if you wanted to create your own you can intialize your own instance:
var mqResponse = new Message<Response>(response);
If you only had the runtime late-bound type info, you can create one with:
var mqResponse = MessageFactory.Create(response);
Use RabbitMQ Message Filters
If you just wanted to log incoming and outgoing messages you can use the RabbitMQ Message Filters, e.g:
var mqServer = new RabbitMqServer("localhost")
{
PublishMessageFilter = (queueName, properties, msg) => {
properties.AppId = "app:{0}".Fmt(queueName);
},
GetMessageFilter = (queueName, basicMsg) => {
var props = basicMsg.BasicProperties;
receivedMsgType = props.Type; //automatically added by RabbitMqProducer
receivedMsgApp = props.AppId;
}
};
Under normal circumstances messages with a response will be published to the response.inq, I understand that and it's a nifty way to notify other parties that "something" has happened. But, when using the RPC pattern, the response goes back to the temp queue and disappears. Is this correct? Is there a convenient way, short of publishing another message, to achieve this behavior, of auto-notification?
The Message Workflow docs in describes the normal message workflow for calling a Service via ServiceStack.RabbitMQ:
Request / Reply
The Request/Reply alters the default message flow by specifying its own ReplyTo address to change the queue where the response gets published, e.g:
const string replyToMq = mqClient.GetTempQueueName();
mqClient.Publish(new Message<Hello>(new Hello { Name = "World" }) {
ReplyTo = replyToMq
});
IMessage<HelloResponse> responseMsg = mqClient.Get<HelloResponse>(replyToMq);
mqClient.Ack(responseMsg);
responseMsg.GetBody().Result //= Hello, World!
When using the Request/Reply pattern no other message is published in any other RabbitMQ topic/queue, to alert other subscribers the client would need to republish the message.
RabbitMqServer callbacks
Another way to find out when a message has been published or received is to use the PublishMessageFilter and GetMessageFilter callbacks on the RabbitMqServer and Client which lets you inspect each message that they sent or received, e.g:
var mqServer = new RabbitMqServer("localhost")
{
PublishMessageFilter = (queueName, properties, msg) => {
//...
},
GetMessageFilter = (queueName, basicMsg) => {
//...
}
};