I've set up an EventHub and Logger in Azure. I configured the EventHub with the default "Log to EventHub" policy:
<log-to-eventhub logger-id="adc-test-logger2">
#( string.Join(",", DateTime.UtcNow, context.Deployment.ServiceName, context.RequestId, context.Request.IpAddress, context.Operation.Name) )
</log-to-eventhub>
The business is interested in logging the following events:
invalid subscription key
invalid endpoint
I hooked into IEventProcessor to capture the info captured by the event hub logger above. However, the info captured with this approach in its default implementation seems fairly rudimentary:
Message received. Partition: '1', Data: '2/1/2016 7:22:17 PM,myapimaangement
.azure-api.net,caf54d02-6890-4e26-b364-b38208d1b565,40.78.99.216,Retrieve resour
ce'
How can I configure EventHub logging or the downstream listener to capture and store information which is more descriptive/useful? What is the set of useful events you have captured using this approach?
The failure conditions you are looking for can be trapped by the on-error policy and are documented here
Related
by Httptrigger Azure function, if you send a POST request you receive something like this as a response:
{
"id": "66ee5d08196874aeb99c9e62ddc7b190",
"statusQueryGetUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190?taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"sendEventPostUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190/raiseEvent/{eventName}?taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"terminatePostUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190/terminate?reason={text}&taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"rewindPostUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190/rewind?reason={text}&taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg==",
"purgeHistoryDeleteUri": "https://asynchttpfunction.azurewebsites.net/runtime/webhooks/durabletask/instances/66ee5d08196945aeb44c9e62ddc7b190?taskHub=Orchestration&connection=Storage&code=FSVfJyGODSeKHPO0cM8Po9e1jMT7MghVMGuJqTaGTN56E1RUHnlVJg=="
}
The statusQueryGetUri provides information of the long running orchestration instance. If you follow this link you will receive a suitable runtimeStatus that describes the status of the orchestration instance along with some other useful information.here
My question is now:
actually we don't send a POST request to an Event grid Azure function trigger, Is there any way to get the status of the Azure function? Complete or is still running?
The Azure Event Grid is an eventing Pub/Sub model where the interest of source is distributed to the subscribed event handler endpoint or resource in the reliable manner with a retry policy and dead-lettering option. The AEG is waiting for delivery response processing max. 60 seconds.
There is no built-in the features what you are asking in the AEG, however you can use the REST API for metrics of the specific subscription to obtain its counters value:
MatchedEventCount,
DeliveryAttemptFailCount,
DeliverySuccessCount,
DroppedEventCount,
DeadLetteredCount
The following GET is an example for getting a subscription metrics:
https://management.azure.com/subscriptions/mysubId/resourceGroups/mygroup/providers/Microsoft.EventGrid/topics/mytester/providers/Microsoft.EventGrid/eventSubscriptions/mysubscription/providers/Microsoft.Insights/metrics?api-version=2018-01-01&interval=PT5M&metricnames=MatchedEventCount,DeliveryAttemptFailCount,DeliverySuccessCount,DroppedEventCount,DeadLetteredCount
Note, that the authorization header with a bearer token is required for this call.
More details about the monitoring an event message delivery can be found here.
I'm using the code to implement Azure service bus topics that can be found here:
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-how-to-use-topics-subscriptions
I've tried to run two instances of the subscriber program, which holds these methods:
private static void RegisterOnMessageHandlerAndReceiveMessages()
{
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
MaxConcurrentCalls = 1,
AutoComplete = false
};
_subscriptionClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
private static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
Console.WriteLine($"Received #{message.SystemProperties.SequenceNumber} message: {Encoding.UTF8.GetString(message.Body)}");
await _subscriptionClient.CompleteAsync(message.SystemProperties.LockToken);
}
However, this doesn't allow both subscribers to recieve the message, it is recieved one message each split across both subscribers.
How does I ensure both subscribers get all the messages coming from the bus?
Azure Service Bus is a broker with Competing Consumer pattern when it comes to retrieving messages. This is expected behaviour. What you have here is a scaled-out processing endpoint, where both are trying to process messages from the same subscription. The messages are distributed between these two competing consumers. If you need to distribute the same message to more than a single subscriber, you should have different subscription entities created and listened to.
It's worth stating - as it's not obvious from any of the documentation - that each topic should be setup with a subscription for each subscriber.
I.e. if I have 3 subscribers:
/myservicebus/mytopic/mysubscription1
/myservicebus/mytopic/mysubscription2
/myservicebus/mytopic/mysubscription3
I was expecting - probably same as OP - that topics and subscribers were automatically configured as one:many.
Essentially, if multiple subscribers are configured to the same subscription, then this becomes a 'first sub wins' scenario - the same as if it were a queue.
I also ran into this.
I resolved it by making an extra micro service just for forwarding the messages. The forwarder service itself must subscribe to the messages that each of your subscribers must receive. Each Subscriber registers itself (with a unique id) with the event forwarder service (by sending a message to it on startup). The subscribers must then use the message name + the unique id to subscribe to the messages. The message forwarder service will receive the messages and forward them to all the registered subscribers by changing the subject/label of the message (the unique id is added to the subject/label).
This workaround also has it's limitations, there can only be one instance of the message forwarder service or otherwise you have the same problem again. And the message forwarder service must also be running before the other subscribers otherwise it doesn't receive the registrations.
I'm testing out Azure Event Grid with a Azure Relay, Hybrid Connection handler. It's not working for me.
I can see messages being published to Azure Event Grid. So far so good.
I've set up a subscription to the Event Grid topic and I've configured it to send events to a handler specified as an Azure Relay with a Hybrid Connection endpoint.
Looking at the metrics for the subscription with the Hybrid Connection handler, I see the following telemetri:
Matched Events
Delivery Failed
Expired Events
… but I see no Delivery Succeeded event???
Also, the Hybrid Connection listener (just a simple Console App) connected to the Azure Relay receives nothing. I've tested the listener by sending some test messages directly to the Relay and that works fine.
The logical conclusion, is that the events published to the Event Grid are not being delivered probably to the Relay Hybrid Connection handler. But why? There are not that many parameters, so I not sure what I'm doing wrong. It seems rather straight forward to configure this.
I'm a the point where I'm beginning to believe that the Event Grid / Hybrid Connection scenario is currently not working. It is in preview after all, so that could explain it.
I know that there is not much to go on here, but I was hoping that others might have some experience with this?
The issue still remains. The issues seems to be related to the formating of the json being passed from the Event Grid Subscription to the Hybrid Connect.
Update
I finally had some time to look more carefully into this.
I set up the Event Grid Tester and every time the Event Grid recieves a message I can see this error in the log:
HybridConnection: Message processing failed - Unable to cast object of type 'Newtonsoft.Json.Linq.JValue' to type 'Newtonsoft.Json.Linq.JObject'.
I'm still not able to fix this, as I do not control the message. The message is generated by Azure Logic App and send to the Event Grid using the Event Grid connector. The Event Grid Connector in Azure Logic Apps is in preview, so that might explain the challenges I'm seeing.
I've successfully tested Event Grid subscriber locally using Azure Relay and documented it here. It is based on a sample Microsoft is providing and implemented using WCF variant of client, not the .NET Standard version.
you can simulate an Event Grid message to the HybridConnection url using a http POST request:
POST https://{myNamespace}.servicebus.windows.net/{myHybridConnectionName}
headers:
content-type: application/json
x-ms-version: 2015-07-08
Aeg-Event-Type: Notification
Authorization: SharedAccessSignature sr=xxx&sig=xxxx&se=11111111&skn=xxxxx
body:
{
"id": "123456",
"eventTime": "2018-07-22T13:09:07.5164877Z",
"eventType": "recordInserted",
"dataVersion": "1.0",
"metadataVersion": "1",
"subject": "/myapp/vehicles/motorcycles",
"data": {
"make": "Ducati",
"model": "Monster"
}
}
Note, that the sasToken for authorization header can be copied from the Azure Event Grid Tester log panel, when the HybridConnection has been opened.
the other option is to generate it by the following code:
using Microsoft.Azure.Relay;
// ...
var tokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider("hybridconnectionPolicyName", "hybridconnectionPrimaryKey");
var token = tokenProvider.GetTokenAsync("https://{myNamespace}.servicebus.windows.net/{myHybridConnectionName}", TimeSpan.FromDays(10)).Result.TokenString;
I use Azure Service Bus to communicate IoT Hub with Node.js Functions. On Function side, I need to have access to message body as well as custom properties.
By retrieving messages elsewhere, I have noticed that my message in Service Bus consists of:
Body (the string set on IoT Device side).
Custom properties (include properties set on IoT Device side, IoT Device ID, and some additional metadata).
I have found online that in C# it is possible to access these custom properties using BrokeredMessage object. However, there is no mention on how to make this happen in Node.js. To give some details, I am printing the message as soon as it arrives to Function, using the following code:
module.exports = function(context, message) {
context.log('Message', message, 'of type', (typeof message));
...
}
What I am getting in the log console is:
message { test: true } of type object
Where "{ test: true }" is the content of a message set by IoT Device. No trace of properties though...
Is there some formal way or at least a trick to receive and extract these properties?
Thank you!
Browsing resources mentioned in the link posted by Ling Toh in a comment I got inspired to have a look at the context object.
Apparently, all of the Service Bus custom properties are available in context.bindingData.properties object.
In my case:
properties:
{
type: 'sometype', // <- this is the property I have set manually in IoT Hub message
'iothub-connection-device-id': 'mydeviceid',
'iothub-connection-auth-method': '{"scope":"somescope","type":"sometype","issuer":"external","acceptingIpFilterRule":null}',
'iothub-connection-auth-generation-id': 'someid' // <- These are added by IoT Hub
}
I am looking for a way to see which device has sent which message in Azure.
Via "IoT-hub" we can get a list of the devices but I cannot seem to find a way to correlate the messages to the devices.
Doe anyone have any idea?
Thanks in advance.
Regards
Have a look at this document for more details about the message format.
The device id is a part of the IoT Hub message system properties such as ConnectionDeviceId.
The following example shows a query of the ASA job. You can see how to get the device id from the telemetry message via the stream pipeline:
WITH subquery as (
SELECT
System.Timestamp as time,
counter,
temperature,
humidity,
EventProcessedUtcTime,
IoTHub.ConnectionDeviceId as deviceId,
IoTHub.MessageId as messageId
FROM iot Timestamp by time
)
SELECT
*
INTO
outBlob
FROM
subquery
another example is for Azure EventHubTrigger Function (AF). The telemetry message from the stream pipeline (events endpoint) is serialized into the EventData object and pushed to the AF:
public static async Task Run(EventData ed, TraceWriter log)
{
log.Info($"Label = {ed.SystemProperties["iothub-message-source"]} -{ed.SystemProperties["iothub-connection-device-id"]}/{ed.SequenceNumber}");
// ...
}
You could try Azure IoT Toolkit extension for VS Code to monitor all the messages sent to Azure IoT Hub. You could see which device has sent what message to Azure IoT Hub.
You could also refer to this blog post for more details about how to use this extension to monitor messages.
When a message arrives in IoT Hub, the hub adds a number of system properties to the message, including the deviceid of the device that sent the message - for more information about message properties, see https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-construct
If you're using C# to read the messages, see the ConnectionDeviceId property of this class: https://learn.microsoft.com/en-us/dotnet/api/microsoft.azure.devices.messagesystempropertynames?view=azure-dotnet