I've recently started working with the Azure-Iot-SDK and wonder how to investigate this behaviour.
My application is ok with sending messages to the IoT hub for some time, but suddenly just stops sending. Without an error or anything else that helps me find the rootcause for this. Just restarting the application (a service to be precise) is enough to get it working again.
Code is like this (on DataChangedEvent) :
try {
deviceClient = DeviceClient.Create(connectionString, x509Certificate, tType);
Log("Start sending message")
await deviceClient.SendEventAsync(message);
DoLogging("Done sending!");
}
catch (Exception e)
{
moreLogging("Error occurred");
}
Somehow the "Done sending!" message stops appearing in the logs, but the "Start sending message" keeps coming.
Does anyone have any suggestions how to proceed?
From SendEventAsync method implementation,
The async operation shall retry until time specified in
OperationTimeoutInMilliseconds property expire or unrecoverable
error(authentication or quota exceed) occurs.
So check the OperationTimeoutInMilliseconds value to see if there is an error occurs after reach the timeout.
Meanwhile, you can monitor the connection status by registering such handler:
deviceClient.SetConnectionStatusChangesHandler(ConnectionStatusChangeHandler);
The handler may look like this:
static void ConnectionStatusChangeHandler(ConnectionStatus status, ConnectionStatusChangeReason reason)
{
Console.WriteLine();
Console.WriteLine("Connection Status Changed to {0}", status);
Console.WriteLine("Connection Status Changed Reason is {0}", reason);
Console.WriteLine();
}
And you can try this way to see if it helps:
await deviceClient.SendEventAsync(eventMessage).ConfigureAwait(false);
If you haven't fixed the timeouts, here's a tip for you: try re-creating your IoT Hub in another region ("West US" works stably for me).
Related
The following is the code for sending batched messages and the code block following is the setting for retry policy while creating the service bus client.
While testing a scenario involving transient network fault, we see the log that the send task was faulted but the messages were seen in the service bus for the message-id that were assigned to the messages of the batch. Is it possible that the retry kept happening in the background even after entering the fault? I am of the understanding that the fault will happen only AFTER the retry procedures are completed by the SDK. Is that correct?
Secondary query - How do I log the internal retry attempts?
//Code for sending. Uses Azure.Messaging.ServiceBus
await tempSender.SendMessagesAsync(batch).ContinueWith(async (tStat) =>
{
if (tStat.IsFaulted || tStat.IsCanceled)
{
_logger.Log($"Service Bus Send Task Faulted for batch {batchId}", EventLevel.Error);
}
else
{
_logger.Log($"Sent batch of size {batch.Count}", EventLevel.LogAlways);
onCompletionCallBack(messages);
}
});
//Retry policy
protected ServiceBusRetryOptions GetRetryPolicy() => new ServiceBusRetryOptions()
{
Delay = TimeSpan.FromSeconds(0.1),
MaxDelay = TimeSpan.FromSeconds(30),
MaxRetries = 5,
Mode = ServiceBusRetryMode.Exponential,
};
Your code could be simplified to the following:
try
{
await tempSender.SendMessagesAsync(batch);
_logger.Log($"Sent batch of size {batch.Count}", EventLevel.LogAlways);
onCompletionCallBack(messages);
}
catch (Exception exception)
{
_logger.Log($"Service Bus Send Task Faulted for batch {batchId}", EventLevel.Error);
}
Service Bus SDK will not throw an exception until all retries are exhausted. There's a chance that the messages have been dispatched, but the acknowledgement hasn't been received, which will explain what you're observing.
Secondary query - How do I log the internal retry attempts?
Transient failures and retry attempts are not directly surfaced to your application. To observe them, you would need to enable SDK logging.
What is the correct way to continuously receive messages from the Azure Service Bus? I do the following by polling the queue infinitely.
const receiver = queueClient.createReceiver(ReceiveMode.peekLock);
while (true) {
const maxMessageCount = 1;
try {
if (!receiver.isReceivingMessages()) {
const message = await receiver.receiveMessages(maxMessageCount);
if (message.length > 0) {
/**
* Note: we are always receiving one message per call
*/
processMessage(message[0]);
}
}
} catch (err) {
processError(err);
}
}
But I am hitting a very weird situation. After some time, half of the messages get lost and only half are received by the above receiver. I wonder what is happening. Is there some bug in this code? I usually start the nodejs script and stop it using ctrl+c. Is there a possibility of some deamon listner running in the background with the above logic even when I have stopped the nodejs script?
Note: I always start a single nodejs script.
Messages are never lost with Azure Service Bus. As your code is using PeekLock mode, most likely the messages get retried and eventually end up in the dead-letter queue. Check the dead letter queue to see if they are there. Alternatively, verify that time to live (TTL) is not set to something short and causes messages to be purged.
i am encountering a problem i'm not familiar with.
So i'm trying to log exceptions from a test Azure Function but when i throw an exception and return a 400 bad request, application insight registers the log as a successful request.
As i understand it it is probably register the function's successful run but what i don't understand is how i then should log the exception.
So what i have done so far is this.
(I Will be referring to Application Insights as AI from here on out)
I started by created an AI-Resource.
Then i took the instrument key and applied it to the app settings of my function.
After that i installed the AI NUGET to my function bu creating a Projet.json file and then pasting something like this which installed the necessary assemblies and such.
{
"frameworks": {
"net46":{
"dependencies": {
"Microsoft.ApplicationInsights": "2.4.0"
}
}
}
}
after this i initialize the TelemetryClient in the function and try to log and exception in a catch:
Initiation:
string key = TelemetryConfiguration.Active.InstrumentationKey = System.Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY", EnvironmentVariableTarget.Process);
TelemetryClient telemetry = new TelemetryClient() {
InstrumentationKey = key
};
Catch:
catch (Exception e)
{
Dictionary<string,string> properties = new Dictionary<string,string>();
properties.Add("Function Payload", data.ToString());
properties.Add("Function Exception", e.ToString());
telemetry.TrackException(e, properties);
return req.CreateResponse(HttpStatusCode.BadRequest, e);
}
Test running the function i get:
2018-03-07T14:24:36.171 [Info] Function started (Id=0292b455-314d-4c4c-872a-2b8137a72305)
2018-03-07T14:24:37.092 [Info] Function completed (Success, Id=0292b455-314d-4c4c-872a-2b8137a72305, Duration=931ms)
In Application insights i can can only see bad requests for StatusCode: 500
but 400 Bad requests gets logged as Successful requests.
And also the TrackException functionality doesn't log any of the custom properties...
So what am i missing?
MORE DETAILS ABOUT EXCEPTION LOGGING:
#Mikhail is right that we treat this as a success because the function is a success. We didn't want to use status codes to guess whether there was a successful operation or not, so we look for whether the function threw an exception.
Your exception isn't appearing in that screen because it hasn't been property correlated with this function execution. If you go to App Insights Analytics and query for that ExceptionTelemetry, you should see it.
In order to correlate it with a specific function, you'd need to set the OperationId, which is the same as the function's InvocationId. There is a sample that shows how to do this with Events, Metrics, and Dependencies, but doing it for Exceptions is the same (you can ignore the User.Id assignment): https://learn.microsoft.com/en-us/azure/azure-functions/functions-monitoring#custom-telemetry-in-c-functions
Update: From what you've shown of your function above, you may be able to get away with doing something like:
catch (Exception e)
{
log.Error("Function Payload " + data.ToString());
throw;
}
That would return a 500 (rather than a 400), and Functions would log the trace to Application Insights, and then log the exception as well as the Request failure. If you're not using your TelemetryClient anywhere else, you could remove that from your code.
The server (Azure Function) processed the request without errors, you returned a result from it, so from the point of view of Function App runtime the request was processed successfully. You can see that from the log too:
... Function completed (Success, ...
So, it makes sense that the Function App registers the call as success in Application Insights too. At least, that's how they chose to implement it.
the "for this operation" not showing exceptions implies that the exception that you sent does not have the same operationId as the azure function. operation
id is how application insights "links" related telemetry together.
your "exeption logging" screenshot is not an exception, but a request, so the custom properties logged on your exception won't be there.
if you want your azure function to fail, and show as a failed request, and log an exception, why are you catching the exception and logging it yourself? doesn't catching the exception then cause the azure function to succeed? why not just let the exception trickle out and let the function runtime do that part for you? (doesn't it?)
I have a Xamarin app using the Microsoft.Azure.Devices.Client.DeviceClient to listen for messages from the Azure IoT hub. Occasionally I get errors thrown that I cannot catch, and deal with. One such exception is:
Microsoft.Azure.Devices.Client.Exceptions.UnauthorizedException
This exception is just an example of an exception I am getting, and one that I can reliably recreate by changing system time. Other IoTHubExceptions are sporadic and difficult to recreate and hence the need to try and catch the exception.
Can anyone tell me how I can catch these IoT hub based errors? The code is:
try
{
await _deviceClientInbound.OpenAsync();
// execution never gets passed this line and the exception is eventually thrown, having reached this point
Message receivedMessage = await _deviceClientInbound.ReceiveAsync();
if (receivedMessage == null) continue;
await _deviceClientInbound.CompleteAsync(receivedMessage);
}
catch (Exception e)
{
// Exception is never caught in here ...
// How can I catch the IoTHubException based exceptions
}
If you are getting unauthorized exception . Then please check your iot hub connection string used in device client to confirm that the shared access polity used with connection string have all the privileges or not, that is needed at your end.
I believe this has to do with the various refactorings done for the supported platforms and you will likely have to catch all exceptions, process the ones you want, and rethrow ones you don't. Depending on platform you might be able to include:
using Microsoft.Azure.Devices.Client.Exceptions;
The sources are here if you feel like spelunking:
https://github.com/Azure/azure-iot-sdk-csharp/search?utf8=%E2%9C%93&q=using+Microsoft.Azure.Devices.Client.Exceptions
I am subscribing to a channel in Pusher on my local machine using the Javascript SDK, and I don't get any error.
However, when I publish an event to that channel it is not received by the subscriber.
I've looked at Pusher's debug console and saw that the message is indeed sent but the subscription never occurs, as the connection is somehow interrupted, apparently prior to the subscription request (i.e I get a disconnection message, as shown in the console screenshot below).
the code is pretty boilerplate:
var pusher = new Pusher('PUSHER_KEY');
channel = pusher.subscribe('game' + game.gameId);
channel.bind('statusChange', function(game) {
console.log("GOT PUSHER - STATUS " + game.status);
$scope.game.status = game.status;
});
Examining the channel.subscribed property shows that the subscription failed as it equals false. I am at the sandbox plan (max 20 connections) and am only using 2 connections.
What can disrupt the connection?
The channel object:
Console screenshot:
I don't know what's the issue exactly but enabling the logs on the client side might help your find it:
Pusher.log = function(message) {
if (window.console && window.console.log) {
window.console.log(message);
}
};
There's some resources on the website to debug that kind of problem too: http://pusher.com/docs/debugging