Azure ServiceBus timeout when Sending Message from Azure Web App - azure

We are facing issue with web API which is hosted in Azure web app. We are inserting a message in Azure Service Bus . Getting intermittent Timeout issue when message is sent to Service Bus queue through Azure web app but when debugging the same code locally it works fine. The service bus is same in both cases. Not able to find the reason for same.Below is code snippet.
Logger.Log(ErrorLevel.Info, "Started Sending Message to Service Bus for Job Id :" + newjobObject.JobId);
ServiceBusRepository.SendMessageToQueue(jobEntity.JobId);
Logger.Log(ErrorLevel.Info, "Completed Sending Message to Service Bus for Job Id :" + newjobObject.JobId);
This is code for SendMessageToQueue method
public static void SendMessageToQueue(string Message)
{
try
{
var message = new BrokeredMessage(Message);
Logger.Log(ErrorLevel.Info,"Before SendMessagetoQueue= " + message);
sendQueueClient.Send(message);
Logger.Log(ErrorLevel.Info,"After SendMessagetoQueue= " + message);
}
catch(Exception ex)
{
Logger.Log(ex.Message.ToString());
}
}
In logs only first logging statement is getting logged not the second one. So it means while sending message some exception is occurring.Anyone can provide pointers to debug this issue.

Intermittent timeout exceptions (System.TimeoutException) are transient exceptions and are acceptable. According to documentation, you should add a retry logic fro any transient exception.
In addition to that, for timeout exceptions, you might want to adjust messaging factory settings (see documentation here).
When the client throws an exception, have a closer at that exception. It will contain info that could help. Would not hurt to post it along with the question.

Related

Is IoT hub message feedback absolute confirmation?

I'm working Azure Function that sends data to few devices via IoT hub. I'm trying to log whole process and I am unsure if my current solution is sufficient.
So far I'm using message feedback(as mentioned in documentation) to log if device received send message.
"The IoT hub doesn't generate a feedback message. If the cloud-to-device message reaches the Completed state, the IoT hub generates a feedback message." As I understand it if I receive said feedback it is confirmation that message was successfully/unsuccessfully received by device.
Is my understanding that this is absolute confirmation that message was or wasn't received by device correct? Or is there another option to get is confirmation?
I recommend reading through the section Receive Cloud to Device Delivery feedback for better understanding on this. The section explains how you can set the Acknowledgment feedback option. The Azure IoT Hub provides feedback in both Positive and Negative scenarios.
If you have set the message Ack to full as indicated in the article using the following code commandMessage.Ack = DeliveryAcknowledgement.Full;, you will receive a message in both Completed as well as Dead lettered scenarios (Positive and Negative outcome).
If you are specifically targeting the success messages, you would need to set the acknowledgement to Positive. The feedback you then receive is a confirmation proving that message was successfully received by device.
Hope this helps!
I followed the below steps to send a message to an IoT device using azure function.
Additional answer wrto #LeelaRajesh_Sayana.
Created a IoT hub and added the device
Add the device name and click on save
Select the device you created and click on send message
Enter the message to send
To create a particular condition, we have code it. I have used C# function time trigger to send message IoT Hub messages and added the below condition .
try
{
log.Loginformation($"Message sent : {message}");
}
catch (Exception ex)
{
log.LogError($"Error sending message :{ex.Message});
}
using System;
using Microsoft.Azure.Devices;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
namespace AzureFunctionIoT
{
public static class SendMessageToIoTDevices
{
[FunctionName("SendMessageToIoTDevices")]
public static void Run([TimerTrigger("0 0 0 * * *")]TimerInfo myTimer, ILogger log)
{
string connectionString = Environment.GetEnvironmentVariable("IoTHubConnectionString");
ServiceClient serviceClient = ServiceClient.CreateFromConnectionString(connectionString);
var message = new Microsoft.Azure.Devices.Message(System.Text.Encoding.ASCII.GetBytes("Hello IoT Devices"));
serviceClient.SendAsync("<DeviceId>", message).GetAwaiter().GetResult();
log.LogInformation("Message sent successfully");
}
}
}
Run the messages with Azure IoT Hub (.NET) Code

How to send messages to the dead letter queue in Azure Service Bus using MassTransit?

I'm new to Azure Service Bus and MassTransit. Now I'm experimenting with messages and I want to send a Message that goes into the Dead Letter Queue of the Azure Service Bus. What should such a message look like? Will it only appear in the Dead Letter Queue if there is no consumer? And how should I consume a Dead Letter Message by creating a dead letter consumer?
Thanks in advance :)
Consumer:
public Task Consume(ConsumeContext<IEventCreated> context)
{
try
{
var serializedMessage = JsonSerializer.Serialize(context.Message, new JsonSerializerOptions());
logger.LogInformation("Event consumed: {serializedMessage}", serializedMessage);
return Task.CompletedTask;
}
catch (Exception ex)
{
logger.LogError(ex, "Error while processing message {messageId}. Retry attempt {retryAttempt}", context.MessageId, context.GetRetryAttempt());
throw;
}
}
My consumer definition:
public class EventCreatedConsumerDefinition : ConsumerDefinition<EventCreatedConsumer>
{
protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator,IConsumerConfigurator<EventCreatedConsumer> consumerConfigurator)
{
endpointConfigurator.UseMessageRetry(r => r.Interval(5, 500));
endpointConfigurator.ConfigureDeadLetterQueueDeadLetterTransport();
endpointConfigurator.ConfigureDeadLetterQueueErrorTransport();
}
}
When configured to use the dead-letter queue (DLQ), MassTransit will move faulted or skipped messages to the DLQ instead of creating separate _error or _skipped queues.
If a message consumer throws an exception, and all retry/redelivery limits are reached, the message is dead-lettered using the Azure SDK along with additional message headers detailing the reason/exceptions that occurred.
As for how to deal with messages in the DLQ, you can use Azure Service Bus Explorer (in the Azure Portal) or the old client Windows application to examine/move the messages, or consider other options like a SaaS service that can monitor/move those messages for you.
Message Aid is an early stage tool that does this, author is the co-founder of MassTransit.

Send alert on failure of webjobs in azure portal

There is one requirement to send an email upon unsuccessful web-jobs only. There are 16 web-jobs which are deployed and running successfully on azure portal. I have suggested to modify the code for existing web-job but client does not want to modify web-jobs. He wants to add something extra which does not require to modify web-jobs anymore. I am confused, without modifying web-jobs, how can I send an email? I searched a lot on google and stack-overflow but didn't get anything.
How can I implement this?
Few of the workarounds for getting the notification to email for WebJobs Status:
Using ErroTrigger and SendGrid extensions. you can do the Notifications sending to email for the WebJob SDK.
check this article on how to set that up which uses both extensions to send an email if an error occurred 10 times in 30 minutes window with a throttle up to 1 hour.
public static void ErrorMonitor(
[ErrorTrigger("0:30:00", 10, Throttle = "1:00:00") TraceFilter filter,
[SendGrid] SendGridMessage message)
{
message.Subject = "WebJobs Error Alert";
message.Text = filter.GetDetailedMessage(5)
}
If you aren't using the WebJob SDK, then unfortunately there aren't any events for continuous webjobs. There is only one for triggered jobs.
Also, visit this MSFT Doc and SO Thread for information on setting up the email alerts with App Services.

Azure Notifcation Hub CreateOrUpdateInstallationAsync not failing but Installation not being created

I have an ASP.NET Web Api that is registering Android and iOS installations in our Azure Notification Hub. It creates a HubClient using a connection string
this.azureHub = NotificationHubClient.CreateClientFromConnectionString(
"Endpoint=sb://OUR-HUB-NS.servicebus.windows.net/;SharedAccessKeyName=DefaultFullSharedAccessSignature;SharedAccessKey=tK/SEXXXXXXXXXXX/7LUtvRoNt+HjToFmP+T++yW5g=",
"OUR-HUB");
We then create an installation using this code
try
{
await azureHub.CreateOrUpdateInstallationAsync(installation, token);
}
catch(Exception e)
{
this.logger.LogError(e, "CreateOrUpdateInstallationAsync failed with" + e.Message);
return false;
}
return true;
When I had the NoticationHubClient incorrectly configured I was getting 401 errors and if I pass in nonsense for the installation data I will get a 400. Currently the call does not throw an error so I asm going to assume what I am passing is correct and it can successfully connect to the hub. However registrations are not being created. If i try and get the installation after it is created with
GetInstallationAsync(installation.InstallationId)
I get an error saying the installation cannot be found and if I try and get all registrations i get an empty list
var registrations = await azureHub.GetAllRegistrationsAsync(0);
Further confirmation that my installations are not being created is that if I use the "Test Send" in Azure Portal I get "Message was successfully sent, but there were no matching targets." for either Apple or Android platform
What is happening to my registration?
I recreated the Notification Hub with a different name, in the same namespace and everything just started working.
I then recreated the non-working hub in Azure Portal and it now all works.
All I can say is that I had previously deleted the hub using the OSS Service Bus Explorer and perhaps that deleted the hub in a way that Azure didn't like.
Not the answer I was expecting.

Bot deployed on azure suddenly became non-responsive

I have a bot deployed on Azure using Bot Framework. Nothing was changed on the code. But today the bot became completely unresponsive. Any message I try to send results invariably in a "Couldn't Send. Retry?" message. Occasionally in past I could see the bot was slower to reply to one or other message, but this time is a different thing, the bot is completely mute. On inspecting the issues for web chat channel on azure dashboard I can see that the errors are all the same "There was an error sending this message to your bot: HTTP status code GatewayTimeout". What may I be doing wrong?
Just in case it is relevant, the bot uses LUIS services and also a database both deployed in azure. Tried to access these services separately and they are working fine with good response time. Anyway, I don't think the bot program reaches the point of trying to communicate with any of them. I doesn't even reach the first IDialogContext.PostAsync() which is the very first instruction on the StartAsync() method in the root dialog. Help with this much appreciated
the errors are all the same "There was an error sending this message to your bot: HTTP status code GatewayTimeout". What may I be doing wrong?
Firstly, I do a test with the following sample, and if the request take a long time to get response, which might cause " Gateway Timeout" issue. You can turn on Application Insights with your bot application to trace the request(s) sent from your bot, and check if some requests take too long time.
Example:
private async Task MessageReceivedAsync(IDialogContext context, IAwaitable<object> result)
{
var activity = await result as Activity;
// calculate something for us to return
int length = (activity.Text ?? string.Empty).Length;
if (activity.Text.ToLower().Contains("timeout test"))
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://xxxxx/api/values/xxx");
request.Method = "GET";
request.ContentType = "application/json";
//I set a 30s delay for returning response in my external api
var response = request.GetResponse();
string content = string.Empty;
using (var stream = response.GetResponseStream())
{
using (var sr = new StreamReader(stream))
{
content = sr.ReadToEnd();
}
}
activity.Text = content.ToString();
// return our reply to the user
await context.PostAsync($"API returned {activity.Text}");
}
else
{
// return our reply to the user
await context.PostAsync($"You sent {activity.Text} which was {length} characters");
}
context.Wait(MessageReceivedAsync);
}
Test result:
Secondly, if your Bot Service pricing tier is free, please check if your bot service reached 10,000 messages/month limit (for Premium channels). In this SO thread, another community member reported reaching that limit would cause " Gateway Timeout" error.
Besides, if possible, you can create a new bot service on Azure portal and then publish your bot application to that corresponding new Azure web app that you specified as messaging endpoint, and check if your bot app can work as expected on new Azure environment.
Note:
I also checked the status history of Azure services and found:
6/27 RCA - App Service - West Europe
Summary of impact: Between 16:00 UTC on 27 Jun 2018 and 13:00 UTC on 28 Jun 2018, a subset of customers using App Service in West Europe may have received HTTP 500-level response codes, timeouts or high latency when accessing App Service (Web, Mobile and API Apps) deployments hosted in this region.
6/25 RCA - Multiple Services - South Central US
Summary of impact: Between 19:40 and 20:52 UTC on 25 Jun 2018, a subset of customers in South Central US may have experienced difficulties connecting to resources and/or 500-level errors hosted in this region. Virtual Machines may have rebooted unexpectedly. Impacted services included: Storage, Virtual Machines, Key Vault, Site Recovery, Machine Learning, Cloud Shell, Logic Apps, Redis Cache, Visual Studio Team Services, Service Bus, ExpressRoute, Application Insights, Backup, Networking, API Management, App Service (Linux) and App Service.
Not sure if above issue causes the problem, if you tried all approaches that you can do to troubleshoot the issue, but the issue with your bot service is still not mitigated, you can try to create support request to report it.
Try enabling [botauthentication] in message controller. Also try to open outgoing connection also from your IIS server to internet. By the way there can be some other problems as well like :
1)Your web.config file contains an App ID that does not match the one you posted.
2)Your server's time is incorrect. Tokens have 20 minutes of validity (5 minutes before the token was issued and 15 minutes after.) Is it possible your server time is different?
3)Your bot encountered a problem retrieving list of signing keys. Check to make sure you can access these URLs:
https://api.aps.skype.com/v1/keys
https://login.botframework.com/v1/.well-known/keys
You can also follow this issue to check if you're doing it correctly https://github.com/Microsoft/BotBuilder/issues/4389

Resources