Bot deployed on azure suddenly became non-responsive - azure

Question

I have a bot deployed on Azure using Bot Framework. Nothing was changed on the code. But today the bot became completely unresponsive. Any message I try to send results invariably in a "Couldn't Send. Retry?" message. Occasionally in past I could see the bot was slower to reply to one or other message, but this time is a different thing, the bot is completely mute. On inspecting the issues for web chat channel on azure dashboard I can see that the errors are all the same "There was an error sending this message to your bot: HTTP status code GatewayTimeout". What may I be doing wrong?
Just in case it is relevant, the bot uses LUIS services and also a database both deployed in azure. Tried to access these services separately and they are working fine with good response time. Anyway, I don't think the bot program reaches the point of trying to communicate with any of them. I doesn't even reach the first IDialogContext.PostAsync() which is the very first instruction on the StartAsync() method in the root dialog. Help with this much appreciated

Answer 1

the errors are all the same "There was an error sending this message to your bot: HTTP status code GatewayTimeout". What may I be doing wrong?
Firstly, I do a test with the following sample, and if the request take a long time to get response, which might cause " Gateway Timeout" issue. You can turn on Application Insights with your bot application to trace the request(s) sent from your bot, and check if some requests take too long time.
Example:
private async Task MessageReceivedAsync(IDialogContext context, IAwaitable<object> result)
{
var activity = await result as Activity;
// calculate something for us to return
int length = (activity.Text ?? string.Empty).Length;
if (activity.Text.ToLower().Contains("timeout test"))
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://xxxxx/api/values/xxx");
request.Method = "GET";
request.ContentType = "application/json";
//I set a 30s delay for returning response in my external api
var response = request.GetResponse();
string content = string.Empty;
using (var stream = response.GetResponseStream())
{
using (var sr = new StreamReader(stream))
{
content = sr.ReadToEnd();
}
}
activity.Text = content.ToString();
// return our reply to the user
await context.PostAsync($"API returned {activity.Text}");
}
else
{
// return our reply to the user
await context.PostAsync($"You sent {activity.Text} which was {length} characters");
}
context.Wait(MessageReceivedAsync);
}
Test result:
Secondly, if your Bot Service pricing tier is free, please check if your bot service reached 10,000 messages/month limit (for Premium channels). In this SO thread, another community member reported reaching that limit would cause " Gateway Timeout" error.
Besides, if possible, you can create a new bot service on Azure portal and then publish your bot application to that corresponding new Azure web app that you specified as messaging endpoint, and check if your bot app can work as expected on new Azure environment.
Note:
I also checked the status history of Azure services and found:
6/27 RCA - App Service - West Europe
Summary of impact: Between 16:00 UTC on 27 Jun 2018 and 13:00 UTC on 28 Jun 2018, a subset of customers using App Service in West Europe may have received HTTP 500-level response codes, timeouts or high latency when accessing App Service (Web, Mobile and API Apps) deployments hosted in this region.
6/25 RCA - Multiple Services - South Central US
Summary of impact: Between 19:40 and 20:52 UTC on 25 Jun 2018, a subset of customers in South Central US may have experienced difficulties connecting to resources and/or 500-level errors hosted in this region. Virtual Machines may have rebooted unexpectedly. Impacted services included: Storage, Virtual Machines, Key Vault, Site Recovery, Machine Learning, Cloud Shell, Logic Apps, Redis Cache, Visual Studio Team Services, Service Bus, ExpressRoute, Application Insights, Backup, Networking, API Management, App Service (Linux) and App Service.
Not sure if above issue causes the problem, if you tried all approaches that you can do to troubleshoot the issue, but the issue with your bot service is still not mitigated, you can try to create support request to report it.

Answer 2

Try enabling [botauthentication] in message controller. Also try to open outgoing connection also from your IIS server to internet. By the way there can be some other problems as well like :
1)Your web.config file contains an App ID that does not match the one you posted.
2)Your server's time is incorrect. Tokens have 20 minutes of validity (5 minutes before the token was issued and 15 minutes after.) Is it possible your server time is different?
3)Your bot encountered a problem retrieving list of signing keys. Check to make sure you can access these URLs:
https://api.aps.skype.com/v1/keys
https://login.botframework.com/v1/.well-known/keys
You can also follow this issue to check if you're doing it correctly https://github.com/Microsoft/BotBuilder/issues/4389