After a deploy to our Azure Web App, we are getting 500 timeouts on any request to the service:
500 - The request timed out.
The web server failed to respond within the specified time.
This has come out of the blue and we cannot determine what's causing it. It seems to take around 230s consistently to time out.
I've enabled all the diagnostic logs in the portal:
But I honestly don't quite know what to look for in the logs. I've scoured through all the files in the following folders but nothing jumps out.
How can I troubleshoot this problem?
The trick to get debug messages is to set stdoutLogFile="D:\home\LogFiles\stdout.log" in your config, instead of the ..\logs path that you had. After changing that, you get an error file under D:\home\LogFiles. Here is the error you get:
Application startup exception: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentNullException: SMTP server password cannot be null or empty.
Parameter name: smtpPassword
at TransitApi.Infrastructure.Modules.Logging.EmailOutput.EmailLogger..ctor(String recipient, String sender, String smtpUsername, String smtpPassword, String smtpHost, Int32 smtpPort, String environmentName, LogLevel minimumLevel)
at TransitApi.Infrastructure.Modules.Logging.EmailOutput.EmailLoggerProvider.CreateLogger(String name)
at Microsoft.Extensions.Logging.Logger.AddProvider(ILoggerProvider provider)
at Microsoft.Extensions.Logging.LoggerFactory.AddProvider(ILoggerProvider provider)
at TransitApi.Api.Startup.Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
So some kind of issue setting up the mail server. That causes the process to crash, and that it behaves poorly.
But I highly suggest that you upgrade to Core RC2, as RC1 is quite obsolete.
Related
Trying to deploy ABP framework 6.0.0 AuthServer at Azure, after sucessful built and run locally.
Got "HTTP Error 500.30 - ASP.NET Core app failed to start"
When trying to get some logs, using Azure Debug Console at myproject.scm.azurewebsites.net/DebugConsole I got following:
C:\home\site\wwwroot>dotnet MyProject.AuthServer.dll
[19:52:57 INF] Starting MyProject.AuthServer.
[19:53:00 FTL] MyProject.AuthServer terminated unexpectedly!
Volo.Abp.AbpInitializationException: An error occurred during ConfigureServicesAsync phase of the module Volo.Abp.OpenIddict.AbpOpenIddictAspNetCoreModule, Volo.Abp.OpenIddict.AspNetCore, Version=6.0.0.0, Culture=neutral, PublicKeyToken=null. See the inner exception for details.
---> Internal.Cryptography.CryptoThrowHelper+WindowsCryptographicException: Access is denied.
at Internal.Cryptography.Pal.StorePal.FromSystemStore(String storeName, StoreLocation storeLocation, OpenFlags openFlags)
at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
at Microsoft.Extensions.DependencyInjection.OpenIddictServerBuilder.AddDevelopmentEncryptionCertificate(X500DistinguishedName subject)
at Microsoft.Extensions.DependencyInjection.OpenIddictServerBuilder.AddDevelopmentEncryptionCertificate()
at Volo.Abp.OpenIddict.AbpOpenIddictAspNetCoreModule.<>c__DisplayClass1_0.<AddOpenIddictServer>b__0(OpenIddictServerBuilder builder)
at Microsoft.Extensions.DependencyInjection.OpenIddictServerExtensions.AddServer(OpenIddictBuilder builder, Action`1 configuration)
at Volo.Abp.OpenIddict.AbpOpenIddictAspNetCoreModule.AddOpenIddictServer(IServiceCollection services)
at Volo.Abp.OpenIddict.AbpOpenIddictAspNetCoreModule.ConfigureServices(ServiceConfigurationContext context)
at Volo.Abp.Modularity.AbpModule.ConfigureServicesAsync(ServiceConfigurationContext context)
at Volo.Abp.AbpApplicationBase.ConfigureServicesAsync()
--- End of inner exception stack trace ---
at Volo.Abp.AbpApplicationBase.ConfigureServicesAsync()
at Volo.Abp.AbpApplicationFactory.CreateAsync[TStartupModule](IServiceCollection services, Action`1 optionsAction)
at Microsoft.Extensions.DependencyInjection.ServiceCollectionApplicationExtensions.AddApplicationAsync[TStartupModule](IServiceCollection services, Action`1 optionsAction)
at Microsoft.Extensions.DependencyInjection.WebApplicationBuilderExtensions.AddApplicationAsync[TStartupModule](WebApplicationBuilder builder, Action`1 optionsAction)
at MyProject.Program.Main(String[] args) in D:\a\1\s\aspnet-core\src\MyProject.AuthServer\Program.cs:line 35
This say nothing to me, and when Googling around, I hit this at abp.io website: https://support.abp.io/QA/Questions/3537/OpenIddict-WindowsCryptographicException-Access-is-denied
But I can´t figure out where I shall do the changes. It feels like this should be noted in the ABP.io docs?
Do you know what can cause this, or have some ideas what can be wrong?
I was having the same issue with having the same setup as you have with the exception of deploying to azure, but it should be the same solution.
Please check this SO question:
WindowsCryptographicException: Access is denied when publishing app in Azure
The problem with OpenIddict (or rather, the goodness we're not used to) is that it is bare bones and you have to implement everything yourself.
OpenIddict must have a certificate in order to function on the server you're deploying to, and to have better security on top of whatever measures you're taking.
Basically what you will do is create a new self-signed certificate, upload it to cert store, add the thumbprint to the code and add it to one WEBSITE_LOAD_CERTIFICATES environment variable on said server.
You can add this environment variable to the code instead of hard-coding the thumbprint.
Here is a not-so-informative part of the abp.io documentation about the issue at hand:
Abp.io doc
And here is what I did in the code:
I hope this helps.
Today I've adjusted the App Service Log settings for my Azure App Serivce: a .NET Core 2.1 Web API. Soon after I changed the Application Logging (file system) setting from Information to Warning, my application stopped responding. I immediately kicked of a .NET Trace using the Diagnose and solve problems feature.
This Trace report showed the following:
system.private.corelib!
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start
microsoft.aspnetcore.hosting.il!Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start
microsoft.aspnetcore.hosting.il!Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync
microsoft.aspnetcore.hosting!
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1+<>c[System.Threading.Tasks.VoidTaskResult,Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__188`1[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context]].<.cctor>b__9_0
system.private.corelib!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__188`1[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context]].MoveNext
system.private.corelib!
system.private.corelib!
system.private.corelib!
system.private.corelib!
system.private.corelib!
Same time, the Azure Blob log showed these lines:
2019-07-06 14:04:06.476 +00:00 [Error] Microsoft.AspNetCore.Server.Kestrel: Connection id "0HLO233NHLK6Q", Request id "0HLO233NHLK6Q:000003BF": An unhandled exception was thrown by the application.
System.OverflowException: Arithmetic operation resulted in an overflow.
at System.IO.StreamWriter.Write(String value)
at Microsoft.Extensions.RazorViews.BaseView.Write(String value)
at Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync()
at Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
2019-07-06 14:04:19.245 +00:00 [Error] Microsoft.AspNetCore.Server.Kestrel: Connection id "0HLO233NHLK6T", Request id "0HLO233NHLK6T:000000EF": An unhandled exception was thrown by the application.
System.ArgumentOutOfRangeException: Index and count must refer to a location within the buffer.
Parameter name: chars
at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Write(String value)
at Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync()
at Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
These stacktraces don't show any of my own code, they seem framework related.
Is there a message or a hint in these error stack traces that I can use to prevent these situations?
Many thanks in advance
I know it's too late but maybe my answer will help someone else.
I received the same exceptions as you. But it was not Azure App Service but IIS. The problem was in incorrect configuration somewhere in Startup.cs. You webhost tried to show error page but at the same time App Service (or IIS in my case) tried to shut down or restart it.
This thread helped me a lot https://github.com/aspnet/KestrelHttpServer/issues/1737
I created a basic web MVC app in vs 2017 and tried to publish it on Azure.
Tried to use existing app service and created a new as well, but both failed.
Received this error -
Severity Code Description Project File Line Suppression State
Error Web deployment task failed. ((20-Jun-19 12:40:32 PM) An error occurred when the request was processed on the remote computer.)
(20-Jun-19 12:40:32 PM) An error occurred when the request was processed on the remote computer.
There is an unclosed literal string. Line 1, position 70.
at System.Xml.XmlTextReaderImpl.Throw(Exception e)
at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
at System.Xml.XmlTextReaderImpl.ParseAttributeValueSlow(Int32 curPos, Char quoteChar, NodeData attr)
at System.Xml.XmlTextReaderImpl.ParseAttributes()
at System.Xml.XmlTextReaderImpl.ParseElement()
at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XPath.XPathDocument.LoadFromReader(XmlReader reader, XmlSpace space)
at System.Xml.XPath.XPathDocument..ctor(TextReader textReader)
at Microsoft.Web.Deployment.PackageDeserializer..ctor(Stream stream, DeploymentBaseContext baseContext)
at Microsoft.Web.Deployment.DeploymentAgent.HandleSync(DeploymentAgentAsyncData asyncData, Nullable`1 passId, String user, String siteName) WebApplicationForAzure 0
To start with, kindly check to see if the app works fine locally.
A deployment issue is an issue that causes the wrong set of files to get deployed to your site folder (typically site\wwwroot), or that causes some files not to get deployed at all.
A runtime issue happens when the files in your wwwroot folder are exactly what they should be, but for some reason the site doesn't run correctly.
Make sure site correctly deploys locally for isolating the issue. Are you deploying to staging slots?
Set WEBSITE_WEBDEPLOY_USE_SCM=false in the Application settings from the Azure Portal, re-download the publish profile and then re-try to see if that helps.
Navigate to your problematic app in the Azure Portal.
Go to Configuration settings blade.
Under Application Settings, set WEBSITE_WEBDEPLOY_USE_SCM = false
Ensure that the ‘false’ key is in the value field.
Reset and re-download the publish settings again in VS.
Also, isolate to see if it is due to network proxy or port on your system.
Since yesterday (2016-02-11 18:32 EST), we get this mysterious error in our Azure Webapps' "New Relic" monitoring logs. It seems to happens every 5 minutes We have five applications, all were deployed yesterday around that exact time, but the deployment process it the same as before...
The applications are all functional. I do not know where this error comes from, but the Stack trace mentions Kudu.Services.Web.Tracing.TraceModule.OnBeginRequest
Error message
System.UriFormatException: Invalid URI: The hostname could not be parsed.
Stack trace
at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
at System.Web.Util.UriUtil.BuildUriImpl(String scheme, String serverName, String port, String path, String queryString, Boolean useLegacyRequestUrlGeneration)
at System.Web.Util.UriUtil.BuildUri(String scheme, String serverName, String port, String path, String queryString)
at System.Web.HttpRequest.BuildUrl(Func`1 pathAccessor)
at System.Web.HttpRequest.get_Url()
at Kudu.Services.Web.Tracing.TraceModule.TryConvertSpecialHeadersToEnvironmentVariable(HttpRequestWrapper request)
at Kudu.Services.Web.Tracing.TraceModule.OnBeginRequest(Object sender, EventArgs e)
at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
Any ideas what is causing this? Is it a Kudu / Azure issue? Can I do anything about it?
There was indeed a new build of Kudu yesterday that caused this regression (build 51.50211.2078). When it was found, and additional build was deployed (build 51.50212.2079) to address it. However, it doesn't take effect until the site is restarted.
Please try restarting your site and confirm that the issue is gone. If you go to the Kudu home page, you can confirm the version before and after the restart.
I'm running into a problem with getting SSL to work in the Development Fabric. I'm running a clean install of Windows 8 Pro with Visual Studio 2012 Ultimate and the October 2012 Azure SDK for .NET. IIS8 is not installed, only IIS Express, which claims to support HTTPS so I'm hoping that's not the issue.
Running VS 12 as administrator, I've created a blank VS solution, added a new (.NET 4.5) cloud service with a new ASP.NET MVC 4 Internet web application project, and hit F5. Everything works fine. Then, when I add an SSL certificate to the web role and replace the HTTP endpoint (port 80) with an HTTPS endpoint (port 443, with the certificate), hitting F5 produces the following error message:
Windows Azure Tools for Microsoft Visual Studio
There was an error attaching the debugger to the role instance 'deployment18(32).WindowsAzureCloudService.Mvc4WebRole_IN_0' with Process Id: 4892'. Unable to attach. Access is denied.
Note, the last part ("Access is denied") comes in a few variations, a particularly pleasant one being "Catastrophic failure". :)
The only message in the VS Output window ('General' output) is:
Windows Azure Tools: Warning: Remapping private port 443 to 444 in role 'Mvc4WebRole' to avoid conflict during emulation.
The Compute Emulator UI is not much help; just before the instance disappears, this is the only console output that I get consistently (sometimes other messages appear, but sporadically every few runs; I'm not sure how to capture these):
[fabric] Role Instance: deployment18(33).WindowsAzureCloudService.Mvc4WebRole.0
[fabric] Role state Unknown
[fabric] Role state Suspended
[fabric] Role state Busy
[fabric] Role state Unhealthy
[fabric] Role state Stopped
The certificate was obtained from a CA and properly imported into the Local Machine/Personal/Certificates store as a .pfx with private key, extended properties, and marked as exportable, for what it's worth.
When I attempt to publish the service to Azure, I get one build (validation) warning about the database connection string (which I assume is irrelevant):
The connection string 'DefaultConnection' is using a local database '(LocalDb)\v11.0' in project 'Mvc4WebRole'. This connection string will not work when you run this application in Windows Azure. To access a different database, you should update the connection string in the web.config file.
Probably more important, the deployment actually fails with the following history in the Windows Azure Activity Log window:
9:00:25 AM - Warning: There are package validation warnings.
9:00:25 AM - Preparing deployment for WindowsAzureCloudService - 1/3/2013 8:59:55 AM with Subscription ID '<...>' using Service Management URL 'https://management.core.windows.net/'...
9:00:25 AM - Connecting...
9:00:26 AM - Object reference not set to an instance of an object.
9:00:26 AM - Deployment failed with a fatal error
Can someone help me troubleshoot this issue? I've rebooted a few times. ;)
Thanks in advance!
EDIT (Jan. 3, 4:44 PM): I have a few ideas that might help me make progress, but some are pretty drastic so any advice would be appreciated:
Is there a way to capture all the output from the Compute Emulator (Dev Fabric) to a log file so I can review it? (System.Diagnostic.Trace calls from my service won't help, since I don't even get as far as the RoleEntryPoint when using HTTPS!) I figured this out; see next edit.
That null pointer exception during the Azure deployment has me worried. Is it worthwhile to try reinstalling the Azure SDK, and if so, how should I go about doing a clean install of it?
Has anyone seen a problem of this sort disappear when switching to using full IIS for the emulator? (That seems unlikely since IIS vs. IIS Express should have no relevance to the Azure deployment.)
EDIT (Jan. 4, 10:15 AM): Bad news: I tried the suggestion to grant Read access to the certificates, but it didn't help in my case. Good news: I managed to capture one of those sporadic messages in the Compute Emulator UI before it shut down; it was a bit of info from some diagnostics. Not helpful in and of itself, but it revealed where the Development Fabric was storing its temporary files:
[Diagnostics] Information: C:\Users\Lars\AppData\Local\dftmp\Resources\0005155d-4592-40f4-812e-18793b26576c\directory\DiagnosticStore\Monitor
The GUID portion gets recreated for every deployment, and it is deleted when the deployment goes away (as it always does in my case). But in the parent directory ('dftmp'), there are a few helpful directories that I then monitored during a new deployment: DevFCLogs, DFAgentLogs, and IISConfiguratorLogs. I guess that answers the first question I had yesterday! :)
DFAgentLogs\DFAgent.log: (41KB) No useful information. A bunch of "Failure to read pipe" messages and failures to get the role/deployment instance ID, which I assume are just noise.
DevFCLogs\DevFabric--2013.01.04--<...>.log: (510 KB) No useful information. I skimmed the file and also searched for 'error', 'failure', 'not found', 'certificate', and 'Mvc4WebRole_IN_0'; none of those showed any hints of what was going on.
IISConfiguratorLogs\IISConfigurator.log: (6 KB) Now we're making progress!! :) Can someone tell me what this means? (In the meantime, I'm off ILSpy-hunting... fun fun...)
IISConfigurator Information: 0 : [00006356:00000005, 2013/01/04 16:07:08.915] Using IIS Express appdomain
(...)
IISConfigurator Information: 0 : [00006356:00000005, 2013/01/04 16:07:08.936] Adding binding 127.255.0.0:444: to site deployment18(40).WindowsAzureCloudService.Mvc4WebRole_IN_0_Web
IISConfigurator Information: 0 : [00006356:00000005, 2013/01/04 16:07:10.484] Caught exception
IISConfigurator Information: 0 : [00006356:00000005, 2013/01/04 16:07:10.487] Exception:System.Runtime.InteropServices.COMException (0x800401F3): Invalid class string (Exception from HRESULT: 0x800401F3 (CO_E_CLASSSTRING))
Server stack trace:
at Microsoft.Web.Administration.Interop.IAppHostProperty.get_Value()
at Microsoft.Web.Administration.ConfigurationElement.GetPropertyValue(IAppHostProperty property)
at Microsoft.Web.Administration.Binding.get_CertificateHash()
at Microsoft.Web.Administration.BindingCollection.Add(Binding binding)
at Microsoft.WindowsAzure.ServiceRuntime.IISConfigurator.WasManager.DeploySite(String roleId, WASite roleSite, String appPoolName, String sitePath, String iisLogsRootFolder, String failedRequestLogsRootFolder, List1 bindings, List1 protocols, FileManager fileManager, WAAppPool defaultAppPoolSettings, String roleGuid, String& appPoolSid, List`1 appPoolsAdded, String configPath)
EDIT (Jan. 4, 11 AM): ILSpy wasn't much help; the exception is being thrown at an interop point (we knew that already) while trying to get the hash of a certificate in order to set up the binding (we knew that too). Does anyone know what COM object would need to be registered in order to get a certificate hash for a binding in Microsoft.Web.Administration? Or how I could intercept the interop call to find out? Bonus points if you can tell me why this is happening in the first place. :)
I've had similar problem on two computers. On both cases installing IIS solved the problem.
It seems to be enough to just install the IIS (via add/remove Windows components). You don't need to start using it. The installation changes something and after that my IIS Express started working again with HTTPS from Visual Studio.
There is a discussion on similar issue on MSDN Social:
http://social.msdn.microsoft.com/Forums/nl-NL/windowsazuredevelopment/thread/ad362016-16f6-459a-8022-9307aa5f910e
And the issue has been also raised on Microsoft connect:
https://connect.microsoft.com/VisualStudio/feedback/details/758533
In my case the error in the log files was:
IISConfigurator Information: 0 : [00007644:00000007, 2013.01.17
00:39:18.523] Exception:System.Runtime.InteropServices.COMException
(0x800401F3): Invalid class string (Exception from HRESULT: 0x800401F3
(CO_E_CLASSSTRING))
I found the log files from C:\Users\\AppData\Local\dftmp\IISConfiguratorLogs directory.
When running locally with a private key cert for SSL, you'll need to give the user the emulator app is running under access to the private key. Open mmc.exe and add the Certificates >> Local Computer Snap-In to view your certificate. Right Click on the certificate, then All Tasks >> Manage Private Keys - then add IUSR and Network Service with at least read access.
For deployment to azure, you'll need to upload the certificate to the Cloud Service and make sure the certificate is valid for the domain.
Follow step 11 from http://www.microsoft.com/en-us/download/details.aspx?id=35448. From this SO post