Sporadically my WebApi application stops working until i do a app pool recycle.
The site gets stuck with this YSOD error:
Detailed Error Information
Module: IIS Web Core
Notification: BeginRequest
Handler: Not yet determined
Error Code: 0x8007000d
Config Error: Configuration file is not well-formed XML
Config File: \\?\{PathToMyVirtualApplicationWebsiteHere}\web.config
Physical Path: {PathToMyVirtualApplicationWebsiteHere}
The oddest issue here is that the YSOD Config Source is reported as blank:
Config Source
-1:
0:
IF the XML web.config was malformed, normally you see the bad line of xml here.
Seemingly Root Cause - I have IIS to recycle at a set time every day, and it seems to go bad:
An application has reported as being unhealthy. The worker process will now request a recycle. Reason given: An error message detailing the cause of this specific request failure can be found in the application event log of the web server. Please review this log entry to discover what caused this error to occur. The data is the error.
A ton more of "the data is the error" events doing recycle at same second as above
One second later. Message: Exception of type 'System.OutOfMemoryException' was thrown.StackTrace: at System.Web.Hosting.HostingEnvironment.StopRegisteredObjects(Boolean immediate)
at System.Web.Hosting.HostingEnvironment.InitiateShutdownWorkItemCallback(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
~ 1 minute later Windows Eror Reporting kicks in ... Faulting application name: w3wp.exe
Then all is quiet in event logs
What i have verified:
ProcMon shows no access to {PathToMyVirtualApplicationWebsiteHere}\web.config
App Pool Recycle will fix the problem (stop/starts immediately), until next time
Update:
I am using Application Initialization 1.0 for IIS 7.5, in conjunction with Url Rewrite Module ... on a different server where this was not installed properly we have the EXACT same error (but without the out of memory problems)
Related
Today I've adjusted the App Service Log settings for my Azure App Serivce: a .NET Core 2.1 Web API. Soon after I changed the Application Logging (file system) setting from Information to Warning, my application stopped responding. I immediately kicked of a .NET Trace using the Diagnose and solve problems feature.
This Trace report showed the following:
system.private.corelib!
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start
microsoft.aspnetcore.hosting.il!Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start
microsoft.aspnetcore.hosting.il!Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync
microsoft.aspnetcore.hosting!
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1+<>c[System.Threading.Tasks.VoidTaskResult,Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__188`1[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context]].<.cctor>b__9_0
system.private.corelib!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__188`1[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context]].MoveNext
system.private.corelib!
system.private.corelib!
system.private.corelib!
system.private.corelib!
system.private.corelib!
Same time, the Azure Blob log showed these lines:
2019-07-06 14:04:06.476 +00:00 [Error] Microsoft.AspNetCore.Server.Kestrel: Connection id "0HLO233NHLK6Q", Request id "0HLO233NHLK6Q:000003BF": An unhandled exception was thrown by the application.
System.OverflowException: Arithmetic operation resulted in an overflow.
at System.IO.StreamWriter.Write(String value)
at Microsoft.Extensions.RazorViews.BaseView.Write(String value)
at Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync()
at Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
2019-07-06 14:04:19.245 +00:00 [Error] Microsoft.AspNetCore.Server.Kestrel: Connection id "0HLO233NHLK6T", Request id "0HLO233NHLK6T:000000EF": An unhandled exception was thrown by the application.
System.ArgumentOutOfRangeException: Index and count must refer to a location within the buffer.
Parameter name: chars
at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Write(String value)
at Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync()
at Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
These stacktraces don't show any of my own code, they seem framework related.
Is there a message or a hint in these error stack traces that I can use to prevent these situations?
Many thanks in advance
I know it's too late but maybe my answer will help someone else.
I received the same exceptions as you. But it was not Azure App Service but IIS. The problem was in incorrect configuration somewhere in Startup.cs. You webhost tried to show error page but at the same time App Service (or IIS in my case) tried to shut down or restart it.
This thread helped me a lot https://github.com/aspnet/KestrelHttpServer/issues/1737
Given:
ASP.Net Core MVC application, deployed in docker containers on linux.
Nginx reverse proxy + load balancer
When the ASP.Net core container gets stuck (after a timeout on nginx), it calls a different instance of the application. At that point, the application should cancel the processing in order to not perform data manipulation operations twice. This works great with a global:
public override async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
{
using (context.HttpContext.RequestAborted.Register(() => context.HttpContext.RequestAborted.ThrowIfCancellationRequested()))
{
await base.OnActionExecutionAsync(context, next);
}
}
The problem here is, that nginx might cancel the request, right after the transaction in the controller and underlying services completed. Which means, the request was successful, but just slightly too long, and nginx will try a second time anyway. In order to solve this, we wanted to set a second timeout which would be few seconds shorter than nginx timeout. The only solution I've found is this:
public override async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
{
var timeout = new CancellationTokenSource();
timeout.CancelAfter(ActionTimeout);
using (timeout.Token.Register(() => timeout.Token.ThrowIfCancellationRequested()))
using (context.HttpContext.RequestAborted.Register(() => context.HttpContext.RequestAborted.ThrowIfCancellationRequested()))
{
await base.OnActionExecutionAsync(context, next);
}
}
But this is throwing the following exception:
System.AggregateException was unhandled
Message: An unhandled exception of type 'System.AggregateException' occurred in System.Private.CoreLib.ni.dll
Additional information: One or more errors occurred.
And afterwards the application is crashing. Putting try/catch anywhere around of course also doesn't help and adding a global exception filter to the middleware doesn't help either.
What is the best way to cancel Requests on linux after a custom timout, without depending on a reverse proxy to do it?
Update - details on exceptions:
At first I see the following exception:
System.OperationCanceledException was unhandled by user code
HResult=-2146233029
Message=The operation was canceled.
Source=System.Private.CoreLib
StackTrace:
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at ...BaseController.<>c__DisplayClass4_0.<OnActionExecutionAsync>b__0() in ...\Controllers\BaseController.cs:line 28
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
InnerException:
Which is fine, that's the cancellation token throwing the exception. But the second one, the one which I can't catch, not sure how I could see the details there, I just see the following screen:
And regardless of what I tried, the process always stopped afterwards (even if I press continue execution)
I am trying to orchestrate a web service deployed in Windows Azure with Apache ODE. I am testing the services with the Eclipse integrated Web Service Explorer. The Azure WS works fine but when I test the Artifacts.wsdl it throws the following error:
14:41:38,777 INFO [DeploymentPoller] Deployment of artifact BPEL_process successful: [{http://artifacts}process-132]
14:41:49,474 WARN [SimpleScheduler] Dispatching jobs with more than 5 minutes delay. Either the server was down for some time or the job load is greater than available capacity
14:41:57,594 WARN [ExternalService] Fault response: faultType=(unkown)
<?xml version='1.0' encoding='utf-8'?><s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body><s:Fault><faultcode xmlns:a="http://schemas.microsoft.com/net/2005/12/windowscommunicationfoundation/dispatcher">a:InternalServiceFault</faultcode><faultstring xml:lang="en-US">The server was unable to process the request due to an internal error. For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.</faultstring></s:Fault></s:Body></s:Envelope>
14:41:57,620 ERROR [INVOKE] Failure during invoke:
14:41:57,623 INFO [BpelRuntimeContextImpl] ActivityRecovery: Registering activity 15, failure reason: on channel 27
Here are the files:
process.bpel
Artifacts.wsdl
mod2.wsdl
mod2.xsd
EDIT This is what I get when I add the IncludeExceptionDetailInFaults in my Web Service:
17:51:49,762 WARN [ExternalService] Fault response: faultType=(unkown)
<?xml version='1.0' encoding='utf-8'?><s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body><s:Fault><faultcode xmlns:a="http://schemas.microsoft.com/net/2005/12/windowscommunicationfoundation/dispatcher">a:InternalServiceFault</faultcode><faultstring xml:lang="en-US">Object reference not set to an instance of an object.</faultstring><detail><axis2ns1:ExceptionDetail xmlns:axis2ns1="http://schemas.datacontract.org/2004/07/System.ServiceModel" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><axis2ns1:HelpLink i:nil="true" /><axis2ns1:InnerException i:nil="true" /><axis2ns1:Message>Object reference not set to an instance of an object.</axis2ns1:Message><axis2ns1:StackTrace> at WCFServiceWebRole1.AzureImpl.mod2(String arg0)
at SyncInvokemod2(Object , Object[] , Object[] )
at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)</axis2ns1:StackTrace><axis2ns1:Type>System.NullReferenceException</axis2ns1:Type></axis2ns1:ExceptionDetail></detail></s:Fault></s:Body></s:Envelope>
17:51:49,798 ERROR [INVOKE] Failure during invoke:
17:51:49,801 INFO [BpelRuntimeContextImpl] ActivityRecovery: Registering activity 15, failure reason: on channel 27
Still no clue of what it means. Eclipse points out en error on my bpel file. The <copy> tag under <assign> throws:
The from-spec of "<xs:simpleType "string">" is not compatible with to-spec of "<xs:complexType>" but I ignored it because following the xsd schemas it doesn't make sense to me and sometimes it even disappears.
EDIT II:
As mentioned by #vanto the <assign> activity was wrong. I had to add the prefix of the namespace of my wsdl to the CDATA tag:
<bpel:query queryLanguage="urn:oasis:names:tc:wsbpel:2.0:sublang:xpath1.0">
<![CDATA[mod2:mod2Result]]>
</bpel:query>
Still don't fully understantand namespaces but it seems it is related to the fact I had to add elementFormDefault="qualified" to the xsd imported by the wsdl of the Azure Service.
The error means that ODE got a soap fault as a response. This indicates that the invoked Web service raised an exception. Please check the logs of your web service or turn on the IncludeExceptionDetailInFaults feature in order to get details about what went wrong. Please not that ODE does not validate outgoing messages, so if your <assign> activities do not create the correct request, ODE won't notice but the invoked service is likely to complain.
I just set up a continuous deployment build (using the azure TFS service a.k.a visualstudio.com) for my project, but it doesn't work right. The build seems to be fine, and the deployment starts, but my worker role always recycles continuously. My manual builds in Visual Studio deployed through the management portal work just fine. When I remoted into one of the worker VMs to see what was going on, I found this in the event log:
An unhandled exception occurred. Type: Autofac.Core.DependencyResolutionException Process ID: 2488
Process Name: WaWorkerHost
Thread ID: 13
AppDomain Unhandled Exception for role MyProject.Worker_IN_1
Exception: An exception was thrown while invoking the constructor 'Void .ctor(Microsoft.WindowsAzure.CloudStorageAccount)' on type 'TemplateRepository'.
at MyProject.Worker.WorkerRole.Run() in c:\a\src\MyProject\Dev\MyProject.Worker\WorkerRole.cs:line 383
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.StartRoleInternal()
at Microsoft.WindowsAzure.ServiceRuntime.Implementation.Loader.RoleRuntimeBridge.<startrole>b__1()
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
Inner Exception: The type initializer for 'Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment' threw an exception.
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.GetConfigurationSettingValue(String configurationSettingName)
at MyProject.Worker.Core.Repositories.TemplateRepository..ctor(CloudStorageAccount account) in c:\a\src\MyProject\Dev\MyProject.Worker.Core\Repositories\TemplateRepository.cs:line 24
at lambda_method(Closure , Object[] )
at Autofac.Core.Activators.Reflection.ConstructorParameterBinding.Instantiate()
Inner Exception: Could not load file or assembly 'msshrtmi, Version=1.8.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT:
0x80131040)
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeEnvironment()
at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment..cctor()
It seems that for some reason, builds performed on the TFS build servers wind up with an incorrect reference to version 1.8 of msshrtmi, rather than 1.7 (which is what all my projects actually reference). I tried adding an explicit reference of msshrtmi, and I also tried the suggestion found here (http://tomkrueger.wordpress.com/2010/07/27/azure-deployment-issue-after-upgrading-to-visual-studio-2010-and-net-4-0/) and neither one seemed to work.
Any other suggestions?
update: here's a link to my cross-post on MSDN forums
Could you be referencing 1.8 versions of the Azure assemblies?
The 1.8 version of Microsoft.WindowsAzure.ServiceRuntime references version 1.8 of msshrtmi.dll and will fail with this message if the target Azure environment does not have the dll.
I have an issue that is occurring after my server reboots. I have some features in SharePoint, for doing various things. One thing they do is add items into a couple of different MSMQ queues. Immediately after a server reboot, if I try to log in to SharePoint and I go to a custom page of mine, and click on a button within that page (which would then create the msmq item) I get taken to a page that displays the following:
The website declined to show this webpage
This error (HTTP 403 Forbidden) means that Internet Explorer was able to connect to the website, but it does not have permission to view the webpage
If I dig in to the Event Logs I find the following errors in Application:
Event ID: 1314
An unhandled access exception has occurred
AND in the Security event logs:
3 of these:
Event ID: 560
Object Open:
Object Server: SC Manager
Object Type: SERVICE OBJECT
Object Name: MSDTC
....
Accesses: Query status of service
Event ID: 560
Object Open:
Object Server: SC Manager
Object Type: SC_MANAGER OBJECT
Object Name: ServicesActive
...
Image File Name: C:\WINDOWS\system32\services.exe
...
Accesses: Connect to service controller
Query service database lock state
Event ID: 560
Object Open:
Object Server: SC Manager
Object Type: SERVICE OBJECT
Object Name: MSDTC
...
Image File Name: C:\WINDOWS\system32\services.exe
...
Accesses: Query service configuration information
ALSO, if I dig in to the SharePoint logs, I find the following errors:
Application error when access /my/site/url/MyPage.aspx, Error=Access
is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Server stack trace: at
System.Transactions.Oletx.IDtcProxyShimFactory.ConnectToProxy(String
nodeName, Guid resourceManagerIdentifier, IntPtr managedIdentifier,
Boolean& nodeNameMatches, UInt32& whereaboutsSize, CoTaskMemHandle&
whereaboutsBuffer, IResourceManagerShim& resourceManagerShim) at
System.Transactions.Oletx.DtcTransactionManager.Initialize() at
System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
at
System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(TransactionOptions
properties) at
System.Transactions.TransactionStatePromoted.EnterState(InternalTransaction
tx) ...
...at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)
at System.Transactions.Transaction.Promote() at
System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction
transaction) at
System.Transactions.TransactionInterop.GetDtcTransaction(Transaction
transaction) at
System.ServiceModel.Channels.MsmqQueue.GetNativeTransaction(MsmqTransactionMode
transactionMode) at
System.ServiceModel.Channels.MsmqQueue.SendDtcTransacted(NativeMsmqMessage
message, MsmqTransactionMode transactionMode) at
System.ServiceModel.Channels.MsmqQueue.Send(NativeMsmqMessage message,
MsmqTransactionMode transactionMode) at
System.ServiceModel.MsmqIntegration.MsmqIntegrationOutputChannel.OnSend(Message
message, TimeSpan timeout) at System.ServiceModel.Chann...
I have two VMs, one for the web (sharepoint) machine, and one of the DB machine (where all the content dbs etc are stored). The DB server is a primary domain controller, and both machines are part of the same domain. I've created a domain user to use for the SharePoint application pool.
Strangely, it is possible to make the errors go away by going to a different part of my application and running that code (which also adds an entry into the msmq). Once that is done, then all of the rest of the application works as normal.
Any help would be incredibly appreciated.
Note: I noticed that the error had something to do with the anonymous user (IUSR...) in IIS for the SharePoint site. I changed that user to be a domain administrator user, and the error no longer happens. So, it's something to do with the permissions of the IUSR.. user, but I don't know what...
O M G
Ok, so I tried something, it was a LONG shot, but I was at my wits end.
I wrapped all of the code that was talking to the MSMQ (adding items to the queue) in the run with elevated privelages thang, and it worked!!!!
SPSecurity.RunWithElevatedPrivileges(delegate()
{
//MSMQCode here
});
Now, I can reboot the server, and do the stuff that was erroring before, and it works fine!
Consider running warm up scripts upon server reboot.
http://blogs.msdn.com/joelo/archive/2006/08/13/697044.aspx
Give everyone read permission to the bin directory under inetpub\wss etc..