I've web API hosted on IIS.
I've been getting "Connection Was Reset" on random times as a response to my API Calls. and after several retrial my request gets a proper response
I've checked HTTPERR and found a lot of "Connection_Abandoned_By_ReqQueue" near the time i got Connection Was Reset Response. also i sometimes get App Crash in Event Viewer contains the following:
*Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0
Problem signature:
P1: w3wp.exe
P2: 8.5.9600.16384
P3: 52157ba0
P4: SAPbobsCOM90.dll
P5: 9.30.190.0
P6: 5c7f4d65
P7: c0000005
P8: 0089a548
P9:
P10:
Attached files:
These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_w3wp.exe_695eddbfadf3c4a0ec181813e91099c0502fcadb_234314c6_7e0a5f90
Analysis symbol:
Rechecking for solution: 0
Report Id: 6975c2ef-e9ad-11e9-80ca-00155d0b1f02
Report Status: 0
Hashed bucket: *
I've managed to export a Dump file from IIS the time the crash happened.
and used windbg and DebugDiag. and found out the exception is System.NullReferenceException Object reference not set to an instance of an object.
but this error makes no sense, since the same request get a proper request after multiple retrials.
Related
I have a BizTalk 2013 orchestration receives an input file, processes it and in the end transfers it to a SharePoint 2010 folder.
It has been working OK earlier, with small (< 1 MB) files. But recently the orchestration has tried to upload a file of 230MB; and the orchestration failed (BizTalk put it to list of Suspended Items). The error was the following:
“ErrorCode 0xc0c0167a (Transmission Failure)
[System.Net.WebException] The remote server returned an error: (400) Bad Request.
This error was triggered by the Windows SharePoint Services receive location or send port with URI wss://intranetazure:80/OnlineParsing/Archive.
Windows SharePoint Services adapter event ID: 12310”.
The URI intranetazure:80/OnlineParsing/Archive is OK. I cannot understand – what went wrong? Could the root cause be the fact that the failed files are large (> 200 MB)?
What should I do to fix it?
Today I've adjusted the App Service Log settings for my Azure App Serivce: a .NET Core 2.1 Web API. Soon after I changed the Application Logging (file system) setting from Information to Warning, my application stopped responding. I immediately kicked of a .NET Trace using the Diagnose and solve problems feature.
This Trace report showed the following:
system.private.corelib!
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start
microsoft.aspnetcore.hosting.il!Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start
microsoft.aspnetcore.hosting.il!Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync
microsoft.aspnetcore.hosting!
microsoft.aspnetcore.hosting!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1+<>c[System.Threading.Tasks.VoidTaskResult,Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__188`1[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context]].<.cctor>b__9_0
system.private.corelib!
System.Private.CoreLib!System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[System.Threading.Tasks.VoidTaskResult,Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol+<ProcessRequests>d__188`1[Microsoft.AspNetCore.Hosting.Internal.HostingApplication+Context]].MoveNext
system.private.corelib!
system.private.corelib!
system.private.corelib!
system.private.corelib!
system.private.corelib!
Same time, the Azure Blob log showed these lines:
2019-07-06 14:04:06.476 +00:00 [Error] Microsoft.AspNetCore.Server.Kestrel: Connection id "0HLO233NHLK6Q", Request id "0HLO233NHLK6Q:000003BF": An unhandled exception was thrown by the application.
System.OverflowException: Arithmetic operation resulted in an overflow.
at System.IO.StreamWriter.Write(String value)
at Microsoft.Extensions.RazorViews.BaseView.Write(String value)
at Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync()
at Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
2019-07-06 14:04:19.245 +00:00 [Error] Microsoft.AspNetCore.Server.Kestrel: Connection id "0HLO233NHLK6T", Request id "0HLO233NHLK6T:000000EF": An unhandled exception was thrown by the application.
System.ArgumentOutOfRangeException: Index and count must refer to a location within the buffer.
Parameter name: chars
at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
at System.IO.StreamWriter.Write(String value)
at Microsoft.AspNetCore.Hosting.Views.ErrorPage.ExecuteAsync()
at Microsoft.Extensions.RazorViews.BaseView.ExecuteAsync(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
These stacktraces don't show any of my own code, they seem framework related.
Is there a message or a hint in these error stack traces that I can use to prevent these situations?
Many thanks in advance
I know it's too late but maybe my answer will help someone else.
I received the same exceptions as you. But it was not Azure App Service but IIS. The problem was in incorrect configuration somewhere in Startup.cs. You webhost tried to show error page but at the same time App Service (or IIS in my case) tried to shut down or restart it.
This thread helped me a lot https://github.com/aspnet/KestrelHttpServer/issues/1737
Worker role instance stops daily and i get following error in WADWindowsEventLogsTable.
Instance status is unhealthy after this event is logged.
Fault bucket , type 0
Event Name: PDUWICA
Response: Not available
Cab Id: 0
Problem signature:
P1: 8
P2: 1.6
P3: 6.3.0.0
P4: 1033
P5: 61
P6:
P7:
P8:
P9:
P10:
Attached files:
These files may be available here:
Analysis symbol:
Rechecking for solution: 0
Report Id: a32d8f81-b668-11e5-80c8-000d3a316974
Report Status: 0
Hashed bucket:
Can you enable Intellitrace, re-deploy your instance (suggest to use local emulator if possible for a staging instance), get a more specific information about any unhealthy experience as posted and report the same? Review this link for a step by step guide on how to enable and use Intellitrace (http://justazure.com/microsoft-azure-cloud-services-part-5-debug-monitor-scale/)
Edit:
Once logged in to the instance with Remote assistance.
Try to access the Schedule Tasks, following is the screenshot on my PC and that might get you some specific information.
Once the scheduler runs, try to dig in and look for anything suspicious
I developed a application site where one facility is FAQ in which user can post text data without any limit.
I hv two server to run the application whenever a single field (question or answer) size is huge (like one page long) one of the server is giving service unavailable. I checked in log the error detail is
-------------------
Event Type: Error
Event Source: W3SVC
Event Category: None
Event ID: 1002
Date: 1/23/2012
Time: 3:29:49 PM
User: N/A
Computer: BA5SWWW006
Description:
Application pool 'pool_name' is being automatically disabled due to a series of failures in the process(es) serving that application pool.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
--------------------
AND ALSO
-------------------
Event Type: Error
Event Source: VsJITDebugger
Event Category: None
Event ID: 4096
Date: 1/23/2012
Time: 3:29:44 PM
User: NT AUTHORITY\NETWORK SERVICE
Computer: BA5SWWW006
Description:
An unhandled win32 exception occurred in w3wp.exe [10896]. Just-In-Time debugging this exception failed with the following error: Debugger could not be started because no user is logged on.
Check the documentation index for 'Just-in-time debugging, errors' for more information.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 02 00 5c 80 ..\
------------------------
but the other server is working fine. i check all d basic setting of both is same.
and also there no such logged in error for any other module.
Even error wont occur for same module if the text in question or answer is less.
when this occur it ask for enter the user credentials i couldn't understand why it prompt for such?????
i use my-sql with longtext field to store question or answer.
May be best to try the IIS Debug Diagnostics Tool to further diagnose the problem.
This SO question has plenty of other suggestions: How to diagnose IIS fatal communication error problem
I have an issue that is occurring after my server reboots. I have some features in SharePoint, for doing various things. One thing they do is add items into a couple of different MSMQ queues. Immediately after a server reboot, if I try to log in to SharePoint and I go to a custom page of mine, and click on a button within that page (which would then create the msmq item) I get taken to a page that displays the following:
The website declined to show this webpage
This error (HTTP 403 Forbidden) means that Internet Explorer was able to connect to the website, but it does not have permission to view the webpage
If I dig in to the Event Logs I find the following errors in Application:
Event ID: 1314
An unhandled access exception has occurred
AND in the Security event logs:
3 of these:
Event ID: 560
Object Open:
Object Server: SC Manager
Object Type: SERVICE OBJECT
Object Name: MSDTC
....
Accesses: Query status of service
Event ID: 560
Object Open:
Object Server: SC Manager
Object Type: SC_MANAGER OBJECT
Object Name: ServicesActive
...
Image File Name: C:\WINDOWS\system32\services.exe
...
Accesses: Connect to service controller
Query service database lock state
Event ID: 560
Object Open:
Object Server: SC Manager
Object Type: SERVICE OBJECT
Object Name: MSDTC
...
Image File Name: C:\WINDOWS\system32\services.exe
...
Accesses: Query service configuration information
ALSO, if I dig in to the SharePoint logs, I find the following errors:
Application error when access /my/site/url/MyPage.aspx, Error=Access
is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Server stack trace: at
System.Transactions.Oletx.IDtcProxyShimFactory.ConnectToProxy(String
nodeName, Guid resourceManagerIdentifier, IntPtr managedIdentifier,
Boolean& nodeNameMatches, UInt32& whereaboutsSize, CoTaskMemHandle&
whereaboutsBuffer, IResourceManagerShim& resourceManagerShim) at
System.Transactions.Oletx.DtcTransactionManager.Initialize() at
System.Transactions.Oletx.DtcTransactionManager.get_ProxyShimFactory()
at
System.Transactions.Oletx.OletxTransactionManager.CreateTransaction(TransactionOptions
properties) at
System.Transactions.TransactionStatePromoted.EnterState(InternalTransaction
tx) ...
...at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)
at System.Transactions.Transaction.Promote() at
System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction
transaction) at
System.Transactions.TransactionInterop.GetDtcTransaction(Transaction
transaction) at
System.ServiceModel.Channels.MsmqQueue.GetNativeTransaction(MsmqTransactionMode
transactionMode) at
System.ServiceModel.Channels.MsmqQueue.SendDtcTransacted(NativeMsmqMessage
message, MsmqTransactionMode transactionMode) at
System.ServiceModel.Channels.MsmqQueue.Send(NativeMsmqMessage message,
MsmqTransactionMode transactionMode) at
System.ServiceModel.MsmqIntegration.MsmqIntegrationOutputChannel.OnSend(Message
message, TimeSpan timeout) at System.ServiceModel.Chann...
I have two VMs, one for the web (sharepoint) machine, and one of the DB machine (where all the content dbs etc are stored). The DB server is a primary domain controller, and both machines are part of the same domain. I've created a domain user to use for the SharePoint application pool.
Strangely, it is possible to make the errors go away by going to a different part of my application and running that code (which also adds an entry into the msmq). Once that is done, then all of the rest of the application works as normal.
Any help would be incredibly appreciated.
Note: I noticed that the error had something to do with the anonymous user (IUSR...) in IIS for the SharePoint site. I changed that user to be a domain administrator user, and the error no longer happens. So, it's something to do with the permissions of the IUSR.. user, but I don't know what...
O M G
Ok, so I tried something, it was a LONG shot, but I was at my wits end.
I wrapped all of the code that was talking to the MSMQ (adding items to the queue) in the run with elevated privelages thang, and it worked!!!!
SPSecurity.RunWithElevatedPrivileges(delegate()
{
//MSMQCode here
});
Now, I can reboot the server, and do the stuff that was erroring before, and it works fine!
Consider running warm up scripts upon server reboot.
http://blogs.msdn.com/joelo/archive/2006/08/13/697044.aspx
Give everyone read permission to the bin directory under inetpub\wss etc..