After failing to get DebugDiag to analyse crash-dump files it was suggested that I try using WinDbg instead.
The crash-dump files have been created on a Windows Server 2016 box, running my ASP.Net 4.5.2 web application on IIS-10. My ASP.Net web application contains several 3rd party components, with their individual DLLs.
I have copied the crash-dump files onto my Windows 10 development machine, and am running WinDbg locally instead of on the server.
The problem is... when I run !analyze -v in WinDbg on any of the crash-dump files, it effectively hangs while "Downloading file xxx.DLL" (xxx.DLL being the name of just one of the 3rd party component DLLs), and eventually cancels itself after a period of time.
I'm running WinDbg on the same machine that I built the website on in the first place... so is there a way of telling WinDbg that it can find the DLL in a particular location on the local machine?
I obviously don't have a .pdb file for any of the 3rd party components, and so I'm not bothered about it loading symbols for those DLLs... but either I somehow tell it to ignore those particular DLLs, or I tell it how to find them locally.
Can anybody point me in the right direction?
You don't have to analyze the dump file with !analyze -v.
If you need to load dll, then .load D:.... is enough.
To maunal analyze a dump file.
Please run .loadb sos clr to load debug module. If the crash server and your machine run different version of .net framework. Then you need to load sos.dll manually.
When you need to debug .net application in IIS, !mex extension is recommened.
https://www.microsoft.com/en-us/download/details.aspx?id=53304
You can load mex.dll via .load c:\.....\mex.dll
!mex.aspxpages can show all requests inside the process and their process
!mex.mthreads show the status of all threads
!mex.clrstack2 will show all exceptions and mananaged call stack in specific thread.
1.You can use ~* k to load the full call stack in all threads and !mex.mthreads check status.
Then you may find something like KERNELBASE!RaiseException in specific thread
2.Then go to this thread via threadid~ like 12~
3.Run !mex.clrstack2 and it will show the crash exception
Basically, no, you cannot speed up the process of loading symbols for DLLs where you don't have symbols. IMHO, the only way of speeding up the symbol process would be to disable the HTTP server, so that symbols are only searched on your local disk.
See also: How to set up symbols in WinDbg if you have not done this often.
Getting a HTTP 404 for those files should not take very long. However, it tries various file endings and pointers etc. Sometimes Microsoft servers are slow. Also, having a lot of 3rd party DLLs may sum up of course. That can be pretty anoying.
I'll start by saying I don't 100% understand everything I had to do, but here are the step I took to discover where the stackoverflow issue was in my application...
The majority of the information came from this blog.
On the server I added the following registry settings to create the crash dump files...
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps]
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\w3wp.exe]
"DumpCount"=dword:00000005
"DumpFolder"=hex(2):43,00,3a,00,5c,00,43,00,72,00,61,00,73,00,68,00,44,00,75,\
00,6d,00,70,00,73,00,5c,00,00,00
(The DumpCount is the number of files to store before it starts overwriting old ones - DumpFolder is where the files are to be saved, is a REG_EXPAND_SZ and in my case represents C:\CrashDumps\)
Waited for crashes to happen
Copied the crash files into a directory on my local machine called C:\WinDbg\CrashDumps\
Create another directory called C:\WinDbg\Symbols, into which I placed...
clr.dll (from the server, taken from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\)
sos.dll (from the server, taken from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\)
all .dll and .pdb files from my local development environment, including third party component .dll files
Installed WinDbg via Windows Store on my Windows 10 development machine
Ran windbgx -y c:\windbg\symbols via Run command (for some reason it's windbgx on my machine but maybe that's because it's via the Store rather than manual download)
In the file menu Open dump file and select one of the dump files in C:\WinDbg\CrashDumps
Ran the following commands...
.symfix
.reload
.load c:\windbg\symbols\sos.dll (see note 1 below)
!clrstack (see note 2 below)
Although this didn't give me all the information I expected, what it did show was that one of my 3rd party components was 100% to blame for the stackoverflow exception.
Note 1 - Lots of places I read said that .loadby sos clr should be used, but that just gave me The call to LoadLibrary(C:\ProgramData\Dbg\sym\clr.dll\5E7D1F3B9eb000\sos.dll) failed and I couldn't figure out how to fix it... so instead I've used .load c:\windbg\symbols\sos.dll.
Note 2 - The !clrstack command worked because WinDbg appeared to pre-select the thread that had the exception. The other option is to use ~*e !clrstack which will show you call stacks for ALL threads.
Related
I have a webjob that I'm debugging locally as a console app. But once I stop the code from within VS2017 I'm unable to recompile the exe as I get the following error
Unable to copy file "obj\Debug\******.******.exe" to "bin\Debug\******.******.exe". Access to the path 'bin\Debug\******.******.exe' is denied.
When I look at the processes that are running theres nothing there that jumps out.
I've closed VS and restarted it, but thats not cleared the issue.
Other than restarting my machine is there anything else I can try?
You can use the Windows Resource Monitor tool. Once opened, navigate to the CPU tab and enter the full path of the .exe (e.g. C:\Project\bin\debug\App.exe) in the Search Handles text box in the bottom-mid right corner and search. You should see the list of processes currently locking on the particular resource (in this case, the .exe). Select all the unwanted processes, right-click and end process. Depending upon on the version of Windows you are running, the experience might differ a bit but the general idea is more or less the same.
When running a UWP project I'm working on I receive the following dialog.
"Unable to activate Windows Store app 'MyAppsMangledName'. The 'MyExeName' process started, but the activation request failed with error 'The App didn't start'."
The Visual Studio output has the following.
The thread 0x3d4c has exited with code -1073741515 (0xc0000135).
The thread 0x3b50 has exited with code -1073741515 (0xc0000135).
The program 'MyExeName' has exited with code -1073741515 (0xc0000135) 'A dependent dll was not found'.
The Event Viewer has 3 events that basically restate the popup dialog in 3 different ways and nothing else.
Running Process Monitor during the startup shows me many dlls being successfully loaded but nothing indicating failure besides some NAMENOTFOUND events which unfortunately don't show what name wasn't found.
In Win32 a helpful dialog usually indicates which dll could not be loaded. And of course with .Net apps the fusion logs make tracing this very straight forward. But for Store/UWP apps I can't seem to find a good way to track down the offending dependency.
This just hit me too on a project I'm working on. And after much digging, someone on my team was able to figure it out. So figured I'd share it for others struggling with the same issue.
We're doing UWP with C++ using VS2015. So with that in mind, there is a program called gflags located for me at C:\program Files (x86)\Windows Kits 10\Debuggers\x64\gflags.exe
So you'll want a cmd window with admin, and run the command gflags.exe -i your-program-name.exe +sls
Note: gflags wasn't in my path so either add the path or navigate to where it is before executing the command.
Just pass in the name of the exe without directories. What it does is sets a registry setting for VS that turns on sls (show loader snaps) for exe's matching that name. Then run your application in VS and and you'll get a large amount of dll loading information including names of the dlls that fail to load in your output window. In our case it was this:
5038:34f4 # 789320468 - LdrpProcessWork - ERROR: Unable to load DLL: "vccorlib140d_app.DLL", Parent Module: "E:\projects---\Source\Builds\vs2015_Debug_UWP_x64\AppX---.exe", Status: 0xc0000135
Another quicker alternative way to test this (YMMV) was to compare the output with another build config that does work. In our case, we can run release builds just fine, but debug builds barf. And the release output showed vccorlib140_app.dll loaded while the debug had it missing.
I've been searching for a while and finding many old articles but also found one with a Regedit to enable the dumping of .dmp files, which isn't doing it.
Instructions I used are from Microsoft, Collecting User-Mode Dumps (MSDN)
According to Symantec no re-boot is required.
I created it for both:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\Windows Error Reporting\LocalDumps and HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps.
All I get is a report.wer file. No dump. The event log is writing exceptions about every minute and about every 30 minutes the app pool dies, so I'm trying to get the details on where the problem is coming from.
Anyone know why those keys aren't working? Does the report.wer provide anything useful for me? One StackOverflow post said SOS extension and when I search for it Microsoft points me to download WinDbg which I already installed. A lot of instructions I find from 2008 and older seem out-dated.
Edit: I managed to get dump files by running the debugging tool on the server. But there's so much wrong information out there on how to get the actual code info that none of it is working.
I've tried adding the symbols path to my pdb files and I've tried
.loadby sos clr
.loadby sos
!CLRStack
!analyze -v
and all I get is wrong symbols.
WER won't catch crash dumps if there is no crash at all. You know that a process can exit normally, when you think it crashes.
If you do want to learn why a process exits (normally or abnormally), the most reliable way I can think of is to utilize Debug Diag (which is built specifically for IIS),
http://msdn.microsoft.com/en-us/library/ff420662.aspx
Note that you probably should use Debug Diag 2.0 at this moment, and must add Kernel32!ExitProcess and Kernel32!TerminateProcess as breakpoints.
I am facing some critical issue which might be interesting for whom , those who are playing with window azure sdk. I have created on EXE which read performance counter data like CPU, memory, asp.net session from system like
queryCollection = ExecuteWMIQuery("SELECT * FROM win32_perfformatteddata_perfdisk_physicaldisk");
and I have aded this EXE in startup task of simple asp.net application which i have uploaded on window Azure. Now when i connecting to RDP of that I can see following errors in my event log as per below.
Disabled performance counter data collection from the
"ASP.NET_64_2.0.50727" service because the performance counter library
for that service has generated one or more errors. The errors that
forced this action have been written to the application event log.
Correct the errors before enabling the performance counters for this
service.
======================================================================
Windows cannot open the 64-bit extensible counter DLL
ASP.NET_64_2.0.50727 in a 32-bit environment. Contact the file vendor
to obtain a 32-bit version. Alternatively if you are running a 64-bit
native environment, you can open the 64-bit extensible counter DLL by
using the 64-bit version of Performance Monitor. To use this tool,
open the Windows folder, open the System32 folder, and then start
Perfmon.exe.
So i am thinking that my EXE trying to fetch performance counter for 32 bit (win32 indicate that) and that will log above error.
So anyone here came across this type of issue , also if my guess is correct then is there any way to implement my EXE logic such way that it can be run smoothly in any environment(32 or 64 bit)?
Hope that this would remain interesting question here!!!
Thanks In Advance
Arun.
That is correct. IIS running in Azure is running 64-bit unless you change it to run 32-bit in a startup task. You could try building it with the Any CPU setting. But most likely the best way is to do something like what the sysinternal tools does. They will spawn a new process that runs in 64-bit mode when needed. Then you can handle both.
I encountered this error while migrating to a Azure VM.
Solved it by using the InstallUtil which is located in the Framework64 folder instead of the one in the Framework folder
I have been looking at an issue for a week straight and have been unable to figure it out and I am desperate for the fix.
On a client site, we have two environments: UAT and PROD. UAT works perfect (Please keep this in mind). We are now trying to deploy the solution to PROD but certain parts of the solution are not working.
We have developed an asp.net application that we provide to clients to allow them to invoke SSIS packages (there are a couple of drop downs that they first select then click a button named "invoke").
When the user clicks the Invoke button, a batch file named InvokeSSIS.bat is called that assembles a command line call to dtexec with the appropriate parameters.
I'm having a problem with a particular package that is responsible for calling an executable which generates a spreadsheet that i will be importing into my system.
The executable is on an mapped H:\ drive.
I have modified the InvokeSSIS.bat batch file to capture the command the batch file is generating. If I execute this command from the command line, it works perfectly. From the webapp Invoker, it executes the package but the tasks responsible for calling the executable doesn't execute as the entire package takes only 1 second to complete (whereas it should take about a minute.)
The executable DOES have a GUI, but it is NOT interactive. This is because when you call the GUI with specific parameters, it automatically runs in batch mode and executes a macro used to generate the desired spreadsheet.
I know this is ok because it works on the UAT server AND it works from the command line!
I have checked the permissions on the executable (bu right-clicking the executable and clicking properties.) I have granted Full Control on the executable to the same user specified as the identity tab of the application pool i am using.
Can someone please help me? As I said I am dying over here!
Please let me know if you have any ideas or what other info you need.
Environment (both UAT and PROD)
OS: Windows Server 2003
IIS 6
asp.net 2.0
SQL Server 2008
Thanks!
Steve
You can't use a mapped drive with IIS.
You must use the \\servername syntax to reach files on other systems.
I agree with user544284 that this is at least in part a mapping issue. I'll ignore for a minute the complete insanity of having a web application call a batch file to start an executable that's on a remote network drive through a drive letter mapping.
Most likely the UAT box has something set up that maps that drive letter for you which Prod is missing.
The only other possibility is a security violation is occurring. Running .exe's from a network drive is generally frowned on. Do the two environments have the exact same version of windows? Are they configured the same with regards to UAC? Any differences here are going to be important.
Which brings up an interesting thought. I wonder if someone logged in to the UAT server using the same account credentials the app pool is using and added the ip address of the machine where the exe lives to the list of "Local Intranet" sites... Or, if they installed SSIS on the UAT server itself.
Just because YOU can log in to the server and run it on the command line means nothing. You have to find out if the drive letter is mapped at all for the user that the web app is running under and whether that user has the required security bits and whether the local OS will allow it regardless.
Okay, I can't ignore it: hairbrained is the nicest adjective I can come up with for this "architecture". Do yourself a favor and go back to the drawing board on this one. It has the word "brittle" written all over it, as you have already found. Instead of building out a batch file to call dtexec, just do it directly either by something like this or this.