C Application crashes using GetOpenFileName. DEP stops the crash - windows-server-2008-r2

Question

We have a C application which is using the GetOpenFileName common dialog to get the user to select a file. We have been having crashes on Windows2008R2. I figured out that if we put an DEP exception on our application the crashes stop
However, I cant figure out what we are doing wrong or what we can do to stop the crash in the first place. I have placed our code below.
typedef struct {
OPENFILENAME ofn;
COUNT nInternal;
COUNT nExternal;
char szDirName[_MAX_DIR];
char szFile[_MAX_PATH];
char szFileTitle[_MAX_PATH];
char szFilter[128];
} OPENFILENAMEINFO;
typedef OPENFILENAMEINFO FAR *LPOPENFILENAMEINFO;
LPOPENFILENAMEINFO RequestFileNameEx(HWND hDlg, LPSTR lpExt, BOOL bSave, LPSTR lpInit)
{
LPOPENFILENAMEINFO lpFileNameInfo;
int i;
DWORD dwError;
DWORD dwSize;
LPSTR lpDir;
LPSTR lpDrive;
lpFileNameInfo = (LPOPENFILENAMEINFO)mballc(1,sizeof(OPENFILENAMEINFO));
strcpy(lpFileNameInfo->szFilter,lpExt);
for (i=0; lpFileNameInfo->szFilter[i] != '\0'; i++) {
if (lpFileNameInfo->szFilter[i] == '|')
lpFileNameInfo->szFilter[i] = '\0';
}
memset(&lpFileNameInfo->ofn, 0, sizeof(OPENFILENAME));
lpFileNameInfo->ofn.lStructSize = sizeof(OPENFILENAME);
lpFileNameInfo->ofn.hwndOwner = hDlg;
lpFileNameInfo->ofn.lpstrFilter = lpFileNameInfo->szFilter;
lpFileNameInfo->ofn.nFilterIndex = 1;
lpFileNameInfo->ofn.lpstrFile = lpFileNameInfo->szFile;
lpFileNameInfo->ofn.nMaxFile = sizeof(lpFileNameInfo->szFile);
lpFileNameInfo->ofn.lpstrFileTitle = lpFileNameInfo->szFileTitle;
lpFileNameInfo->ofn.nMaxFileTitle = sizeof(lpFileNameInfo->szFileTitle);
lpFileNameInfo->ofn.lpstrInitialDir = _getcwd(lpFileNameInfo->szDirName, _MAX_DIR);
if (bSave) {
lpFileNameInfo->ofn.Flags = OFN_SHOWHELP | OFN_PATHMUSTEXIST | OFN_OVERWRITEPROMPT | OFN_NOCHANGEDIR;
dwError = GetSaveFileName(&lpFileNameInfo->ofn);
} else {
lpFileNameInfo->ofn.Flags = OFN_SHOWHELP | OFN_PATHMUSTEXIST | (bDir==FALSE?OFN_FILEMUSTEXIST:0) | OFN_NOCHANGEDIR;
dwError = GetOpenFileName(&lpFileNameInfo->ofn);
}
if (!dwError) {
dwError = CommDlgExtendedError();
if (dwError)
ResourceHandleError(GETOPENFAIL, dwError);
mbfree(lpFileNameInfo);
return(NULL);
}
return(lpFileNameInfo);
}
a crash dump stack trace looks like
0023:73E61FFF (0x080D7974 0x080D7970 0x078B62F0 0x00000000) msxml6.dll
0023:73E68165 (0x080D7970 0x080D78F0 0x078B62F0 0x080D78F0) msxml6.dll, DllCanUnloadNow()+22084 byte(s)
0023:73E67D08 (0x078B62F0 0x080D7970 0x00000000 0x080D78F0) msxml6.dll, DllCanUnloadNow()+20967 byte(s)
0023:73E6827A (0x080D78F0 0x080D7970 0x080D7950 0x59E489BA) msxml6.dll, DllCanUnloadNow()+22361 byte(s)
0023:73E68241 (0x080D7970 0x080D7950 0x59E489BA 0x00000000) msxml6.dll, DllCanUnloadNow()+22304 byte(s)
0023:73E69DDF (0x00000000 0x080D7950 0x00000000 0x0762FAE0) msxml6.dll, DllCanUnloadNow()+29374 byte(s)
0023:73E6BF9F (0x080D7970 0x080D7950 0x71932915 0x078B5E90) msxml6.dll, DllGetClassObject()+5125 byte(s)
0023:73E6BF83 (0x73E81B38 0x080D39C0 0x080D39C0 0x080D3980) msxml6.dll, DllGetClassObject()+5097 byte(s)
0023:73E6C318 (0x71932881 0x06148CB8 0x06148CB8 0x00000000) msxml6.dll, DllGetClassObject()+6014 byte(s)
0023:73E6CD18 (0x720B35A0 0x0762FBD8 0x06148CB8 0x0762FD68) msxml6.dll, DllGetClassObject()+8574 byte(s)
0023:73E78671 (0x720B35A0 0x0762FBD8 0x0762FD68 0x00000000) msxml6.dll, DllGetClassObject()+56023 byte(s)
0023:73E6AAE5 (0x73E6AC28 0x00000000 0x720B35A0 0x0762FBD8) msxml6.dll, DllCanUnloadNow()+32708 byte(s)
0023:74B0A0E1 (0x00000000 0x00000000 0x00000000 0x00000001) ole32.dll, CoCreateInstanceEx()+0915 byte(s)
0023:74B09FA1 (0x720B3614 0x00000000 0x00000017 0x00000000) ole32.dll, CoCreateInstanceEx()+0595 byte(s)
0023:74B09E25 (0x720B3614 0x00000000 0x00000017 0x00000000) ole32.dll, CoCreateInstanceEx()+0215 byte(s)
0023:74B09D86 (0x720B3614 0x00000000 0x00000017 0x00000000) ole32.dll, CoCreateInstanceEx()+0056 byte(s)
0023:74B09D3F (0x720B3614 0x00000000 0x00000017 0x720B35A0) ole32.dll, CoCreateInstance()+0052 byte(s)
0023:720B352B (0x0553A7C0 0x00000000 0x0070E2DC 0x0070E288) FunDisc.dll
0023:720B9470 (0x0553A7C0 0x00000000 0x00000001 0x00000001) FunDisc.dll, DllGetClassObject()+21871 byte(s)
0023:720C3B69 (0x00000001 0x0070E288 0x8007000E 0x00000000) FunDisc.dll, DllUnregisterServer()+20504 byte(s)
0023:720B75AA (0x73751590 0x00000000 0x00000001 0x00000000) FunDisc.dll, DllGetClassObject()+13993 byte(s)
0023:720B1CE9 (0x73751590 0x00000000 0x00000001 0x055874F8) FunDisc.dll
0023:720B1C39 (0x00709310 0x73751590 0x00000000 0x00000001) FunDisc.dll
0023:73752F84 (0x055E2F28 0x00709310 0x73751590 0x00000000) NetworkItemFactory.dll
0023:737530A5 (0x055E2F28 0x0762FF88 0x763643C0 0x055E2F28) NetworkItemFactory.dll
0023:73753144 (0x055E2F28 0x00000000 0x00000000 0x03EDFB9C) NetworkItemFactory.dll
0023:763643C0 (0x03EDFB9C 0x0762FFD4 0x77029EF2 0x03EDFB9C) SHLWAPI.dll, IUnknown_QueryService()+0346 byte(s)
0023:74C9339A (0x03EDFB9C 0x13BB74FB 0x00000000 0x00000000) kernel32.dll, BaseThreadInitThunk()+0018 byte(s)
0023:77029EF2 (0x763642ED 0x03EDFB9C 0xFFFFFFFF 0x770B736F) ntdll.dll, RtlInitializeExceptionChain()+0099 byte(s)
0023:77029EC5 (0x00000000 0x00000000 0x00000000 0x00000000) ntdll.dll, RtlInitializeExceptionChain()+0054 byte(s)

Answer 1

I had the same problem and found a possible solution by Synastry on
http://social.msdn.microsoft.com/Forums/en-US/vcmfcatl/thread/5037519a-78e2-42f4-94cd-bbe88e0f16d6/
All of us suffering this problem has a call stack in the function 'CoUninitialize'. This function is automatically called when a worker thread of COM ends up. This worker thread is not created directly by the user but when some function using COM library is called, it creates it(or them). And wagscallion and I commonly called the function 'GetOpenFileName'. I guess that GSansoucie also called some COM automated function too.
Ending up the worker thread and being called 'CoUninitialize' is legal and normal action of COM library. That's not the reason. The exception that we met is caused by uninitializing COM Server while it's still in use. But our(or at least my) code is also legal and proper except one thing. I did never called 'CoInitialize' or 'CoInitializeEx' function in my code.
COM library has a internal count(similar to a reference counter) and it's incremented by calling CoInitialize or CoInitializeEx and decremented by calling CoUninitialize. But I didn't call any initializer function. Although I didn't call it, GetOpenFileName function calls it in GetOpenFileName's implementation for it's worker threads. And after the function returns, the worker threads wait for another COM job for a while. This is why the exception is occured not instantly when GetOpenFileName function returns. But the worker threads decided to end up themselves, they call CoUninitialize and now the internal count of COM library Server goes 0 and CoUninitialize frees all resources from memory.
But after GetOpenFileName function returns, some of resources should remain in memory(I can't sure about this but if this assumption is not true, we would never meet an exception). To maintain them not to be freed, we need to call CoInitialize or CoInitializeEx(MSDN recommends later one) in the initialization of our program. Also we need to call CoUninitialize before our program finishes.
IN SHORT, WE NEED TO CALL 'CoInitialize' or 'CoInitializeEx' at the start of our program and 'CoUninitialize' at the end of program. But MSDN doesn't describe about this for GetOpenFileName or any other functions using COM library. :-(
In my case, by calling initializer and uninitializer the problem is gone and now everythings work well. Take a look and apply it to your code. If there's another reason causing this exception you know, please let us know too. :-)
Thank you for reading.
For myself this was not the solution but a hint where to look at. I have used CoInitialize in my multi-thread application which internally calls CoInitializeEx with COINIT_APARTMENTTHREADED.
I changed the call now from CoInitialize(NULL) to CoInitializeEx(NULL, COINIT_MULTITHREADED) and my problems seem to be gone.