We have an application where we have several Azure WebJobs set up, and we have NLog set up to report on the entry and exit of those webjobs. We have NLog hooked into Stackify so that we can use the Stackify logger to watch things move from WebJob to WebJob. The logging works (so we know the configuration is OK). The problem we are hitting is that the logging is intermittent - we have independent confirmation of messages travelling along several WebJobs, but we will only get partial logging. For example, the first log will record that a message was read off of it, but won't show the exit message. The next log will show the entry and exit. The problem is that they all leverage the exact same code, so the same log message should appear every single time. Has anyone else encountered this issue before? Before you answer, we have taken these diagnostic steps:
(1) We have verified the proper configuration settings in all webjob config files.
(2) We have verified that all of the webjob config files are properly posted in Azure.
(3) We have independently verified (via the webjob consoles in Azure) that the messages are getting to the different webjobs, processing, and then moving on to the next webjob, meaning that they would be invoking the logging code.
(4) We have verified that our Stackify log data is not being throttled (this might have made sense if the logs had simply stopped - but the fact that we just get partial logging in some cases and full logging in others during a single message transmission makes this unlikely).
(5) We have pored over the Stackify documentation, which seems to ensure that configuration is set up properly so that you get logging at all. It covers the all-or-nothing scenario, but not the some scenario.
I will be happy to provide more clarification as needed.
We discovered what is going on, and it is rather strange. The issue ended up being with our use of NLog.Targets.Stackify. We were using version 1.25.4. When I ran a test using the NLog debugger suggested by Julian, this is what showed up:
2017-01-09 10:14:43.5079 Info Loading assembly name: NLog.Targets.Stackify
2017-01-09 10:14:43.5449 Debug ScanAssembly('NLog, Version=4.0.0.0, Culture=neutral, PublicKeyToken=5120e14c03d0593c')
2017-01-09 10:14:43.5929 Debug Start auto loading, location: C:\Dev\AffinityMain\platform\Integrity.WebJob.Rating\bin\Debug
2017-01-09 10:14:43.5929 Info Auto loading assembly file: C:\Dev\AffinityMain\platform\Integrity.WebJob.Rating\bin\Debug\NLog.Targets.Stackify.dll
2017-01-09 10:14:43.6039 Info NLog.Targets.Stackify, Version=1.18.6200.39247, Culture=neutral, PublicKeyToken=null. File version: 1.18.*. Product version: 1.25.4.
2017-01-09 10:14:43.6039 Debug ScanAssembly('NLog.Targets.Stackify, Version=1.18.6200.39247, Culture=neutral, PublicKeyToken=null')
2017-01-09 10:14:43.6249 Warn Type load exception. Exception: System.IO.FileLoadException: Could not load file or assembly 'NLog, Version=5.0.0.0, Culture=neutral, PublicKeyToken=5120e14c03d0593c' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'NLog, Version=5.0.0.0, Culture=neutral, PublicKeyToken=5120e14c03d0593c'
This was followed shortly by this error when it was parsing the config:
System.ArgumentException: Target cannot be found: 'StackifyTarget'
The documentation indicates that version 4.4 of NLog is sufficient to run this library. However, the NLog debugger indicates that it is looking for version 5.0, and not finding it (which it wouldn't because we aren't using it). Now, what makes this strange is that it did work at one time, with this version, so somehow a reference to NLog 5.0 is stuck somewhere in the system, but we can't find it because everything in our solution is running NLog 4.4. We've checked the csproj, the packages config, the app config, the actual installed nuget packages - no reference to version 5.0.
The answer to the problem was to downgrade to 1.25.3. As soon as I did that, it worked. I then tried to bump it back to 1.25.4, and had the same problem again. Matt - to your point about Shutdown - we are going to add that to our webjobs, and then I will monitor to see if we are seeing complete logs. Thank you everyone for your suggestions!
You should be able to fix this problem by adding one line of code at the end of your application to StackifyLib to flush.
StackifyLib.Logger.Shutdown();
https://github.com/stackify/stackify-api-dotnet/blob/master/README.md
I hope this fixes it for you, if not, please contact Stackify support.
Could be nice with an example of the NLog-config. If using the async-wrapper, then by default the overflow action is to discard "random" messages.
Related
D:\Program Files (x86)\MSBuild\14.0\bin\Microsoft.Common.CurrentVersion.targets(1819,5): warning MSB3277: Found conflicts between different versions of the same dependent assembly that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed.
Now, I know what to do if this error message had come from Visual Studio: Found conflicts between different versions of the same dependent assembly that could not be resolved
The problem is that this error message comes from drilling into "Logs" in the Deployment Centre of my Azure web app (from clicking "Show Logs..." next to "Running deployment command..." I can't figure out where these reference conflicts are listed for an Azure app service despite trying to turn on as much logging as possible and searching for a solution to this problem.
Could someone please show me where to go to see where these conflicts are listed in an Azure app service? Where is the build log? How do I turn it "on"?
EDIT: I just found this: What is the default location for MSBuild logs?. I don't know if it applies to Azure.
Update:
The code is being continuously pulled into my Azure app from my Azure DevOps repository whenever I push a new code change. It's building (which includes compiling) on the server, somewhere. I just don't think there's any way to actually see the build logs. In Visual Studio, all we have is the output window, and there's no equivalent on the server.
Update:
Before viewing logs on Azure, be aware that azure-web-sites do not provide compilation capabilities, so compilation-related warnings and errors are not displayed in the logs. Azure's log feature is used to track runtime warnings or run-time errors, not compile-time.
Since you have deployed to Azure, please have a look of this:
Log stream:
From your description, you need to change the assembly which is mismatch with others.
If you still can't resolve this problem, please add the error and the assembly you used in your app service to this question text.
Our Azure Web App Service in production was working fine and suddenly on 15 November exactly at 019-11-15 06:22:56.2653 we saw the app service being down due to the following error. We have not changed anything neither deployed anything new on the said app service. Wanted to know if there was any upgrade from Microsoft which caused this?
ERROR|Stopped program because of exception Could not load file or assembly 'Microsoft.ApplicationInsights, Version=2.11.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file specified.
Also, looking at the error we took new nuget package for appinsight and deployed and it seems to be working fine. But wanted to know what caused this and how can we prevent this in future.
There is actually an pre-release of Microsoft.ApplicationInsights, version 2.12.0-beta2 on 15 November. But this should does nothing to do with your issue.
You should nav to your azure web site -> kudu, to see if the specified dll is removed by accident.
I have a web app on Azure. When I publish an update to it using WebDeply it doesn't work. I get a 502.5 Process Failure error. Upon investigation the content of the error is:
Unhandled Exception: System.IO.FileLoadException: Could not load file or assembly 'Microsoft.AspNetCore.Hosting.Abstractions, Version=2.0.2.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
at Web.Program.Main(String[] args)
Does anyone know why I would be getting this?
It works locally but not when published. The most recent update I did to the AspNetCore ddls was done as part of this release where I upgraded the NuGet packages reference to be the latest patch fixes ie. from 2.0.0 to 2.0.2. Is it something I did wrong?
I eventually solved the issue by using an FTP, deleting all of the DLLs in the website folder and re-publishing the application.
Seems Web Publish or VS decided not to update those files for some reason.
Addional information for Adam Stapleton.
As Adam Stapleton mentioned it seems that Web publish or VS doesn't update the related files. If we update the AspNetCore, I recommend that we could remove the Azure WebApp files during republish the WebApp. We also could do this with VS. For more detail information, please refer to the screenshot.
I'm having an issue as below:
Could not load file or assembly 'msshrtmi, Version=2.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.IO.FileNotFoundException: Could not load file or assembly 'msshrtmi, Version=2.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Assembly Load Trace: The following information can be helpful to determine why the assembly 'msshrtmi, Version=2.3.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' could not be loaded.
WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].
The issue happens only when publishing to Windows Azure Websites (WAWS). I want to detect whether I am running in Windows Azure mode or not. Is this possible in WAWS? This error is happening only when I call Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.IsAvailable.
I'd assume that the WAWS environment would have the Azure SDK installed, right? Or is this only available for Cloud services?
I've looked at some solutions telling you to manually include it in x64 or x86 version, but I would like not to be limited that way, or similar workarounds.
I encountered this same error when publishing a Web App. After trying every other solution on the web, the following finally worked for me:
In the Azure Portal, select your Web App. Select 'Settings > Application settings'. Change the Platform value from 32-bit to 64-bit. Re-publish.
We're using log4net in a number of our services. Most of the time it works flawlessly. However, we'll occasionally see log4net "freezes" for a period of time before continuing to work normally. For example, the logs below shows that log4net was "frozen" for over 4 hours. We know this application is verbose, so we expect to see logs written every second.
2010-11-04 04:02:09,393 DEBUG Some message...
2010-11-04 08:48:07,114 DEBUG Some other message...
Another problem we'll occasionally see is that entire log files get deleted. We have log4net configured to roll over by date. Sometimes we'll see that an entire weeks worth of logs go missing. Is log4net mistakenly deleting these logs files? We do not have log4net configured to keep a number of logs.
Has anyone encountered this before?
I have not seen this, but remember that Log4net is designed to continue even if it can not log its information - See this FAQ Article
If anything locks the log file (eg Backup/AntiVirus) then you will get missing information.
You could trying running DebugView to see if any such warning appears when the logging fails or perhaps configure log4net to keep its own logs (see another FAQ Article )