After updating to Azure SDK 2.6 (from 2.5.1) I have not been able to publish my Azure Worker Role instance. It had been working previously with the 2.5.1 SDK. The worker process fails to start in Azure and hangs with the status of:
Busy (Waiting for role to start... Application startup tasks are running. [2015- 06-02T23:27:56Z] Last role error: exit code - 0, exit time - [2015/06/02, 23:27:20.218])
The worker process does make use of Azure Diagnostics - configured in my service definition file like:
<Imports>
<Import moduleName="RemoteAccess" />
<Import moduleName="RemoteForwarder" />
<Import moduleName="Diagnostics"/>
</Imports>
and I do see an error in the Event Log of the worker VM:
System.UnauthorizedAccessException: Access to the path 'DiagnosticPlugin-ShutdownEvent' is denied.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.Threading.EventWaitHandle..ctor(Boolean initialState, EventResetMode mode, String name)
at Microsoft.WindowsAzure.Plugins.Launcher.Program.MainImpl(String[] args)
at Microsoft.WindowsAzure.Plugins.Launcher.Program.Main(String[] args)
I'm just not sure how to correct the issue. Any ideas?
Azure Diagnostics Module should no longer be present in either SDK 2.5 or 2.6 of Azure deployments. Azure Diagnostics in SDK2.5+ world is now configured as an extension. (Diagnostics v1.3 version)
Here is more information: https://azure.microsoft.com/en-us/documentation/articles/cloud-services-dotnet-diagnostics/
Related
What I am trying to achieve
Hello, I have dotnetcore web Api which is running in Azure App Service server farm. For logging I have used both ILogger and Log4net. I am trying to send the logs of service to application-insights but the logs are not streaming when deployed to Azure but when tested locally it works fine and I can see logs from both the library ILogger/Log4net getting streamed to Application Insights.
What I have tried so far
Below configuration I have added in my service for application insights.
Runtime version: netcoreapp3.1 version-2.31.0.1
Hosting environment: Azure App Service
app-service.csproj
<ItemGroup>
<PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.20.0" />
<PackageReference Include="Microsoft.ApplicationInsights.Log4NetAppender" Version="2.20.0" />
...
<ItemGroup>
log4net.config
<appender name="aiAppender" type="Microsoft.ApplicationInsights.Log4NetAppender.ApplicationInsightsAppender, Microsoft.ApplicationInsights.Log4NetAppender">
<InstrumentationKey name="AppInsightsKey" value="abcdefgh-abcd-abcd-abcd-abcdefghijkl" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%message%newline"/>
</layout>
</appender>
startup.cs
public Startup(IConfiguration configuration)
{
Configuration = configuration;
Logger.ConfigureLog4Net("./logs/app.log", Configuration)
...
}
public void ConfigureServices(IServiceCollection services)
{
// The following line enables Application Insights telemetry collection.
services.AddApplicationInsightsTelemetry(
new Microsoft.ApplicationInsights.AspNetCore.Extensions.ApplicationInsightsServiceOptions
{
EnableActiveTelemetryConfigurationSetup = true,
InstrumentationKey = Configuration["ApplicationInsights:InstrumentationKey"]
}) ;
services.AddMvc();
...
}
appsettings.json
"ApplicationInsights": {
"InstrumentationKey": "xxxxx"
}
I have also added ApplicationInsights.config file as suggested in elsewhere for Log4net to work as below
<?xml version="1.0" encoding="utf-8"?>
<ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings">
<InstrumentationKey>xxxxx</InstrumentationKey>
</ApplicationInsights>
I have also added the key ApplicationInsights:InstrumentationKey in App Service Configuration App Settings. As well tried with default app settings key APPINSIGHTS_INSTRUMENTATIONKEY.
I have made sure all the required App settings keys are present in App service configuration too.
Followed this MS document on common issues Troubleshooting Application Insights Agent (formerly named Status Monitor v2) and as it quotes below
If any of these DLLs are present in the bin directory, monitoring
might fail:
Microsoft.ApplicationInsights.dll
Microsoft.AspNet.TelemetryCorrelation.dll
System.Diagnostics.DiagnosticSource.dll
But it's really confusing cause the required Nugets like "Microsoft.ApplicationInsights.AspNetCore" has "Microsoft.ApplicationInsights.dll" as dependency which in-turn has "System.Diagnostics.DiagnosticSource.dll" as dependency; and without the required AspNetCore nuget Application Insight won't work.
Essentially, this is a strange issue am facing where-in all logs and exceptions getting streamed perfectly fine in local run but once deployed to Azure App service no logs are streaming at all. Can someone shed some light here? Am I missing something obvious??
There are 5 Nuget packages that depend onSystem.Diagnostics.DiagnosticSource.dll , which belongs to logging of data.
This particular dll produces more conflicts while logging ApplicationInsights data.
Delete the System.Diagnostics.DiagnosticSource.dll from the bin folder ,build and deploy the application once and check the logs in Portal.
As it is a Default template added with the
Microsoft.ApplicationInsights.AspNetCore , I didn't found any build issues after deleting it from bin.
Here is the workaround which cleared the same issue.
I have created a Cloud Service project with following roles:
One Web role
One Worker role
For both the roles, I have added Azure Application Insights SDK/ Nuget pkg. I have also added the Status Monitor to both of them, StartUp task. Two roles point to different AI resources. For Worker role I also added the below three environment variables in csdef file.
<Environment>
<Variable name="COR_ENABLE_PROFILING" value="1" />
<Variable name="COR_PROFILER" value="{324F817A-7420-4E6D-B3C1-143FBED6D855}" />
<Variable name="MicrosoftInstrumentationEngine_Host" value="{CA487940-57D2-10BF-11B2-A3AD5A13CBC0}" />
</Environment>
The problems is unlike Web role, Worker role doesn't populate the CPU usage/Available memory in Application Insights, as can be shown below.
Application Insights resource of Worker Role:
What can be the issue and can I use any GUID in the environment variables shown above?
Also, it seems that for worker roles we need to set the ApplicationInsights.config file to be copied always to the output directory. However, when I do that, worker role throws the task exception and keeps on restarting, which finally fails the deployment.
1) In order to collect the performance counters account that you worker role process (WaWorkerHost.exe) running under should be part of Performance Monitor users group.
Take a look at the thread below: it explains how to modify start up task (InstallAgent.ps1) to give the right permissions. It's done for the web role, so you'll need to replace $user = "Network Service" with the right account in your case.
add Azure Application insight Status Monitor from command line
2) No, you cannot use any GUID for env variable. It's predefined COM object GUID of AppInsights profiler.
Service Bus Gateway is not starting.
First of all I have tried completely uninstalling all Azure/Fabric/Service Bus type SDKs and installations. I have installed both Windows Azure Pack: Service Bus 1.1 and Windows Azure Pack: Security Update for Service Bus 1.1 (KB2972621) from scratch.
After deleting all service bus related databases I start the Service Bus Configuration wizard and begin starting a new farm with custom settings.
I leave everything default except adding my own certificates, and of course entering my password.
After clicking go the the service bus configuration wizard eventually gets stuck in progress:
Starting
Created and configured Service Bus farm management database.
Created and configured Service Bus gateway database.
Creating default container.
Processing completed
Validating input and configuration parameters.
Granting 'Log on as Service' privilege to the run as account.
Windows Fabric configuration started.
Running Windows Fabric deployment.
Windows Fabric starting.
Service Bus configuration started.
Updating database.
Service Bus services starting.
The event viewer logs for Microsoft-Service Bus > Operational has the following errors. Top error received first...
Exception during fabric service creation for container 1, Exception System.ArgumentException: At least one address must be provided if hostEndpoints is non-null
Parameter name: hostEndpoints
at System.Fabric.FabricClient.InitializeFabricClient(SecurityCredentials credential, TimeSpan keepAliveInterval, String[] hostEndpoints)
at System.Fabric.FabricClient..ctor(SecurityCredentials credential, String[] hostEndpoints)
at Microsoft.ServiceBus.Commands.ServiceBusGetCommands.CreateFabricClient()
at Microsoft.ServiceBus.Commands.ServiceBusCommandBase.RegisterWinFabricService(Int64 containerId)
And then:
Service Bus Gateway service failed to start, retry count 1. Exception message: An error occurred creating the configuration section handler for namespacePolicyDataStoreFactory: Could not load file or assembly 'Microsoft.Cloud.Common.AzureStorage, Version=2.1.0.0, Culture=neutral, PublicKeyToken=4fe77f22fa8374f3' or one of its dependencies. The system cannot find the file specified.. Stack Trace: at System.Configuration.BaseConfigurationRecord.CallCreateSection(Boolean inputIsTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentConfig, ConfigXmlReader reader, String filename, Int32 line)
at System.Configuration.BaseConfigurationRecord.CreateSectionDefault(String configKey, Boolean getRuntimeObject, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object& result, Object& resultRuntimeObject)
at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)
at Microsoft.Cloud.ServiceBus.ServiceRegistryManagerContext.CreateNamespacePolicyDataManager(IComponentSite site)
at Microsoft.Cloud.ServiceBus.ServiceRegistryManagerContext.LoadServices(IComponentSite site)
at Microsoft.Cloud.ServiceBus.Common.Components.ComponentFactoryBase`1.CreateComponent()
at Microsoft.Cloud.HostingModel.ComponentHost.CreateComponent(IComponentFactory componentFactory)
at Microsoft.Cloud.HostingModel.ComponentHost.CreateComponents()
at Microsoft.Cloud.HostingModel.ComponentHost.Open()
at Microsoft.ServiceBus.Gateway.Gateway.OnStart(String[] args)
In Services the relevant services have the following statuses:
Service Bus Gateway: Starting
Service Bus Message Broker: Stopped
Service Bus Resource Provider: Stopped
Thanks
If you received this error after updating to NET framework v4.6+ then
Microsoft have a fix for this now, download this Update for Service Bus Server 1.1 (KB3086798) http://www.microsoft.com/en-us/download/details.aspx?id=49496
TL;DR;
Download dropbox.com/s/cb5fro1rv... - thanks Arash Rahimi
Navigate to your SDK tools
sn -Vr Microsoft.Cloud.Common.AzureStorage.dll
gacutil /i Microsoft.Cloud.Common.AzureStorage.dll
This issue was caused by installing the Visual Studio 2015 RC. Thankfully someone else has already experienced this and an answer has been posted.
Please see this answer on How do I report a bug in Windows Server Service Bus?
For whatever reason if the Dropbox link becomes obsolete. I have added it to Github.
Make sure any/all RabbitMQ service(s) are all stopped. (Or anything that uses the AMQP protocol)
I am new to the Azure platform and simply trying to pull my IIS logs into my storage account.
When running locally and using the storage emulator, I can see the logs without issue. However, when deploying the application, the log files are never created and the only container I see in blob storage is "vsdeploy".
I have followed the steps outlined here: http://www.windowsazure.com/en-us/develop/net/common-tasks/diagnostics/
I have imported the diagnostics module in my ServiceDefinition.csdef
...<Imports>
<Import moduleName="Diagnostics" />...
I have created a WebRole.cs class and configured directories for a scheduled transfer every two minutes.
public class WebRole : RoleEntryPoint
{
public override bool OnStart()
{
// For information on handling configuration changes
// see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
DiagnosticMonitorConfiguration diagConfig = DiagnosticMonitor.GetDefaultInitialConfiguration();
diagConfig.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(2.0); //IIS LOGS
DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", diagConfig);
return base.OnStart();
}
}
I have also verified my storage account connection string for cloud deployment is correct.
Still, when I deploy, nothing is ever created in the storage account. What piece am I missing or not configuring correctly that is keeping me from my logs?
Thanks.
I'm using Windows Azure to host my python project and I'm trying to enable the diagnostics without good results.
As I'm using python and not .NET, the only way I can actually configure it is through config files.
Below my config files:
ServiceDefinition.csdef
...
<Imports>
<Import moduleName="Diagnostics" />
</Imports>
...
ServiceConfiguration.Cloud.cscfg
....
<Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="DefaultEndpointsProtocol=https;AccountName=<my-account-name>;AccountKey=<my-account-key"/>
....
diagnostics.wadcfg:
<DiagnosticMonitorConfiguration xmlns="http://schemas.microsoft.com/ServiceHosting/2010/10/DiagnosticsConfiguration"
configurationChangePollInterval="PT10M"
overallQuotaInMB="1200">
<DiagnosticInfrastructureLogs bufferQuotaInMB="100"
scheduledTransferLogLevelFilter="Warning"
scheduledTransferPeriod="PT5M" />
<Logs bufferQuotaInMB="200"
scheduledTransferLogLevelFilter="Warning"
scheduledTransferPeriod="PT5M" />
<Directories bufferQuotaInMB="600"
scheduledTransferPeriod="PT5M">
<CrashDumps container="wad-crash-dumps" directoryQuotaInMB="200" />
<FailedRequestLogs container="wad-frq" directoryQuotaInMB="200" />
<IISLogs container="wad-iis" directoryQuotaInMB="200" />
</Directories>
<WindowsEventLog bufferQuotaInMB="200"
scheduledTransferLogLevelFilter="Warning"
scheduledTransferPeriod="PT5M">
<DataSource name="System!*" />
</WindowsEventLog>
</DiagnosticMonitorConfiguration>
In Diagnostics Manager, I can't actually see any data.
Thanks.
May i ask where your diagnostics.wadcfg located? For a regular worker role the diagnostics.wadcfg must be in the root folder and because you don't have worker role module in your project the location of the architecture of your role folder is very important. Be sure to have exact same folder structure in your Python application as a regular worker role and then drop the diagnostics.wadcfg in the role root folder. (add that info back to your question to verify)
Do you see a diagnostics configuration XML is created in your Windows Azure Blob storage which is configured in the *.Diagnostics.ConnectionString. This is a check which suggests that the diagnostics component in the Azure role was able to read the provided configuration and could create the configuration XML at destination blob stroage (same Azure Storage will be use to write log Azure Table storage). Please verify.
Finally your diagnostics.wadcfg need some more work. As this is a non .net worker role you have configured IIS logging (do you really have IIS running in worker role? ) and also have System event log scheduled to transfer "warning only" so if there are no warnings. Finally the log transfer time is set to 5 minutes which is long during test.
What i can suggest as below to test if diagnostics is working or not:
Remove the IIS log if you dont have IIS running the Azure VM
Replace event log DataSource from System!* to Application!* and set filter to Info level
Change the log transfer time to less then a minutes
Run the exact same code in Development Fabric with Diagnostics connection string connected to Actual Azure Storage.
Add custom event log in your machine and see if they are transferred within the time limit to Azure Table storage and specific tables are created
Above should help you to troubleshoot the problem.