Retrieving IIS Logs from Azure - iis

I have been trying to get IIS Logs from Azure, and I was able to get to get it going once - now, no matter what I try, I can't get it to transfer logs to my Storage account.
I was trying to do this without re-deploying my code, which after reading around seemed possible. And, as I mentioned, I was successful. But this is driving me insane, it just won't do it anymore. Although, it does create the Queue in my storage account when I start a transfer, but that's all it seems to do.
The basic steps I am doing are:
Adding the storage name and key to my config as "DiagnosticsConnectionString"*.
Setting a DiagnosticMonitorConfiguration for one minute, with a DirectoriesBufferConfiguration.
Starting an OnDemand Transfer with a new queue name.
I've done all of the above both programmatically, and through the cmdaplets for PowerShell. As soon as I start a transfer, it just stays with a status of "Not Yet Published (Do not end/cancel)".
I have tried Logs, Directories and even deleted and recreated my storage account. Nothing seems to be working. It appeared to work when I directly added my storage account info to my role config via the azure portal; after it Updated the deployment I saw the logs. But this is not working anymore. Does anyone have some good advice/material? I just want to transfer my IIS logs to my storage account - I've been at it for days.
Update:*This is my config: . My WebRole.cs contained the following, when it worked:
DiagnosticMonitor.Start("DiagnosticsConnectionString");
I've updated it to start transfers:
var diag = new DiagnosticMonitorConfiguration()
{
ConfigurationChangePollInterval = TimeSpan.FromMinutes(1),
Directories = new DirectoriesBufferConfiguration()
{
ScheduledTransferPeriod = TimeSpan.FromMinutes(1)
},
Logs = new BasicLogsBufferConfiguration()
{
ScheduledTransferLogLevelFilter = LogLevel.Verbose,
ScheduledTransferPeriod = TimeSpan.FromMinutes(1)
}
};
DiagnosticMonitor.Start("DiagnosticsConnectionString", diag);

Change one line:
From
var diag = new DiagnosticMonitorConfiguration()
to
var diag = DiagnosticMonitor.GetDefaultInitialConfiguration()
Afterwords, use the existing objects within the diag and not add your own.
This is my OnStart:
var config = DiagnosticMonitor.GetDefaultInitialConfiguration();
config.DiagnosticInfrastructureLogs.ScheduledTransferLogLevelFilter = LogLevel.Information;
config.DiagnosticInfrastructureLogs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
config.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
DiagnosticMonitor.Start("DiagnosticsConnectionString", config);

It could be that the logs are not being generated, rather than a problem at the download time.
There is a progam called AzureLogFetcher that may help, tips on getting logging to work can be found here

Related

How to get FileTrigger to work with Azure file storage in Webjob

I have a webjob that I have set up to be triggered when a file is added to a directory:
[FileTrigger(#"<DIR>\<dir>\{name}", "*", WatcherChangeTypes.Created, autoDelete: true)] Stream file,
I have it configured:
var config = new JobHostConfiguration
{
JobActivator = new NinjectActivator(kernel)
};
var filesConfig = new FilesConfiguration();
#if DEBUG
filesConfig.RootPath = #"C:\Temp\";
#endif
config.UseFiles(filesConfig);
config.UseCore();
The path is for working locally and I was expecting that commenting out the FilesConfiguration object leaving it default would allow it to pick up the connection string I have set up and trigger when files are added. This does not happen it turns out that by default the RootPath is set to "D:\Home" and produces an InvalidOperationException
System.InvalidOperationException : Path 'D:\home\data\<DIR>\<dir>' does not exist.
How do I get the trigger to point at the File storage area of the storage account I have set up for it. I have tried removing the FilesConfiguration completely from Program.cs in the hope that it would work against the settings but it only produces the same Exception.
System.InvalidOperationException : Path 'D:\home\data\\' does not exist.
When you publish to azure, the default directory is D:\HOME\DATA, so when you run webjob it could not find the path so you get the error message.
How do I get the trigger to point at the File storage area of the storage account I have set up for it.
The connectionstring you have set have two applies: one is used for dashboard logging and the other is used for application functionality (queues, tables, blobs).
It seems that you could not get filetrigger working with azure file storage.
So, if you want to invoke your filetrigger when you create new file, you could go to D:\home\data\ in KUDU to create a DIR folder and then create new .txt file in it.
The output is as below:
BTW, it seems that you'd better not use autoDelete when you create file, if use you will get error like:
NotSupportedException: Use of AutoDelete is not supported when using change type 'Changed'.

Azure WebJobs Connection Strings configuration ( AzureWebJobsDashboard?? )

I'm trying to work with Azure Webjobs, I understand the way its works but I don't understand why I need to use two connection strings, one is for the queue for holding the messages but
why there is another one called "AzureWebJobsDashboard" ?
What its purpose?
And where I get this connection string from ?
At the moment I have one Web App and one Webjob at the same solution, I'm experiment only locally ( without publishing anything ), the one thing I got up in the cloud is the Storage account that holds the queue.
I even try to put the same connection string in both places ( AzureWebJobsDashboard,AzureWebJobsStorage) but its throw exception :
"Cannot bind parameter 'log' when using this trigger."
Thank you.
There are two connection strings because the WebJobs SDK writes some logs in the storage account. It gives you the possibility of having one storage account just for data (AzureWebJobsStorage) and the another one for logs (AzureWebJobsDashboard). They can be the same. Also, you need two of them because you can have multiple job hosts using different data accounts but sending logs to the same dashboard.
The error you are getting is not related to the connection strings but to one of the functions in your code. One of them has a log parameter that is not of the right type. Can you share the code?
Okay, anyone coming here looking for an actual answer of "where do I get the ConnectionString from"... here you go.
On the new Azure portal, you should have a Storage Account resource; mine starts with "portalvhds" followed by a bunch of alphanumerics. Click that so see a resource Dashboard on the right, followed immediately by a Settings window. Look for the Keys submenu under General -- click that. The whole connection string is there (actually there are two, Primary and Secondary; I don't currently understand the difference, but let's go with Primary, shall we?).
Copy and paste that in your App.config file on the connectionString attribute of the AzureWebJobsDashboard and AzureWebJobsStorage items. This presumes for your environment you only have one Storage Account, and so you want that same storage to be used for data and logs.
I tried this, and at least the WebJob ran without throwing an error.
#RayHAz - Expanding upon your above answer (thanks)...
I tried this https://learn.microsoft.com/en-us/azure/app-service/webjobs-sdk-get-started
but in .Net Core 2.1, was getting exceptions about how it couldn't find the connection string.
Long story short, I ended up with the following, which worked for me:
appsettings.json, in a .Net Core 2.1 Console app:
{
"ConnectionStrings": {
"AzureWebJobsStorage": "---your Azure storage connection string here---",
"AzureWebJobsDashboard":"---the same connectionstring---"
}
}
... and my Program.cs file...
using System;
using System.IO;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
namespace YourWebJobConsoleAppProjectNamespaceHere
{
public class Program
{
public static IConfiguration Configuration;
static void Main(string[] args)
{
var builder = new ConfigurationBuilder()
.SetBasePath(Path.Combine(AppContext.BaseDirectory))
.AddJsonFile("appsettings.json", true);
Configuration = builder.Build();
var azureWebJobsStorageConnectionString = Configuration.GetConnectionString("AzureWebJobsStorage");
var azureWebJobsDashboardConnectionString = Configuration.GetConnectionString("AzureWebJobsDashboard");
var config = new JobHostConfiguration
{
DashboardConnectionString = azureWebJobsDashboardConnectionString,
StorageConnectionString = azureWebJobsStorageConnectionString
};
var loggerFactory = new LoggerFactory();
config.LoggerFactory = loggerFactory.AddConsole();
var host = new JobHost(config);
host.RunAndBlock();
}
}
}

how to view azure diagnostics Log

I am not able to find out how to see azure diagnostics logs. Code which I wrote is as follows.
DiagnosticMonitorConfiguration config = iagnosticMonitor.GetDefaultInitialConfiguration();
System.Diagnostics.Trace.Listeners.Add(new Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener());
config.Logs.ScheduledTransferLogLevelFilter = LogLevel.Information;
config.WindowsEventLog.ScheduledTransferPeriod = System.TimeSpan.FromMinutes(1.0);
DiagnosticMonitor.Start("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString", config);
then I added Trace.WriteLine("some message"); in the code. Now where do I find those messages. I checked in Visual Studio server explorer, where I added my storage account reference. wad-control-container has only config files.
You might want to take a look at this blog post by Michael S. Collier. A setting in your cloud project might cause the logs to end up in an other place than you would expect them to:
http://michaelcollier.wordpress.com/2012/04/02/where-is-my-windows-azure-diagnostics-data/
Update:
Note that you'll have to take care of every small detail to make everything work.
Where are you writing Trace.WriteLine("some message"); ? Is it in your WebRole.cs? If that's the case, you'll need to configure the trace listener for the WebRole.cs (this runs in an other process, different than your actual web application).
Here is an example how you can set up the trace listener in the WebRole.cs class:
System.Diagnostics.Trace.Listeners.Add(new Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener());
System.Diagnostics.Trace.AutoFlush = true;
After setting that up, you can use Trace.WriteLine.
This is a good step by step on how to enable diagnostics:
http://www.windowsazure.com/en-us/develop/net/common-tasks/diagnostics/?sec=commontasks
I wrote a tool that allows you to view Azure Diagnostic information.. check it out
AzTools - Azure Diagnostic Viewer
Click here to see how you could use this tool

Azure Diagnostics - runtime def vs. wadcfg

I'm trying to understand the various ways to configure the Diagnostics in Windows Azure.
So far I've set a diagnostics.wadcfg that is properly used by Azure as I retrieve its content in the xml blob stored by Diagnostics in the wad-control-container (and the tables are updated at the correct refresh rate).
Now I would like to override some fields from the cscfg, in order to boost the log transfer period for all instances, for example (without having to update each wad-control-container file, which will be erased in case of instance recycle btw).
So in my WebRole.Run(), I get a parameter from RoleEnvironment.GetConfigurationSettingValue() and try to apply it to the current config ; but my problem is that the values I read from DiagnosticMonitor.GetDefaultInitialConfiguration() do not correspond to the content of my diagnostics.wadcfg, and setting new values in there doesn't seem to have any effect.
Can anyone explain the relationship between what's taken from diagnostics.wadcfg and the values you can set at run-time?
Thanks
GetDefaultInitialConfiguration() will not return you your current settings, becasue as its name states it takes a default configuration. You have to use the GetCurrentConfiguration method if you need to take the configuration that is in place.
However, if you need to just boost the transfer, you could use for example the Cerebrata's Azure Diagnostics Manager to quickly kick off on-demand transfer of your roles.
You could also use the Windows Azure Diagnostics Management cmdlets for powershell. Check out this article.
Hope this helps!
In order to utilize values in wadcfg file the following code code could be used to access current DiagnosticsMonitorConfiguration:
var cloudStorageAccount = CloudStorageAccount.Parse(
RoleEnvironment.GetConfigurationSettingValue(WADStorageConnectionString));
var roleInstanceDiagnosticManager = cloudStorageAccount.CreateRoleInstanceDiagnosticManager(
RoleEnvironment.DeploymentId,
RoleEnvironment.CurrentRoleInstance.Role.Name,
RoleEnvironment.CurrentRoleInstance.Id);
var dmc = roleInstanceDiagnosticManager.GetCurrentConfiguration();
// Set different logging settings
dmc.Logs....
dmc.PerformanceCounters....
// don't forget to update
roleInstanceDiagnosticManager.SetCurrentConfiguration(dmc);
The code by Boris Lipshitz doesn't work now (Breaking Changes in Windows Azure Diagnostics (SDK 2.0)): "the DeploymentDiagnosticManager constructor now accepts a connection string to the storage account instead of a CloudStorageAccount object".
Updated code for SDK 2.0+:
var roleInstanceDiagnosticManager = new RoleInstanceDiagnosticManager(
// Add StorageConnectionString to your role settings for this to work
CloudConfigurationManager.GetSetting("StorageConnectionString"),
RoleEnvironment.DeploymentId,
RoleEnvironment.CurrentRoleInstance.Role.Name,
RoleEnvironment.CurrentRoleInstance.Id);
var dmc = roleInstanceDiagnosticManager.GetCurrentConfiguration();
// Set different logging settings
dmc.Logs....
dmc.PerformanceCounters....
// don't forget to update
roleInstanceDiagnosticManager.SetCurrentConfiguration(dmc)

How-to: Create role instances on emulator

How do I create new instances of some role via C# using Azure emulator? Is there some guide about that? There are some manuals about creating instances in the cloud, not in emulator.
So far I know that:
I need to change config-file. Is it config in sln-file or in some temp-delpoyment folder?
I need to use csrun tool. How to pick params?
UPD
Got it.
To change count or instances on emulator, you have to:
update 'ServiceConfiguration.cscfg' file in bin-folder
run 'csrun' tool with params: string.Format("/update:{0};\"{1}\"", deploymentId, "<path to ServiceConfiguration.cscfg>")
where deploymentId:
// get id from RoleEnvironment with regex
var patternt = Regex.Escape("(") + #"\d+" + Regex.Escape(")");
var input = RoleEnvironment.DeploymentId;
var m = Regex.Match(input, patternt);
var deploymentId = m.ToString().Replace("(", string.Empty).Replace(")", string.Empty);
If you have troubles running csrun via code, read this:
http://social.msdn.microsoft.com/Forums/en/windowsazuredevelopment/thread/62ca1372-2388-4181-9dbd-8fbba470ea77
In local emulator, you need to modify the CSCFG file under the deployment .csx folder, instead of your source code folder, since the local emulator will fire your application up from that folder.
Once you modified the saved your CSCFG file, for example the count of the instances you can retrieve the new value from your code immediately. But if you want the local emulator detect this changes and perform the related actions, such as increase the VMs or invoke the Configuration_Changed method, you need to execute
csrun /update:;
You can retrieve the deployment id from the compute emulator UI.
You can find the instance count in the ServiceConfiguration.cscfg in your Azure project

Resources