I have a wcf service hosted on azure. When I deploy it and start the autoscaler object the web service roles are constantly being recycled and in an unhealthy state. If I do not start the autoscaler I have no issues, however I would like to use WASABi.
Here is my WebRole.cs
ublic class WebRole : RoleEntryPoint
{
private Autoscaler autoscaler;
public override bool OnStart()
{
// To enable the AzureLocalStorageTraceListner, uncomment relevent section in the web.config
DiagnosticMonitorConfiguration diagnosticConfig = DiagnosticMonitor.GetDefaultInitialConfiguration();
diagnosticConfig.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(1);
diagnosticConfig.Directories.DataSources.Add(AzureLocalStorageTraceListener.GetLogDirectory());
// For information on handling configuration changes
// see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
autoscaler = EnterpriseLibraryContainer.Current.GetInstance<Autoscaler>();
autoscaler.Start();
return base.OnStart();
}
public override void OnStop()
{
autoscaler.Stop();
}
}
Have you tried using IntelliTrace to diagnose the reason for recycling? Here is a good article descirbing how to setup and trouble shoot with IntelliTrace.
What does your Run method look like? It needs to keep the role alive so it should be something like this:
public override void Run()
{
Trace.TraceInformation("ScalerRole entry point called", "Information");
while (true)
{
Thread.Sleep(100000);
Trace.TraceInformation("Working", "Information");
}
}
Related
I am maintaining a legacy Cloud Services application hosted on Azure targeting .net 4.6.1. Inside the Application_Start method of the Global.asax on the Web Role we are registering an event handler for RoleEnvironment.StatusCheck however our logs are demonstrating that this event call back is never being called or triggered.
According to this blog: https://convective.wordpress.com/2010/03/18/service-runtime-in-windows-azure/ we were expecting this event to be triggered every 15 seconds and we believe this was happening however has since stopped. We expect that the stopped working around the time we installed some new DLLs into the solution (some of these dlls include: Microsoft.Rest.ClientRuntime.dll, Microsoft.Azure.Storage.Common.dll, Microsoft.Azure.Storage.Blob.dll, Microsoft.Azure.KeyVault.dll)
We've tried RDP-ing onto the VM to check the event logs but nothing obvious is there. Any suggestions on where we may be able to search for clues?
It seems your event handler is not registered. Try below code with a different approach:
public class WorkerRole : RoleEntryPoint
{
public override bool OnStart()
{
RoleEnvironment.StatusCheck += RoleEnvironmentStatusCheck;
return base.OnStart();
}
// Use the busy object to indicate that the status of the role instance must be Busy
private volatile bool busy = true;
private void RoleEnvironmentStatusCheck(object sender, RoleInstanceStatusCheckEventArgs e)
{
if (this.busy)
{
// Sets the status of the role instance to Busy for a short interval.
// If you want the role instance to remain busy, add code to
// continue to call the SetBusy method
e.SetBusy();
}
}
public override void Run()
{
Trace.TraceInformation("Worker entry point called", "Information");
while (true)
{
Thread.Sleep(10000);
}
}
public override void OnStop()
{
base.OnStop();
}
}
I wrote a custom logger for Application Insights in my app. I don't see any exceptions or ANY events when viewing App Insights in Azure Portal. Here is the logger class code, when I debug the code I do see a key assigned to the InstrumentationKey property, any ideas what I am doing wrong here? Do I need to attach other info to the client or configuration?
public class AppInsightsLogger:ILogger
{
private TelemetryClient ai;
public AppInsightsLogger()
{
ai = new TelemetryClient();
if (string.IsNullOrEmpty(ai.InstrumentationKey))
{
// attempt to load instrumentation key from app settings
var appSettingsTiKey = AppSettings.InsightsKey;
if (!string.IsNullOrEmpty(appSettingsTiKey))
{
TelemetryConfiguration.Active.InstrumentationKey = appSettingsTiKey;
ai.InstrumentationKey = appSettingsTiKey;
}
else
{
throw new Exception("Could not find instrumentation key for Application Insights");
}
}
}
public void LogException(Exception ex)
{
ai.TrackException(ex);
}
}
I created a new Console Application, installed latest stable ApplicationInsights SDK and pretty much kept your example, with minor but important difference - I either let it wait before shutting down after calling TrackException or added TelemetryClient.Flush()
namespace logtest
{
class Program
{
static void Main(string[] args)
{
AppInsightsLogger logger = new AppInsightsLogger();
logger.LogException(new InvalidOperationException("Is data showing?"));
// either wait for a couple of minutes for the batch to be sent of add ai.Flush() after ai.TrackException() to send the batch immediately
Console.ReadLine();
}
}
public class AppInsightsLogger
{
private TelemetryClient ai;
public AppInsightsLogger()
{
ai = new TelemetryClient();
if (string.IsNullOrEmpty(ai.InstrumentationKey))
{
// attempt to load instrumentation key from app settings
var appSettingsTiKey = "<ikey>";
if (!string.IsNullOrEmpty(appSettingsTiKey))
{
TelemetryConfiguration.Active.InstrumentationKey = appSettingsTiKey;
ai.InstrumentationKey = appSettingsTiKey;
}
else
{
throw new Exception("Could not find instrumentation key for Application Insights");
}
}
}
public void LogException(Exception ex)
{
ai.TrackException(ex);
// ai.Flush();
}
}
}
First I could see telemetry item sent in Visual Studio debug output window:
Then I could see telemetry leaving my machine in Fiddler, I also could see it was successfully accepted by our data collection endpoint.
And finally I could see it in the portal:
Nowadays, the question and the accepted answer are outdated. The contemporary approach is to add Microsoft.ApplicationInsights.AspNet NuGet package, which has logging out-of-the-box.
Please refer to the official docs.
The gist from the docs:
ApplicationInsightsLoggerProvider is enabled by default.
Captured log types can be configured in appsettings.json (or programmatically if you really need it)
"Logging": {
"LogLevel": {
"Default": "Warning"
},
"ApplicationInsights": {
"LogLevel": {
"Default": "Error"
}
}
}
For a version 3 webjob, I tried instantiating a TelemetryClient and calling TrackException(), as well as calling Flush() and waiting up to 180 seconds, but none of that worked. What does work is using the ILogger logger object and passing the exception into LogError().
I am working with an Azure Service Bus Queue (or potentially a topic if required), and would like to know how a Web Job can be used with the Queue.
When a message comes onto the queue it represents a process that will run within the web job (or be started from the webjob). This process might be quick, 30 seconds, or it might be slow, 1 hour etc.
Can I use a single Web Job for this and somehow say that it should be running no more than 10 of these processes at a time?
Yes you can use a WebJob. I have created a simple WebJob with Storage Queue to just guide how it can be done. The below workflow will run only ten process at a time and keep all the other requests in memory of ConcurrentQueue. You will have to implement the logic to dequeue it and consume it
public class Functions
{
public delegate void CompletedProcessHandler(object sender, CompletedProcessHandlerArgs args);
static readonly Dictionary<int, CustomProcess> _dictionary =
new Dictionary<int, CustomProcess>();
static readonly ConcurrentQueue<ProcessEntity> _remaining =
new ConcurrentQueue<ProcessEntity>();
// This function will get triggered/executed when a new message is written
// on an Azure Queue called queue.
public static void ProcessQueueMessage([QueueTrigger("testqueue")] ProcessEntity msg,
TextWriter log)
{
if (_dictionary.Count <= 10)
{
var newProcess = new CustomProcess((_dictionary.Last().Key) + 1,
msg.Duration);
}
else
{
_remaining.Enqueue(msg);
}
}
public static void CompletedProcess(object sender, CompletedProcessHandlerArgs args)
{
_dictionary[Int32.Parse(args.ProcessID)].Dispose();
_dictionary.Remove(Int32.Parse(args.ProcessID));
}
}
public class CustomProcess : IDisposable
{
public event Functions.CompletedProcessHandler OnProcessCompleted;
private CancellationTokenSource _token;
private string _id;
private Timer _timer;
public CustomProcess(int i, int duration)
{
_timer = new Timer { Enabled = true, Interval = duration * 1000 };
_timer.Elapsed += Timer_Elapsed;
_id = i.ToString();
_token = new CancellationTokenSource();
Task.Factory.StartNew(() => WriteMessages());
_timer.Start();
OnProcessCompleted += Functions.CompletedProcess;
}
private void Timer_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
{
_token.Cancel();
OnProcessCompleted?.Invoke(this, new CompletedProcessHandlerArgs(_id));
}
private void WriteMessages()
{
while (!_token.Token.IsCancellationRequested)
{
Console.WriteLine("Test Message from process " + _id);
}
}
public void Dispose()
{
_token.Dispose();
_timer.Dispose();
}
}
public class CompletedProcessHandlerArgs : EventArgs
{
public string ProcessID { get; set; }
public CompletedProcessHandlerArgs(string ID)
{
ProcessID = ID;
}
}
public class ProcessEntity
{
public int Duration { get; set; }
}
In the app.config of the web job you need to provide the two app settings
<add name="AzureWebJobsDashboard"
connectionString="DefaultEndpointsProtocol=https;AccountName=[AccountName];AccountKey=[AccountKey]" />
<add name="AzureWebJobsStorage"
connectionString="DefaultEndpointsProtocol=https;AccountName=[AccountName];AccountKey=[AccountKey]" />
The Program file is the default one from the Visual Studio template
public class Program
{
// Please set the following connection strings in app.config for this WebJob to run:
// AzureWebJobsDashboard and AzureWebJobsStorage
static void Main()
{
var host = new JobHost();
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
}
WebJob will keep dequeue the message the moment it comes. Since you want only 10 to run at a time you will have to enqueue the message in memory and wait for running process to complete before you start a new one
As #Rick has mentioned you can set the is_Singleton property to true in settings.job file of the web job
Yes, you can trigger a web job with an Azure Service Bus Queue or Topic. A good example to look at to get you going would be the Service Bus quick start project template in Visual Studio.
In particular, you want to look at the ServiceBusTrigger attribute that the Web Jobs SDK provides.
As for the scalability of the web job, this will scale according to your web app instances. So, if you had say 5 instances of your web app with always on enabled, then you would have 5 instances of your web job. As an additional comment on this, if you wanted just one instance of the web job in an environment of 5 web app instances, then you could set the is_singleton property to true in the settings.job file.
Azure role have RoleEnvironment.Stopping event that is raised when the role is being stopped. I discovered some issue in some unrelated code that needs special treatment in cases when the role is being stopped. Something like:
public class SomeFarAwayClass {
void someFarAwayFunction()
if( roleIsBeingStopped ) {
workSpecially();
} else {
workUsually();
}
}
}
Now I want to subscribe to RoleEnvironment.Stopping and in the event handler raise the roleIsBeingStopped permanently. Something like this:
public class SomeFarAwayClass {
//
private static bool roleIsBeingStopped = false;
public void SetBeingStopped() { roleIsBeingStopped = true; }
}
class MyRoleClass : RoleEntryPoint {
overribe bool OnStart()
{
RoleEnvironment.Stopping += stopping;
return base.OnStart();
}
void stopping(object sender, RoleEnvironmentStoppingEventArgs args)
{
SomeFarAwayClass.SetBeingStopped();
}
}
This solution implies that the role is never restarted in the same process, otherwise I'll need to reset the flag at some point. So far I've never seen Azure roles being restarted in the same process, it's a new process every time.
Can I be sure that once Azure role Stopping has been raised the role is never rerun in the same process?
I think you probably can, but at the same time you don't need to, because you also have the OnStart call you could use to re-set the flag.
I generally prefer to not rely on thing out of my control where I don't have to (of which there are many!), this would be one I'd avoid personally.
I have a simple WebRole class in my Azure solution:
public class WebRole : RoleEntryPoint
{
public override bool OnStart()
{
DiagnosticMonitor.Start("DiagnosticsConnectionString");
// For information on handling configuration changes
// see the MSDN topic at http://go.microsoft.com/fwlink/?LinkId=166357.
RoleEnvironment.Changing += RoleEnvironmentChanging;
CloudStorageAccount.SetConfigurationSettingPublisher(
(configName, configSettingPublisher) =>
{
string connectionString = RoleEnvironment.GetConfigurationSettingValue(configName);
configSettingPublisher(connectionString);
}
);
return base.OnStart();
}
// ...
}
For some reason, I can't get breakpoints in OnStart() to be hit when I run the project. Why might this be?
One possibility is that your startup project is your web application instead of the cloud project. Make sure that the cloud project is the startup project, and then verify your app is running in the development fabric.