I have an WebApp running on Azure, the purpose of the WebApp is to process jobs from an Azure Storage Queue. It is set to Always On. The problem is that when I stop the WebApp in the Azure management portal the job doesn't actually stop.
I instantiated the job like:
class Program
{
static void Main()
{
string connectionString = ConfigurationManager.ConnectionStrings["StorageConnectionString"].ConnectionString;
JobHostConfiguration config = new JobHostConfiguration(connectionString);
config.NameResolver = new QueueNameResolver();
JobHost host = new JobHost(config);
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
}
The function that processes the queue is:
public static void ProcessQueueMessage([QueueTrigger("%apilogeventqueue%")] CloudQueueMessage messageEnvelope, TextWriter log, CancellationToken token)
{
}
I have trace logging so I can see the job never actually gets stopped.
WebJobs run under a special SCM (source control management) site of the host WebApp. When you stop the WebApp in the portal this SCM portion of the WebApp continues to run, because it supplies management capabilities to the WebApp that need to be available even when stopped.
To stop a WebJob from running, you can disable it in the portal UI by right clicking on the job in the jobs list and choosing "Stop". You can also stop ALL jobs in the WebApp by setting the WEBJOBS_STOPPED app setting to 1 (details here).
Related
I was configured the Multi-step web test for monitoring the Azure Web Jobs health using Azure Application Insights by following this documentation. But this multi-step web test will check only the status of Azure Web Job, whether it is “Running, Failed and Aborted”.
Sometimes, Azure Web Job was Aborted. But the job runs inside it. So, I need to monitor the status of Azure Web Job based on error in the logs like shown in below figure using Multi-step web test.
You could use Application Insights Integration to implement it. The LogCategoryFilter has a Default property with an initial value of Information, meaning that any messages with levels of Information, Warning, Error or Critical will be logged.
You need three packages in total:
Microsoft.Azure.WebJobs.Logging.ApplicationInsights
Microsoft.Extensions.Logging
Microsoft.Extensions.Logging.Console
Configure the JobHostConfiguration
string instrumentationKey = Environment.GetEnvironmentVariable("APPINSIGHTS_INSTRUMENTATIONKEY");
if (!string.IsNullOrEmpty(instrumentationKey))
{
// build up a LoggerFactory with ApplicationInsights and a Console Logger
config.LoggerFactory = new LoggerFactory().AddApplicationInsights(instrumentationKey, null).AddConsole();
config.Tracing.ConsoleLevel = TraceLevel.Off;
}
Note: Don't forget adding the APPINSIGHTS_INSTRUMENTATIONKEY in your application setting.
I test ProcessQueueMessage webjob.
public static void ProcessQueueMessage([QueueTrigger("myqueue")] string message, ILogger logger)
{
logger.LogInformation("it is a error......");
logger.LogError(new Exception(), "it is a test error...");
}
This is my webjob log.
And this is the Application Insight page. You could find the Information, Warning and Exception are all shown there.
I have an Azure timer trigger webjob, deployed in two different environments with two different storage accounts. The webjobs have been deployed for about two months now, and were running as expected for quite some time. However, for about a week now, the function is being recognised but not being executed.
Here is how the logs look:
Found the following functions:
ADFSchedulerWebJob.Functions.ProcessTimerMessage
Job host started
Nothing happens after the Job host is started, i.e., the function ProcessTimerMessage is not called/executed. This is how the function looks:
public static void ProcessTimerMessage([TimerTrigger("0 */2 * * * *", RunOnStartup = true)] TimerInfo info, TextWriter log)
{
//function logic
}
This is how the Main method looks like:
static void Main()
{
JobHostConfiguration config = new JobHostConfiguration();
config.UseTimers();
var host = new JobHost(config);
host.RunAndBlock();
}
The same is occurring even when I try to debug the webjob locally. What could be the possible reason/resolution?
Any help appreciated.
What could be the possible reason/resolution?
As Thoms mentioned that it seems that the same storage account is used for another webjob or others.
We could debug it with following ways:
1.Try to use another storage account
2.Check the webjob log from the storage
We could get more info about Webjob log from the blog.
My queue trigger works only when the azure queue is being populated with messages and when simultaneously the webjob is running.(in my local dev environment)
But when i start the webjob (having the queue trigger) first , and then after a few seconds put new messages into the queue, the trigger doesn't detect them. Its like the trigger stops listening once there is no new messages. is this a normal behaviour for the trigger ? If not how do i resolve this issue ?
Main method :
static void Main()
{
InitializeQueue();
var config = new JobHostConfiguration();
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}
var host = new JobHost();
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
Function with queue trigger
public static void ProcessQueueMessage([QueueTrigger("myqueue")] CloudQueueMessage message)
{
Debug.Write(message.AsString);
}
For on-demand WebJobs, Azure has a configuration setting called WEBJOBS_IDLE_TIMEOUT that is denoted in seconds and defaults to two minutes.
If your on-demand WebJob isn’t triggered within the WEBJOBS_IDLE_TIMEOUT interval, Azure will kill your WebJob.
I recommend setting your Web App’s WEBJOBS_IDLE_TIMEOUT variable to a healthy number of seconds to prevent this from happening.
You can set this variable on the Web App’s Application Settings screen by adding it to the App Settings section of the Web App instance that is hosting your WebJob.
In my small job, I just use same Storage Account for AzureWebJobsDashboard and AzureWebJobsStorage configs.
But what happened when we use same connection string for local Debugging and Published job equally? Are they treated in isolated manner? Or do they have any conflict issue?
I looked into blobs of published job and found azure-webjobs-dashboad/functions/instances or azure-webjobs-dashboad/functions/recent/by-job-run/{jobname}, or azure-webjobs-hosts/output-logs directories; they have no discriminator specified among jobs while some other directories have GUID with job name.
Note that my job will be run in continuous.
Or do they have any conflict issue?
No, there is no conflict issue. Base on my experience, it is not recommended to local debugging while Published job is running in the azure with the same connection string. Take Azure Storage queue for example, we can't control which queue should be executed in the local or in the azure. If we try to use debug it locally, please have a try to stop the continue WebJob from Azure Portal.
If we try to know WebJob is executed from which instance we could log the instance info in the code with the environment variable WEBSITE_INSTANCE_ID.The following is the code sample:
public static void ProcessQueueMessage([QueueTrigger("queue")] string message, TextWriter log)
{
string instance = Environment.GetEnvironmentVariable("WEBSITE_INSTANCE_ID");
string newMsg = $"WEBSITE_INSTANCE_ID：{instance}, timestamp：{DateTime.Now}, Message：{message}";
log.WriteLine(newMsg);
Console.WriteLine(newMsg);
}
More info please refer to how to use azure queue storage with the WebJob SDK. The following is snipped from the document.
If your web app runs on multiple instances, a continuous WebJob runs on each machine, and each machine will wait for triggers and attempt to run functions. The WebJobs SDK queue trigger automatically prevents a function from processing a queue message multiple times; functions do not have to be written to be idempotent
Update :
About timer trigger we could find more explanation of timer trigger in the WebJob from the GitHub document.So if you want to debug locally, please have a try to stop the WebJob from the azure portal.
only a single instance of a particular timer function will be running across all instances (you don't want multiple instances to process the same timer event).
I want to migrate typical windows service which is written in .net to Azure using Service fabric.
To implement this , I am creating one service fabric application containing one micro service as guest executable which uses .exe of windows service and deploying application package to service fabric cluster.
After deploying service fabric application on cluster I want windows service should install & start automatically on all nodes however at any time application is running on any single node. I want windows service should run on only one node at a time.
Please kindly help to implement this.
You can certainly run your service as a guest executable. Making sure it only runs on one node can be done by setting the instance count to 1 in the manifest, like so:
<Parameters>
<Parameter Name="GuestService_InstanceCount" DefaultValue="-1" />
</Parameters>
...
<DefaultServices>
<Service Name="GuestService">
<StatelessService ServiceTypeName="GuestServiceType"
InstanceCount="[GuestService_InstanceCount]">
<SingletonPartition />
</StatelessService>
</Service>
</DefaultServices>
Or, you could actually migrate it, not just re-host it in the SF environment...
If your Windows Service is written in .NET and the you wan't to benefit from Service Fabric then the job of migrating the code from a Windows Service to a Reliable Service in Service Fabric should not be to big.
Example for a basic service:
If you start by creating a Stateless Service in a Service Fabric application you end up with a service implementation that looks like (comments removed):
internal sealed class MigratedService : StatelessService
{
public MigratedService(StatelessServiceContext context)
: base(context)
{ }
protected override IEnumerable<ServiceInstanceListener> CreateServiceInstanceListeners()
{
return new ServiceInstanceListener[0];
}
protected override async Task RunAsync(CancellationToken cancellationToken)
{
// TODO: Replace the following sample code with your own logic
// or remove this RunAsync override if it's not needed in your service.
long iterations = 0;
while (true)
{
cancellationToken.ThrowIfCancellationRequested();
ServiceEventSource.Current.ServiceMessage(this.Context, "Working-{0}", ++iterations);
await Task.Delay(TimeSpan.FromSeconds(1), cancellationToken);
}
}
The RunAsync method starts running as soon as the Service is up and running on a node in the cluster. It will continue to run until the cluster, for some reason, decides to stop the service, or move it to another node.
In your Windows Service code you should have a method that is run on start. This is usually where you set up a Timer or similar to start doing something on a continuous basis:
protected override void OnStart(string[] args)
{
System.Timers.Timer timer = new System.Timers.Timer();
timer.Interval = 60000; // 60 seconds
timer.Elapsed += new System.Timers.ElapsedEventHandler(this.OnTimer);
timer.Start();
}
public void OnTimer(object sender, System.Timers.ElapsedEventArgs args)
{
...
DoServiceStuff();
Console.WriteLine("Windows Service says hello");
}
Now grab that code in OnTimer and put it in your RunAsync method (and any other code you need):
protected override async Task RunAsync(CancellationToken cancellationToken)
{
while (true)
{
cancellationToken.ThrowIfCancellationRequested();
DoServiceStuff();
ServiceEventSource.Current.ServiceMessage(this.Context,
"Reliable Service says hello");
await Task.Delay(TimeSpan.FromSeconds(60), cancellationToken);
}
}
Note the Task.Delay(...), it should be set to the same interval as your Windows Service had for it's Timer.
Now, if you have logging in your Windows Service and you use ETW, then that should work out of the box for you. You simply need to set up some way of looking at those logs from Azure now, for instance using Log Analytics (https://learn.microsoft.com/en-us/azure/log-analytics/log-analytics-service-fabric).
Other things you might have to migrate is if you run specific code on shut down, on continue, and if you have any parameters sent to the service on startup (for instance connection strings to databases). Those need to be converted to configuration settings for the service, look at SO 33928204 for a starting point for that.
The idea behind service fabric is so that it manages your services, from deployment and running. Once you've deployed your service/application to the service fabric instance it will be just like running a windows service (kinda) so you wont need to install your windows service. If you're using something like TopShelf you can just run the exe and everything will run totally fine within service fabric.