I have an Azure webjob that is triggered by a queue (inherited, not originally written by me). The queue usually only has one item placed on it at a time, but on the first of every month has many items.
On these occasions it has always processed one queue item at a time, finishing processing before picking up the next.
I noticed, however, that as of a couple of months ago it started processing two files at any one time, which is causing problems.
Whilst i could refactor the code to allow for this, I really don't have the time, and the return would be minimal. I simply want it to process one item at a time again, but i cant find anything that may have caused this to change.
Are there any settings in the azure portal I should be aware of? I don't believe any code relating to the trigger itself has changed.
Thank you in advance
Sure, this can be done. Note that a WebJob can be triggered by either a Service Bus Queue or an Azure Storage Queue. Here's info for both.
For Azure Storage Queues
By default, a QueueTrigger will grab 16 messages at a time and process them in parallel. If you don't want this, you need to set the JobHostConfiguration instance's BatchSize property to 1 in your WebJob's static void Main method. Example:
static void Main(string[] args)
{
JobHostConfiguration config = new JobHostConfiguration();
config.Queues.BatchSize = 8;
JobHost host = new JobHost(config);
host.RunAndBlock();
}
For Service Bus Queues
Similarly, you'll set properties in the JobHostConfiguration. If you're using Service Bus, there's a little more setup. Example:
static void Main(string[] args)
{
JobHostConfiguration config = new JobHostConfiguration();
ServiceBusConfiguration serviceBusConfig = new ServiceBusConfiguration();
serviceBusConfig.MessageOptions.MaxConcurrentCalls = 1;
config.UseServiceBus(serviceBusConfig);
JobHost host = new JobHost(config);
host.RunAndBlock();
}
In my case, the problem was that someone had scaled out the Azure app service to 2 instances. However, I'm marking Rob's answer as the most helpful.
Related
I already have a WebJob that I created in .NET Core 2.1 which also uses DI. This particular WebJob runs continuously and the new one I'm trying to create will run at set intervals.
In the following code, I'm telling the WebJob to run continuously and need to remove those lines but I want to make sure I'm doing it right. I'm putting //REMOVE on the lines that I need to remove. Could someone please verify I'm doing it right?
Again, the idea is to create a WebJob that will run at set intervals so I need to remove the lines that indicate a continuously running WebJob.
static void Main(string[] args)
{
IServiceCollection serviceCollection = new ServiceCollection();
ConfigureServices(serviceCollection);
var configuration = new JobHostConfiguration();
configuration.Queues.MaxPollingInterval = TimeSpan.FromSeconds(1); // REMOVE
configuration.Queues.BatchSize = 1; // REMOVE
configuration.JobActivator = new CustomJobActivator(serviceCollection.BuildServiceProvider());
configuration.UseTimers();
var host = new JobHost(configuration);
host.RunAndBlock(); // REMOVE
}
private static void ConfigureServices(IServiceCollection services)
{
var configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
.Build();
// Resolve repositories
services.AddTransient<IMyRepository, MyRepository>();
// Create instances of clients
services.AddSingleton(new MyCustomClient(configuration));
// Azure connection strings for the WebJob
Environment.SetEnvironmentVariable("AzureWebJobsDashboard", configuration.GetConnectionString("WebJobsDashboard"));
Environment.SetEnvironmentVariable("AzureWebJobsStorage", configuration.GetConnectionString("WebJobsStorage"));
}
I'm putting //REMOVE on the lines that I need to remove. Could someone please verify I'm doing it right?
AFAIK, the code lines you want to remove could not tell webjob to run at set intervals.
config.Queues.MaxPollingInterval = TimeSpan.FromSeconds(1);
MaxPollingInterval is the max amount of time the WebJob will check the queue. If the queue is empty the WebJob will start checking less frequently up to a max of 10 min.
config.Queues.BatchSize = 2; //the amount of items your WebJob will process at the same time
For more details, you could refer to this article about webjob JobHostConfiguration.
When using the Azure WebJobs SDK you can use TimerTrigger to declare job functions that run on a schedule.
public static void StartupJob(
[TimerTrigger("0 0 */2 * * *", RunOnStartup = true)] TimerInfo timerInfo)
{
Console.WriteLine("Timer job fired!");
}
You can get TimerTrigger and other extensions by installing the Microsoft.Azure.WebJobs.Extensions nuget package.
When using the TimerTrigger, be sure to add a call to config.UseTimers() to your startup code to register the extension.
config.UseTimers(); //allows us to use a timer trigger in our functions.
When using the Azure WebJobs SDK, you deploy your code to a Continuous WebJob, with AlwaysOn enabled. You can then add many scheduled functions you desire in that WebJob.
I have a Webjob that I want to be time triggered:
public class ArchiveFunctions
{
private readonly IOrderArchiver _orderArchiver;
public ArchiveFunctions(IOrderArchiver orderArchiver)
{
_orderArchiver = orderArchiver;
}
public async Task Archive([TimerTrigger("0 */5 * * * *")] TimerInfo timer, TextWriter log)
{
log.WriteLine("Hello world");
}
}
My program.cs:
public static void Main()
{
var config = new JobHostConfiguration
{
JobActivator = new AutofacJobActivator(RegisterComponents())
};
config.UseTimers();
var host = new JobHost(config);
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
my publish-setting.json:
{
"$schema": "http://schemastore.org/schemas/json/webjob-publish-settings.json",
"webJobName": "OrdersArchiving",
"runMode": "OnDemand"
}
Here is what it looks like on azure portal:
My problem is that the job runs, I have the hello world, but the job keeps in run state and it get to a time out error message:
[02/05/2018 15:34:05 > f0ea5f: ERR ] Command 'cmd /c ""Ores.Contr ...' was aborted due to no output nor CPU activity for 121 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.
What can I do to fix this?
I have a wild guess RunAndBlock could be a problem.. but I do not see a solution..
Thanks!
Edit:
I have tested Rob Reagan answer, it does help with the error, thank you!
On my same service, I have one other time triggerd job (was done in core, while mine is not).
You can see the Webjob.Missions is 'triggered', and status update on last time it ran. You can see as well the schedule on it.
I would like to have the same for mine 'OrdersArchiving'.
How can I achieve that?
Thanks!
Change your run mode to continuous and not triggered. The TimerTrigger will handle executing the method you've placed it on.
Also, make sure that you're not using a Free tier for hosting your WebJob. After twenty minutes of inactivity, the app will be paused and will await a new HTTP request to wake it up.
Also, make sure you've enabled Always On on your Web App settings to prevent the same thing from happening to a higher service tier web app.
Edit
Tom asked how to invoke methods on a schedule for a Triggered WebJob. There are two options to do so:
Set the job up as triggered and use a settings.json file to set up the schedule. You can read about it here.
Invoke a method via HTTP using an Azure Scheduler. The Azure Scheduler is a separate Azure service that you can provision. It has a free tier which may be sufficient for your use. Please see David Ebbo's post on this here.
Is there any way to configure triggers without attributes? I cannot know the queue names ahead of time.
Let me explain my scenario here.. I have one service bus queue, and for various reasons (complicated duplicate-suppression business logic), the queue messages have to be processed one at a time, so I have ServiceBusConfiguration.OnMessageOptions.MaxConcurrentCalls set to 1. So processing a message holds up the whole queue until it is finished. Needless to say, this is suboptimal.
This 'one at a time' policy isn't so simple. The messages could be processed in parallel, they just have to be divided into groups (based on a field in message), say A and B. Group A can process its messages one at a time, and group B can process its own one at a time, etc. A and B are processed in parallel, all is good.
So I can create a queue for each group, A, B, C, ... etc. There are about 50 groups, so 50 queues.
I can create a queue for each, but how to make this work with the Azure Webjobs SDK? I don't want to copy-paste a method for each queue with a different ServiceBusTrigger for the SDK to discover, just to enforce one-at-a-time per queue/group, then update the code with another copy-paste whenever another group is needed. Fetching a list of queues at startup and tying to the function is preferable.
I have looked around and I don't see any way to do what I want. The ITypeLocator interface is pretty hard-set to look for attributes. I could probably abuse the INameResolver, but it seems like I'd still have to have a bunch of near-duplicate methods around. Could I somehow create what the SDK is looking for at startup/runtime?
(To be clear, I know how to use INameResolver to get queue name as at How to set Azure WebJob queue name at runtime? but though similar this isn't my problem. I want to setup triggers for multiple queues at startup for the same function to get the one-at-a-time per queue processing, without using the trigger attribute 50 times repeatedly. I figured I'd ask again since the SDK repo is fairly active and it's been a year..).
Or am I going about this all wrong? Being dumb? Missing something? Any advice on this dilemma would be welcome.
The Azure Webjob Host discovers and indexes the functions with the ServiceBusTrigger attribute when it starts. So there is no way to set up the queues to trigger at the runtime.
The simpler solution for you is to create a long time running job and implement it manually:
public class Program
{
private static void Main()
{
var host = new JobHost();
host.CallAsync(typeof(Program).GetMethod("Process"));
host.RunAndBlock();
}
[NoAutomaticTriggerAttribute]
public static async Task Process(TextWriter log, CancellationToken token)
{
var connectionString = "myconnectionstring";
// You can also get the queue name from app settings or azure table ??
var queueNames = new[] {"queueA", "queueA" };
var messagingFactory = MessagingFactory.CreateFromConnectionString(connectionString);
foreach (var queueName in queueNames)
{
var receiver = messagingFactory.CreateMessageReceiver(queueName);
receiver.OnMessage(message =>
{
try
{
// do something
....
// Complete the message
message.Complete();
}
catch (Exception ex)
{
// Log the error
log.WriteLine(ex.ToString());
// Abandon the message so that it can be retry.
message.Abandon();
}
}, new OnMessageOptions() { MaxConcurrentCalls = 1});
}
// await until the job stop or restart
await Task.Delay(Timeout.InfiniteTimeSpan, token);
}
}
Otherwise, if you don't want to deal with multiple queues, you can have a look at azure servicebus topic/subscription and create SqlFilter to send your message to the right subscription.
Another option could be to create your own trigger: The azure webjob SDK provides extensibility points to create your own trigger binding :
Binding Extensions Overview
Good Luck !
Based on my understanding, your needs seems to be building a message batch system in parallel. The #Thomas solution is good, but I think Azure Batch service with Table storage may be better and could be instead of the complex solution of ServiceBus queue + WebJobs with a trigger.
Using Azure Batch with Table storage, you can control the task creation and execute the task in parallel and at scale, even monitor these tasks, please refer to the tutorial to know how to.
I am developing a triggered webjob that use TimerTrigger.
Before the webjob stops, I need to dispose some objects but I don't know how to trigger the "webjob stop".
Having a NoAutomaticTrigger function, I know that I can use the WebJobsShutdownWatcher class to handle when the webjob is stopping but with a triggered job I need some help...
I had a look at Extensible Triggers and Binders with Azure WebJobs SDK 1.1.0-alpha1.
Is it a good idea to create a custom trigger (StopTrigger) that used the WebJobsShutdownWatcher class to fire action before the webjob stops ?
Ok The answer was in the question :
Yes I can use the WebJobsShutdownWatcher class because it has a Register function that is called when the cancellation token is canceled, in other words when the webjob is stopping.
static void Main()
{
var cancellationToken = new WebJobsShutdownWatcher().Token;
cancellationToken.Register(() =>
{
Console.Out.WriteLine("Do whatever you want before the webjob is stopped...");
});
var host = new JobHost();
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
EDIT (Based on Matthew comment):
If you use Triggered functions, you can add a CancellationToken parameter to your function signatures. The runtime will cancel that token when the host is shutting down automatically, allowing your function to receive the notification.
public static void QueueFunction(
[QueueTrigger("QueueName")] string message,
TextWriter log,
CancellationToken cancellationToken)
{
...
if(cancellationToken.IsCancellationRequested) return;
...
}
I was recently trying to figure out how to do this without the
WebJobs SDK which contains the WebJobShutdownWatcher, this is what I
found out.
What the underlying runtime does (and what the WebJobsShutdownWatcher referenced above checks), is create a local file at the location specified by the environment variable %WEBJOBS_SHUTDOWN_FILE%. If this file exists, it is essentially the runtime's signal to the webjob that it must shutdown within a configurable wait period (default of 5 seconds for continuous jobs, 30 for triggered jobs), otherwise the runtime will kill the job.
The net effect is, if you are not using the Azure WebJobs SDK, which contains the WebJobsShutdownWatcher as described above, you can still achieve graceful shutdown of your Azure Web Job by monitoring for the shutdown file on an interval shorter than the configured wait period.
Additional details, including how to configure the wait period, are described here: https://github.com/projectkudu/kudu/wiki/WebJobs#graceful-shutdown
I would like to have my queue retry failed webjobs every 90 minutes and only for 3 attempts.
When creating the queue i use the following code
CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();
IRetryPolicy linearRetryPolicy = new LinearRetry(TimeSpan.FromSeconds(5400), 3);
queueClient.DefaultRequestOptions.RetryPolicy = linearRetryPolicy;
triggerformqueue = queueClient.GetQueueReference("triggerformqueue");
triggerformqueue.CreateIfNotExists();
However when simulating a failed webjob attempt the queue uses the default retry policy.
I'm i missing something.
I think you might be thinking about this backwards. Queues don't actually perform behavior. Instead what I am guessing you want to do is have a web job that is configured to pull messages from a queue and then if it fails to process the message from a queue for some reason have the web job retry 90 minutes later. In this case you just need to set the invisibility timeout to be 90 minutes (default is 30 seconds) which will ensure that if the message isn't fully processed (ie - GetMessage and DeleteMessage are both called) then the message will reappear on the queue 90 minutes later.
Take a look at this Getting Started with Queue Storage document for more information.
There is something like Azure WebJobs SDK Extensions and ErrorTriggerAttribute (it isn't yet available in nuget 1.0.0-beta1 package, but you have access to public repository)
public static void ErrorMonitor(
[ErrorTrigger("0:30:00", 10, Throttle = "1:00:00")] TraceFilter filter,
TextWriter log)
https://github.com/Azure/azure-webjobs-sdk-extensions#errortrigger
You need to use your RetryPolicy when you add an item to the queue, not on the queue itself, eg.
var queue = queueClient.GetQueueReference("myQueue");
queue.CreateIfNotExists();
options = new QueueRequestOptions { RetryPolicy = linearRetryPolicy };
await queue.AddMessageAsync(yourMessage, null, new TimeSpan(0, delayMinutes, 0), options, null);