I have a few Azure WebJobs that run to completion, once my business logic is done I call await StopAsync(stoppingToken);
However, Azure Portal continues to show their status as "Running" until eventually the jobs terminated after the default 120 second timeout.
How can I correctly tell Azure Portal/Kudu that the job is in fact finished?
Here is an example that shows the issue:
namespace MyService
{
public class MyService : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
// same issue whether or not I call this:
await StopAsync(stoppingToken);
}
}
}
A week after posting this question my jobs are now mysteriously completing successfully even though no changes were made, I assume someone at Microsoft fixed something on their end.
I have an azure function that runs off of a queue trigger. The repository has method to grab the connection string from the ConnectionStrings collection.
return System.Configuration.ConfigurationManager.ConnectionStrings["MyDataBase"].ToString();
This works great for the most part but I see intermittently that this returns a null exception error.
Is there a way I can make this more robust?
Do azure functions sometimes fail to get the settings?
Should I store the setting in a different section?
I also want to say that this runs thousands of times a day but I see this popup about a 100 times.
Runtime version: 1.0.12299.0
Are you reading the configuration for every function call? You should consider reading it once (e.g. using a Lazy<string> and static) and reusing it for all function invocations.
Maybe there is a concurrency issue when multiple threads access the code. Putting a lock around the code could help as well. ConfigurationManager.ConnectionStrings should be tread-safe, but maybe it isn't in the V1 runtime.
A similar problem was posted here, but this concerned app settings and not connection strings. I don't think using CloudConfigurationManager should be the correct solution.
You can also try putting the connection string into the app settings, unless you are using Entity Framework.
Connection strings should only be used with a function app if you are using entity framework. For other scenarios use App Settings. Click to learn more.
(via Azure Portal)
Not sure if this applies to the V1 runtime as well.
The solution was to add a private static string for the connection string. Then only read from the configuration if it fails. I then added a retry that paused half a second. This basically removed this from happening.
private static string connectionString = String.Empty;
private string getConnectionString(int retryCount)
{
if (String.IsNullOrEmpty(connectionString))
{
if (System.Configuration.ConfigurationManager.ConnectionStrings["MyEntity"] != null)
{
connectionString = System.Configuration.ConfigurationManager.ConnectionStrings["MyEntity"].ToString();
}
else
{
if (retryCount > 2)
{
throw new Exception("Failed to Get Connection String From Application Settings");
}
retryCount++;
getConnectionString(retryCount);
}
}
return connectionString;
}
I don't know if this perfect but it works. I went from seeing this exception 30 times a day to none.
I wanted to create a console app as a WebJob using .NET Core but the WebJobs SDK is not yet available in .NET Core.
I've been advised to handle the scenario of reading messages from Azure Storage Queue manually. Looks like all the WebJobs SDK does is to keep polling the queue anyway.
Is the following code the basic idea in doing this? It doesn't look very sophisticated but not sure how it can be more sophisticated.
static void Main(string[] args)
{
var runContinuously = true;
while (runContinuously)
{
ReadAndProcessMessage();
System.Threading.Thread.Sleep(1000);
};
}
private static void ReadAndProcessMessage()
{
// Read message
ReadMessage();
// Process message and handle the work
HandleWork();
}
That will work. And I like simplicity.
The QueueTriggerAttribute makes use of a random exponential back-off algorithm to help minimize your transaction costs. If you'd like to trace through the logic of how this is accomplished, starting with the QueueListener class is a good way to go. Clone the project and then hop over to the RandomizedExponentialBackoffStrategy class.
I've got an Azure webjob with several queue-triggered functions. The SDK documentation at https://learn.microsoft.com/en-us/azure/app-service-web/websites-dotnet-webjobs-sdk-storage-queues-how-to#config defines the MaxDequeueCount property as:
The maximum number of retries before a queue message is sent to a
poison queue (default is 5).
but I'm not seeing this behavior. In my webjob I've got:
JobHostConfiguration config = new JobHostConfiguration();
config.Queues.MaxDequeueCount = 1;
JobHost host = new JobHost(config);
host.RunAndBlock();
and then I've got a queue-triggered function in which I throw an exception:
public void ProcessQueueMessage([QueueTrigger("azurewejobtestingqueue")] string item, TextWriter logger)
{
if ( item == "exception" )
{
throw new Exception();
}
}
Looking at the webjobs dashboard I see that the SDK makes 5 attempts (5 is the default as stated above):
and after the 5th attempt the message is moved to the poison queue. I expect to see 1 retry (or no retries?) not 5.
UPDATE: Enabled detailed logging for the web app and opted to save those logs to an Azure blob container. Found some logs relevant to my problem located in the azure-jobs-host-archive container. Here's an example showing an item with a dequeue count of 96:
{
"Type": "FunctionCompleted",
"EndTime": "2017-02-22T00:07:40.8133081+00:00",
"Failure": {
"ExceptionType": "Microsoft.Azure.WebJobs.Host.FunctionInvocationException",
"ExceptionDetails": "Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: ItemProcessor.ProcessQueueMessage ---> MyApp.Exceptions.MySpecialAppExceptionType: Exception of type 'MyApp.Exceptions.MySpecialAppExceptionType' was thrown.
},
"ParameterLogs": {},
"FunctionInstanceId": "1ffac7b0-1290-4343-8ee1-2af0d39ae2c9",
"Function": {
"Id": "MyApp.Processors.ItemProcessor.ProcessQueueMessage",
"FullName": "MyApp.Processors.ItemProcessor.ProcessQueueMessage",
"ShortName": "ItemProcessor.ProcessQueueMessage",
"Parameters": [
{
"Type": "QueueTrigger",
"AccountName": "MyStorageAccount",
"QueueName": "stuff-processor",
"Name": "sourceFeedItemQueueItem"
},
{
"Type": "BindingData",
"Name": "dequeueCount"
},
{
"Type": "ParameterDescriptor",
"Name": "logger"
}
]
},
"Arguments": {
"sourceFeedItemQueueItem": "{\"SourceFeedUpdateID\":437530,\"PodcastFeedID\":\"2d48D2sf2\"}",
"dequeueCount": "96",
"logger": null
},
"Reason": "AutomaticTrigger",
"ReasonDetails": "New queue message detected on 'stuff-processor'.",
"StartTime": "2017-02-22T00:07:40.6017341+00:00",
"OutputBlob": {
"ContainerName": "azure-webjobs-hosts",
"BlobName": "output-logs/1ffd3c7b012c043438ed12af0d39ae2c9.txt"
},
"ParameterLogBlob": {
"ContainerName": "azure-webjobs-hosts",
"BlobName": "output-logs/1cf2c1b012sa0d3438ee12daf0d39ae2c9.params.txt"
},
"LogLevel": "Info",
"HostInstanceId": "d1825bdb-d92a-4657-81a4-36253e01ea5e",
"HostDisplayName": "ItemProcessor",
"SharedQueueName": "azure-webjobs-host-490daea03c70316f8aa2509438afe8ef",
"InstanceQueueName": "azure-webjobs-host-d18252sdbd92a4657d1a436253e01ea5e",
"Heartbeat": {
"SharedContainerName": "azure-webjobs-hosts",
"SharedDirectoryName": "heartbeats/490baea03cfdfd0416f8aa25aqr438afe8ef",
"InstanceBlobName": "zd1825bdbdsdgga465781a436q53e01ea5e",
"ExpirationInSeconds": 45
},
"WebJobRunIdentifier": {
"WebSiteName": "myappengine",
"JobType": "Continuous",
"JobName": "ItemProcessor",
"RunId": ""
}
}
What I'm further looking for though are logs which would show me detail for a particular queue item where processing succeeds (and hence is removed from the queue) or fails due to an exception and is placed in the poison queue. I so far haven't found any logs showing that detail. The log files referenced in the output above do not contain data of this sort.
UPDATE 2: Looked at the state of my poison queue and it seems like it could be a smoking gun but I'm too dense to put 2 and 2 together. Looking at the screenshot of the queue below you can see the message with the ID (left column) 431210 in there many times. The fact that it appears multiple times says to me that the message in the original queue is failing improperly.
As mentioned by Rob W, this issue exists when using WindowsAzure.Storage > 7.1.2. The issue has apparently been fixed in issue #1141 but this has not yet made it into a release.
Contributer asifferman has shared a code snippet in a comment post on issue #985. that appears to resolve the problem (it worked perfectly for me).
In case of link rot, and to meet SO rules, here's the post along with the code snippet:
For those (like me) who cannot wait the next release to get the
WebJobs SDK to work with the latest releases of Azure Storage, and
based on the explanations of #brettsam, you can simply write a custom
CustomQueueProcessorFactory to create a new CloudQueueMessage in
CopyMessageToPoisonQueueAsync.
namespace ConsoleApplication1
{
using Microsoft.Azure.WebJobs.Host.Queues;
using Microsoft.WindowsAzure.Storage.Queue;
using System.Threading;
using System.Threading.Tasks;
public class CustomQueueProcessorFactory : IQueueProcessorFactory
{
public QueueProcessor Create(QueueProcessorFactoryContext context)
{
return new CustomQueueProcessor(context);
}
private class CustomQueueProcessor : QueueProcessor
{
public CustomQueueProcessor(QueueProcessorFactoryContext context)
: base(context)
{
}
protected override Task CopyMessageToPoisonQueueAsync(CloudQueueMessage message, CloudQueue poisonQueue, CancellationToken cancellationToken)
{
var newMessage = new CloudQueueMessage(message.Id, message.PopReceipt);
newMessage.SetMessageContent(message.AsBytes);
return base.CopyMessageToPoisonQueueAsync(newMessage, poisonQueue, cancellationToken);
}
}
}
}
Then in your Main, you just have to set the custom queue processor
factory in the job host configuration:
var config = new JobHostConfiguration();
config.Queues.QueueProcessorFactory = new CustomQueueProcessorFactory();
I could get it work with WindowsAzure.Storage 8.1.1 and
Microsoft.Azure.WebJobs 2.0.0. Hope that helps!
If you are still seeking an answer, we tried some of the answers listed without success. It turns out that it was a version issue with the Storage sdk (WindowsAzure.Storage) and the Webjob sdk (Microsoft.Azure.WebJobs). To fix it, we ended up having to downgrade our version of the Storage sdk to 7.2.1 (we had recently upgraded to 8.1.1). Based on the article below, the engineers are now aware of the problems and will hopefully have it fixed soon:
https://github.com/Azure/azure-webjobs-sdk/issues/1045
MaxDequeueCount property works correctly for me if I configure it.
So it is very odd that it is not working for you. When I set
config.Queues.MaxDequeueCount = 2; then I get the expected result please refer to the screenshot.
And we also could use dequeueCount to control the retry times. The following is the demo code for no try.
public void ProcessQueueMessage([QueueTrigger("queue")] string item, int dequeueCount, TextWriter logger)
{
if (dequeueCount == 1)
{
if (item == "exception")
{
throw new Exception();
}
logger.WriteLine($"NewMsge: {item}");
Console.WriteLine($"NewMsge: {item}");
}
}
Log info please refer to the screenshot
I suspect it's because you're not actually running the binaries that you think you are in Azure. This one threw me for a loop as well.
When you're running triggered WebJobs on Azure, publishing a new version of the WebJob doesn't cause the old triggered WebJob to be immediately unloaded and the new one started. If you look at your WebJob logs, I suspect you will not see a restart when you republished.
This is because Kudu by default copies all of your WebJob files to a temp directory and executes them. From the Kudu WebJob docs:
The WebJob is copied to a temporary directory under %TEMP%\jobs{job
type}{job name}{random name} and will run from there This option
prevents the original WebJob binaries from being locked which might
cause issues redeploying the WebJob. For example updating an .exe file
that is currently running.
The only success I've had in making sure that a newly published triggered WebJob is actually running is to do the following:
Log into the Kudu console. It's https://yourappname.scm.azurewebsites.net. You'll use the same credentials that you do when logging into the Azure Portal.
Once logged in, click on the Process Explorer menu option at the top. Find your WebJob process that's currently running, and kill it.
FTP into your Web App. Browse to the directory containing your WebJob code, and delete it. It should be under /app_data/jobs/triggered/[your webjob name].
I then hop over to the portal, browse to by Web App management blade that hosts the WebJob, click on the WebJobs menu option, and confirm that the old WebJob is no longer there.
Publish my new WebJob from Visual Studio.
That should guarantee you that you're running the code that you publish. Hope this helps.
I am seeing the same thing where messages go way past the max dequeue count. I will post more details in a bit, but I am also seeing what appears to be a very large number end up in poison queue. So I suspect that it is adding to poison queue after 5, but that trying more which ends up in lots in poison queue (hundreds).
For anyone using the Azure WebJobs v3.x SDK:
In v3.x, hosts.json does not work for WebJob.
Instead, version 3.x uses the standard ASP.NET Core APIs, so you need to configure it using the ConfigureWebJobs method:
static async Task Main()
{
var builder = new HostBuilder();
builder.ConfigureWebJobs(b =>
{
b.AddAzureStorageCoreServices();
b.AddAzureStorage(a => {
a.BatchSize = 8;
a.NewBatchThreshold = 4;
a.MaxDequeueCount = 4;
a.MaxPollingInterval = TimeSpan.FromSeconds(15);
});
});
var host = builder.Build();
using (host)
{
await host.RunAsync();
}
}
Docs: https://learn.microsoft.com/pt-pt/azure/app-service/webjobs-sdk-how-to#queue-storage-trigger-configuration
I have created my first azure webjob that runs continously;
I'm not so sure this is a code issue, but for the sake of completeness here is my code:
static void Main()
{
var host = new JobHost();
host.CallAsync(typeof(Functions).GetMethod("ProcessMethod"));
host.RunAndBlock();
}
And for the function:
[NoAutomaticTrigger]
public static async Task ProcessMethod(TextWriter log)
{
log.WriteLine(DateTime.UtcNow.ToShortTimeString() + ": Started");
while (true)
{
Task.Run(() => RunAllAsync(log));
await Task.Delay(TimeSpan.FromSeconds(60));
}
log.WriteLine(DateTime.UtcNow.ToShortTimeString() + "Shutting down..");
}
Note that the async task fires off a task of its own. This was to ensure they were started quite accurately with the same interval. The job itself is to download an url, parse and input some data in a db, but that shouldnt be relevant for the multiple instance issue I am experiencing.
My problem is that once this has been running for roughly 5 minutes a second ProcessMethod is called which makes me have two sessions simoultaniously doing the same thing. The second method says it is "started from Dashboard" even though I am 100% confident I did not click anything to start it off myself.
Anyone experienced anything like it?
Change the instance count to 1 from Scale tab of WebApp in Azure portal. By default it is set to 2 instances which is causing it to run two times.
I can't explain why it's getting called twice, but I think you'd be better served with a triggered job using a CRON schedule (https://azure.microsoft.com/en-us/documentation/articles/web-sites-create-web-jobs/#CreateScheduledCRON), instead of a Continuous WebJob.
Also, it doesn't seem like you are using the WebJobs SDK, so you can completely skip that. Your WebJob can be as simple as a Main that directly does the work. No JobHost, no async, and generally easier to get right.