I have a WebJob:
public class StandardizeFileFunction : FunctionBase
{
public static async Task StandardizeFile([BlobTrigger("foobar/Raw/{blobName}")] Stream input, string blobName)
{ ... }
}
The AzureWebJobsDashboard and AzureWebJobsStorage connection strings point to Azure, but I am running locally in debug mode.
In the past, the blob trigger was set up to work against "foo/Raw/{blobName}" and "foo2/Raw/{blobName}", but I have since done a clean build and deleted the bin and obj directories. Even so, I still see the WebJob accessing the old containers in addition to the new container. In the below, you'll see that containers foo and foo2 are being accessed despite the function being set up for foobar:
StandardizeFileFunction.StandardizeFile
Job host started
Executing 'StandardizeFileFunction.StandardizeFile' (Reason='New blob detected: foo/Raw/Study/062014.txt', Id=98c90b27-b1b4-464a-898c-8d9137c564a1)
Exception while executing function: StandardizeFileFunction.StandardizeFile
Executing 'StandardizeFileFunction.StandardizeFile' (Reason='New blob detected: foo2/Raw/Study/Temperature/HF732 1310-Room-3.txt', Id=90060f17-9a6f-47f2-a09d-b39784f5152f)
Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: StandardizeFileFunction.StandardizeFile ---> System.InvalidOperationException: Exception binding parameter 'blobName' ---> System.InvalidOperationException: Binding data does not contain expected value 'blobName'.
at Microsoft.Azure.WebJobs.Host.Bindings.Data.ClassDataBinding`1.BindAsync(BindingContext context)
at Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding`1.<BindCoreAsync>d__8.MoveNext()
--- End of inner exception stack trace ---
Why does it keep accessing the old containers and how do I stop it?
UPDATE
I repro-ed this with a new WebJob project. Here's what I did:
Use Azure Storage Explorer to add hundreds of files to container "foo"
Run the WebJob (it doesn't need to do anything)
After a couple dozen blobs are processed, stop the WebJob
Change the container name in the blob trigger (like "fooed" or "abcde")
Run the WebJob
At this point you may see errors, if not, keep switching back and forth between "foo" and "fooed" until you do.
Executing 'StandardizeFileFunction.StandardizeFile' (Reason='New blob detected: foo/Raw/Study/062014.txt', Id=98c90b27-b1b4-464a-898c-8d9137c564a1)
public static async Task StandardizeFile([BlobTrigger("foobar/Raw/{blobName}")] Stream input, string blobName)
According to your description and error message, I found in your error message your function is triggered by blob container “foo/foo2”.
But your code is triggered by blob container “foobar”.
I think maybe something wrong with your solution projects.
I suggest you could start a new webjob and try again. Make sure you have started the right project.
Related
I have function app where I have one HttpTrigger and 3 BlobTrigger functions. After I deployed it, http trigger is working fine but for others functions which are blob triggers, it gives following errors
"Stopping the listener 'Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobListener' for function " for one function
Stopping the listener 'Microsoft.Azure.WebJobs.Host.Listeners.CompositeListener' for function
" for another two
I verified with other environments and config values are same/similar so not sure why we are getting this issue in one environment only. I am using consumption mode.
Update: When file is placed in a blob function is not getting triggered.
Stopping the listener 'Microsoft.Azure.WebJobs.Host.Blobs.Listeners.BlobListener' for function
I was observed the same message when working on the Azure Functions Queue Trigger:
This message doesn't mean the error in function. Due to timeout of Function activity, this message will appear in the App Insights > Traces.
I have stopped sending the messages in the Queue for some time and has been observed the traces like Web Job Host Stopped and if you run the function again or any continuous activity is present in the Function, then this message will not appear in the traces.
If you are using elastic Premium and has VNET integrated, the non-http trigers needs Runtime scale monitoring enabled.
You can find Function App-->Configuration--> Function runtime settings and turn on Runtime scale monitoring.
If function app and storage account which holds the metadata of the function Private linked, you will need to add the app settings WEBSITE_CONTENTOVERVNET = 1.
Also, make sure you have private linked for blob, file, table and queue on storage account.
I created ticket with MS to fix this issue. After analysis I did some code changes as
Function was async but returning void so changed to return Task.
For the trigger I was using connection string from app settings. But then I changed it to azureWebJobStorage(even though bobth were same) in function trigger attribute param
It started working. So posting here in case it is helpful for others
I am trying to run a Console Application in Azure Batch. I have created the Client Application having responsibility of creating and running Job & Tasks in the Azure Batch so far. My ClientApplication successfully creates a Job, a Task within the Job but when the application code uploaded on the AzureStorage get executed in Azure the Pool, the Task fails and exits with an Exit code -532462766. I could not find any information for this exit code. Please help me to fix this.
Here is my code for creating the task and it's configuration
private static async Task<List<CloudTask>> AddTasksAsync(BatchClient batchClient, string jobId)
{
Console.WriteLine("Adding {0} tasks to job [{1}]...", 1, jobId);
List<CloudTask> tasks = new List<CloudTask>();
string taskId = "topNtask";
string taskCommandLine = String.Format("cmd /c %AZ_BATCH_NODE_SHARED_DIR%\\SelfHostedWebAPI.exe {0} {1} {2}", 11109, "MSS_F_MST_______", "Normal");
//string taskCommandLine = String.Format("cmd /c %AZ_BATCH_NODE_SHARED_DIR%\\SelfHostedWebAPI.exe");
CloudTask task = new CloudTask(taskId, taskCommandLine);
tasks.Add(task);
await batchClient.JobOperations.AddTaskAsync(jobId, tasks);
return tasks;
}
And the task failure report from my Azure Portal
Corresponding hex value for -532462766 is 0xE0434352; more information about this error code/exception can be found on this SO post. Common culprits are missing assemblies (see the Assembly Binding Log Viewer). Additionally, the Windows Event Viewer might be able to provide you with more information prior to having to resort to WinDbg and SOS.
Finally I got the solution of this problem and to save one's precious time here is the solution.
We need to upload each and every .dll files and dependencies of our code that run in Azure Pool to the linked AzureStorage account. In addition, when the pool is being created and it's StartTask is initialized, a ResourceFile with SAS(Shared Access Signature, for the target .dll that it points) for every .dll file or any other type, should be given to it as a parameter.
I am working with Azure Queues, Controlling them using WebJob, In the functions file i have one QueueTrigger function, which fires up when AzureQueue receives some message.
Now the problem is that function (QueueTrigger) executes successfully, I have setup proper exception handling inside queue trigger function, everything executed without any errors. But when QueueTrigger function ends, code is throwing exception. Storage Exception 404 Not found.
and that message is not getting deleted from queue. Next time when i run job it is still fetching the old message. I have manually created storage containers
azure-jobs-host-archive
azure-jobs-host-output
azure-webjobs-dashboard
azure-webjobs-hosts
I have seen in one post answer but this does not help at all.
I have no idea how to tackle this exception or why this exception in throwing in the code.
Thank you
Eman
After updating the following packages the problem get solved.
Microsoft.WindowAzure.Storage
Microsoft.Azure.Webjobs
Microsoft.Azure.WebJobs.Host
I am developing a triggered webjob that use TimerTrigger.
Before the webjob stops, I need to dispose some objects but I don't know how to trigger the "webjob stop".
Having a NoAutomaticTrigger function, I know that I can use the WebJobsShutdownWatcher class to handle when the webjob is stopping but with a triggered job I need some help...
I had a look at Extensible Triggers and Binders with Azure WebJobs SDK 1.1.0-alpha1.
Is it a good idea to create a custom trigger (StopTrigger) that used the WebJobsShutdownWatcher class to fire action before the webjob stops ?
Ok The answer was in the question :
Yes I can use the WebJobsShutdownWatcher class because it has a Register function that is called when the cancellation token is canceled, in other words when the webjob is stopping.
static void Main()
{
var cancellationToken = new WebJobsShutdownWatcher().Token;
cancellationToken.Register(() =>
{
Console.Out.WriteLine("Do whatever you want before the webjob is stopped...");
});
var host = new JobHost();
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
}
EDIT (Based on Matthew comment):
If you use Triggered functions, you can add a CancellationToken parameter to your function signatures. The runtime will cancel that token when the host is shutting down automatically, allowing your function to receive the notification.
public static void QueueFunction(
[QueueTrigger("QueueName")] string message,
TextWriter log,
CancellationToken cancellationToken)
{
...
if(cancellationToken.IsCancellationRequested) return;
...
}
I was recently trying to figure out how to do this without the
WebJobs SDK which contains the WebJobShutdownWatcher, this is what I
found out.
What the underlying runtime does (and what the WebJobsShutdownWatcher referenced above checks), is create a local file at the location specified by the environment variable %WEBJOBS_SHUTDOWN_FILE%. If this file exists, it is essentially the runtime's signal to the webjob that it must shutdown within a configurable wait period (default of 5 seconds for continuous jobs, 30 for triggered jobs), otherwise the runtime will kill the job.
The net effect is, if you are not using the Azure WebJobs SDK, which contains the WebJobsShutdownWatcher as described above, you can still achieve graceful shutdown of your Azure Web Job by monitoring for the shutdown file on an interval shorter than the configured wait period.
Additional details, including how to configure the wait period, are described here: https://github.com/projectkudu/kudu/wiki/WebJobs#graceful-shutdown
I am running an azure webjobs SDK console application (continuous) with the recommended setup:
public static void ProcessQueueMessage([QueueTrigger("logqueue")] string logMessage, TextWriter logger)
The azure queue I am running against has ~6000 messages in it and I am running the web-job locally, as a console application.
The problem I'm having is that the processing randomly stops after processing between zero and ~30 messages. The console stays open, but no more console messages are displayed.
For example, it might just process 2 messages:
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
And then, nothing. There doesn't seem to be anything wrong with my internet connection and I can't trace the issues down to any particular messages.
Has anyone else had issues with this SDK?
Update:
I made sure that I was using the right versions of all of the dependencies by removing the nuget packages and then re-running install-package Microsoft.Axure.Webjobs. I am now using webjobs version 1.1.0 which has pulled in version 4.3 of azure storage.
As recommended by Matthew, I have pulled down the source code for azure webjobs to determine where the process is freezing up. Once the freez-up occurs, I pause execution and checked the running threads for what I believe is the culprit within Microsoft.Azure.WebJobs.Host.CompositeTraceWriter
protected virtual void InvokeTextWriter(TraceEvent traceEvent)
{
if (_innerTextWriter != null)
{
string message = traceEvent.Message;
if (!string.IsNullOrEmpty(message) &&
message.EndsWith("\r\n", StringComparison.OrdinalIgnoreCase))
{
// remove any terminating return+line feed, since we're
// calling WriteLine below
message = message.Substring(0, message.Length - 2);
}
_innerTextWriter.WriteLine(message);
if (traceEvent.Exception != null)
{
_innerTextWriter.WriteLine(traceEvent.Exception.ToDetails());
}
}
}
The line it freezes on is line 66 : _innerTextWriter.WriteLine(message);
_innerTextWriter is an instance of System.IO.TextWriter.SyncTextWriter
Is it possible there is some deadlock issue with this class or the way it is being used?
Some notes:
I am running in the debugger, so in this case I believe the textwriter is forwarding to the console internally
I have my batchsize set to 1 via config.Queues.BatchSize = 1;, not sure if that could matter
I'm currently working on setting up an environment on another computer so that I can see if it is reproducible somewhere other than this machine (surface book).
Update
The issue was me not understanding how the new windows 10 command prompt works. Any time you click on the command window, it goes into "select" mode which completely pauses execution of the process.
Basically: https://superuser.com/questions/419717/windows-command-prompt-freezing-randomly?newreg=ece53f5584254346be68f85d1fd2f18d
You can tell it is in this state because it will prefix the window title with the word "Select":
You have to press enter or click again to get it going once again.
So, two final comments:
1) What an incredibly confusing and un-intuitive behavior for a command window!
2) I hope some admin will come take pity on the shame I have brought upon myself and my family by deleting this question.
To get rid of this strange behavior, you can disable QuickEdit mode:
Strange. When it is in this stuck state, can you try adding a new queue message to the queue and see if that triggers? Are you sure your function isn't hanging internally? What version of the SDK are you using? You might also try upgrading to v1.1.0 which we just released last week. If there are really a bunch of messages in the queue waiting to be processed, I can't think of anything that would cause this. The queue listener in the SDK should chug along, reading batches of messages in parallel and dispatching them to your function. Have you changed any of the JobHostConfiguration.Queues configuration knobs? You haven't force updated the version of the Azure SDK have you to something higher than the WebJobs SDK supports?
Another option if you can't figure this out might be to clone the SDK, build it and debug it locally. The repo is here. The main queue processing loop is here.