I'm using Azure Batch .NET API. After tasks completion, I try to retrieve the output files (or any files) from the tasks via CloudTask.GetNodeFile() but this throws a "not found" exception. Same for stdout t.GetNodeFile(Microsoft.Azure.Batch.Constants.StandardOutFileName) and stderr t.GetNodeFile(Microsoft.Azure.Batch.Constants.StandardErrorFileName). There is simply no files of any sort on the node (verified via Azure portal). Job and pool are not yet deleted when I do the check and nodes are in "Idle" state.
Here is a relevant snippet of code, note that this is basically one of the examples from the official azure-batch-samples at github:
IPagedEnumerable<CloudTask> ourTasks = job.ListTasks(new ODATADetailLevel(selectClause: "id,state,url"));
client.Utilities.CreateTaskStateMonitor().WaitAll(ourTasks, TaskState.Completed, TimeSpan.FromMinutes(config.TimeoutMinutes));
Console.WriteLine("Tasks are done.");
foreach (CloudTask t in ourTasks)
{
t.ListNodeFiles(recursive: true).ToList(); // always of size 1, with only element being "/wd"
}
Related
I have an azure queue trigger associated to a queue, and I want to ensure that the trigger only reads and executes one message at the time. So, when the message is executed (successfuly or not) it processes the next message.
What is happening is that the queue executes one message, yet it begins to execute other message. My host.json:
"queues": {
"maxPollingInterval": 20000,
"visibilityTimeout": "00:01:00",
"batchSize": 1,
"maxDequeueCount": 5,
"newBatchThreshold": 1
}
Following instructions from MS link:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-queue#trigger---configuration
If you want to avoid parallel execution for messages received on one
queue, you can set batchSize to 1
So it would be expected to run only one message at the time (I'm using consumption plan).
This is critical because I need to ensure that only one message it processed at the time.
Is there any setting that I could change?
Or is queue trigger not a good option to address this requirement?
Storage Queue does NOT guarantee ordering - so if needing sequential processing because order of delivery matters you need to consider Azure Service Bus and set in Function setting (host.json)
maxConcurrentCalls = 1
Even if you do the trick by setting maximum number of instances that a function app can scale to as follow, ordering is still not guaranteed with Azure Storage Queue.
WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT = 1
Microsoft documentation is not the perfect one. It's being continuously updated.
If you want to minimize parallel execution for queue-triggered functions in a function app, you can set the batch size to 1. But this setting eliminates concurrency only so long as your function app runs on a single virtual machine (VM).
If you have multiple Virtual Machines and function instances on each VM, there will be one message processed for each function instance running in each virtual machine.
This microsoft document explains concurrency on triggers.
For those coming across this question looking to debug locally and getting issue with multiple queue items making this difficult, you can add the following to your local.settings.json to override the default functionality on your machine only:
{
"IsEncrypted": false,
"Values": {
"AzureFunctionsJobHost__extensions__queues__batchSize": 1
}
}
Documentation
I am trying to run a Console Application in Azure Batch. I have created the Client Application having responsibility of creating and running Job & Tasks in the Azure Batch so far. My ClientApplication successfully creates a Job, a Task within the Job but when the application code uploaded on the AzureStorage get executed in Azure the Pool, the Task fails and exits with an Exit code -532462766. I could not find any information for this exit code. Please help me to fix this.
Here is my code for creating the task and it's configuration
private static async Task<List<CloudTask>> AddTasksAsync(BatchClient batchClient, string jobId)
{
Console.WriteLine("Adding {0} tasks to job [{1}]...", 1, jobId);
List<CloudTask> tasks = new List<CloudTask>();
string taskId = "topNtask";
string taskCommandLine = String.Format("cmd /c %AZ_BATCH_NODE_SHARED_DIR%\\SelfHostedWebAPI.exe {0} {1} {2}", 11109, "MSS_F_MST_______", "Normal");
//string taskCommandLine = String.Format("cmd /c %AZ_BATCH_NODE_SHARED_DIR%\\SelfHostedWebAPI.exe");
CloudTask task = new CloudTask(taskId, taskCommandLine);
tasks.Add(task);
await batchClient.JobOperations.AddTaskAsync(jobId, tasks);
return tasks;
}
And the task failure report from my Azure Portal
Corresponding hex value for -532462766 is 0xE0434352; more information about this error code/exception can be found on this SO post. Common culprits are missing assemblies (see the Assembly Binding Log Viewer). Additionally, the Windows Event Viewer might be able to provide you with more information prior to having to resort to WinDbg and SOS.
Finally I got the solution of this problem and to save one's precious time here is the solution.
We need to upload each and every .dll files and dependencies of our code that run in Azure Pool to the linked AzureStorage account. In addition, when the pool is being created and it's StartTask is initialized, a ResourceFile with SAS(Shared Access Signature, for the target .dll that it points) for every .dll file or any other type, should be given to it as a parameter.
I am new to Azure Worker Role ,I have created a new Azure Cloud Service project and added a worker role.In which I have a thread that makes function call to sampledll where I am creating EventHub listener.
I found error like below.
Severity Code Description Project File Line Suppression State
Error The process cannot access the file 'D:\Azure\roles\simpleWorkerRole\approot\sampledll.dll' because it is being used by another process. simpleWorkerRole.WorkerRole C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Windows Azure Tools\2.9\Microsoft.WindowsAzure.targets 1057
The process cannot access the file 'D:\Azure\roles\simpleWorkerRole\approot\sampledll.dll' because it is being used by another process.
The error message indicates the sampledll.dll is being used by another process, you could try to use Process Monitor tool to find which process is using (or locking) sampledll.dll and then you could manually end the process and rebuild the solution.
I am running an azure webjobs SDK console application (continuous) with the recommended setup:
public static void ProcessQueueMessage([QueueTrigger("logqueue")] string logMessage, TextWriter logger)
The azure queue I am running against has ~6000 messages in it and I am running the web-job locally, as a console application.
The problem I'm having is that the processing randomly stops after processing between zero and ~30 messages. The console stays open, but no more console messages are displayed.
For example, it might just process 2 messages:
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
And then, nothing. There doesn't seem to be anything wrong with my internet connection and I can't trace the issues down to any particular messages.
Has anyone else had issues with this SDK?
Update:
I made sure that I was using the right versions of all of the dependencies by removing the nuget packages and then re-running install-package Microsoft.Axure.Webjobs. I am now using webjobs version 1.1.0 which has pulled in version 4.3 of azure storage.
As recommended by Matthew, I have pulled down the source code for azure webjobs to determine where the process is freezing up. Once the freez-up occurs, I pause execution and checked the running threads for what I believe is the culprit within Microsoft.Azure.WebJobs.Host.CompositeTraceWriter
protected virtual void InvokeTextWriter(TraceEvent traceEvent)
{
if (_innerTextWriter != null)
{
string message = traceEvent.Message;
if (!string.IsNullOrEmpty(message) &&
message.EndsWith("\r\n", StringComparison.OrdinalIgnoreCase))
{
// remove any terminating return+line feed, since we're
// calling WriteLine below
message = message.Substring(0, message.Length - 2);
}
_innerTextWriter.WriteLine(message);
if (traceEvent.Exception != null)
{
_innerTextWriter.WriteLine(traceEvent.Exception.ToDetails());
}
}
}
The line it freezes on is line 66 : _innerTextWriter.WriteLine(message);
_innerTextWriter is an instance of System.IO.TextWriter.SyncTextWriter
Is it possible there is some deadlock issue with this class or the way it is being used?
Some notes:
I am running in the debugger, so in this case I believe the textwriter is forwarding to the console internally
I have my batchsize set to 1 via config.Queues.BatchSize = 1;, not sure if that could matter
I'm currently working on setting up an environment on another computer so that I can see if it is reproducible somewhere other than this machine (surface book).
Update
The issue was me not understanding how the new windows 10 command prompt works. Any time you click on the command window, it goes into "select" mode which completely pauses execution of the process.
Basically: https://superuser.com/questions/419717/windows-command-prompt-freezing-randomly?newreg=ece53f5584254346be68f85d1fd2f18d
You can tell it is in this state because it will prefix the window title with the word "Select":
You have to press enter or click again to get it going once again.
So, two final comments:
1) What an incredibly confusing and un-intuitive behavior for a command window!
2) I hope some admin will come take pity on the shame I have brought upon myself and my family by deleting this question.
To get rid of this strange behavior, you can disable QuickEdit mode:
Strange. When it is in this stuck state, can you try adding a new queue message to the queue and see if that triggers? Are you sure your function isn't hanging internally? What version of the SDK are you using? You might also try upgrading to v1.1.0 which we just released last week. If there are really a bunch of messages in the queue waiting to be processed, I can't think of anything that would cause this. The queue listener in the SDK should chug along, reading batches of messages in parallel and dispatching them to your function. Have you changed any of the JobHostConfiguration.Queues configuration knobs? You haven't force updated the version of the Azure SDK have you to something higher than the WebJobs SDK supports?
Another option if you can't figure this out might be to clone the SDK, build it and debug it locally. The repo is here. The main queue processing loop is here.
I am having some problems with the Azure Media Service. I have successfully managed to upload a file to Azure Media Service and encoded it as mp4, but this will only work on localhost in visual studio. When I deploy the following code to a Azure Virtual Machine, Windows Datacenter 2012 edition, the file is uploaded but the encoding job is never created.
Any help is much appreciated!
This is the code I use to create the job:
// Declare a new job.
job= _context.Jobs.Create("New Job");
// Get a media processor reference, and pass to it the name of the
// processor to use for the specific task.
IMediaProcessor processor = GetLatestMediaProcessorByName("Windows Azure Media Encoder");
// Create a task with the encoding details, using a string preset.
ITask task = job.Tasks.AddNew("LSVEncodingTask_" + v.CompanyId + "_user" + v.UserId,
processor,
"H264 Broadband 1080p",
TaskOptions.ProtectedConfiguration);
// Specify the input asset to be encoded.
task.InputAssets.Add(asset);
// Add an output asset to contain the results of the job.
// This output is specified as AssetCreationOptions.None, which
// means the output asset is not encrypted.
task.OutputAssets.AddNew("output asset",
AssetCreationOptions.None);
// Use the following event handler to check job progress.
job.StateChanged += new
EventHandler<JobStateChangedEventArgs>(StateChanged);
// Launch the job.
job.Submit();
// Check job execution and wait for job to finish.
Task progressJobTask = job.GetExecutionProgressTask(CancellationToken.None);
EDIT:
Did some digging (logging...) and found the stack trace:
System.Security.Cryptography.CryptographicException: Access is denied.
at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
at Microsoft.WindowsAzure.MediaServices.Client.EncryptionUtils.SaveCertificateToStore(X509Certificate2 certToStore)
at Microsoft.WindowsAzure.MediaServices.Client.ContentKeyBaseCollection.GetCertificateForProtectionKeyId(IMediaDataServiceContext dataContext, String protectionKeyId)
at Microsoft.WindowsAzure.MediaServices.Client.JobData.ProtectTaskConfiguration(TaskData task, X509Certificate2& certToUse, IMediaDataServiceContext dataContext)
at Microsoft.WindowsAzure.MediaServices.Client.JobData.InnerSubmit(IMediaDataServiceContext dataContext)
at Microsoft.WindowsAzure.MediaServices.Client.JobData.SubmitAsync()
at Microsoft.WindowsAzure.MediaServices.Client.JobData.Submit()
at Livescreen.Domain.Implementation.AzureMediaManager.CreateEncodingJob(IAsset asset, String inputMediaFilePath, String outputFolder, Int32 videoId)
The solution was found here:
When running on IIS7 - you can specify to load the user profile of the App Pool. This gives you access to the per user store of the app
pool account.
Solution: