Task failure and Exit code -532462766 in Azure Batch - azure

I am trying to run a Console Application in Azure Batch. I have created the Client Application having responsibility of creating and running Job & Tasks in the Azure Batch so far. My ClientApplication successfully creates a Job, a Task within the Job but when the application code uploaded on the AzureStorage get executed in Azure the Pool, the Task fails and exits with an Exit code -532462766. I could not find any information for this exit code. Please help me to fix this.
Here is my code for creating the task and it's configuration
private static async Task<List<CloudTask>> AddTasksAsync(BatchClient batchClient, string jobId)
{
Console.WriteLine("Adding {0} tasks to job [{1}]...", 1, jobId);
List<CloudTask> tasks = new List<CloudTask>();
string taskId = "topNtask";
string taskCommandLine = String.Format("cmd /c %AZ_BATCH_NODE_SHARED_DIR%\\SelfHostedWebAPI.exe {0} {1} {2}", 11109, "MSS_F_MST_______", "Normal");
//string taskCommandLine = String.Format("cmd /c %AZ_BATCH_NODE_SHARED_DIR%\\SelfHostedWebAPI.exe");
CloudTask task = new CloudTask(taskId, taskCommandLine);
tasks.Add(task);
await batchClient.JobOperations.AddTaskAsync(jobId, tasks);
return tasks;
}
And the task failure report from my Azure Portal

Corresponding hex value for -532462766 is 0xE0434352; more information about this error code/exception can be found on this SO post. Common culprits are missing assemblies (see the Assembly Binding Log Viewer). Additionally, the Windows Event Viewer might be able to provide you with more information prior to having to resort to WinDbg and SOS.

Finally I got the solution of this problem and to save one's precious time here is the solution.
We need to upload each and every .dll files and dependencies of our code that run in Azure Pool to the linked AzureStorage account. In addition, when the pool is being created and it's StartTask is initialized, a ResourceFile with SAS(Shared Access Signature, for the target .dll that it points) for every .dll file or any other type, should be given to it as a parameter.

Related

WebJob accessing wrong container

I have a WebJob:
public class StandardizeFileFunction : FunctionBase
{
public static async Task StandardizeFile([BlobTrigger("foobar/Raw/{blobName}")] Stream input, string blobName)
{ ... }
}
The AzureWebJobsDashboard and AzureWebJobsStorage connection strings point to Azure, but I am running locally in debug mode.
In the past, the blob trigger was set up to work against "foo/Raw/{blobName}" and "foo2/Raw/{blobName}", but I have since done a clean build and deleted the bin and obj directories. Even so, I still see the WebJob accessing the old containers in addition to the new container. In the below, you'll see that containers foo and foo2 are being accessed despite the function being set up for foobar:
StandardizeFileFunction.StandardizeFile
Job host started
Executing 'StandardizeFileFunction.StandardizeFile' (Reason='New blob detected: foo/Raw/Study/062014.txt', Id=98c90b27-b1b4-464a-898c-8d9137c564a1)
Exception while executing function: StandardizeFileFunction.StandardizeFile
Executing 'StandardizeFileFunction.StandardizeFile' (Reason='New blob detected: foo2/Raw/Study/Temperature/HF732 1310-Room-3.txt', Id=90060f17-9a6f-47f2-a09d-b39784f5152f)
Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: StandardizeFileFunction.StandardizeFile ---> System.InvalidOperationException: Exception binding parameter 'blobName' ---> System.InvalidOperationException: Binding data does not contain expected value 'blobName'.
at Microsoft.Azure.WebJobs.Host.Bindings.Data.ClassDataBinding`1.BindAsync(BindingContext context)
at Microsoft.Azure.WebJobs.Host.Triggers.TriggeredFunctionBinding`1.<BindCoreAsync>d__8.MoveNext()
--- End of inner exception stack trace ---
Why does it keep accessing the old containers and how do I stop it?
UPDATE
I repro-ed this with a new WebJob project. Here's what I did:
Use Azure Storage Explorer to add hundreds of files to container "foo"
Run the WebJob (it doesn't need to do anything)
After a couple dozen blobs are processed, stop the WebJob
Change the container name in the blob trigger (like "fooed" or "abcde")
Run the WebJob
At this point you may see errors, if not, keep switching back and forth between "foo" and "fooed" until you do.
Executing 'StandardizeFileFunction.StandardizeFile' (Reason='New blob detected: foo/Raw/Study/062014.txt', Id=98c90b27-b1b4-464a-898c-8d9137c564a1)
public static async Task StandardizeFile([BlobTrigger("foobar/Raw/{blobName}")] Stream input, string blobName)
According to your description and error message, I found in your error message your function is triggered by blob container “foo/foo2”.
But your code is triggered by blob container “foobar”.
I think maybe something wrong with your solution projects.
I suggest you could start a new webjob and try again. Make sure you have started the right project.

Build Error: while running worker Role in local system

I am new to Azure Worker Role ,I have created a new Azure Cloud Service project and added a worker role.In which I have a thread that makes function call to sampledll where I am creating EventHub listener.
I found error like below.
Severity Code Description Project File Line Suppression State
Error The process cannot access the file 'D:\Azure\roles\simpleWorkerRole\approot\sampledll.dll' because it is being used by another process. simpleWorkerRole.WorkerRole C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Windows Azure Tools\2.9\Microsoft.WindowsAzure.targets 1057
The process cannot access the file 'D:\Azure\roles\simpleWorkerRole\approot\sampledll.dll' because it is being used by another process.
The error message indicates the sampledll.dll is being used by another process, you could try to use Process Monitor tool to find which process is using (or locking) sampledll.dll and then you could manually end the process and rebuild the solution.

Azure Batch - No files on node after task completion

I'm using Azure Batch .NET API. After tasks completion, I try to retrieve the output files (or any files) from the tasks via CloudTask.GetNodeFile() but this throws a "not found" exception. Same for stdout t.GetNodeFile(Microsoft.Azure.Batch.Constants.StandardOutFileName) and stderr t.GetNodeFile(Microsoft.Azure.Batch.Constants.StandardErrorFileName). There is simply no files of any sort on the node (verified via Azure portal). Job and pool are not yet deleted when I do the check and nodes are in "Idle" state.
Here is a relevant snippet of code, note that this is basically one of the examples from the official azure-batch-samples at github:
IPagedEnumerable<CloudTask> ourTasks = job.ListTasks(new ODATADetailLevel(selectClause: "id,state,url"));
client.Utilities.CreateTaskStateMonitor().WaitAll(ourTasks, TaskState.Completed, TimeSpan.FromMinutes(config.TimeoutMinutes));
Console.WriteLine("Tasks are done.");
foreach (CloudTask t in ourTasks)
{
t.ListNodeFiles(recursive: true).ToList(); // always of size 1, with only element being "/wd"
}

How can I keep my Azure WebJob running without "Always On"

I have a continous webjob associated with a website and I am running that website on the Shared mode. I don't want to go to Always On option as there is no real need for my application. I only want to process the message when the calls are made to my website.
My issue is that the job keeps stopping after few minutes even though I am continuously calling a dummy keep alive method on my website at every 5 minute that posts a message to the queue that is monitored by that webjob.
My webjob is a simple console application built using the WebJob SDK that has a code like this
JobHost host = new JobHost(new JobHostConfiguration(storageConnictionSttring));
host.RunAndBlock();
and the message processing function looks like below:
public static void ProcessKeepAliveMessages([QueueTrigger("keepalive")] KeepAliveTrigger message)
{
Console.WriteLine("Keep Alive message called on :{0}", message.MessageTime);
}
The message log for the job basically says says
[03/05/2015 18:51:02 > 4660f6: SYS INFO] WebJob is stopping due to website shutting down
I don't mind if that happen this way, but when the website starts with the next call to keep alive, the webjob is not started. All the messages are queued till I go to the management dashboard or the SCM portal as shown below
https://mysite.scm.azurewebsites.net/api/continuouswebjobs
I can see the status like this:
[{"status":"Starting","detailed_status":"4660f6 - Starting\r\n","log_url":"https://mysite.scm.azurewebsites.net/vfs/data/jobs/continuous/WebJobs/job_log.txt","name":"WebJobs","run_command":"mysite.WebJobs.exe","url":"https://mysite.scm.azurewebsites.net/api/continuouswebjobs/WebJobs","extra_info_url":"https://mysite.scm.azurewebsites.net/azurejobs/#/jobs/continuous/WebJobs","type":"continuous","error":null,"using_sdk":true,"settings":{}}]
I would really appreciate if someone can help me understand what is going wrong here.
I've run into a similar problem. I have a website (shared mode) and an associated webjob (continuous type). Looking at webjob logs, I found that the job enters stopped state after about 15 min. of inactivity and stops reacting to trigger messages. It seems contradictory to the concept of continuous job concept but, apparently, to get it running truly continuously you have to subscribe to a paid website. You get what you pay for...
That said, my website needs to be used only about every few days and running in a shared mode makes perfect sense. I don't mind that the site needs a bit extra time to get started - as long as it restarts automatically. The problem with the webjob is that once stopped it won't restart by itself. So, my goal was to restart it with the website.
I have noticed that a mere look at the webjob from Azure Management Portal starts it. Following this line of thinking, I have found that fetching webjob properties is enough to switch it to the running state. The only trick is how to fetch the properties programmatically, so that restarting the website will also restart the webjob.
Because the call to fetch webjob properties must be authenticated, the first step is to go to Azure Management Portal and download the website publishing profile. In the publishing profile you can find the authentication credentials: username (usually $<website_name>) and userPWD (hash of the password). Copy them down.
Here is a function that will get webjob properties and wake it up (if not yet running):
class Program
{
static void Main(string[] args)
{
string websiteName = "<website_name>";
string webjobName = "<webjob_name>";
string userName = "<from_publishing_profile>";
string userPWD = "<from_publishing_profile>";
string webjobUrl = string.Format("https://{0}.scm.azurewebsites.net/api/continuouswebjobs/{1}", websiteName, webjobName);
var result = GetWebjobState(webjobUrl, userName, userPWD);
Console.WriteLine(result);
Console.ReadKey(true);
}
private static JObject GetWebjobState(string webjobUrl, string userName, string userPWD)
{
HttpClient client = new HttpClient();
string auth = "Basic " + Convert.ToBase64String(Encoding.UTF8.GetBytes(userName + ':' + userPWD));
client.DefaultRequestHeaders.Add("authorization", auth);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var data = client.GetStringAsync(webjobUrl).Result;
var result = JsonConvert.DeserializeObject(data) as JObject;
return result;
}
}
You can use a similar function to get all webjobs in your website (use endpoint https://<website_name>.scm.azurewebsites.net/api/webjobs). You may also look at the returned JObject to verify the actual state of the webjob and other properties.
If you want the WebJob to not stop you need to make sure your scm site is alive.
So the keep-alive requests should go to https://sitename.scm.azurewebsites.net and these requests need to be authenticated (basic auth using your deployment credentials).

Creation of an Azure Media Service Job fails after deploying to server (iis8)

I am having some problems with the Azure Media Service. I have successfully managed to upload a file to Azure Media Service and encoded it as mp4, but this will only work on localhost in visual studio. When I deploy the following code to a Azure Virtual Machine, Windows Datacenter 2012 edition, the file is uploaded but the encoding job is never created.
Any help is much appreciated!
This is the code I use to create the job:
// Declare a new job.
job= _context.Jobs.Create("New Job");
// Get a media processor reference, and pass to it the name of the
// processor to use for the specific task.
IMediaProcessor processor = GetLatestMediaProcessorByName("Windows Azure Media Encoder");
// Create a task with the encoding details, using a string preset.
ITask task = job.Tasks.AddNew("LSVEncodingTask_" + v.CompanyId + "_user" + v.UserId,
processor,
"H264 Broadband 1080p",
TaskOptions.ProtectedConfiguration);
// Specify the input asset to be encoded.
task.InputAssets.Add(asset);
// Add an output asset to contain the results of the job.
// This output is specified as AssetCreationOptions.None, which
// means the output asset is not encrypted.
task.OutputAssets.AddNew("output asset",
AssetCreationOptions.None);
// Use the following event handler to check job progress.
job.StateChanged += new
EventHandler<JobStateChangedEventArgs>(StateChanged);
// Launch the job.
job.Submit();
// Check job execution and wait for job to finish.
Task progressJobTask = job.GetExecutionProgressTask(CancellationToken.None);
EDIT:
Did some digging (logging...) and found the stack trace:
System.Security.Cryptography.CryptographicException: Access is denied.
at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
at Microsoft.WindowsAzure.MediaServices.Client.EncryptionUtils.SaveCertificateToStore(X509Certificate2 certToStore)
at Microsoft.WindowsAzure.MediaServices.Client.ContentKeyBaseCollection.GetCertificateForProtectionKeyId(IMediaDataServiceContext dataContext, String protectionKeyId)
at Microsoft.WindowsAzure.MediaServices.Client.JobData.ProtectTaskConfiguration(TaskData task, X509Certificate2& certToUse, IMediaDataServiceContext dataContext)
at Microsoft.WindowsAzure.MediaServices.Client.JobData.InnerSubmit(IMediaDataServiceContext dataContext)
at Microsoft.WindowsAzure.MediaServices.Client.JobData.SubmitAsync()
at Microsoft.WindowsAzure.MediaServices.Client.JobData.Submit()
at Livescreen.Domain.Implementation.AzureMediaManager.CreateEncodingJob(IAsset asset, String inputMediaFilePath, String outputFolder, Int32 videoId)
The solution was found here:
When running on IIS7 - you can specify to load the user profile of the App Pool. This gives you access to the per user store of the app
pool account.
Solution:

Resources