Azure Batch with C# Application using System.Diagnostics.Process - azure

I am using Azure Batch with a C# application. The overall C# application is responsible for doing many things. Unfortunately, 1 operation that works on my local machine does not work within an Azure Batch application.
The operation that works locally but not as an Azure C# application is programmatically starting a System.Diagnostics. A process that executes a FFmpeg argument to take a screenshot of a video. The process seems to run, but, the jpg file is not created. Locally, the code will create a jpg but as an Azure Batch app, the file is not created.
I have set the image file to be created in the AZ_BATCH_TASK_WORKING_DIR folder. I have reduced the code to just do the 1 operation that does not work as an Azure application and here it is:
using System;
using System.Diagnostics;
using System.IO;
namespace BatchAppWithFfmpegProcess
{
internal class Program
{
static void Main(string[] args)
{
#if DEBUG
string workingDirectory = Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName);
#else
/*
Within each Tasks directory, the Batch service creates a working directory (wd) whose unique path is
specified by the AZ_BATCH_TASK_WORKING_DIR environment variable. This directory provides read/write access
to the task. The task can create, read, update, and delete files under this directory.
This directory is retained based on the RetentionTime constraint that is specified for the task.
The stdout.txt and stderr.txt files are written to the Tasks folder during the execution of the task.
*/
string workingDirectory = Environment.GetEnvironmentVariable("AZ_BATCH_TASK_WORKING_DIR");
#endif
var applicationPath = Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName);
string ffmpegPath = Path.Combine(applicationPath, #"lib\");
string videoFilePath = Path.Combine(applicationPath, "MichaelJacksonSmoothCriminal_Trimmed.mp4");
string thumbnailFilePath = Path.Combine(workingDirectory, "MichaelJacksonSmoothCriminal_Trimmed.jpg");
if (File.Exists(thumbnailFilePath))
{
File.Delete(thumbnailFilePath);
}
string arguments = $"-i \"{videoFilePath}\" -ss 00:00:01.000 -frames:v 1 \"{thumbnailFilePath}\"";
var startInfo = new ProcessStartInfo
{
FileName = ffmpegPath + $"\\ffmpeg.exe",
Arguments = arguments,
RedirectStandardError = false,
RedirectStandardOutput = true,
CreateNoWindow = true,
UseShellExecute = false,
WorkingDirectory = ffmpegPath
};
using (var process = new Process { StartInfo = startInfo })
{
process.Start();
process.WaitForExit();
}
if (File.Exists(thumbnailFilePath))
{
Console.WriteLine("Hurray, it worked!!!");
}
else
{
Console.WriteLine("File was not created.");
}
}
}
}
Perhaps it is impossible to use System.Diagnostics.Process to create files? I have tried to make this as easy as possible to reproduce with the following:
clone the code at:
https://github.com/Dapp3rDanH/AzBatchFfmpegProcess.git
Using Visual Studio 2022, "Publish" BatchAppWithFfmpegProcess code to a Folder using
"Deployment mode" = Self-Contained
Target Framework of net5.0
Target runtime = win-x64.
Create a BatchAppWithFfmpegProcess.zip zip file of the publish folder. Make sure the BatchAppWithFfmpegProcess.exe is in the root of the zip.
Create a Batch account.
Add a Batch application using the BatchAppWithFfmpegProcess.zip file with appId of BatchAppWithFfmpegProcess and a version of 1.
Add a Pool called "Pool1" with 1 dedicated node using microsoftwindowsserver windowsserver 2022-datacenter-core (latest). Add BatchAppWithFfmpegProcess version 1 to Pool1.
Create a new Job with a Task with the command of:
cmd /c %AZ_BATCH_APP_PACKAGE_BatchAppWithFfmpegProcess#1%\BatchAppWithFfmpegProcess.exe
If you check out the stdout.txt file of the task, you will see "File was not created". Any way to get this to work?
Thanks!
Dan

The issue was related to my pool being based upon "microsoftwindowsserver windowsserver 2022-datacenter-core (latest)".
Switching to microsoftwindowsserver windowsserver 2016-datacenter (latest) fixed the issue.

Related

Unable to get environment variable in azure batch VM

Here is my Azure batch configuration which I am trying to create from microsoft tutorials here and here
I am trying to get env variables defined here
CloudPool pool = batchClient.PoolOperations.CreatePool(
poolId: PoolId,
targetDedicatedComputeNodes: PoolNodeCount,
virtualMachineSize: PoolVMSize,
virtualMachineConfiguration: vmConfiguration);
// Specify the application and version to install on the compute nodes
pool.ApplicationPackageReferences = new List<ApplicationPackageReference>
{
new ApplicationPackageReference {
ApplicationId = "7Zip",
Version = "19.00" }
};
// Commit the pool so that it's created in the Batch service. As the nodes join
// the pool, the specified application package is installed on each.
await pool.CommitAsync();
CloudJob job = batchClient.JobOperations.CreateJob();
job.Id = JobId;
job.PoolInformation = new PoolInformation { PoolId = PoolId };
await job.CommitAsync();
string taskId = "blendertask01";
string commandLine =
#"cmd /c echo %AZ_BATCH_APP_PACKAGE_7Zip%";
CloudTask blenderTask = new CloudTask(taskId, commandLine);
batchClient.JobOperations.AddTask(JobId, blenderTask);
I am expecting the output of cmd /c echo %AZ_BATCH_APP_PACKAGE_7Zip% to give me the path where I can find my application 7zip so that i can install it however I dont get that.
Instead i get %AZ_BATCH_APP_PACKAGE_7Zip%
Instead of using AZ_BATCH_APP_PACKAGE_7Zip we should specify the version also
thus it would become %AZ_BATCH_APP_PACKAGE_7Zip#19.00%\7z1900-x64.exe
This information is not clearly defined however after few hit and tries I discovered it
Also that the env variables are not visible when we log on using remote account.

How to check whether a Node.js service is running - in my case Azurite emulator?

I am developing a C# application that should run in Azure. I want to use the Azurite emulator to test it locally. What I want to achieve is: Have my tests detect whether Azurite is running and abort quickly with a nice error message if it is not running.
Apparently Azurite runs on Node.js.
With the old Microsoft Azure Storage Emulator, I can check it like this:
public static class AzureStorageEmulatorDetector
{
public static bool IsRunning()
{
const string exePath = #"C:\Program Files (x86)\Microsoft SDKs\Azure\Storage Emulator\AzureStorageEmulator.exe";
if (!File.Exists(exePath))
return false;
var processStartInfo = new ProcessStartInfo {FileName = exePath, Arguments = "status", RedirectStandardOutput = true};
var process = new Process {StartInfo = processStartInfo};
process.Start();
process.WaitForExit();
var processOutput = process.StandardOutput.ReadToEnd();
return processOutput.Contains("IsRunning: True");
}
}
I want to accomplish something similar with Azurite.
I have installed Azurite like this:
npm install -g azurite
I run it like this:
azurite --silent --location C:\temp\Azurite --debug c:\temp\Azurite\debug.log
I notice that the Azurite command-line application has no parameter that tells me whether it is already running. And when I start Azurite from the console I don't see any process or service in Task Explorer called anything like "azurite". So I don't know what process I'm supposed to check for.
EDIT: Apparently Azurite runs on Node.js. There is indeed a process called node.exe running, but that's not a sufficient condition. Can I query my running Node.js instance and get it to tell me what it is doing?
I am on Windows.
Does anyone know?
Inspired by the comment by Ivan Yang and this answer I did this:
private static bool IsAzuriteRunning()
{
// If Azurite is running, it will run on localhost and listen on port 10000 and/or 10001.
IPAddress expectedIp = new(new byte[] {127, 0, 0, 1});
var expectedPorts = new[] {10000, 10001};
var activeTcpListeners = IPGlobalProperties.GetIPGlobalProperties().GetActiveTcpListeners();
var relevantListeners = activeTcpListeners.Where(t =>
expectedPorts.Contains(t.Port) &&
t.Address.Equals(expectedIp))
.ToList();
return relevantListeners.Any();
}
EDIT: Alternatively, check out this thread on their GitHub for other possibilities: https://github.com/Azure/Azurite/issues/734

Azure Linux App Service with .Net Core Stack. Unable to use NodeJS

I am hosting a .NET Core Application on MS Azure (on a Linux Service Plan) and I want to run some NodeJS code in the .NET Core Application. I did this a while ago on a Windows Service Plan, there it was working. Now I am trying with a Linux Plan and it is not working.
First I was trying to use "Jering.Javascript.NodeJS" and then also "INodeServices" from Microsoft (which is obsolete). But "node" was not found. I also tried to start directly a Process (Code below), but also not working. "node" is not found.
var proc = new System.Diagnostics.Process
{
StartInfo = new System.Diagnostics.ProcessStartInfo
{
FileName = "node",
Arguments = " -v",
RedirectStandardOutput = true
}
};
result += "RUN: " + proc.StartInfo.FileName;
proc.Start();
var reader = proc.StandardOutput;
NodeJS is installed on the server and also the command works there but it seems that the .NET Core app is hosted as docker and does not have any access outside to run NodeJS. Image
I found a useful information here.
The problem is that Node is not present in the container so it is
necessary to have a script to install and start it before starting the
app itself.
Reproduceļ¼š
Here is my script:
//using System.Diagnostics;
ProcessStartInfo startinfo = new ProcessStartInfo();
startinfo.FileName = "bash";
//startinfo.FileName = "/etc/opt/nodejs/14.15.0/bin/node"; //it's no use even node package located here.
Process process = new Process();
process.StartInfo = startinfo;
process.StartInfo.UseShellExecute = false;
process.StartInfo.RedirectStandardInput = true;
process.StartInfo.RedirectStandardOutput = true;
process.Start();
//install and start nodejs
process.StandardInput.WriteLine("apt-get install curl");
process.StandardInput.WriteLine("curl -sL https://deb.nodesource.com/setup_12.x | bash");
process.StandardInput.WriteLine("apt-get install -y nodejs");
//Run "node -v"
process.StandardInput.WriteLine("node -v");
string line = string.Empty;
while (!process.StandardOutput.EndOfStream)
{
line = process.StandardOutput.ReadLine();
_logger.LogInformation(line);
}
process.WaitForExit();
return string.Empty;
It works on my .net Core app based on Linux.
I think I found a better solution ;)
In an app service you can mount a storage. In my case I mounted a storage, which contains the nodeJS lib.
Azure Portal Screenshot
Now i can execute the following code:
string result = "";
var proc = new System.Diagnostics.Process
{
StartInfo = new System.Diagnostics.ProcessStartInfo
{
FileName = "/externallibs/node/bin/node",
Arguments = " -v",
RedirectStandardOutput = true
}
};
result += "RUN: " + proc.StartInfo.FileName;
proc.Start();
var reader = proc.StandardOutput;
return result + reader.ReadToEnd();
You can create on azure portal an environment var named POST_BUILD_COMMAND with a command to fix your environment path.
Linux Service Plans runs on Oryx which is documented here
POST_BUILD_COMMAND=PATH=/usr/bin/node:$PATH

Windows UWP CreateFIle2 cannot read file in ApplicationData.LocalFolder

I am trying to build UWP app in C#. My app has a native library written in C++. Whenever the app tries to read a file in ApplicationData.LocalFolder, CreateFile2 api is returning ERROR_NOT_SUPPORTED_IN_APPCONTAINER. The file exists in the path specified to this api.
This is the sequence of operation in my app.
Launch app. App creates file & writes some data
Later on based on user input app tries to read data in this file
Step 1 is working fine. App is able to create the file & write data in it. Only when app tries to access it later on, does it get this error.
I get the path to ApplicationData.LocalFolder using
Windows.Storage.ApplicationData.Current.LocalFolder.Path
This is the actual path I see in the app:
C:\Users\xxxxx\AppData\Local\Packages\ac7a11e4-c1d6-4d37-b9eb-a4b0dc8f67b8_yyjvd81p022em\LocalState\temp.txt
My code is as below:
CREATEFILE2_EXTENDED_PARAMETERS ms_param = {0};
ms_param.dwSize = sizeof(CREATEFILE2_EXTENDED_PARAMETERS);
ms_param.dwFileAttributes = FILE_ATTRIBUTE_READONLY;
ms_param.dwFileFlags = FILE_FLAG_NO_BUFFERING;
ms_param.dwSecurityQosFlags = SECURITY_DELEGATION;
ms_param.lpSecurityAttributes = NULL;
ms_param.hTemplateFile = NULL;
g_hfile = CreateFile2(filename, GENERIC_READ, FILE_SHARE_READ|FILE_SHARE_WRITE, OPEN_EXISTING, &ms_param);
if (g_hfile == INVALID_HANDLE_VALUE)
{
return GetLastError();
}
I tried CreateFile2 with both OPEN_EXISTING & OPEN_ALWAYS option for dwCreationDisposition parameter, but I see the same error in either case.
I had similar issue with CreateFile2 earlier. But that was an problem with my app & I have fixed that issue. This time though the file is available within the LocalFolder, still I get the error.
The problem here is related to the dwSecurityQosFlags you've set in CREATEFILE2_EXTENDED_PARAMETERS.
When called from a Windows Store app, CreateFile2 is simplified. You can open only files or directories inside the ApplicationData.LocalFolder or Package.InstalledLocation directories. You can't open named pipes or mailslots or create encrypted files (FILE_ATTRIBUTE_ENCRYPTED).
The dwSecurityQosFlags parameter specifies SQOS information. In Windows Stroe app, we can only set it to SECURITY_ANONYMOUS. Using other flag will raise ERROR_NOT_SUPPORTED_IN_APPCONTAINER exception. This indicates that it is not supported in UWP app.
Following is the code I used to test:
StorageFolder^ localFolder = ApplicationData::Current->LocalFolder;
String^ path = localFolder->Path;
path += L"\\MyFile.txt";
CREATEFILE2_EXTENDED_PARAMETERS ms_param = { 0 };
ms_param.dwSize = sizeof(CREATEFILE2_EXTENDED_PARAMETERS);
ms_param.dwFileAttributes = FILE_ATTRIBUTE_READONLY;
ms_param.dwFileFlags = FILE_FLAG_NO_BUFFERING;
ms_param.dwSecurityQosFlags = SECURITY_ANONYMOUS;
ms_param.lpSecurityAttributes = NULL;
ms_param.hTemplateFile = NULL;
HANDLE g_hfile = CreateFile2(path->Data(), GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_WRITE, OPEN_EXISTING, &ms_param);
DWORD error = GetLastError();
If I don't have "MyFile.txt" under LocalFolder, I will get ERROR_FILE_NOT_FOUND exception, otherwise it will be ERROR_SUCCESS.

Path for getting the folder when program run as Service

I have an executable that is to be run as a Service in windows. Since the service runs as Local system, what should be the folder I should be writing any data the program uses. As currently, I use %LocalAppData% but when the exe runs as Service it points me to
C:\Windows\System32\config\systemprofile\AppData
I used following code:
std::string GetLocalAppDataPath()
{
HANDLE hfile;
TCHAR szPath[MAX_PATH];
if(SUCCEEDED(SHGetFolderPath(NULL,CSIDL_LOCAL_APPDATA,NULL,0, szPath)))
{
std::string path = boost::lexical_cast<std::string>(szPath);
boost::replace_all(path, "\\", "\\\\");
return path;
}
}
If I call the above code as:
std::string app_data_path = GetLocalAppDataPath();
std::string log_folder_path = app_data_path + "\\\\lpa\\\\output\\\\";
I get C:\WINDOWS\system32\config\systemprofile\AppData\Local\lpa\output\ instead of my own Local App Data Folder. So Should I be using other folder that LocalSystem can access.?
did you try this:
Get CSIDL_LOCAL_APPDATA path for any user on Windows

Resources