Azure ARM uniqueString function mimic - arm-template

I need to deploy Sql Databases into an Azure Sql Server using to ways: the ARM template way, and a more custom way using C# code. There's a ARM template function called uniqueString(string) that generate a pseudo random hash of a given string. It's a deterministic pure function.
I need to find a way to exactly mimic the behaviour of this function from my C# code. ie I need to reproduce this function into my C# code.
Where can i find the algorithm used by the ARM Api ?
MSDN reference for uniqueString()

Update 2023-01-05 - As suggested by other answers, there's an easier way now - just reference the Azure.Deployments.Expression nuget package which contains all of the Arm functions and then use the following convenience wrapper:
using Azure.Deployments.Expression.Expressions;
using Newtonsoft.Json.Linq;
public static class ArmFunctions
{
public static string? UniqueString(params string[] values)
{
var parameters = values.Select(
arg => new FunctionArgument(
JToken.FromObject(arg)
)
).ToArray();
var result = ExpressionBuiltInFunctions.Functions
.EvaluateFunction("uniqueString", parameters, null);
return result.Value<string>();
}
}
// "zcztcwvu6iyg6"
var unique = ArmFunctions.UniqueString("tyeth");
Original answer for posterity:
I've been researching this myself on and off for a few years now, and I've finally hit paydirt...
// "zcztcwvu6iyg6"
var unique = ArmUniqueString("tyeth");
My ArmUniqueString function is a wrapper around some dlls that are distributed with the Azure Stack Hub Development Kit which is basically a virtual machine image that contains the Azure server-side platform that you can run locally...
private static string ArmUniqueString(string originalString)
{
var assembly = Assembly.GetAssembly(
typeof(Microsoft.WindowsAzure.ResourceStack.Frontdoor.Templates.Engines.TemplateEngine)
);
var functions = assembly.GetType(
"Microsoft.WindowsAzure.ResourceStack.Frontdoor.Templates.Expressions.TemplateExpressionBuiltInFunctions"
);
var uniqueString = functions.GetMethod(
"UniqueString",
BindingFlags.Static | BindingFlags.NonPublic
);
var parameters = new object[] {
"uniqueString",
new JToken[] {
(JToken)originalString
}
};
var result = uniqueString.Invoke(null, parameters).ToString();
return result;
}
You'll need to download the Azure Stack Hub Development Kit and unpack it to get the dlls:
Download the Azure Stack Hub Development Kit - warning: it's about 22Gb!
Run the installer to unpack a 55Gb *.vhdx
Mount the *.vhdx, or expand / unpack it locally
Inside the *.vhdx, find this file and unzip it somewhere:
CloudBuilder\CloudDeployment\NuGetStore\Microsoft.AzureStack.Setup.Services.ResourceManager.5.20.1335.300.nupkg
The content\Website\bin folder inside the *.nupkg contains the necessary dlls
To use them, add an assembly reference to Microsoft.WindowsAzure.ResourceStack.Frontdoor.Templates.dll (it has some dependencies on other files in the bin folder) and that contains the TemplateExpressionBuiltInFunctions class. The code above just uses reflection to invoke the private UniqueString function from that assembly, with a little bit of work to marshal the parameters into appropriate JToken types.
If you wanted to dig into the implementation details you could probably run a decompiler against the assembly to find out what it's doing under the covers...
Note - credits go to this blog article for pointing me in the right direction:
https://the.agilesql.club/2017/12/azure-arm-template-function-internals/

I found some PowerShell code to do this here: https://blogs.technet.microsoft.com/389thoughts/2017/12/23/get-uniquestring-generate-unique-id-for-azure-deployments/
I converted this code to C#:
public string GetUniqueString(string id, int length = 13)
{
string result = "";
var buffer = System.Text.Encoding.UTF8.GetBytes(id);
var hashArray = new System.Security.Cryptography.SHA512Managed().ComputeHash(buffer);
for(int i = 1; i <= length; i++)
{
var b = hashArray[i];
var c = Convert.ToChar((b % 26) + (byte)'a');
result = result + c;
}
return result;
}

This function is released in nuget: https://www.nuget.org/packages/Azure.Deployments.Expression/
The implementation:
https://msazure.visualstudio.com/One/_git/AzureUX-Deployments?path=%2Fsrc%2FExpressions%2FExpressions%2FExpressionBuiltInFunctions.cs&version=GBmaster&_a=contents
Example:
using Azure.Deployments.Expression.Expressions;
using Newtonsoft.Json.Linq;
var funcs = ExpressionBuiltInFunctions.Functions;
var jt = new JTokenExpression("test");
var output = funcs.EvaluateFunction("uniqueString", new JToken[] { jt.Value }).ToString();

Sam Cogan wrote a blog post on how to do this in C# here: https://samcogan.com/using-bicep-functions-in-c-if-you-really-want-to/
Inspired by Sam, I wrote a PowerShell module (for PowerShell 7) that does the same. You can install the module by running Install-Module -Name AzExpression which will give you command called New-AzUniqueString.
Here is an example on how to use it:
New-AzUniqueString -InputStrings 'test', 'value'
Which will output: bhxq2thzm5dym

I finally found a workaround. I used a very simple ARM template which goal is to only output the result of the uniqueStringcommand. Then I fetch this output in my C# code. This solution is not really the quickest one ;-), but it works as desired.

Here is a bicep template which will output a uniqueString for a resource group:
output unique string = uniqueString(resourceGroup().id)

Related

Azure Batch with C# Application using System.Diagnostics.Process

I am using Azure Batch with a C# application. The overall C# application is responsible for doing many things. Unfortunately, 1 operation that works on my local machine does not work within an Azure Batch application.
The operation that works locally but not as an Azure C# application is programmatically starting a System.Diagnostics. A process that executes a FFmpeg argument to take a screenshot of a video. The process seems to run, but, the jpg file is not created. Locally, the code will create a jpg but as an Azure Batch app, the file is not created.
I have set the image file to be created in the AZ_BATCH_TASK_WORKING_DIR folder. I have reduced the code to just do the 1 operation that does not work as an Azure application and here it is:
using System;
using System.Diagnostics;
using System.IO;
namespace BatchAppWithFfmpegProcess
{
internal class Program
{
static void Main(string[] args)
{
#if DEBUG
string workingDirectory = Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName);
#else
/*
Within each Tasks directory, the Batch service creates a working directory (wd) whose unique path is
specified by the AZ_BATCH_TASK_WORKING_DIR environment variable. This directory provides read/write access
to the task. The task can create, read, update, and delete files under this directory.
This directory is retained based on the RetentionTime constraint that is specified for the task.
The stdout.txt and stderr.txt files are written to the Tasks folder during the execution of the task.
*/
string workingDirectory = Environment.GetEnvironmentVariable("AZ_BATCH_TASK_WORKING_DIR");
#endif
var applicationPath = Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.FileName);
string ffmpegPath = Path.Combine(applicationPath, #"lib\");
string videoFilePath = Path.Combine(applicationPath, "MichaelJacksonSmoothCriminal_Trimmed.mp4");
string thumbnailFilePath = Path.Combine(workingDirectory, "MichaelJacksonSmoothCriminal_Trimmed.jpg");
if (File.Exists(thumbnailFilePath))
{
File.Delete(thumbnailFilePath);
}
string arguments = $"-i \"{videoFilePath}\" -ss 00:00:01.000 -frames:v 1 \"{thumbnailFilePath}\"";
var startInfo = new ProcessStartInfo
{
FileName = ffmpegPath + $"\\ffmpeg.exe",
Arguments = arguments,
RedirectStandardError = false,
RedirectStandardOutput = true,
CreateNoWindow = true,
UseShellExecute = false,
WorkingDirectory = ffmpegPath
};
using (var process = new Process { StartInfo = startInfo })
{
process.Start();
process.WaitForExit();
}
if (File.Exists(thumbnailFilePath))
{
Console.WriteLine("Hurray, it worked!!!");
}
else
{
Console.WriteLine("File was not created.");
}
}
}
}
Perhaps it is impossible to use System.Diagnostics.Process to create files? I have tried to make this as easy as possible to reproduce with the following:
clone the code at:
https://github.com/Dapp3rDanH/AzBatchFfmpegProcess.git
Using Visual Studio 2022, "Publish" BatchAppWithFfmpegProcess code to a Folder using
"Deployment mode" = Self-Contained
Target Framework of net5.0
Target runtime = win-x64.
Create a BatchAppWithFfmpegProcess.zip zip file of the publish folder. Make sure the BatchAppWithFfmpegProcess.exe is in the root of the zip.
Create a Batch account.
Add a Batch application using the BatchAppWithFfmpegProcess.zip file with appId of BatchAppWithFfmpegProcess and a version of 1.
Add a Pool called "Pool1" with 1 dedicated node using microsoftwindowsserver windowsserver 2022-datacenter-core (latest). Add BatchAppWithFfmpegProcess version 1 to Pool1.
Create a new Job with a Task with the command of:
cmd /c %AZ_BATCH_APP_PACKAGE_BatchAppWithFfmpegProcess#1%\BatchAppWithFfmpegProcess.exe
If you check out the stdout.txt file of the task, you will see "File was not created". Any way to get this to work?
Thanks!
Dan
The issue was related to my pool being based upon "microsoftwindowsserver windowsserver 2022-datacenter-core (latest)".
Switching to microsoftwindowsserver windowsserver 2016-datacenter (latest) fixed the issue.

Can asynchronous module definitions be used with abstract syntax trees on v8 engine to read third party dependencies? This is for Cloudflare Workers

I understand eval string-to-function is impossible to use on the browsers' application programming interfaces, but there must be another strategy to use third party dependencies without node.js on v8 engine, given Cloudflare does it in-house, unless they disable the exclusive method by necessity or otherwise on their edge servers for Workers. I imagine I could gather the AST of the commonjs module, as I was able to by rollup watch, but what might the actual steps be, by tooling? I mention AMD for it seems to rely on string-to-function (to-which I've notice Mozilla MDN says nothing much about it).
I have been exploring the require.js repositories, and they either use eval or AST
function DEFNODE(type, props, methods, base) {
if (arguments.length < 4) base = AST_Node;
if (!props) props = [];
else props = props.split(/\s+/);
var self_props = props;
if (base && base.PROPS) props = props.concat(base.PROPS);
var code = "return function AST_" + type + "(props){ if (props) { ";
for (var i = props.length; --i >= 0; ) {
code += "this." + props[i] + " = props." + props[i] + ";";
}
var proto = base && new base();
if ((proto && proto.initialize) || (methods && methods.initialize))
code += "this.initialize();";
code += "}}";
//constructor
var cnstor = new Function(code)();
if (proto) {
cnstor.prototype = proto;
cnstor.BASE = base;
}
if (base) base.SUBCLASSES.push(cnstor);
cnstor.prototype.CTOR = cnstor;
cnstor.PROPS = props || null;
cnstor.SELF_PROPS = self_props;
cnstor.SUBCLASSES = [];
if (type) {
cnstor.prototype.TYPE = cnstor.TYPE = type;
}
if (methods)
for (i in methods)
if (HOP(methods, i)) {
if (/^\$/.test(i)) {
cnstor[i.substr(1)] = methods[i];
} else {
cnstor.prototype[i] = methods[i];
}
}
//a function that returns an object with [name]:method
cnstor.DEFMETHOD = function (name, method) {
this.prototype[name] = method;
};
if (typeof exports !== "undefined") exports[`AST_${type}`] = cnstor;
return cnstor;
}
var AST_Token = DEFNODE(
"Token",
"type value line col pos endline endcol endpos nlb comments_before file raw",
{},
null
);
https://codesandbox.io/s/infallible-darwin-8jcl2k?file=/src/mastercard-backbank/uglify/index.js
https://www.youtube.com/watch?v=EF7UW9HxOe4
Is it possible to make a C++ addon just to add a default object for
node.js named exports or am I Y’ing up the wrong X
'.so' shared library for C++ dlopen/LoadLibrary (or #include?)
“I have to say that I'm amazed that there is code out there that loads one native addon from another native addon! Is it done by acquiring and then calling an instance of the require() function, or perhaps by using uv_dlopen() directly?”
N-API: An api for embedding Node in applications
"[there is no ]napi_env[ just yet]."
node-api: allow retrieval of add-on file name - Missing module in Init
Andreas Rossberg - is AST parsing, or initialize node.js abstraction for native c++, enough?
v8::String::NewFromUtf8(isolate, "Index from C++!");
Rising Stack - Node Source
"a macro implicit" parameter - bridge object between
C++ and JavaScript runtimes
extract a function's parameters and set the return value.
#include <nan.h>
int build () {
NAN_METHOD(Index) {
info.GetReturnValue().Set(
Nan::New("Index from C++!").ToLocalChecked()
);
}
}
// Module initialization logic
NAN_MODULE_INIT(Initialize) {
/*Export the `Index` function
(equivalent to `export function Index (...)` in JS)*/
NAN_EXPORT(target, Index);
}
New module "App" Initialize function from NAN_MODULE_INIT (an atomic?-macro)
"__napi_something doesn't exist."
"node-addon-API module for C++ code (N-API's C code's headers)"
NODE_MODULE(App, Initialize);
Sep 17, 2013, 4:42:17 AM to v8-u...#googlegroups.com "This comes up
frequently, but the answer remains the same: scrap the idea. ;)
Neither the V8 parser nor its AST are designed for external
interfacing. In particular (1) V8's AST does not necessarily reflect
JavaScript syntax 1-to-1, (2) we change it all the time, and (3) it
depends on various V8 internals. And since all these points are
important for V8, don't expect the situation to change.
/Andreas"
V8 c++: How to import module via code to script context (5/28/22, edit)
"The export keyword may only be used in a module interface unit.
The keyword is attached to a declaration of an entity, and causes that
declaration (and sometimes the definition) to become visible to module
importers[ - except for] the export keyword in the module-declaration, which is just a re-use of the keyword (and does not actually “export” ...entities)."
SyntheticModule::virtual
ScriptCompiler::CompileModule() - "Corresponds to the ParseModule abstract operation in the ECMAScript specification."
Local<Function> foo_func = ...;//external
Local<Module> module = Module::CreateSyntheticModule(
isolate, name,
{String::NewFromUtf8(isolate, "foo")},
[](Local<Context> context, Local<Module> module) {
module->SetSyntheticModuleExport(
String::NewFromUtf8(isolate, "foo"), foo_func
);
});
Context-Aware addons from node.js' commonjs modules
export module index;
export class Index {
public:
const char* app() {
return "done!";
}
};
import index;
import <iostream>;
int main() {
std::cout << Index().app() << '\n';
}
node-addon-api (new)
native abstractions (old)
"Thanks to the crazy changes in V8 (and some in Node core), keeping native addons compiling happily across versions, particularly 0.10 to 0.12 to 4.0, is a minor nightmare. The goal of this project is to store all logic necessary to develop native Node.js addons without having to inspect NODE_MODULE_VERSION and get yourself into a macro-tangle[ macro = extern atomics?]."
Scope Isolate (v8::Isolate), variable Local (v8::Local)
typed_array_to_native.cc
"require is part of the Asynchronous Module Definition AMD API[, without "string-to-function" eval/new Function()],"
node.js makes objects, for it is written in C++.
"According to the algorithm, before finding
./node_modules/_/index.js, it tried looking for express in the
core Node.js modules. This didn’t exist, so it looked in node_modules,
and found a directory called _. (If there was a
./node_modules/_.js, it would load that directly.) It then
loaded ./node_modules/_/package.json, and looked for an exports
field, but this didn’t exist. It also looked for a main field, but
this didn’t exist either. It then fell back to index.js, which it
found. ...require() looks for node_modules in all of the parent directories of the caller."
But java?
I won't accept this answer until it works, but this looks promising:
https://developer.oracle.com/databases/nashorn-javascript-part1.html
If not to run a jar file or something, in the Worker:
https://github.com/nodyn/jvm-npm
require and build equivalent in maven, first, use "dist/index.js".
Specifically: [ScriptEngineManager][21]
https://stackoverflow.com/a/15787930/11711280
Actually: js.commonjs-require experimental
https://docs.oracle.com/en/graalvm/enterprise/21/docs/reference-manual/js/Modules/
Alternatively/favorably: commonjs builder in C (v8 and node.js)
https://www.reddit.com/r/java/comments/u7elf4/what_are_your_thoughts_on_java_isolates_on_graalvm/
Here I will explore v8/node.js src .h and .cc for this purpose
https://codesandbox.io/s/infallible-darwin-8jcl2k?file=/src/c.cpp
I'm curious why there is near machine-level C operability in Workers if not to use std::ifstream, and/or build-locally, without node.js require.

How to load types in the System namespace with Mono.Cecil?

I'm using Mono.Cecil (0.9.5.4) to inject code into some of my assemblies. Some of the calls I need to make are to objects in the System.ComponentModel namespace. How can I find those 'MethodReferences' that I need to call?
What I tried:
AssemblyDefinition assembly = AssemblyDefinition.ReadAssembly(dllPath);
var objectModelRef = assembly.MainModule.AssemblyReferences.First(i => i.Name == "System.ObjectModel")
var objectModelAssembly = assembly.MainModule.AssemblyResolver.Resolve(objectModelRef);
But then objectmodelAssembly.MainModule.Types has no actual types in it.
I also tried this:
AssemblyDefinition assembly = AssemblyDefinition.ReadAssembly(dllPath);
var system = new DefaultAssemblyResolver().Resolve("System");
var objectIWantToInject = assembly.MainModule.Import(FindType(...));
This works fine on a machine with the full .net 4.5 installed. But since my assembly is a PCL, when I try executing on WinPhone, I get FileNotFound for 'System'.
So if I wanted to get an instance of TypeDefinition for System.ComponentModel.ProgressChangedEventArgs that I could then make calls to some of the methods on, how would I?
In your code:
AssemblyDefinition assembly = AssemblyDefinition.ReadAssembly(dllPath);
var system = new DefaultAssemblyResolver().Resolve("System");
var objectIWantToInject = assembly.MainModule.Import(FindType(...));
I think you are missing at least one import, e.g.
assembly.MainModule.Import(typeof(System.ObjectModel);
I believe you will also need to "Resolve" it to walk further down the tree.
Here is a working sample for anyone that is feeling the same pain:
assembly.MainModule.Import(typeof(string));
var methodBaseRef = assembly.MainModule.Import(typeof(System.Reflection.MethodBase));
var getMemberInfo = assembly.MainModule.Import(typeof(System.Reflection.MemberInfo));
var getMemberInfoResolver = getMemberInfo.Resolve();
var getCurrentMethodRef = assembly.MainModule.Import(typeof(System.Reflection.MethodBase).GetMethod("GetCurrentMethod"));
var get_DeclaringType = assembly.MainModule.Import(getMemberInfoResolver.Methods.Single(m => m.Name == "get_DeclaringType"));
var getTypeInfo = assembly.MainModule.Import(typeof(Type));
var name = assembly.MainModule.Import(typeof(Type).GetMethod("Name"));

Finding builds in Team Explorer's "My Builds"

I'm writing a Visual Studio 2012 add-in which extends Build Explorer - basically, I add a context menu option for every build (completed or running, but not queued). Following a blog post about doing this in VS2010, I managed to do so for builds that appear in Builder Explorer - hooray!
Now, my context menu also appear in Team Explorer's Builds pages, My Builds section. However, when I get the callback, I can't find the actual builds anywhere!
Here's my beforeQueryStatus event handler, where I try to find out whether I have a build to show or not:
private void OpenCompletedInBuildExplorerBeforeQueryStatus(object sender, EventArgs e)
{
var cmd = (OleMenuCommand)sender;
var vsTfBuild = (IVsTeamFoundationBuild)GetService(typeof(IVsTeamFoundationBuild));
// This finds builds in Build Explorer window
cmd.Enabled = (vsTfBuild.BuildExplorer.CompletedView.SelectedBuilds.Length == 1
&& vsTfBuild.BuildExplorer.QueuedView.SelectedBuilds.Length == 0); // No build _requests_ are selected
// This tries to find builds in Team Explorer's Builds page, My Builds section
var teamExplorer = (ITeamExplorer)GetService(typeof(ITeamExplorer));
var page = teamExplorer.CurrentPage as Microsoft.TeamFoundation.Controls.WPF.TeamExplorer.TeamExplorerPageBase;
var vm = page.ViewModel;
// does not compile: 'Microsoft.TeamFoundation.Build.Controls.BuildsPageViewModel' is inaccessible due to its protection level
var vm_private = vm as Microsoft.TeamFoundation.Build.Controls.BuildsPageViewModel;
// But debugger shows that if it did, my builds would be here:
var builds = vm_private.MyBuilds;
}
Is there a way to get the list of builds?
More generally, is there a way to get some "window which this context menu belong to"? Currently I'm just looking around in parts of VS I assume would have builds...
I managed to get the build using reflection:
var teamExplorer = (ITeamExplorer)GetService(typeof(ITeamExplorer));
var BuildsPage = teamExplorer.CurrentPage as Microsoft.TeamFoundation.Controls.WPF.TeamExplorer.TeamExplorerPageBase;
var PageViewModel = BuildsPage.ViewModel as Microsoft.TeamFoundation.Controls.WPF.TeamExplorer.TeamExplorerPageViewModelBase;
// PageViewModel is actually Microsoft.TeamFoundation.Build.Controls.BuildsPageViewModel. But, it's private, so get SelectedBuilds through reflection
var SelectedBuilds = PageViewModel.GetType().GetProperty("SelectedBuilds").GetValue(PageViewModel) as System.Collections.IList;
if (SelectedBuilds.Count != 1)
{
cmd.Enabled = false;
return;
}
object BuildModel = SelectedBuilds[0];
// BuildModel is actually Microsoft.TeamFoundation.Build.Controls.BuildModel. But, it's private, so get UriToOpen through reflection
var BuildUri = BuildModel.GetType().GetProperty("UriToOpen").GetValue(BuildModel) as Uri;
// TODO: Use BuildUri...
cmd.Enabled = true;

Using an emscripten compiled C library from node.js

After following instructions on the emscripten wiki I have managed to compile a small C library. This resulted in an a.out.js file.
I was assuming that to use functions from this library (within node.js) something like this would have worked:
var lib = require("./a.out.js");
lib.myFunction('test');
However this fails. Can anyone help or point me to some basic tutorial related to this?
Actually, all the functions are already exported. Generated JavaScript contains following lines:
var ENVIRONMENT_IS_NODE = typeof process === 'object' && typeof require === 'function';
// …
if (ENVIRONMENT_IS_NODE) {
// …
module['exports'] = Module;
}
If you got a function called my_fun in your C code, then you'll have Module._my_fun defined.
There are some problems with this approach, though.
Optimizer may remove or rename some functions, so always specify them passing -s EXPORTED_FUNCTIONS="['_main','_fun_one','_fun_two']". Function signatures in C++ are bit mangled, so it's wise to extern "C" { … } the ones which you want to export.
Furthermore, such a direct approach requires JS to C type conversions. You may want to hide it by adding yet another API layer in file added attached with --pre-js option:
var Module = {
my_fun: function(some_arg) {
javascript to c conversion goes here;
Module._my_fun(converted_arg) // or with Module.ccall
}
}
Module object will be later enhanced by all the Emscripten-generated goodies, so don't worry that it's defined here, not modified.
Finally, you will surely want to consider Embind which is a mechanism for exposing nice JavaScript APIs provided by Emscripten. (Requires disabling newest fastcomp backend.)
The problem here is that your a.out.js file is going to look like this
function myFunction() {
...
}
Not like this
function myFunction() {
...
}
exports.myFunction = myFunction;
You need to write a build script that lists the tokens you want to publically export from each C program and appends exports.<token> = <token>;\n to the end of your file for each token.

Resources