According to the documentation, we should be able to add custom libraries as follows:
az synapse spark pool update --name testpool \
--workspace-name testsynapseworkspace --resource-group rg \
--package-action Add --package package1.jar package2.jar
However, when I try this with my python package whl files, I get an error message that the package does not exist.
> $new_package_names = "PACKAGE1-1.0.1-py3-none-any.whl PACKAGE2-1.0.6.3-py3-none-any.whl PACKAGE3-1.0.0-py3-none-any.whl PACKAGE4-1.0.1-py3-none-any.whl"
> az synapse spark pool update --name $pool_name --workspace-name $workspace_name --resource-group $resource_group --package-action Add --package $new_package_names
I receive the following error:
(LibraryDoesNotExistInWorkspace) The LibraryArtifact PACKAGE1-1.0.1-py3-none-any.whl PACKAGE2-1.0.6.3-py3-none-any.whl PACKAGE3-1.0.0-py3-none-any.whl PACKAGE4-1.0.1-py3-none-any.whl does not exist.
Code: LibraryDoesNotExistInWorkspace
Message: The LibraryArtifact PACKAGE1-1.0.1-py3-none-any.whl PACKAGE2-1.0.6.3-py3-none-any.whl PACKAGE3-1.0.0-py3-none-any.whl PACKAGE4-1.0.1-py3-none-any.whl does not exist.
The same works if I have only one package in the variable $new_package_names.
It looks to me like Azure thinks it's all one package instead of four different ones. All four are uploaded to the synapse workspace and available for selection when I do the same process manually. Does anyone know of a fix for this issue? Does it only work for .jar files for some reason?
Turns out that it really comes down to the format in which I pass the package names to the function. Something apparently changed internally as the previous way did not work anymore.
As MartinJaffer from Microsoft answered in the MS Q&A forum:
"""
If you are using az in powershell, there is a better way to go about this.
$new_package_names = "PACKAGE1-1.0.1-py3-none-any.whl" , "PACKAGE2-1.0.6.3-py3-none-any.whl" , "PACKAGE3-1.0.0-py3-none-any.whl" , "PACKAGE4-1.0.1-py3-none-any.whl"
az synapse spark pool update --name $pool_name --workspace-name $workspace_name --resource-group $resource_group --package-action Add --package #new_package_names
Here we changed new_package_names into an array type, and use the # splatter operator to seperate them.
As simpler example, it makes the following two excerpts be equivalent:
Copy-Item "test.txt" "test2.txt" -WhatIf
$ArrayArguments = "test.txt", "test2.txt"
Copy-Item #ArrayArguments -WhatIf
"""
Utilizing the splatter operator when passing the parameters worked perfectly.
Related
got a folder called data-asset which contains a yaml file with the following
type: uri_folder
name: <name_of_data>
description: <description goes here>
path: <path>
In a pipeline am referencing this using azure cli inline script using the following command az ml data create -f .yml but getting error
full error-D:\a\1\s\ETL\data-asset>az ml data create -f data-asset.yml
ERROR: 'ml' is misspelled or not recognized by the system.
Examples from AI knowledge base:
az extension add --name anextension
Add extension by name
trying to implement this https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-register-data-assets?tabs=CLI
how can a resolve this?
One of the workaround you can follow to resolve the above issue;
Based on this GitHub issue as suggested by #adba-msft .
Please make sure that you have upgraded your azure cli to latest and
Azure CLI ML extension v2 is being used.
To check and upgrade the cli we can use the below cmdlts:
az version
az upgrade
For more information please refer this similar SO THREAD|'create' is misspelled or not recognized by the system on az ml dataset create .
I did observe the same issue after trying the aforementioned suggestion by #Dor Lugasi-Gal it works for me with (in my case az ml -h) after installed the extension with az extension add -n ml -y can able to get the result of az ml -h without any error.
SCREENSHOT FOR REFERENCE:-
I've learned how to deploy .sh scripts to Azure with Azure CLI. But it seems like I have no clear understanding of how they work.
I'm creating the script that simply unarchives a .tgz archive in a current directory of Azure Web App, and then just deletes it. Quite simple:
New-Item ./startup.sh
Set-Content ./startup.sh '#!/bin/sh'
Add-Content ./startup.sh 'tar zxvf archive.tgz; rm-rf ./archive.tgz'
And then I deploy the script like this:
az webapp deploy --resource-group Group
--name Name
--src-path ./startup.sh
--target-path /home/site/wwwroot/startup.sh
--type=startup
Supposedly, it should appear in /home/site/wwwroot/, but for some reason it never does. No matter how I try. I thought it just gets executed and then deleted automatically (since I specified it as a startup script), but the archive is there, not unarchived at all.
My stack is .NET Core.
What am I doing wrong, and what's the right way to do what I need to do? Thank you.
I don't know if it makes sense, but I think the problem might be that you're using the target-path parameter while you should be using path instead.
From the documentation you cited, when describing the Azure CLI functionality, they state:
The CLI command uses the Kudu publish API to deploy the package and can be
fully customized.
The Kudu publish API reference indicates, when describing the different values for type and especially startup:
type=startup: Deploy a script that App Service automatically uses as the
startup script for your app. By default, the script is deployed to
D:\home\site\scripts\<name-of-source> for Windows and
home/site/wwwroot/startup.sh for Linux. The target path can be specified
with path.
Note the use of path:
The absolute path to deploy the artifact to. For example,
"/home/site/deployments/tools/driver.jar", "/home/site/scripts/helper.sh".
I never tested it, I am aware that the option is not described when taking about the az webapp deploy command itself, and it may be just an error in the documentation, but it may work:
az webapp deploy --resource-group Group
--name Name
--src-path ./startup.sh
--path /home/site/wwwroot/startup.sh
--type=startup
Note that the path you are providing is the default one; as a consequence, you could safely delete it if required:
az webapp deploy --resource-group Group
--name Name
--src-path ./startup.sh
--type=startup
Finally, try including some debug or echo commands in your script: perhaps the problem can be motivated for any permissions issue and having some traces in the logs could be helpful as well.
I am creating one ACR task with below script in azure cli to patch my azure container image when base image is updated and its working fine.
az acr task create
--registry Myregistry
--name myacrtask
--image myimage:{{.Run.ID}}
--context https://dev.azure.com/testaccount/myproject/_git/acr-build-helloworld-node.git#master
--file Dockerfile-app
--commit-trigger-enabled true
--base-image-trigger-enabled true
--git-access-token *****************************
Now my image having tag with "Run ID" as you can see in my command which is getting generated when task is running.
Now I want to create a custom tag like CurrentDate and some text like below.
if today's date is 09032020 then tag should be like
09032020_sometext
I am not sure how i can generate this kind of tag on the place of run id. I tried like
--image myimage:{{$(date +'%m%d%Y-BAU')}}
but no luck..
Any suggestion will be really apprecited.
Thanks
Rajiv
You can change the tag like this in the command:
--image myimage:$(date +%m%d%Y)-BAU
Then it will work fine. And the tag will like this:
And if you want the tag like 09032020_sometext, maybe you need to change the - into _.
I tried to add a custom script to VM through extensions. I have observed that when vm is created, Microsoft.Azure.Extensions.CustomScript type is created with name "cse-agent" by default. So I try to update extension by encoding the file with script property
az vm extension set \
--resource-group test_RG \
--vm-name aks-agentpool \
--name CustomScript \
--subscription ${SUBSCRIPTION_ID} \
--publisher Microsoft.Azure.Extensions \
--settings '{"script": "'"$value"'"}'
$value represents the script file encoded in base 64.
Doing that gives me an error:
Deployment failed. Correlation ID: xxxx-xxxx-xxx-xxxxx.
VM has reported a failure when processing extension 'cse-agent'.
Error message: "Enable failed: failed to get configuration: invalid configuration:
'commandToExecute' and 'script' were both specified, but only one is validate at a time"
From the documentation, it is mentioned that when script attribute is present,
there is no need for commandToExecute. As you can see above I haven't mentioned commandToExecute, it's somehow taking it from previous extension. Is there a way to update it without deleting it? Also it will be interesting to know what impact will cse-agent extension will create when deleted.
FYI: I have tried deleting 'cse-agent' extension from VM and added my extension. It worked.
the CSE-AGENT vm extension is crucial and manages all of the post install needed to configure the nodes to be considered a valid Kubernetes nodes. Removing this CSE will break the VMs and will render your cluster inoperable.
IF you are interested in applying changes to nodes in an existing cluster, while not officially supported, you could leverage the following project.
https://github.com/juan-lee/knode
This allows you to configure the nodes using a DaemonSet, which helps when you node pools have the auto-scaling feature enabled.
for simple Node alteration of the filesystem, a privilege pod with host path will also work
https://dev.to/dannypsnl/privileged-pod-debug-kubernetes-node-5129
How can I create an Azure container instance and configure it with an environment variables file?
Something that'd be equivalent to Docker's --env-file flag for the run command. I couldn't find a way to do that but I'm new to both Azure and Docker.
So it'd look something like: az container create <...> --env-file myEnvFile where myEnvFile is stored somewhere on Azure so I could grab it, like how Docker can grab such a file locally.
You can find what you want here https://learn.microsoft.com/en-us/cli/azure/container?view=azure-cli-latest#az-container-create
i.e.
az container create -g MyResourceGroup --name myapp --image myimage:latest --environment-variables key1=value1 key2=value2
Apologies, realised you want it from a file, if running in a script can you not have the file set local environment variables or parse the file to set them and then run the command above?
I'm really sure there is no parameter to set the environment variables of the Azure container instance from a file only through one command.
You can take a look at the parameter --environment-variables in the command az container create:
A list of environment variables for the container. Space-separated
values in 'key=value' format.
It requires the value of a list. So you can read from the file to create a list and then use the list as the value of the parameter --environment-variables in the create command.
As far as I'm aware, from answers and my research, this is currently not supported.