I want to automatically deploy artifacts (custom python packages / .whl files) from the Azure DevOps artifact feed to a Synapse spark pool.
Currently I have to manually:
download the .whl file from the artifact feed
upload the .whl file to the synapse workspace
add the package/.whl to the packages of a spark pool (> "Select from workspace packages").
I have so far not found any option to do this as part of a release/pipeline in Azure DevOps or via the AzureCLI and also have not found any documentation on it. I was wondering whether anyone has found a solution concerning automating this step? It is quite cumbersome to have to do it manually.
PS: I already asked this question on the MS forum, but have gotten no official answer yet.
I managed to update the pool spark packages by using both the Download Packages task and the Azure CLI task in the release pipeline on devops.
First use the Download Packages task to download your feed to the pipeline $(System.DefaultWorkingDirectory) then use the Azure CLI to update the sparkpool.
In the Azure CLI task write in the inline command field the two following lines. Use the $(System.DefaultWorkingDirectory) as working directory, also I recommand to check the Use global Azure CLI configuration checkbox.
First command to upload your .whl to the workspace. Here is the doc
az synapse workspace-package upload --workspace-name yourworkspace --package yourpackage.whl
Then upload it to the pool spark. Here is the doc.
az synapse spark pool update --name yourpoolspark --workspace-name yourworkspace --resource-group yourresourcegroupe --package-action Add --package yourpackage.whl
Those two commands will add the package to your workspace and then upload it to the poolspark.
You can achieve the same things with the Azure Powershell task on a Windows agent. Here is the doc to explore : link.
Hope it will help you !
Related
I need to run snyk scan for Azure container and set it to fail only when there are new vulnerabilities found as compared to previous image
I did follow the snyk-delta document to configure task in azure but the tasks are getting failed.
suggestions are to use snyk cli to scan in the azure pipeline
not sure how to configure it
We want to add snyk-delta scan in out pipeline, add CLI script without using Snyk extension.
any suggestion to do this
Currently, we can use the snyk Security task to scan azure pipeline.
https://docs.snyk.io/integrations/ci-cd-integrations/azure-pipelines-integration/add-the-snyk-security-task-to-your-pipelines
And snyk cli, you can try: https://docs.snyk.io/more-info/getting-started/snyk-integrations/docker/scanning-with-the-docker-cli
I am writing a program that uploads a file to Azure blob storage, creates a virtual machine, and now I want to download and execute that file within the VM. For that I am trying to execute the command az storage blob download <parameters> in the VM, but unfortunately az is not recognized. How can I enable that the Azure CLI is (pre)installed on each new VM? Is there such a possibility provided by Azure? Or should I install the Azure CLI with yum for each VM within my script? Any information or ideas is highly appreciated, thank you.
You need to install the Azure CLI yourself. See the steps here. You can install it following the steps one by one, or put the commands inside a script and execute the script in the VM extension or cloud-init in the creation time.
This question already has an answer here:
How to create new build pipeline using Azure DevOps REST API?
(1 answer)
Closed 2 years ago.
I was looking for a solution of creating automation process for creation of pipelines to eliminate all the manual process we do while creating a pipeline in Azure devops portal.
Example:
If i want to deploy my Python based application and it doesn't have any pipelines defined previously. I want to create a pipeline and deploy the app to the Azure server whole process should be done with out manual intervention.
You still need to create somehow your YAML file or use what you have already. Of course you can prepare some templates and reuse them and if you define folder structure for your project and you will keep it then you will minimize yoour manual tasks. Once you have your YAML file defines you can use for instance az cli command:
az pipelines create --name 'Your pipeline name' --description 'Your pipeline description' --repository https://github.com/your-repo --branch master --yml-path your-file-in-repo.yaml --org https://dev.azure.com/your-organizaton --project 'your project'
From what I see this is not possible to use REST API to define YAML pipelines. Please take a look here.
So if you use calssic build it will be possible to use REST API. You can take a look here.
POST https://dev.azure.com/{organization}/{project}/_apis/build/definitions?definitionToCloneId={definitionToCloneId}&definitionToCloneRevision={definitionToCloneRevision}&api-version=5.1
You can for instance indicate here which definition should be cloned as a base for your new build.
I'm having trouble following this guide section 3.6.5.3 "Writing Logs to Azure Blob Storage"
The documentation states you need an active hook to Azure Blob storage. I'm not sure how to create this. Some sources say you need to create the hook in the UI, and some say you can use an environment variable. Either way, none of my logs are getting written to blob store and I'm at my wits end.
Azure Blob Store hook(or any hook for that matter) tells overflow how to write to into Azure Blob Store. This is already included in recent versions of airflow, wasb_hook.
You will need to make sure that the hook is able to write to Azure Blob Store. Just mention the REMOTE_BASE_LOG_FOLDER bucket should be named like wasb-xxx. Once you take care of these two things instructions works without a hitch,
I achieved writing logs to blob using below steps
Create folder named config inside airflow folder
Create empty __init__.py and log_config.py files inside config folder
Search airflow_local_settings.py in your machine
/home/user/env/lib/python2.7/site-packages/airflow/config_templates/airflow_local_settings.py
/home/user/env/lib/python2.7/site-packages/airflow/config_templates/airflow_local_settings.pyc
run
cp /home/user/env/lib/python2.7/site-packages/airflow/config_templates/airflow_local_settings.py config/log_config.py
Edit airflow.cfg [core] section
remote_logging = True
remote_log_conn_id = log_sync
remote_base_log_folder=wasb://airflow-logs#storage-account.blob.core.windows.net/logs/
logging_config_class =log_config.DEFAULT_LOGGING_CONFIG
Add log_sync connection object as below
install airflow azure dependency
pip install apache-airflow[azure]
Restart webserver and scheduler
I'm trying to create an Azure VM and then copy an install file to the VM and then silently installing it. I have created a basic Azure Resource Group project, and can create and deploy the VM, but I can't figure out how to do everything from the powershell script.
It sounds like you could use a custom script extension to do what you want. In your ARM template, you can specify the url for a file and the command to run; Azure will handle getting the file onto your VM and running it based on your command. Here is an example from the Azure Quickstart Templates: https://github.com/Azure/azure-quickstart-templates/tree/master/windows-vm-custom-script
Hope this helps! :)