I've installed the latest Pulumi azuread module and I have this error when I try a pulumi preview:
Previewing update (int):
Type Name Plan Info
pulumi:pulumi:Stack test-int
└─ azuread:index:Application test 1 error
Diagnostics:
azuread:index:Application (test):
error: Error obtaining Authorization Token from the Azure CLI: Error waiting for the Azure CLI: exit status 1
my index.ts is very basic:
import * as pulumi from "#pulumi/pulumi";
import * as azure from "#pulumi/azure";
import * as azuread from "#pulumi/azuread";
const projectName = pulumi.getProject();
const stack = pulumi.getStack();
const config = new pulumi.Config(projectName);
const baseName = `${projectName}-${stack}`;
const testRg = new azure.core.ResourceGroup(baseName, {
name: baseName
});
const test = new azuread.Application("test", {
availableToOtherTenants: false,
homepage: "https://homepage",
identifierUris: ["https://uri"],
oauth2AllowImplicitFlow: true,
replyUrls: ["https://replyurl"],
type: "webapp/api",
});
Creating resources and AD application with the old module azure.ad works fine.
I have no clue what I am missing now....
EDIT:
index.ts the old way
import * as pulumi from "#pulumi/pulumi";
import * as azure from "#pulumi/azure";
const projectName = pulumi.getProject();
const stack = pulumi.getStack();
const config = new pulumi.Config(projectName);
const baseName = `${projectName}-${stack}`;
const testRg = new azure.core.ResourceGroup(baseName, {
name: baseName
});
const test = new azure.ad.Application("test", {
homepage: "https://homepage",
availableToOtherTenants: false,
identifierUris: ["https://uri"],
oauth2AllowImplicitFlow: true,
replyUrls: ["https://replyurl"]
});
Result of pulumi preview:
Previewing update (int):
Type Name Plan Info
pulumi:pulumi:Stack test-int
+ └─ azure:ad:Application test create 1 warning
Diagnostics:
azure:ad:Application (test):
warning: urn:pulumi:int::test::azure:ad/application:Application::test verification warning: The Azure Active Directory resources have been split out into their own Provider.
Information on migrating to the new AzureAD Provider can be found here: https://terraform.io/docs/providers/azurerm/guides/migrating-to-azuread.html
As such the Azure Active Directory resources within the AzureRM Provider are now deprecated and will be removed in v2.0 of the AzureRM Provider.
Resources:
+ 1 to create
2 unchanged
EDIT 2:
I'm running this on Windows 10:
az cli = 2.0.68
pulumi cli = 0.17.22
#pulumi/azure = 0.19.2
#pulumi/azuread = 0.18.2
#pulumi/pulumi = 0.17.21
Here are my principal permissions for Azure Active Directory Graph:
And the permissions for Microsoft Graph:
I ran into this issue and after hours I realized Fiddler was somehow interfering with the Az CLI running
Related
Background
I am using Azure DevOps for hosting the source of my web application and building/deploying the application to an Azure Static Web App.
I am using the "branch preview environments" of Static Web App like this (source):
steps:
...
- task: AzureStaticWebApp#0
inputs:
...
production_branch: 'main'
This works fine so far. For example, if I use a branch "dev", a corresponding branch environment is being created.
Question
How can I automatically delete the Azure static web app branch preview environment once the branch it was created for is being deleted?
Use Azure cli?
The only approach I found so far is using Azure CLI - but how to automate?
az staticwebapp environment delete --name my-static-app \
--environment-name an-env-name --subscription my-sub
I solved it by creating a separate pipeline triggered by the main branch. The pipeline removes all deployments that don't have an open pull request.
Here is the pipeline, basically just calling a node script that takes care of the cleanup:
name: Cleanup static web apps
trigger:
- main
# Add the following variables into devops:
# - DEVOPS_PAT: your personal access token for DevOps
# - AZURE_SUBSCRIPTION: the subscription in azure under which your swa lives
variables:
NPM_CONFIG_CACHE: $(Pipeline.Workspace)/.npm
DEVOPS_ORG_URL: "https://dev.azure.com/feedm3"
DEVOPS_PROJECT: "azure-playground"
AZURE_STATIC_WEBAPP_NAME: "react-app"
jobs:
- job: cleanup_preview_environments_job
displayName: Cleanup
pool:
vmImage: ubuntu-latest
steps:
- task: Cache#2
inputs:
key: 'npm | "$(Agent.OS)" | package-lock.json'
restoreKeys: |
npm | "$(Agent.OS)"
path: $(NPM_CONFIG_CACHE)
displayName: "Cache npm"
- script: |
npm ci
displayName: "Install dependencies"
- task: AzureCLI#2
inputs:
azureSubscription: "test-service-connection-name"
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
npm run ci:cleanup-deployments
displayName: "Cleanup outdated deployments"
This is the actual script that removes the deployments:
import { getPersonalAccessTokenHandler, WebApi } from "azure-devops-node-api";
import { exec as callbackExec } from 'child_process';
import { promisify } from 'util';
const exec = promisify(callbackExec);
const DEVOPS_ORG_URL = process.env["DEVOPS_ORG_URL"] as string;
const DEVOPS_PROJECT = process.env["DEVOPS_PROJECT"] as string;
const DEVOPS_PAT = process.env["DEVOPS_PAT"] as string;
const AZURE_SUBSCRIPTION = process.env["AZURE_SUBSCRIPTION"] as string;
const AZURE_STATIC_WEBAPP_NAME = process.env["AZURE_STATIC_WEBAPP_NAME"] as string;
const ALWAYS_DEPLOYED_BRANCHES = ['main'];
const REPO_ID = process.env['BUILD_REPOSITORY_ID'] as string;
const getAllStaticWebAppDeployments = async (): Promise<{ name: string; sourceBranch: string, hostname: string }[]> => {
const { stdout, stderr } = await exec(`az staticwebapp environment list --name ${AZURE_STATIC_WEBAPP_NAME} --subscription ${AZURE_SUBSCRIPTION}`);
if (stderr) {
console.error('Command failed!', stderr);
throw new Error(stderr);
}
return JSON.parse(stdout);
}
const run = async () => {
console.log(`Cleanup outdated deployments ${{REPO_ID, DEVOPS_PROJECT, AZURE_STATIC_WEBAPP_NAME}}...`)
const webAppDeployments = await getAllStaticWebAppDeployments();
// post comment
const authHandler = getPersonalAccessTokenHandler(DEVOPS_PAT);
const connection = new WebApi(DEVOPS_ORG_URL, authHandler);
await connection.connect();
const gitApi = await connection.getGitApi(`${DEVOPS_ORG_URL}/${DEVOPS_PROJECT}`);
// status 1 is active (PullRequestStatus type)
const activePullRequests = await gitApi.getPullRequests(REPO_ID, { status: 1 });
const activePullRequestBranches = activePullRequests.map(pr => pr.sourceRefName).filter(Boolean).map(fullBranchName => fullBranchName!.split('/')[2]);
// main deployment should always be alive
activePullRequestBranches.push(...ALWAYS_DEPLOYED_BRANCHES);
const outdatedDeployments = webAppDeployments.filter(deployment => {
return !activePullRequestBranches.includes(deployment.sourceBranch);
})
console.log('Deployments to delete:', outdatedDeployments);
for await (const deployment of outdatedDeployments) {
const deploymentName = deployment.name;
console.log(`Deleting deployment ${deploymentName}...`);
/**
* Deletion works, but ends with an irrelevant error.
*/
try {
const { stderr } = await exec(`az staticwebapp environment delete --name ${AZURE_STATIC_WEBAPP_NAME} --subscription ${AZURE_SUBSCRIPTION} --environment-name ${deploymentName} --yes`);
if (stderr) {
console.error('Could not delete deployment ', deploymentName);
} else {
console.log('Deleted deployment ', deploymentName);
}
} catch (e) {
console.log('Deleted deployment ', deploymentName);
}
}
console.log('Outdated deployments cleared!')
}
await run();
The full repo can be found here: https://github.com/feedm3/learning-azure-swa-devops
I've followed the documentation pretty well as outlined here.
I've setup my azure machine learning environment the following way:
from azureml.core import Workspace
# Connect to the workspace
ws = Workspace.from_config()
from azureml.core import Environment
from azureml.core import ContainerRegistry
myenv = Environment(name = "myenv")
myenv.inferencing_stack_version = "latest" # This will install the inference specific apt packages.
# Docker
myenv.docker.enabled = True
myenv.docker.base_image_registry.address = "myazureregistry.azurecr.io"
myenv.docker.base_image_registry.username = "myusername"
myenv.docker.base_image_registry.password = "mypassword"
myenv.docker.base_image = "4fb3..."
myenv.docker.arguments = None
# Environment variable (I need python to look at folders
myenv.environment_variables = {"PYTHONPATH":"/root"}
# python
myenv.python.user_managed_dependencies = True
myenv.python.interpreter_path = "/opt/miniconda/envs/myenv/bin/python"
from azureml.core.conda_dependencies import CondaDependencies
conda_dep = CondaDependencies()
conda_dep.add_pip_package("azureml-defaults")
myenv.python.conda_dependencies=conda_dep
myenv.register(workspace=ws) # works!
I have a score.py file configured for inference (not relevant to the problem I'm having)...
I then setup inference configuration
from azureml.core.model import InferenceConfig
inference_config = InferenceConfig(entry_script="score.py", environment=myenv)
I setup my compute cluster:
from azureml.core.compute import ComputeTarget, AksCompute
from azureml.exceptions import ComputeTargetException
# Choose a name for your cluster
aks_name = "theclustername"
# Check to see if the cluster already exists
try:
aks_target = ComputeTarget(workspace=ws, name=aks_name)
print('Found existing compute target')
except ComputeTargetException:
print('Creating a new compute target...')
prov_config = AksCompute.provisioning_configuration(vm_size="Standard_NC6_Promo")
aks_target = ComputeTarget.create(workspace=ws, name=aks_name, provisioning_configuration=prov_config)
aks_target.wait_for_completion(show_output=True)
from azureml.core.webservice import AksWebservice
# Example
gpu_aks_config = AksWebservice.deploy_configuration(autoscale_enabled=False,
num_replicas=3,
cpu_cores=4,
memory_gb=10)
Everything succeeds; then I try and deploy the model for inference:
from azureml.core.model import Model
model = Model(ws, name="thenameofmymodel")
# Name of the web service that is deployed
aks_service_name = 'tryingtodeply'
# Deploy the model
aks_service = Model.deploy(ws,
aks_service_name,
models=[model],
inference_config=inference_config,
deployment_config=gpu_aks_config,
deployment_target=aks_target,
overwrite=True)
aks_service.wait_for_deployment(show_output=True)
print(aks_service.state)
And it fails saying that it can't find the environment. More specifically, my environment version is version 11, but it keeps trying to find an environment with a version number that is 1 higher (i.e., version 12) than the current environment:
FailedERROR - Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: 0f03a025-3407-4dc1-9922-a53cc27267d4
More information can be found here:
Error:
{
"code": "BadRequest",
"statusCode": 400,
"message": "The request is invalid",
"details": [
{
"code": "EnvironmentDetailsFetchFailedUserError",
"message": "Failed to fetch details for Environment with Name: myenv Version: 12."
}
]
}
I have tried to manually edit the environment JSON to match the version that azureml is trying to fetch, but nothing works. Can anyone see anything wrong with this code?
Update
Changing the name of the environment (e.g., my_inference_env) and passing it to InferenceConfig seems to be on the right track. However, the error now changes to the following
Running..........
Failed
ERROR - Service deployment polling reached non-successful terminal state, current service state: Failed
Operation ID: f0dfc13b-6fb6-494b-91a7-de42b9384692
More information can be found here: https://some_long_http_address_that_leads_to_nothing
Error:
{
"code": "DeploymentFailed",
"statusCode": 404,
"message": "Deployment not found"
}
Solution
The answer from Anders below is indeed correct regarding the use of azure ML environments. However, the last error I was getting was because I was setting the container image using the digest value (a sha) and NOT the image name and tag (e.g., imagename:tag). Note the line of code in the first block:
myenv.docker.base_image = "4fb3..."
I reference the digest value, but it should be changed to
myenv.docker.base_image = "imagename:tag"
Once I made that change, the deployment succeeded! :)
One concept that took me a while to get was the bifurcation of registering and using an Azure ML Environment. If you have already registered your env, myenv, and none of the details of the your environment have changed, there is no need re-register it with myenv.register(). You can simply get the already register env using Environment.get() like so:
myenv = Environment.get(ws, name='myenv', version=11)
My recommendation would be to name your environment something new: like "model_scoring_env". Register it once, then pass it to the InferenceConfig.
I have created in terraform the following logging metric resource
resource "google_logging_metric" "proservices_run" {
name = "user/proservices-run"
filter = "resource.type=gae_app AND severity>=ERROR"
project = "${google_project.service.project_id}"
metric_descriptor {
metric_kind = "DELTA"
value_type = "INT64"
}
}
I have also on Stackdriver a custom metric named user/proservices-run.
However the following two import attempts fail:
$ terraform import google_logging_metric.proservices_run proservices-run
google_logging_metric.proservices_run: Importing from ID "proservices-run"...
google_logging_metric.proservices_run: Import complete!
Imported google_logging_metric (ID: proservices-run)
google_logging_metric.proservices_run: Refreshing state... (ID: proservices-run)
Error: google_logging_metric.proservices_run (import id: proservices-run): 1 error occurred:
* import google_logging_metric.proservices_run result: proservices-run: google_logging_metric.proservices_run: project: required field is not set
$ terraform import google_logging_metric.proservices_run user/proservices-run
google_logging_metric.proservices_run: Importing from ID "user/proservices-run"...
google_logging_metric.proservices_run: Import complete!
Imported google_logging_metric (ID: user/proservices-run)
google_logging_metric.proservices_run: Refreshing state... (ID: user/proservices-run)
Error: google_logging_metric.proservices_run (import id: user/proservices-run): 1 error occurred:
* import google_logging_metric.proservices_run result: user/proservices-run: google_logging_metric.proservices_run: project: required field is not set
Using
Terraform v0.11.14
and
provider.google = 2.11.0
provider.google-beta 2.11.0
edit: I noticed the project: required field is not set in the error message, I added the field project in my TF code, however the outcome is still the same.
I ran into the same issue trying to import a log-based metrics.
The solution was to set the env-var GOOGLE_PROJECT=<your-project-id> when running the command.
GOOGLE_PROJECT=MyProjectId \
terraform import \
"google_logging_metric.create_user_count" \
"create_user_count"
My terraform-modules repo location is like this:
https://teamabc.visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster
I have three directories/modules at root level, namely compute, resourcegroup and sqlserver.
However, when I run terraform init. terraform is unable to download the required modules.
main.tf
module "app_vms" {
source = "https://teamabc.visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster"
rg_name = var.resource_group_name
location = module.resource_group.external_rg_location
vnet_name = var.virtual_network_name
subnet_name = var.sql_subnet_name
app_nsg = var.application_nsg
vm_count = var.count_vm
base_hostname = var.app_host_basename
sto_acc_suffix = var.storage_account_suffix
vm_size = var.virtual_machine_size
vm_publisher = var.virtual_machine_image_publisher
vm_offer = var.virtual_machine_image_offer
vm_sku = var.virtual_machine_image_sku
vm_img_version = var.virtual_machine_image_version
username = var.username
password = var.password
allowed_source_ips = var.ip_list
}
module "resource_group" {
source = "https://teamabc.visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fresourcegroup&version=GBmaster"
rg_name = "test_rg"
}
module "azure_paas_sqlserver" {
source = "https://teamabc.visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fsqlserver&version=GBmaster"
}
It gives me a series of errors like below:(sample only give not all the errors as they are same)
Error: Failed to download module
Could not download module "sql_vms" (main.tf:1) source code from
"https://teamabc.visualstudio.com/dummpproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster":
error downloading
'https://teamabc.visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster':
no source URL was returned
Error: Failed to download module
Could not download module "sql_vms" (main.tf:1) source code from
"https://teamabc.visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster":
error downloading
'https://teamabc.visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster':
no source URL was returned
I tried to remove https:// part but no luck. The repo does require username and password to login.
Wondering if I should be making a public repo in github? but push within the organization is to use Azure Repos.
Post First comment
Thanks for the lead, I did tried but still no charm.
My source url now looks like below
source = "git::https://teamabc:lfithww4xpp4eksvoimgzkpi3ugu6xvrkf26mfq3jth3642jgyoa#visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster"
I get error below:
Error: Failed to download module
Could not download module "sql_vms" (main.tf:1) source code from
"git::https://teamabc:lfithww4xpp4eksvoimgzkpi3ugu6xvrkf26mfq3jth3642jgyoa#visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster":
error downloading
'https://teamabc:lfithww4xpp4eksvoimgzkpi3ugu6xvrkf26mfq3jth3642jgyoa#visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster':
/usr/bin/git exited with 128: Cloning into '.terraform/modules/sql_vms'...
fatal: repository
'https://teamabc:lfithww4xpp4eksvoimgzkpi3ugu6xvrkf26mfq3jth3642jgyoa#visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster/'
not found
Here:
teamabc.visuastudio.com is the parent azure devops url
dummyproject is the project name
After Charles Response
Error: Failed to download module
Could not download module "sql_vms" (main.tf:1) source code from
"git::https://teamabc:lfithww4xpp4eksvoimgzkpi3ugu6xvrkf26mfq3jth3642jgyoa#visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster.git":
error downloading
'https://teamabc:lfithww4xpp4eksvoimgzkpi3ugu6xvrkf26mfq3jth3642jgyoa#visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster.git':
/usr/bin/git exited with 128: Cloning into '.terraform/modules/sql_vms'...
fatal: repository
'https://teamabc:lfithww4xpp4eksvoimgzkpi3ugu6xvrkf26mfq3jth3642jgyoa#visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster.git/'
not found
You can take a look at Generic Git Repository, the URL should be a Git URL. And finally, it should like this:
source = "git::https://teamabc:lfithww4xpp4eksvoimgzkpi3ugu6xvrkf26mfq3jth3642jgyoa#visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster.git"
Or you can select a branch from your Git Repository like this:
source = "git::https://teamabc:lfithww4xpp4eksvoimgzkpi3ugu6xvrkf26mfq3jth3642jgyoa#visualstudio.com/dummyproject/_git/terraform-modules?path=%2Fcompute&version=GBmaster.git?ref=<branch>"
Finally, got it working by below command:
git::https://<PAT TOKEN>#<Azure DevOps URL>/DefaultCollection/<PROJECT NAME>/_git/<REPO NAME>//<sub directory>
I am new to docker as well as azure batch. The problem i am having currently is i have 2 dotnet console applications one of them runs locally (which creates the pool, job and task on azure batch programmatically) and for second one i have created a docker image and pushed to azure container registry. Now the things is when i create the cloudtTask from locally running application as monetione below
TaskContainerSettings cmdContainerSettings = new TaskContainerSettings(
imageName: "myrepository.azurecr.io/pipeline:latest",
containerRunOptions: "--rm"
);
CloudTask containerTask = new CloudTask(
id: "task1",
commandline: cmdLine);
containerTask.ContainerSettings = cmdContainerSettings;
Console.WriteLine("Task created");
await batchClient.JobOperations.AddTaskAsync(newJobId, containerTask);
Console.WriteLine("-----------------------");
and add it to the BatchClient, the expcetion i get in azure batch (Azure portal) is this:
System.UnauthorizedAccessException: Access to the path '/home/_azbatch/.dotnet' is denied. ---> System.IO.IOException: Permission denied
--- End of inner exception stack trace ---
What can be the problem? Thank you.
As the comment ended up being the answer, I'm posting it here for clarity for future viewers:
The task needs to be run with elevated rights.
eg.
containerTask.UserIdentity = new UserIdentity(new AutoUserSpecification(elevationLevel: ElevationLevel.Admin, scope: AutoUserScope.Task));
See the docs for more info
i am still not able to pull image from docker, i am using nodejs .. following are configs for creating task
const taskConfig = {
"id": "task-new-2",
"commandLine": "bash -c 'node index.js'",
"containerSettings": {
"imageName": "xxx.xx.io/xx-test:latest",
"containerRunOptions": "--rm",
"username": "xxx",
"password": "tfDlZ",
"registryServer": "xxx.xx.io",
// "workingDirectory": "AZ_BATCH_NODE_ROOT_DIR"
},
"userIdentity": {
"autoUser": {
"scope": "pool",
"elevationLevel": "admin"
}
}
}