#provider azurem.mgmt is Subscription A.
#prodiver azurem.corpapps is Subscription B.
I am trying to create nsg rule in Subscription A with Provider azurerm.mgmt. Here the destination application security group is in Subscription B with Provider azurerm.corpapps in this subscription.
provider "azurerm" {
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
tenant_id = "${var.tenant_id}"
subscription_id = "${var.subscription}"
alias = "mgmt"
}
provider "azurerm" {
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
tenant_id = "${var.tenant_id}"
subscription_id = "${var.subscription_B}"
alias = "corpapps"
}
Then i use the provider to get my asg from Subscription B as shown:
Then i use that reference in my nsg rule
However, i get error - saying the ASG is not found:
But, in azure portal the resource is already there as shown:
I have tried to assign the SP which has owner role on both subscriptions or using Azure account with CLI but it's no luck. Also, as the comment points out, there is a limitation that NSG does not reference ASG in different location. After my validation, you can not add the ASG from another subscription even it's in the same region as the NSG or targets VNet.
Moreover, when you add this ASG as the target source or destination in the NSG rules, you will see
Select an application security group (ASG) as the security rule
source. ASGs enable fine-grained network security policies based on
workloads or applications instead of IP addresses or CIDR blocks.
Rules specifying an application security group are only applied to
network interfaces that are members of the application security group
on the same virtual network.
Related
I am trying to implement a strategy where I can create a NSG in one Azure subscription and use the same NSG resource to attach to any VMs or NICs created in other subscriptions and resource groups.
How can this implementation work via Terraform where I want to attach a single (default) NSG (created in a separate subscription) to multiple VMs and NICs in other subscriptions?
Default NSG for all Azure Subscriptions via Terraform:
Rules defined for a certain network security group with some network security rules will only apply to that resource group. As a result of this limitation for network security groups, it is not feasible to access an NSG in subscriptions other than the existing ones.
You cannot access an NSG that exists in one subscription in another, even though it is provided in the same region.
If you need to add network security in other subscriptions, you can consider the following methods:
Add multiple subscriptions in provider using alias while deploying Terraform code, as mentioned article by #Jeff Brown.
provider "azurerm"{
alias = "xx"
subscription = "subscription1"
features{}
}
provider "azurerm"{
alias = "xxdev"
subscription = "subscription2"
features{}
}
resource "azurerm_network_security_group" "example"{
//Add configuration
}
Note: Include azurerm providers to deploy the same NSG or any Azure resource across multiple subscriptions provided by subscription Ids.
terraform import can be used to import existing resources from anywhere.
terraform import azurerm_network_security_group.<NSG> <ResourceID>
Output:
we deploy resources in our Azure tenant through Jenkins which uses terraform to provision infra resources.
and we use service principal for authentication and infra provisioning which are in same tenant. in our infra deployment we also create VNET peering with the new Vnet which get deployed and our central VNET which has all the infra resources like monitoring and logging platform.
now we have a use case where by using the same Jenkins and terraform scripts we want to provision resources on different tenant. this can be done by using the service principal of remote tenant.
but now issue is service principal of TenantB do not have rights to create network resources in TenantA. to make this happen service principal of TenantB should have access on Vnet in TenantA.
i am looking for documentation or guidance how we can give access to service principal of TenantB in our TenantA?
Hoping that you have created a service principal a service principal
using multi-tenant authentication (if single-tenant please change the
authentication method to multi-tenant) , add a redirect uri
https://www.microsoft.com.
After you have created the service principal you can open the below url in a private browser for adding it on another tenant:
https://login.microsoftonline.com/<Tenant B ID>/oauth2/authorize?client_id=<Application (client in tenant A)
ID>&response_type=code&redirect_uri=https%3A%2F%2Fwww.microsoft.com%2F
It will ask for authorization on behalf of organization , you can
accept it.
After the above is done , then you can login to portal of that tenant
and go to enterprise application you will see that , provide role
assignment for that subscription (owner/contributor).
After this is done you can use something like the below terraform
script:
provider "azurerm" {
alias = "tenantA"
subscription_id = "b83c1ed3-xxxxx-xxxxxx-xxxxxx-xxxxxx" #subid for tenant A
tenant_id = "72f988bf-xxxxxx-xxxxx-xxxxxxx-xxxxxx"#tenantid of tenant A
client_id = "f6a2f33d-xxxx-xxxx-xxxxx-xxxxxxxx"#client id of service principal in tenant A
client_secret = "y5L7Q~oiMOoGCxm7fK~xxxxxxxxxxxxxxx"#client secret of service principal in tenant A
auxiliary_tenant_ids = ["ab078f81-xxxxxx-xxxxxxxx-xxxxxx"]# tenant id of tenant B
features {}
}
provider "azurerm"{
alias = "tenantB"
subscription_id = "88073b30-xxx-xxxxx-xxxxx-xxxxxxx"#sub id of tenant B
tenant_id = "ab078f81-xxxxx-xxxxxxx-xxxxxxxxx" # tenant id of tenant B
client_id = "f6a2f33d-xxxx-xxxxxx-xxxxxx-xxxxxx" #client id of service principal in tenant A
client_secret = "y5L7Q~oiMOoGCxm7fK~xxxxxxxxxxxxxxxx" #client secret of service principal in tenant A
auxiliary_tenant_ids = ["72f988bf-xxxx-xxxxx-xxxxxxxxxx-xx"] # tenant id of tenant A
features {}
}
data "azurerm_resource_group" "tenantARG"{
provider = azurerm.tenantA
name = "reswourcegroup"
}
data "azurerm_resource_group" "tenantBRG"{
provider = azurerm.tenantB
name = "ansuman-resourcegroup"
}
data "azurerm_virtual_network" "GlobalVnet"{
provider = azurerm.tenantA
name = "ansuman-vnet"
resource_group_name= data.azurerm_resource_group.tenantARG.name
}
data "azurerm_virtual_network" "tenantBVnet"{
provider = azurerm.tenantB
name = "test-vnet"
resource_group_name= data.azurerm_resource_group.tenantBRG.name
}
resource "azurerm_virtual_network_peering" "example-1" {
provider= azurerm.tenantA
name = "peer1to2"
resource_group_name = data.azurerm_resource_group.tenantARG.name
virtual_network_name = data.azurerm_virtual_network.GlobalVnet.name
remote_virtual_network_id = data.azurerm_virtual_network.tenantBVnet.id
}
resource "azurerm_virtual_network_peering" "example-2" {
provider = azurerm.tenantB
name = "peer2to1"
resource_group_name = data.azurerm_resource_group.tenantBRG.name
virtual_network_name = data.azurerm_virtual_network.tenantBVnet.name
remote_virtual_network_id = data.azurerm_virtual_network.GlobalVnet.id
}
Output:
Note: In my test case , I have used 2 vnets present in different tenants. I created a service principal in tenant A and provided contributor permissions to it in tenant B using the above methods and then used terraform to perform the vnet peering.
I want to deploy some resources on Azure with Terraform.
On Azure, I have an account with "Owner rights" on one Resource Group only(RGName). Not at the subscription level.
From my linux server, I installed "az cli" and I did "az login". At this step, everything is OK.
The problem appears when I want to execute terraform to create one resource.
Content of provider.tf (the only one .tf file for now) :
provider "azurerm" {
}
If I do a "terraform plan", it works.
If I add the following line, it fails. Please see the error at the end :
resource "azurerm_virtual_network" "myterraformnetwork" {
name = "myVnet"
address_space = ["10.0.0.0/16"]
location = "eastus"
resource_group_name = "RGName"
tags = {
environment = "Terraform Demo"
}
}
I do not have right on subscription level but I do not need to.
With the Azure WebUI I can create resource on this Resource Group without problem.
The error :
Error: Error ensuring Resource Providers are registered: Cannot register provider Microsoft.DevSpaces with Azure Resource Manager: resources.ProvidersClient#Register: Failure responding to request: StatusCode=403 -- Original Error: autor
est/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client 'accountName' with object id 'IDaccountName' does not have authorization to perform action 'Microsoft.DevSpaces/r
egister/action' over scope '/subscriptions/subscriptionID' or the scope is invalid. If access was recently granted, please refresh your credentials.".
on provider.tf line 1, in provider "azurerm":
1: provider "azurerm" {
Thank you all !
If anyone else has this issue in a corporate (restricted) Azure environment, and doesn't have the patience to register the provider (which may not be necessary if you don't use the specified terraform resource) - have a look at https://github.com/terraform-providers/terraform-provider-azurerm/issues/4440
Specifically, this may help:
provider "azurerm" {
skip_provider_registration = "true"
It obviously won't help if you actually need the resource that fails to get registered (in our case it was Cannot register provider Microsoft.DevSpaces with Azure Resource Manager, but the resource will be variable depending on your environment and what Terraform decides to support)
For your issue, when you have the Owner role of the resource group, you can create new resources or manage the existing resources as you want. So permission is no problem. With the test on my side, it works well using a user has the Owner role of the resource group.
As the error shows, I think the possible reason is that you have multiple subscriptions in the tenant and the current subscription is not the right one which the user has the right permission. You can try to take a check and set the right subscription via the command:
az account set --subscription subscription_id
Thank you for your answer.
I got this when I execute "az account list" :
"cloudName": "AzureCloud",
"id": "***********0d43",
"isDefault": true,
"name": "BU*******",
"state": "Enabled",
"tenantId": "TENANTID",
"user": {
"name": "LOGINNAME",
"type": "user"
I do not have rights on this subscription but it is the only one that I know.
On Azure WebUI I can see that the RGName is on the same subscription.
This is a capture from Azure WebUI on the RGName :
Azure WebUI
Thank you
You may need to register the Resource provider by clicking on register as shown in below screenshot under subscription id.
I am configuring my infrastructure in one Azure Directory with terraform and so I have set the Azure CLI to use that subscription:
az account set --subscription xxxxxxxx-0000-xxxx-YYYY-zzzzzzzzzzzz
Part of the setup is to add Active Directory Groups to APIM Management.
I create the groups and query AD for their ids using az cli in local-exec.
However, Active Directory is in a different subscription to where the infrastructure is being created so this step fails.
How can I switch directory/subscription for this one call?
I create the groups and query AD for their ids using az cli in local-exec.
You can query for groups with datasource azuread_group and/or manage resource azuread_group instead of az cli.
To use datasource/resource with multiple subscriptions you should authenticate multiple providers with aliases with different subscription_id like that:
provider "azuread" {
subscription_id = "xxxxxxxx-0000-xxxx-YYYY-zzzzzzzzzzzz"
}
provider "azuread" {
subscription_id = "another-subscription-id"
alias = "custom"
}
resource "azurerm_api_management" "test" {
...
}
resource "azuread_group" "mygroup" {
provider = azuread.custom
name = "my-group"
}
I'm deploying AKS clusters with Terraform and it's working fine. However, when trying to add security rules to the AKS network security group in the automatically created MC* group, it fails with errors such as:
Creating/Updating Network Security Rule "myRule" (NSG "" / Resource Group "MC_terraform-aks-rg_terraform-aks_westeurope"): network.SecurityRulesClient#CreateOrUpdate: Failure sending request: StatusCode=404 -- Original Error: Code="ResourceNotFound" Message="The Resource 'Microsoft.Network/networkSecurityGroups/securityRules' under resource group 'MC_terraform-aks-rg_terraform-aks_westeurope' was not found."[0m
If I run terraform apply again, it works and the rules get created. I'm thinking there is a race condition somewhere and I even added a null_resource that executes a sleep command for a couple of minutes, but still errors out on the first try.
main.tf
resource "azurerm_kubernetes_cluster" "aks" {
....................................
}
resource "azurerm_network_security_rule" "https" {
name = "myRule"
priority = 101
direction = "Inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "443"
destination_port_range = "*"
source_address_prefixes = "${var.imperva_ips}"
destination_address_prefix = "${azurerm_public_ip.ingress.ip_address}"
resource_group_name = "${azurerm_kubernetes_cluster.aks.node_resource_group}"
network_security_group_name = "${data.external.aks_nsg_name.result.output}"
#depends_on = ["azurerm_resource_group.aks", "azurerm_mysql_virtual_network_rule.mysql", "helm_release.ingress"]
depends_on = [null_resource.delay]
}
resource "null_resource" "delay" {
provisioner "local-exec" {
command = "sleep 60"
}
depends_on = [helm_release.ingress]
}
# get the auto-generated NSG name
data "external" "aks_nsg_id" {
program = [
"bash",
"${path.root}/scripts/aks_nsg_name.sh"
]
depends_on = [azurerm_resource_group.aks]
}
The bash script that pulls the NSG name:
#!/bin/bash
OUTPUT=$(az network nsg list --query [].name -o tsv | grep aks | head -n 1)
jq -n --arg output "$OUTPUT" '{"output":$output}'
For AKS cluster, it's not recommended to create the NSG rules manually, Azure will create the appropriate rules for you automatically. If you create the rules manually, it could cause problems. See the description below:
A network security group filters traffic for VMs, such as the AKS
nodes. As you create Services, such as a LoadBalancer, the Azure
platform automatically configures any network security group rules
that are needed. Don't manually configure network security group rules
to filter traffic for pods in an AKS cluster. Define any required
ports and forwarding as part of your Kubernetes Service manifests, and
let the Azure platform create or update the appropriate rules. You can
also use network policies, as discussed in the next section, to
automatically apply traffic filter rules to pods.
So I would not suggest you create the rules yourself. For more details, see AKS Network Security Group. You'd better use the network policy rather than the NSG rules, and on my side, the network policy is more recommended.
Update:
And the error you got shows that it did not find the rules in the node group. As I see, you need to change the command with a group name of your AKS cluster in the bash. The command you use without a group name will list all the NSG in the subscription, it will not find your NSG if there is not only your AKS cluster.