Terraform AKS nsg security rules error out - azure

I'm deploying AKS clusters with Terraform and it's working fine. However, when trying to add security rules to the AKS network security group in the automatically created MC* group, it fails with errors such as:
Creating/Updating Network Security Rule "myRule" (NSG "" / Resource Group "MC_terraform-aks-rg_terraform-aks_westeurope"): network.SecurityRulesClient#CreateOrUpdate: Failure sending request: StatusCode=404 -- Original Error: Code="ResourceNotFound" Message="The Resource 'Microsoft.Network/networkSecurityGroups/securityRules' under resource group 'MC_terraform-aks-rg_terraform-aks_westeurope' was not found."[0m
If I run terraform apply again, it works and the rules get created. I'm thinking there is a race condition somewhere and I even added a null_resource that executes a sleep command for a couple of minutes, but still errors out on the first try.
main.tf
resource "azurerm_kubernetes_cluster" "aks" {
....................................
}
resource "azurerm_network_security_rule" "https" {
name = "myRule"
priority = 101
direction = "Inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "443"
destination_port_range = "*"
source_address_prefixes = "${var.imperva_ips}"
destination_address_prefix = "${azurerm_public_ip.ingress.ip_address}"
resource_group_name = "${azurerm_kubernetes_cluster.aks.node_resource_group}"
network_security_group_name = "${data.external.aks_nsg_name.result.output}"
#depends_on = ["azurerm_resource_group.aks", "azurerm_mysql_virtual_network_rule.mysql", "helm_release.ingress"]
depends_on = [null_resource.delay]
}
resource "null_resource" "delay" {
provisioner "local-exec" {
command = "sleep 60"
}
depends_on = [helm_release.ingress]
}
# get the auto-generated NSG name
data "external" "aks_nsg_id" {
program = [
"bash",
"${path.root}/scripts/aks_nsg_name.sh"
]
depends_on = [azurerm_resource_group.aks]
}
The bash script that pulls the NSG name:
#!/bin/bash
OUTPUT=$(az network nsg list --query [].name -o tsv | grep aks | head -n 1)
jq -n --arg output "$OUTPUT" '{"output":$output}'

For AKS cluster, it's not recommended to create the NSG rules manually, Azure will create the appropriate rules for you automatically. If you create the rules manually, it could cause problems. See the description below:
A network security group filters traffic for VMs, such as the AKS
nodes. As you create Services, such as a LoadBalancer, the Azure
platform automatically configures any network security group rules
that are needed. Don't manually configure network security group rules
to filter traffic for pods in an AKS cluster. Define any required
ports and forwarding as part of your Kubernetes Service manifests, and
let the Azure platform create or update the appropriate rules. You can
also use network policies, as discussed in the next section, to
automatically apply traffic filter rules to pods.
So I would not suggest you create the rules yourself. For more details, see AKS Network Security Group. You'd better use the network policy rather than the NSG rules, and on my side, the network policy is more recommended.
Update:
And the error you got shows that it did not find the rules in the node group. As I see, you need to change the command with a group name of your AKS cluster in the bash. The command you use without a group name will list all the NSG in the subscription, it will not find your NSG if there is not only your AKS cluster.

Related

Default NSG for all Azure Subscriptions via Terraform

I am trying to implement a strategy where I can create a NSG in one Azure subscription and use the same NSG resource to attach to any VMs or NICs created in other subscriptions and resource groups.
How can this implementation work via Terraform where I want to attach a single (default) NSG (created in a separate subscription) to multiple VMs and NICs in other subscriptions?
Default NSG for all Azure Subscriptions via Terraform:
Rules defined for a certain network security group with some network security rules will only apply to that resource group. As a result of this limitation for network security groups, it is not feasible to access an NSG in subscriptions other than the existing ones.
You cannot access an NSG that exists in one subscription in another, even though it is provided in the same region.
If you need to add network security in other subscriptions, you can consider the following methods:
Add multiple subscriptions in provider using alias while deploying Terraform code, as mentioned article by #Jeff Brown.
provider "azurerm"{
alias = "xx"
subscription = "subscription1"
features{}
}
provider "azurerm"{
alias = "xxdev"
subscription = "subscription2"
features{}
}
resource "azurerm_network_security_group" "example"{
//Add configuration
}
Note: Include azurerm providers to deploy the same NSG or any Azure resource across multiple subscriptions provided by subscription Ids.
terraform import can be used to import existing resources from anywhere.
terraform import azurerm_network_security_group.<NSG> <ResourceID>
Output:

Must Azure NSGs be in the same resource group as NICs which attach to the subnet protected by the NSG?

I'm having some trouble attaching a NIC (in resource group A) to a subnet belonging to a Vnet and NSG in a different resource group (say B). I have Contributor role in resource group A, but only Reader role in resource group B. Is this possible? If so, what am I doing wrong? Here's what it looks like (with UIDs shortened).
% az network nic create --resource-group A --name bastion-nic --vnet-name VN-B --subnet SubnetB
(InvalidResourceReference) Resource /subscriptions/40ef-b75f-c05a034bf2ff/resourceGroups/A/providers/Microsoft.Network/virtualNetworks/VN-B/subnets/SubnetB referenced by resource /subscriptions/b75f-c05a034bf2ff/resourceGroups/A/providers/Microsoft.Network/networkInterfaces/bastion-nic was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.
Code: InvalidResourceReference
I tested the same scenario in my environment .
Scenario: I created a user , 2 resource groups i.e. contributorTest with Contributor access for the user and readerTest with Reader access for the user.
If I use the command you are using then it gives me the same error message as you. To describe the issue when you are using vnet-name, the command thinks that the vnet is also present in the same resource which has been mentioned in the command.
az network nic create --resource-group contributorTest --name bastion-nic --vnet-name ansumantest-vnet --subnet default
So , for example in the above command resource group is contributorTest and we have just provided vnet name and subnet name , which it thinks is present in the same group. So it throws the error as below:
As a Solution you can use the below command to create NIC if the VNet is in different resource group:
az network nic create --resource-group contributorTest --name bastion-nic --subnet /subscriptions/subID/resourceGroups/readerTest/providers/Microsoft.Network/virtualNetworks/ansumantest-vnet/subnets/default
In the above command , we are not providing vnet name & subnet name , as a alternative we have provided the resourceID of the subnet.
Note: The above solution should work only if you have contributor access on both the resource group , in your case you will be getting the below error:
To describe the issue here, while you are creating a NIC it requires to join that NIC to the Subnet which you have specified but as you have reader access only on the VNET resource group it doesn't allow you to join the NIC and subnet.
So , Final solution can be :
Either have the VNET and subnet in the same resource group you are creating NIC on and have a Contributor access on it and use the command you are using .
Grant Contributor Access to the user for the second resource group and use the second Command that I have mentioned as a solution.
Output for the second command after providing contributor access for both the resource groups:

Terraform : How to attach NSG and Service Endpoints in subnet provision

Is there a way in Azurerm to provision a subnet with an NSG attachment and also provision service endpoints as well . This is considering we have a policy to enforce NSG on subnet . So - - if subnet is a separate resource (as a separate resource nsg_id is not a parameter anymore) - it would fail the policy.
As a sub resrouce under vnet , can attach a nsg_id - but does not have parameters to provision a service endpoint or service delegation.
You could deploy a service endpoint or service delegation with Terraform Language local-exec Provisioner or remote-exec provisioner to invoke PowerShell or CLI scripts after a resource is created.
For example, you can use az network vnet subnet update to enable a service endpoint for the subnet.
resource "null_resource" "example" {
provisioner "local-exec" {
command = "az network vnet subnet update -g ${azurerm_resource_group.example.name} -n 'subnet1' --vnet-name ${azurerm_virtual_network.example.name} --service-endpoints 'Microsoft.Sql'"
interpreter = ["PowerShell", "-Command"]
}
}
For more references, you could read this blog- Multi-line PowerShell in Terraform.

NSG rule across subscription in azure via terraform

#provider azurem.mgmt is Subscription A.
#prodiver azurem.corpapps is Subscription B.
I am trying to create nsg rule in Subscription A with Provider azurerm.mgmt. Here the destination application security group is in Subscription B with Provider azurerm.corpapps in this subscription.
provider "azurerm" {
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
tenant_id = "${var.tenant_id}"
subscription_id = "${var.subscription}"
alias = "mgmt"
}
provider "azurerm" {
client_id = "${var.client_id}"
client_secret = "${var.client_secret}"
tenant_id = "${var.tenant_id}"
subscription_id = "${var.subscription_B}"
alias = "corpapps"
}
Then i use the provider to get my asg from Subscription B as shown:
Then i use that reference in my nsg rule
However, i get error - saying the ASG is not found:
But, in azure portal the resource is already there as shown:
I have tried to assign the SP which has owner role on both subscriptions or using Azure account with CLI but it's no luck. Also, as the comment points out, there is a limitation that NSG does not reference ASG in different location. After my validation, you can not add the ASG from another subscription even it's in the same region as the NSG or targets VNet.
Moreover, when you add this ASG as the target source or destination in the NSG rules, you will see
Select an application security group (ASG) as the security rule
source. ASGs enable fine-grained network security policies based on
workloads or applications instead of IP addresses or CIDR blocks.
Rules specifying an application security group are only applied to
network interfaces that are members of the application security group
on the same virtual network.

Terraform "azurerm_storage_share" file share gives 403

From vsts release definition terraform task, create rg, vnet, subnet with:
service_endpoints = ["Microsoft.Storage"]
Then public ip, nsg, vnic
Then create az storage account, with:
network_rules {
virtual_network_subnet_ids = ["${azurerm_subnet.main.id}"]
}
when try to create a storage share:
resource "azurerm_storage_share" "someshare" { ...
this gives a 403. Even giving the service principal owner permissions makes no difference. Resorted to using a powershell follow on script as another task to add the file shares. Can terraform do this?

Resources