Is there a way in Azurerm to provision a subnet with an NSG attachment and also provision service endpoints as well . This is considering we have a policy to enforce NSG on subnet . So - - if subnet is a separate resource (as a separate resource nsg_id is not a parameter anymore) - it would fail the policy.
As a sub resrouce under vnet , can attach a nsg_id - but does not have parameters to provision a service endpoint or service delegation.
You could deploy a service endpoint or service delegation with Terraform Language local-exec Provisioner or remote-exec provisioner to invoke PowerShell or CLI scripts after a resource is created.
For example, you can use az network vnet subnet update to enable a service endpoint for the subnet.
resource "null_resource" "example" {
provisioner "local-exec" {
command = "az network vnet subnet update -g ${azurerm_resource_group.example.name} -n 'subnet1' --vnet-name ${azurerm_virtual_network.example.name} --service-endpoints 'Microsoft.Sql'"
interpreter = ["PowerShell", "-Command"]
}
}
For more references, you could read this blog- Multi-line PowerShell in Terraform.
Related
I am trying to implement a strategy where I can create a NSG in one Azure subscription and use the same NSG resource to attach to any VMs or NICs created in other subscriptions and resource groups.
How can this implementation work via Terraform where I want to attach a single (default) NSG (created in a separate subscription) to multiple VMs and NICs in other subscriptions?
Default NSG for all Azure Subscriptions via Terraform:
Rules defined for a certain network security group with some network security rules will only apply to that resource group. As a result of this limitation for network security groups, it is not feasible to access an NSG in subscriptions other than the existing ones.
You cannot access an NSG that exists in one subscription in another, even though it is provided in the same region.
If you need to add network security in other subscriptions, you can consider the following methods:
Add multiple subscriptions in provider using alias while deploying Terraform code, as mentioned article by #Jeff Brown.
provider "azurerm"{
alias = "xx"
subscription = "subscription1"
features{}
}
provider "azurerm"{
alias = "xxdev"
subscription = "subscription2"
features{}
}
resource "azurerm_network_security_group" "example"{
//Add configuration
}
Note: Include azurerm providers to deploy the same NSG or any Azure resource across multiple subscriptions provided by subscription Ids.
terraform import can be used to import existing resources from anywhere.
terraform import azurerm_network_security_group.<NSG> <ResourceID>
Output:
I'm having some trouble attaching a NIC (in resource group A) to a subnet belonging to a Vnet and NSG in a different resource group (say B). I have Contributor role in resource group A, but only Reader role in resource group B. Is this possible? If so, what am I doing wrong? Here's what it looks like (with UIDs shortened).
% az network nic create --resource-group A --name bastion-nic --vnet-name VN-B --subnet SubnetB
(InvalidResourceReference) Resource /subscriptions/40ef-b75f-c05a034bf2ff/resourceGroups/A/providers/Microsoft.Network/virtualNetworks/VN-B/subnets/SubnetB referenced by resource /subscriptions/b75f-c05a034bf2ff/resourceGroups/A/providers/Microsoft.Network/networkInterfaces/bastion-nic was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.
Code: InvalidResourceReference
I tested the same scenario in my environment .
Scenario: I created a user , 2 resource groups i.e. contributorTest with Contributor access for the user and readerTest with Reader access for the user.
If I use the command you are using then it gives me the same error message as you. To describe the issue when you are using vnet-name, the command thinks that the vnet is also present in the same resource which has been mentioned in the command.
az network nic create --resource-group contributorTest --name bastion-nic --vnet-name ansumantest-vnet --subnet default
So , for example in the above command resource group is contributorTest and we have just provided vnet name and subnet name , which it thinks is present in the same group. So it throws the error as below:
As a Solution you can use the below command to create NIC if the VNet is in different resource group:
az network nic create --resource-group contributorTest --name bastion-nic --subnet /subscriptions/subID/resourceGroups/readerTest/providers/Microsoft.Network/virtualNetworks/ansumantest-vnet/subnets/default
In the above command , we are not providing vnet name & subnet name , as a alternative we have provided the resourceID of the subnet.
Note: The above solution should work only if you have contributor access on both the resource group , in your case you will be getting the below error:
To describe the issue here, while you are creating a NIC it requires to join that NIC to the Subnet which you have specified but as you have reader access only on the VNET resource group it doesn't allow you to join the NIC and subnet.
So , Final solution can be :
Either have the VNET and subnet in the same resource group you are creating NIC on and have a Contributor access on it and use the command you are using .
Grant Contributor Access to the user for the second resource group and use the second Command that I have mentioned as a solution.
Output for the second command after providing contributor access for both the resource groups:
I have a resource group for my vnet (rg-private-vnet) and one for my webapp and sql server (rg-webapp). Now I want to create a private endpoint for my SQl server in the rg-webapp.
This works in case the private endpoint is created in the resource group from he vnet. But I want it to be located in the resource group of the sql server.
I used:
az network private-endpoint create --name $sqlPrivateEndpoint --resource-group $resourceGroupVnet --vnet-name $vnetName --subnet $sqlSubnetName --private-connection-resource-id $id --connection-name mySqlConnection --group-id sqlServer
In case I use the rg-webapp I got the error:
(InvalidResourceReference) Resource /subscriptions/XXX/[...} referenced by resource /subscriptions/XXX/[...] was not found. Please make sure that the referenced resource exists, and that both resources are in the same region.
with the rg-private-vnet it works. In the azure portal it is no problem to create the endpoint in the rg-webapp resource group. I don't see any more parameters to create it in a diffrent rg.
According to the documentation you need to provide the resource id of the subnet to the --subnet parameter (instead of the subnet name) and skip the --vnet-name parameter if the subnet resides in a different resource group:
--subnet Name or ID of an existing subnet. If name specified, also specify --vnet-name. If you want to use an existing subnet in other resource group or subscription, please provide the ID instead of the name of the subnet and do not specify the --vnet-name. (az network private-endpoint | Microsoft Docs)
I'm deploying AKS clusters with Terraform and it's working fine. However, when trying to add security rules to the AKS network security group in the automatically created MC* group, it fails with errors such as:
Creating/Updating Network Security Rule "myRule" (NSG "" / Resource Group "MC_terraform-aks-rg_terraform-aks_westeurope"): network.SecurityRulesClient#CreateOrUpdate: Failure sending request: StatusCode=404 -- Original Error: Code="ResourceNotFound" Message="The Resource 'Microsoft.Network/networkSecurityGroups/securityRules' under resource group 'MC_terraform-aks-rg_terraform-aks_westeurope' was not found."[0m
If I run terraform apply again, it works and the rules get created. I'm thinking there is a race condition somewhere and I even added a null_resource that executes a sleep command for a couple of minutes, but still errors out on the first try.
main.tf
resource "azurerm_kubernetes_cluster" "aks" {
....................................
}
resource "azurerm_network_security_rule" "https" {
name = "myRule"
priority = 101
direction = "Inbound"
access = "Allow"
protocol = "Tcp"
source_port_range = "443"
destination_port_range = "*"
source_address_prefixes = "${var.imperva_ips}"
destination_address_prefix = "${azurerm_public_ip.ingress.ip_address}"
resource_group_name = "${azurerm_kubernetes_cluster.aks.node_resource_group}"
network_security_group_name = "${data.external.aks_nsg_name.result.output}"
#depends_on = ["azurerm_resource_group.aks", "azurerm_mysql_virtual_network_rule.mysql", "helm_release.ingress"]
depends_on = [null_resource.delay]
}
resource "null_resource" "delay" {
provisioner "local-exec" {
command = "sleep 60"
}
depends_on = [helm_release.ingress]
}
# get the auto-generated NSG name
data "external" "aks_nsg_id" {
program = [
"bash",
"${path.root}/scripts/aks_nsg_name.sh"
]
depends_on = [azurerm_resource_group.aks]
}
The bash script that pulls the NSG name:
#!/bin/bash
OUTPUT=$(az network nsg list --query [].name -o tsv | grep aks | head -n 1)
jq -n --arg output "$OUTPUT" '{"output":$output}'
For AKS cluster, it's not recommended to create the NSG rules manually, Azure will create the appropriate rules for you automatically. If you create the rules manually, it could cause problems. See the description below:
A network security group filters traffic for VMs, such as the AKS
nodes. As you create Services, such as a LoadBalancer, the Azure
platform automatically configures any network security group rules
that are needed. Don't manually configure network security group rules
to filter traffic for pods in an AKS cluster. Define any required
ports and forwarding as part of your Kubernetes Service manifests, and
let the Azure platform create or update the appropriate rules. You can
also use network policies, as discussed in the next section, to
automatically apply traffic filter rules to pods.
So I would not suggest you create the rules yourself. For more details, see AKS Network Security Group. You'd better use the network policy rather than the NSG rules, and on my side, the network policy is more recommended.
Update:
And the error you got shows that it did not find the rules in the node group. As I see, you need to change the command with a group name of your AKS cluster in the bash. The command you use without a group name will list all the NSG in the subscription, it will not find your NSG if there is not only your AKS cluster.
From vsts release definition terraform task, create rg, vnet, subnet with:
service_endpoints = ["Microsoft.Storage"]
Then public ip, nsg, vnic
Then create az storage account, with:
network_rules {
virtual_network_subnet_ids = ["${azurerm_subnet.main.id}"]
}
when try to create a storage share:
resource "azurerm_storage_share" "someshare" { ...
this gives a 403. Even giving the service principal owner permissions makes no difference. Resorted to using a powershell follow on script as another task to add the file shares. Can terraform do this?