So i can encrypt the os disk with Terrafrom from what i have seen on this site. But how do i encrypt the data disks as well? I thought maybe "VolumeType": "All" would cover all disks but that did not happen. This code works for encrypting os disk... what do i need to do for multiple disks? I am stuck.
Thanks!
provider "azurerm" {
features {}
}
data "azurerm_key_vault" "keyvault" {
name = "testkeyvault1"
resource_group_name = "testRG1"
}
resource "azurerm_virtual_machine_extension" "vmextension" {
name = "DiskEncryption"
virtual_machine_id = "/subscriptions/<sub id>/resourceGroups/TESTRG1/providers/Microsoft.Compute/virtualMachines/testvm-1"
publisher = "Microsoft.Azure.Security"
type = "AzureDiskEncryption"
type_handler_version = "2.2"
#auto_upgrade_minor_version = true
settings = <<SETTINGS
{
"EncryptionOperation": "EnableEncryption",
"KeyVaultURL": "${data.azurerm_key_vault.keyvault.vault_uri}",
"KeyVaultResourceId": "${data.azurerm_key_vault.keyvault.id}",
"KeyEncryptionKeyURL": "https://testkeyvault1-1.vault.azure.net/keys/testKey/314c507de8a047a5bfeeb477efcbff60",
"KekVaultResourceId": "${data.azurerm_key_vault.keyvault.id}",
"KeyEncryptionAlgorithm": "RSA-OAEP",
"VolumeType": "All"
}
SETTINGS
tags = {
Environment = "test"
}
}
I tested your code for a newly created VM with 2 Data Disks and it was the same for me as well , If I keep "Volume: ALL" then also only OS Disk get ADE enabled and not the data disks if I verify from portal or Azure CLI.
Solution for it will be as below :
Please make sure that the attached data disks are added as volumes and are formatted from within the VM before adding the extension from Terraform.
Once the above is done and you do a terraform apply to your code , After successful apply it will reflect on Portal and as well as inside the VM.
Related
When I configure Azure Monitoring using the OMS solution for VMs with this answer Enable Azure Monitor for existing Virtual machines using terraform, I notice that this feature is being deprecated and Azure prefers you move to the new monitoring solution (Not using the log analytics agent).
Azure allows me to configure VM monitoring using this GUI, but I would like to do it using terraform.
Is there a particular setup I have to use in terraform to achieve this? (I am using a Linux VM btw)
Yes, that is correct. The omsagent has been marked as legacy and Azure now has a new monitoring agent called "Azure Monitor agent" . The solution given below is for Linux, Please check the Official Terraform docs for Windows machines.
We need three things to do the equal UI counterpart in Terraform.
azurerm_log_analytics_workspace
azurerm_monitor_data_collection_rule
azurerm_monitor_data_collection_rule_association
Below is the example code:
data "azurerm_virtual_machine" "vm" {
name = var.vm_name
resource_group_name = var.az_resource_group_name
}
resource "azurerm_log_analytics_workspace" "workspace" {
name = "${var.project}-${var.env}-log-analytics"
location = var.az_location
resource_group_name = var.az_resource_group_name
sku = "PerGB2018"
retention_in_days = 30
}
resource "azurerm_virtual_machine_extension" "AzureMonitorLinuxAgent" {
name = "AzureMonitorLinuxAgent"
publisher = "Microsoft.Azure.Monitor"
type = "AzureMonitorLinuxAgent"
type_handler_version = 1.0
auto_upgrade_minor_version = "true"
virtual_machine_id = data.azurerm_virtual_machine.vm.id
}
resource "azurerm_monitor_data_collection_rule" "example" {
name = "example-rules"
resource_group_name = var.az_resource_group_name
location = var.az_location
destinations {
log_analytics {
workspace_resource_id = azurerm_log_analytics_workspace.workspace.id
name = "test-destination-log"
}
azure_monitor_metrics {
name = "test-destination-metrics"
}
}
data_flow {
streams = ["Microsoft-InsightsMetrics"]
destinations = ["test-destination-log"]
}
data_sources {
performance_counter {
streams = ["Microsoft-InsightsMetrics"]
sampling_frequency_in_seconds = 60
counter_specifiers = ["\\VmInsights\\DetailedMetrics"]
name = "VMInsightsPerfCounters"
}
}
}
# associate to a Data Collection Rule
resource "azurerm_monitor_data_collection_rule_association" "example1" {
name = "example1-dcra"
target_resource_id = data.azurerm_virtual_machine.vm.id
data_collection_rule_id = azurerm_monitor_data_collection_rule.example.id
description = "example"
}
Reference:
monitor_data_collection_rule
monitor_data_collection_rule_association
I am currently looking to deploy the SentinelOne agent via Terraform. There does not appear to be much documentation online for VM extension usage in terms of Terraform. Has anyone successfully deployed the S1 agent via Terraform extension? I am unclear on what to add to the settings/protected_settings blocks. Any help is appreciated.
"azurerm_virtual_machine_extension" "example" {
name = "hostname"
virtual_machine_id = azurerm_virtual_machine.example.id
publisher = "SentinelOne.LinuxExtension"
type = "LinuxExtension"
type_handler_version = "1.0"
To add to the settings/protected settings blocks in terraform
resource "azurerm_virtual_machine_extension" "example" {
name = "hostname"
virtual_machine_id = azurerm_virtual_machine.example.id
publisher = "SentinelOne.LinuxExtension"
type = "LinuxExtension"
type_handler_version = "1.0"
settings = <<SETTINGS
{
"commandToExecute": "powershell.exe -Command \"${local.powershell_command}\""
}
SETTINGS
tags = {
environment = "Production"
}
depends_on = [
azurerm_virtual_machine.example
]
}
Settings - The extension's settings are provided as a string-encoded JSON object.
protected settings In the same way that settings are supplied as a JSON object in a string, the protected settings passed to the extension are also.
The keys in the settings and protected settings blocks must be case sensitive according to some VM Extensions. Make sure they are consistent with how Azure expects them (for example, the keys for the JsonADDomainExtension extension the keys are supposed to be in TitleCase)
Reference: azurerm_virtual_machine_extension
Installing the plugin manually and checking the JSON output gives the following settings block:
{
"LinuxAgentVersion": "22.4.1.2",
"SiteToken": "<your_site_token_here"
}
Unfortunately, this leaves the one critical field required for installation out, since it's a protected setting. That is the field name for the "Sentinel One Console API token".
UPDATE:
Working extension example after finding the correct JSON key value:
resource "azurerm_virtual_machine_extension" "testserver-sentinelone-extension" {
name = "SentinelOneLinuxExtension"
virtual_machine_id = azurerm_linux_virtual_machine.testserver.id
publisher = "SentinelOne.LinuxExtension"
type = "LinuxExtension"
type_handler_version = "1.2"
automatic_upgrade_enabled = false
settings = <<SETTINGS
{
"LinuxAgentVersion": "22.4.1.2",
"SiteToken": "<your_site_token_here>"
}
SETTINGS
protected_settings = <<PROTECTEDSETTINGS
{
"SentinelOneConsoleAPIKey": "${var.sentinel_one_api_token}"
}
PROTECTEDSETTINGS
}
EDIT: Figured it out by once again manually installing the extension on another test system, and then digging into the waagent logs on that VM to see what value was being queried by the enable.sh script.
# cat /var/lib/waagent/SentinelOne.LinuxExtension.LinuxExtension-1.2.0/scripts/enable.sh | grep Console
api_token=$(echo "$protected_settings_decrypted" | jq -r ".SentinelOneConsoleAPIKey")
I have a VM template I'm deploying an Azure Virtual Desktop environment with terraform (via octopus deploy) to Azure. On top of the Virtual Machines, I'm installing a number of extensions which culminates with a vm extension to register the VM with the Host Pool.
I'd like to rebuild the VM each time the custom script extension is applied (Extension #2, after domain join). But in rebuilding the VM, I'd like to build out a new VM, complete with the host pool registration before any part of the existing VM is destroyed.
Please accept the cut down version below to understand what I am trying to do.
I expect the largest number of machine recreations to come from enhancements to the configuration scripts that configure the server on creation. Not all of the commands are expected to be idempotent and we want the AVD vms to be ephemeral. If an issue is encountered, the support team is expected to be able to drain a server and destroy it once empty to get a replacement by terraform apply. In a case where the script gets updated though, we want to be able to replace all VMs quickly in an emergency, or at the very least minimize the nightly maintenance window.
Script Process: parameterized script > gets filled out as a template file > gets stored as an az blob > called by custom script extension > executed on the machine.
VM build process: VM is provisioned > currently 8 extensions get applied one at a time, starting with the domain join, then the custom script extension, followed by several Azure monitoring extensions, and finally the host pool registration extension.
I've been trying to use the create_before_destroy lifecycle feature, but I can't get it to spin up the VM, and apply all extensions before it begins removing the hostpool registration from the existing VMs. I assume there's a way to do it using the triggers, but I'm not sure how to do it in such a way that it always has at least the current number of VMs.
It would also need to be able to stop if it encounters an error on the new vm, before destroying the existing vm (or better yet, be authorized to rebuild VMs if an extension fails part way through).
resource "random_pet" "avd_vm" {
prefix = var.client_name
length = 1
keepers = {
# Generate a new pet name each time we update the setup_host script
source_content = "${data.template_file.setup_host.rendered}"
}
}
data "template_file" "setup_host" {
template = file("${path.module}\\scripts\\setup-host.tpl")
vars = {
storageAccountName = azurerm_storage_account.storage.name
storageAccountKey = azurerm_storage_account.storage.primary_access_key
domain = var.domain
aad_group_name = var.aad_group_name
}
}
resource "azurerm_storage_blob" "setup_host" {
name = "setup-host.ps1"
storage_account_name = azurerm_storage_account.scripts.name
storage_container_name = time_sleep.container_rbac.triggers["name"]
type = "Block"
source_content = data.template_file.setup_host.rendered #"${path.module}\\scripts\\setup-host.ps1"
depends_on = [
azurerm_role_assignment.account1_write,
data.template_file.setup_host,
time_sleep.container_rbac
]
}
data "template_file" "client_r_drive_mapping" {
template = file("${path.module}\\scripts\\client_r_drive_mapping.tpl")
vars = {
storageAccountName = azurerm_storage_account.storage.name
storageAccountKey = azurerm_storage_account.storage.primary_access_key
}
}
resource "azurerm_windows_virtual_machine" "example" {
count = length(random_pet.avd_vm)
name = "${random_pet.avd_vm[count.index].id}"
...
lifecycle {
ignore_changes = [
boot_diagnostics,
identity
]
}
}
resource "azurerm_virtual_machine_extension" "first-domain_join_extension" {
count = var.rdsh_count
name = "${var.client_name}-avd-${random_pet.avd_vm[count.index].id}-domainJoin"
virtual_machine_id = azurerm_windows_virtual_machine.avd_vm.*.id[count.index]
publisher = "Microsoft.Compute"
type = "JsonADDomainExtension"
type_handler_version = "1.3"
auto_upgrade_minor_version = true
settings = <<SETTINGS
{
"Name": "${var.domain_name}",
"OUPath": "${var.ou_path}",
"User": "${var.domain_user_upn}#${var.domain_name}",
"Restart": "true",
"Options": "3"
}
SETTINGS
protected_settings = <<PROTECTED_SETTINGS
{
"Password": "${var.admin_password}"
}
PROTECTED_SETTINGS
lifecycle {
ignore_changes = [settings, protected_settings]
}
depends_on = [
azurerm_virtual_network_peering.out-primary,
azurerm_virtual_network_peering.in-primary,
azurerm_virtual_network_peering.in-secondary
]
}
# Multiple scripts called by ./<scriptname referencing them in follow-up scripts
# https://web.archive.org/web/20220127015539/https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-windows
# https://learn.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-windows#using-multiple-scripts
resource "azurerm_virtual_machine_extension" "second-custom_scripts" {
count = var.rdsh_count
name = "${random_pet.avd_vm[count.index].id}-setup-host"
virtual_machine_id = azurerm_windows_virtual_machine.avd_vm.*.id[count.index]
publisher = "Microsoft.Compute"
type = "CustomScriptExtension"
type_handler_version = "1.10"
auto_upgrade_minor_version = "true"
protected_settings = <<PROTECTED_SETTINGS
{
"storageAccountName": "${azurerm_storage_account.scripts.name}",
"storageAccountKey": "${azurerm_storage_account.scripts.primary_access_key}"
}
PROTECTED_SETTINGS
settings = <<SETTINGS
{
"fileUris": ["https://${azurerm_storage_account.scripts.name}.blob.core.windows.net/scripts/setup-host.ps1","https://${azurerm_storage_account.scripts.name}.blob.core.windows.net/scripts/client_r_drive_mapping.ps1"],
"commandToExecute": "powershell -ExecutionPolicy Unrestricted -file setup-host.ps1"
}
SETTINGS
depends_on = [
azurerm_virtual_machine_extension.first-domain_join_extension,
azurerm_storage_blob.setup_host
]
}
resource "azurerm_virtual_machine_extension" "last_host_extension_hp_registration" {
count = var.rdsh_count
name = "${var.client_name}-${random_pet.avd_vm[count.index].id}-avd_dsc"
virtual_machine_id = azurerm_windows_virtual_machine.avd_vm.*.id[count.index]
publisher = "Microsoft.Powershell"
type = "DSC"
type_handler_version = "2.73"
auto_upgrade_minor_version = true
settings = <<-SETTINGS
{
"modulesUrl": "https://wvdportalstorageblob.blob.core.windows.net/galleryartifacts/Configuration_3-10-2021.zip",
"configurationFunction": "Configuration.ps1\\AddSessionHost",
"properties": {
"HostPoolName":"${azurerm_virtual_desktop_host_pool.pooleddepthfirst.name}"
}
}
SETTINGS
protected_settings = <<PROTECTED_SETTINGS
{
"properties": {
"registrationInfoToken": "${azurerm_virtual_desktop_host_pool_registration_info.pooleddepthfirst.token}"
}
}
PROTECTED_SETTINGS
lifecycle {
ignore_changes = [settings, protected_settings]
}
depends_on = [
azurerm_virtual_machine_extension.second-custom_scripts
]
}
I'm having a really difficult time trying to deploy a CoreOS virtual machine on vsphere using Terraform.
So far this is the terraform file I'm using:
# Configure the VMware vSphere Provider. ENV Variables set for Username and Passwd.
provider "vsphere" {
vsphere_server = "192.168.105.10"
allow_unverified_ssl = true
}
provider "ignition" {
version = "1.0.0"
}
data "vsphere_datacenter" "dc" {
name = "Datacenter"
}
data "vsphere_datastore" "datastore" {
name = "vol_af01_idvms"
datacenter_id = "${data.vsphere_datacenter.dc.id}"
}
data "vsphere_resource_pool" "pool" {
name = "Cluster_rnd/Resources"
datacenter_id = "${data.vsphere_datacenter.dc.id}"
}
data "vsphere_network" "network" {
name = "VM Network"
datacenter_id = "${data.vsphere_datacenter.dc.id}"
}
data "vsphere_virtual_machine" "template" {
name = "coreos_production"
datacenter_id = "${data.vsphere_datacenter.dc.id}"
}
# Create a folder
resource "vsphere_folder" "TestPath" {
datacenter_id = "${data.vsphere_datacenter.dc.id}"
path = "Test"
type = "vm"
}
#Define ignition data
data "ignition_networkd_unit" "vmnetwork" {
name = "00-ens192.network"
content = <<EOF
[Match]
Name=ens192
[Network]
DNS=8.8.8.8
Address=192.168.105.27/24
Gateway=192.168.105.1
EOF
}
data "ignition_config" "node" {
networkd = [
"${data.ignition_networkd_unit.vmnetwork.id}"
]
}
# Define the VM resource
resource "vsphere_virtual_machine" "vm" {
name = "terraform-test"
folder = "${vsphere_folder.TestPath.path}"
resource_pool_id = "${data.vsphere_resource_pool.pool.id}"
datastore_id = "${data.vsphere_datastore.datastore.id}"
num_cpus = 2
memory = 1024
guest_id = "other26xLinux64Guest"
network_interface {
network_id = "${data.vsphere_network.network.id}"
}
disk {
name = "terraform-test.vmdk"
size = "${data.vsphere_virtual_machine.template.disks.0.size}"
eagerly_scrub = "${data.vsphere_virtual_machine.template.disks.0.eagerly_scrub}"
thin_provisioned = "${data.vsphere_virtual_machine.template.disks.0.thin_provisioned}"
}
clone {
template_uuid = "${data.vsphere_virtual_machine.template.id}"
}
extra_config {
guestinfo.coreos.config.data.encoding = "base64"
guestinfo.coreos.config.data = "${base64encode(data.ignition_config.node.rendered)}"
}
}
I'm using terraform vsphere provier to create the virtual machine and ignition provider to pass customization details of the virtual machine such as network configuration.
It is not quite clear to me if I'm using correctly the extra_config property on the virtual machine definition. You can find documentation about that property here.
Virtual machine gets created, but network settings never are applied, meaning that ignition provisioning is not correctly working.
I would appreciate any guidance on how to properly configure Terraform for this particular scenario (Vsphere environment and CoreOS virtual machine), specially regarding guestinfo configuration.
Terraform v0.11.1, provider.ignition v1.0.0, provider.vsphere v1.1.0
VMware ESXi, 6.5.0, 5310538
CoreOS 1520.0.0
EDIT (2018-03-02)
As of version 1.3.0 of terraform vsphere provider, there is available a new vApp property. Using this property, there is no need to tweak the virtual machine using VMware PowerCLI as I did in the first answer.
There is a complete example of using this property here
The machine definition now would look something like this:
...
clone {
template_uuid = "${data.vsphere_virtual_machine.template.id}"
}
vapp {
properties {
"guestinfo.coreos.config.data.encoding" = "base64"
"guestinfo.coreos.config.data" = "${base64encode(data.ignition_config.node.rendered)}"
}
...
OLD ANSWER
Finally got this working.
The workflow I have used to create a CoreOS machine on vSphere using Terraform is as follows:
Download the latest Container Linux Stable OVA from
https://stable.release.core-os.net/amd64-usr/current/coreos_production_vmware_ova.ova.
Import coreos_production_vmware_ova.ova into vCenter.
Edit machine settings as desired (number of CPUs, disk size etc.)
Disable "vApp Options" of virtual machine.
Convert the virtual machine into a virtual machine template.
Once you have done this, you've got a CoreOS virtual machine template that is ready to be used with Terraform.
As I said in a comment to the question, some days ago I found this and that lead me to understand that my problem could be related to not being able to perform step 4.
The thing is that to be able to disable "vApp Options" (i.e. to see in the UI the "vApp Options" tab of the virtual machine) you need DSR enabled in your vSphere cluster, and, to be able to enable DSR, your hosts must be licensed with a key that supports DRS. Mine weren't, so I was stuck in that 4th step.
I wrote to VMware support, and they told me an alternate way to do this, without having to buy a different license.
This can be done using VMware PowerCLI. Here are the steps to install PowerCLI, and here is the reference. Once you got PowerCLI installed, this is the script I used to disable "vApp Options" in my machines:
Import-Module VMware.PowerCLI
#connect to vcenter
Connect-VIServer -Server yourvCenter -User yourUser -Password yourPassword
#Use this to disable the vApp functionality.
$disablespec = New-Object VMware.Vim.VirtualMachineConfigSpec
$disablespec.vAppConfigRemoved = $True
#Use this to enable
$enablespec = New-Object VMware.Vim.VirtualMachineConfigSpec
$enablespec.vAppConfig = New-Object VMware.Vim.VmConfigSpec
#Get the VM you want to work against.
$VM = Get-VM yourTemplate | Get-View
#Disables vApp Options
$VM.ReconfigVM($disablespec)
#Enables vApp Options
$VM.ReconfigVM($enablespec)
I executed that on a Powershell and managed to reconfigure the virtual machine, performing that 4th step. With this, I finally got my CoreOS virtual machine template correctly configured for this scenario.
I've tested this with terraform vSphere provider versions v0.4.2 and v1.1.0 (the syntax changes) and the machine gets created correctly; Ignition provisioning works and everything you put on your ignition file (network configs, users etc.) is applied on the newly created machine.
Receiving an error when creating a AzureRM VM with a Marketplace Image and Managed Disk, I am missing something syntax ally, maybe a plan block {} how would one find information to put in Plan block ? - Does anyone have a working example of a managed disk based AzureRM VM using a 3rd Party Marketplace image from Azure in Terraform ?
Error applying plan:
1 error(s) occurred:
* azurerm_virtual_machine.VMns-vpx-1: 1 error(s) occurred:
* azurerm_virtual_machine.VMns-vpx-1: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=200 -- Original Error: Long running operation terminated with status 'Failed': Code="VMMarketplaceInvalidInput" Message="Creating a virtual machine fro
m Marketplace image requires Plan information in the request. OS disk name is 'myosdisk1'."
Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.
PS C:\Users\eagle\Downloads\citrix\testingvpx>
In Addition Azure portal shows an error around the OS Disk:
General
Provisioning state Provisioning failed. Creating a virtual machine from Marketplace image requires Plan information in the request. OS disk name is 'myosdisk1'.. VMMarketplaceInvalidInput
Guest agent Unknown
Disks
myosdisk1 Provisioning failed. Creating a virtual machine from Marketplace image requires Plan information in the request. OS disk name is 'myosdisk1'.. VMMarketplaceInvalidInput
Terraform script being used:
resource "azurerm_virtual_machine" "VMns-vpx-1" {
name = "VMns-vpx-1"
location = "${var.azu_location}"
resource_group_name = "${azurerm_resource_group.rgVMns-vpx.name}"
network_interface_ids = ["${azurerm_network_interface.VMns-vpx-1-nic.id}"]
vm_size = "Standard_DS3_v2"
storage_image_reference {
publisher = "citrix"
offer = "netscalervpx120"
sku = "netscalerbyol"
version = "latest"
}
storage_os_disk {
name = "myosdisk1"
caching = "ReadWrite"
create_option = "FromImage"
managed_disk_type = "Premium_LRS"
}
os_profile {
computer_name = ""
admin_username = ""
admin_password = ""
}
boot_diagnostics {
enabled = "true"
storage_uri = "${azurerm_storage_account.xxxx.primary_blob_endpoint}"
}
tags {
Name = "xxxx"
Service = "xxx"
}
}
just like it says, you are missing a "plan" property definition. Marketplace images tend to have that.
Everything else in the vm definition looks fine, unless Terraform needs something else.