My query is related to azurerm_log_analytics_data_export_rule. I have created Log Analytics Workspace and Eventhub in portal followed all the steps in below link.
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/log_analytics_data_export_rule
Both Terraform Plan and Apply are successful. But the expected tables are not created in Eventhub. For example (as per above link) “Heartbeat” table is not created Eventhub after export_rule created. The below Microsoft documentation mentions that the tables will be automatically created in EH or Storage account once export rule creation successful.
https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logs-data-export?tabs=portal
Will be helpful if I get some info on this rule.
The Hashicrop template you are following will create new resource group, storage account, log analytics workspace & a export rule.
Since the above terraform template is creating new environment & there will be no heart beat logs present by default so that is reason why there were no heart beat logs container was created.
When we have tested in our environment, exporting heart beat logs of log analytics workspace data to storage account it took nearly 30 minutes to get the data to be reflected in our storage account.
Data completeness
Data export will continue to retry sending data for up to 30 minutes in the event that the destination is unavailable. If it's still unavailable after 30 minutes then data will be discarded until the destination becomes available.
provider "azurerm" {
features{}
}
resource "azurerm_resource_group" "data_export_resource_group" {
name = "test_data_export_rg"
location = "centralus"
}
resource "azurerm_log_analytics_workspace" "data_export_log_analytics_workspace" {
name = "testdataexportlaw"
location = azurerm_resource_group.data_export_resource_group.location
resource_group_name = azurerm_resource_group.data_export_resource_group.name
sku = "PerGB2018"
retention_in_days = 30
}
resource "azurerm_storage_account" "data_export_azurerm_storage_account" {
name = "testdataexportazurermsa"
resource_group_name = azurerm_resource_group.data_export_resource_group.name
location = azurerm_resource_group.data_export_resource_group.location
account_tier = "Standard"
account_replication_type = "LRS"
}
resource "azurerm_eventhub_namespace" "data_export_azurerm_eventhub_namespace" {
name = "testdataexportehnamespace"
location = azurerm_resource_group.data_export_resource_group.location
resource_group_name = azurerm_resource_group.data_export_resource_group.name
sku = "Standard"
capacity = 1
tags = {
environment = "Production"
}
}
resource "azurerm_eventhub" "data_export_eventhub" {
name = "testdataexporteh1"
namespace_name = azurerm_eventhub_namespace.data_export_azurerm_eventhub_namespace.name
resource_group_name = azurerm_resource_group.data_export_resource_group.name
partition_count = 2
message_retention = 1
}
```
resource "azurerm_log_analytics_data_export_rule" "example" {
name = "testdataExport1"
resource_group_name = azurerm_resource_group.data_export_resource_group.name
workspace_resource_id = azurerm_log_analytics_workspace.data_export_log_analytics_workspace.id
destination_resource_id = azurerm_eventhub.data_export_eventhub.id
table_names = ["Usage","StorageBlobLogs"]
enabled = true
}
```
Related
I'm deploying an AKS cluster via Terraform.
I set an oms_agent block within my aks resource block:
resource "azurerm_kubernetes_cluster" "tfdemo-cluster" {
resource_group_name = var.resourcegroup_name
location = var.location
name = "${var.projectname}-aks"
node_resource_group = "${var.resourcegroup_name}-node"
... omitted to shorten ...
oms_agent {
log_analytics_workspace_id = var.log_analytics_workspace_id
}
Like this it works as aspected.
But when I add an additional resource of type diagnostic_settings like so
resource "azurerm_monitor_diagnostic_setting" "aks-diagnostics" {
name = "aks-logs"
storage_account_id = var.storage_account_id
target_resource_id = azurerm_kubernetes_cluster.tfdemo-cluster.id
log {
category = "kube-audit"
enabled = true
}
metric {
category = "AllMetrics"
retention_policy {
days = 30
enabled = true
}
}
}
I run into an error that says:
"diagnosticsettings.DiagnosticSettingsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=nil nil"
When I tried to google that error messages I found issues related to other Azure services where the sku of that service wasn't matching a specified feature or capacity but I'm don't see that here.
Why I want log analytics workspace AND logs dumped into a storage account: My thinking was just that a log anal. ws is really expensive compared to storage in a storage account. So I thought I send say the audit data for long time retention to the cheap storage account (my settings in the given example might not 100% represent that but it's not the point here I'd say) and still have the "expensive" log analytics service to dig into the cluster performance.
Thanks a lot for any input!
I Tried to reproduce the same in my environment to Create an Azure AKS cluster with OMS Agent and Diagnostic Setting using Terraform:
Sending long-term data retention logs to a Azure Storage Account can be more cost-effective than keeping them in a Azure Log Analytics workspace. However, the Azure Log Analytics workspace can still be useful for real-time analysis and performance monitoring.
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "aksgroup" {
name = "aks-rg"
location = "East US"
}
resource "azurerm_log_analytics_workspace" "oms" {
name = "oms-workspace"
location = azurerm_resource_group.aksgroup.location
resource_group_name = azurerm_resource_group.aksgroup.name
sku = "PerGB2018"
}
resource "azurerm_kubernetes_cluster" "aks" {
name = "cluster-aks1"
location = azurerm_resource_group.aksgroup.location
resource_group_name = azurerm_resource_group.aksgroup.name
dns_prefix = "aks1"
default_node_pool {
name = "default"
node_count = 1
vm_size = "standard_a2_v2"
}
identity {
type = "SystemAssigned"
}
tags = {
Environment = "Production"
}
addon_profile {
oms_agent {
enabled = true
log_analytics_workspace_id = azurerm_log_analytics_workspace.oms.id
}
}
}
output "client_certificate" {
value = azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate
sensitive = true
}
output "kube_config" {
value = azurerm_kubernetes_cluster.aks.kube_config_raw
sensitive = true
}
resource "azurerm_monitor_diagnostic_setting" "aks" {
name = "aks-diagnostic-setting"
target_resource_id = azurerm_kubernetes_cluster.aks.id
storage_account_id = azurerm_storage_account.aks.id
log_analytics_workspace_id = azurerm_log_analytics_workspace.oms.id
log {
category = "kube-audit"
enabled = true
}
metric {
category = "AllMetrics"
retention_policy {
days = 30
enabled = true
}
}
}
resource "azurerm_storage_account" "aks" {
name = "aksdiagnostic"
resource_group_name = azurerm_resource_group.aksgroup.name
location = azurerm_resource_group.aksgroup.location
account_tier = "Standard"
account_replication_type = "LRS"
}
Terraform Apply:
Once ran the code resources are created, like below.
Azure AKS Diagnostic settings created with Log Analytics settings.
Log Analytics settings- created.
Terraform Version
1.2.3
AzureRM Provider Version
v3.13.0
Affected Resource(s)/Data Source(s)
Azure data factory, SQL Database
Terraform Configuration Files
resource "azurerm_data_factory_linked_service_azure_sql_database" "sqldatabase_linked_service_10102022" {
count = (var.subResourcesInfo.sqlDatabaseName != "") ? 1 : 0
depends_on = [azurerm_data_factory_integration_runtime_azure.autoresolve_integration_runtime,
azurerm_data_factory_managed_private_endpoint.sqlserver_managed_endpoint]
name = "AzureSqlDatabase10102022"
data_factory_id = azurerm_data_factory.datafactory.id
integration_runtime_name = "AutoResolveIntegrationRuntime"
use_managed_identity = true
connection_string = format("Integrated Security=False;Data Source=%s.database.windows.net;Initial Catalog=%s;",
var.subResourcesInfo.sqlServerName,
var.subResourcesInfo.sqlDatabaseName)
}
Expected Behaviour
Issue is ADF to DB connectivity, error:
Operation on target DWH_DF_aaa failed: {'StatusCode':'DFExecutorUserError','Message':'Job failed due to reason: com.microsoft.dataflow.broker.InvalidOperationException: Only one valid authentication should be used for AzureSqlDatabase. ServicePrincipalAuthentication is invalid. One or two of servicePrincipalId/key/tenant is missing.','Details':''}
When we created this LS using TF, we get tenant="" in ADF LS Json file which we suspect that causing issue of above error.
When we created the same LS directly on ADF UI, then there is no field of tenant="" in its json file, and if we use this LS in dataflow/pipeline then communication works from ADF to DB.
Expected behavior should be, if we don't provide tenant_id parameter in TF code then in json also should not show tenant="" which then works for connectivity.
I tried to reproduce the scenario in my environment:
With below code , I could create a Linked Service (connection) between Azure SQL Database and Azure Data Factory.
Code:
resource "azurerm_data_factory" "example" {
name = "kaADFexample"
location = data.azurerm_resource_group.example.location
resource_group_name = data.azurerm_resource_group.example.name
managed_virtual_network_enabled = true
}
resource "azurerm_storage_account" "example" {
name = "kaaaexample"
resource_group_name = data.azurerm_resource_group.example.name
location = data.azurerm_resource_group.example.location
account_kind = "BlobStorage"
account_tier = "Standard"
account_replication_type = "LRS"
}
resource "azurerm_data_factory_managed_private_endpoint" "example" {
name = "example"
data_factory_id = azurerm_data_factory.example.id
target_resource_id = azurerm_storage_account.example.id
subresource_name = "blob"
}
resource "azurerm_user_assigned_identity" "main" {
depends_on = [data.azurerm_resource_group.example]
name = "kasupports01-mid"
resource_group_name = data.azurerm_resource_group.example.name
location = data.azurerm_resource_group.example.location
}
resource "azurerm_data_factory_integration_runtime_azure" "test" {
name = "AutoResolveIntegrationRuntime"
data_factory_id = azurerm_data_factory.example.id
location = "AutoResolve"
virtual_network_enabled = true
}
resource "azurerm_data_factory_linked_service_azure_sql_database" "linked_service_azure_sql_database" {
name = "kaexampleLS"
data_factory_id = azurerm_data_factory.example.id
connection_string = "data source=serverhostname;initial catalog=master;user id=testUser;Password=test;integrated security=False;encrypt=True;connection timeout=30"
use_managed_identity = true
integration_runtime_name = azurerm_data_factory_integration_runtime_azure.test.name
depends_on = [azurerm_data_factory_integration_runtime_azure.test,
azurerm_data_factory_managed_private_endpoint.example]
}
output "id" {
value = azurerm_data_factory_linked_service_azure_sql_database.linked_service_azure_sql_database.id
}
Executed: terraform plan
Output:
id = "/subscriptions/xxxxxxxxx/resourceGroups/xxxxxx/providers/Microsoft.DataFactory/factories/kaADFexample/linkedservices/kaexampleLS"
If the error persists in your case ,try removing the tenant attribute in the data_factory just after deployment is done in terraform.
Please check this known issue and mentioned by #chgenzel in terraform-provider-azurerm issues | Github
ADF:
Managed Identity
Linked service : azure sql
Reference: data_factory_linked_service_azure_sql_database | terraformregistry
The azure Database for PostgreSQL Flexible server automatically back up the databases. In case of any accidental deletion of any databases we can restore the database by creating a new flexible server for the recovery process from the back up database .I know how do it from azure portal.Does the terraform code can also configure "backup and restore" for PostgreSQL Flexible server - Restore server.
The exact summary of the manual task documented in the azure doc:https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/how-to-restore-server-portal. Just i want do the task using terraform . In addition to that ensure appropriate login and database level permission
I really appreciate any support and help
It is possible to create the azure database for PostgreSQL flexible server backup using terraform
Please use the below terraform code to restore the server
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "example" {
name = "RG_NAME"
location = "EASTUS"
}
resource "azurerm_virtual_network" "example" {
name = "example-vn"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
address_space = ["10.0.0.0/16"]
}
resource "azurerm_subnet" "example" {
name = "example-sn"
resource_group_name = azurerm_resource_group.example.name
virtual_network_name = azurerm_virtual_network.example.name
address_prefixes = ["10.0.2.0/24"]
service_endpoints = ["Microsoft.Storage"]
delegation {
name = "fs"
service_delegation {
name = "Microsoft.DBforPostgreSQL/flexibleServers"
actions = [
"Microsoft.Network/virtualNetworks/subnets/join/action",
]
}
}
}
resource "azurerm_private_dns_zone" "example" {
name = "example.postgres.database.azure.com"
resource_group_name = azurerm_resource_group.example.name
}
resource "azurerm_private_dns_zone_virtual_network_link" "example" {
name = "exampleVnetZone.com"
private_dns_zone_name = azurerm_private_dns_zone.example.name
virtual_network_id = azurerm_virtual_network.example.id
resource_group_name = azurerm_resource_group.example.name
}
resource "azurerm_postgresql_flexible_server" "example" {
name = "example-psqlflexibleserver"
resource_group_name = azurerm_resource_group.example.name
location = azurerm_resource_group.example.location
version = "12"
delegated_subnet_id = azurerm_subnet.example.id
private_dns_zone_id = azurerm_private_dns_zone.example.id
administrator_login = "psqladmin"
administrator_password = "H#Sh1CoR3!"
zone = "1"
storage_mb = 32768
backup_retention_days = 30
geo_redundant_backup_enabled = true
sku_name = "GP_Standard_D4s_v3"
depends_on = [azurerm_private_dns_zone_virtual_network_link.example]
}
Here I have mentioned the RG_name, subnet, VM, Vnet, db name, password and backup policy days
I have given the backup policy retention days are 30 the policy retention days should be in between 1 to 35 and the defaults value is 7 days
Before running the script we have to check the appropriate login server details
After the follow the below steps to execute the file
terraform init
It will initialize the file
Terraform plan
This will creates an execution plan and it will preview the changes that terraform plans to make the infrastructure
Terraform apply
This will creates or updates the infrastructure depending on the configuration
Previously it was default and the geo_redundant_backup_enabled is false I have set it to true and backup policy will be 30 days
For reference you can use this documentation
When I configure Azure Monitoring using the OMS solution for VMs with this answer Enable Azure Monitor for existing Virtual machines using terraform, I notice that this feature is being deprecated and Azure prefers you move to the new monitoring solution (Not using the log analytics agent).
Azure allows me to configure VM monitoring using this GUI, but I would like to do it using terraform.
Is there a particular setup I have to use in terraform to achieve this? (I am using a Linux VM btw)
Yes, that is correct. The omsagent has been marked as legacy and Azure now has a new monitoring agent called "Azure Monitor agent" . The solution given below is for Linux, Please check the Official Terraform docs for Windows machines.
We need three things to do the equal UI counterpart in Terraform.
azurerm_log_analytics_workspace
azurerm_monitor_data_collection_rule
azurerm_monitor_data_collection_rule_association
Below is the example code:
data "azurerm_virtual_machine" "vm" {
name = var.vm_name
resource_group_name = var.az_resource_group_name
}
resource "azurerm_log_analytics_workspace" "workspace" {
name = "${var.project}-${var.env}-log-analytics"
location = var.az_location
resource_group_name = var.az_resource_group_name
sku = "PerGB2018"
retention_in_days = 30
}
resource "azurerm_virtual_machine_extension" "AzureMonitorLinuxAgent" {
name = "AzureMonitorLinuxAgent"
publisher = "Microsoft.Azure.Monitor"
type = "AzureMonitorLinuxAgent"
type_handler_version = 1.0
auto_upgrade_minor_version = "true"
virtual_machine_id = data.azurerm_virtual_machine.vm.id
}
resource "azurerm_monitor_data_collection_rule" "example" {
name = "example-rules"
resource_group_name = var.az_resource_group_name
location = var.az_location
destinations {
log_analytics {
workspace_resource_id = azurerm_log_analytics_workspace.workspace.id
name = "test-destination-log"
}
azure_monitor_metrics {
name = "test-destination-metrics"
}
}
data_flow {
streams = ["Microsoft-InsightsMetrics"]
destinations = ["test-destination-log"]
}
data_sources {
performance_counter {
streams = ["Microsoft-InsightsMetrics"]
sampling_frequency_in_seconds = 60
counter_specifiers = ["\\VmInsights\\DetailedMetrics"]
name = "VMInsightsPerfCounters"
}
}
}
# associate to a Data Collection Rule
resource "azurerm_monitor_data_collection_rule_association" "example1" {
name = "example1-dcra"
target_resource_id = data.azurerm_virtual_machine.vm.id
data_collection_rule_id = azurerm_monitor_data_collection_rule.example.id
description = "example"
}
Reference:
monitor_data_collection_rule
monitor_data_collection_rule_association
I have a terraform code that creates a stream analytics job, An input and output for the job too.
Below is my terraform code:
provider "azurerm" {
version = "=1.44"
}
resource "azurerm_stream_analytics_job" "test_saj" {
name = "test-stj"
resource_group_name = "myrgname"
location = "Southeast Asia"
compatibility_level = "1.1"
data_locale = "en-US"
events_late_arrival_max_delay_in_seconds = 60
events_out_of_order_max_delay_in_seconds = 50
events_out_of_order_policy = "Adjust"
output_error_policy = "Drop"
streaming_units = 3
tags = {
environment = "test"
}
transformation_query = var.query
}
resource "azurerm_stream_analytics_output_blob" "mpl_saj_op_jk_blob" {
name = var.saj_jk_blob_output_name
stream_analytics_job_name = "test-stj"
resource_group_name = "myrgname"
storage_account_name = "mystaname"
storage_account_key = "mystakey"
storage_container_name = "testupload"
path_pattern = myfolder/{day}"
date_format = "yyyy-MM-dd"
time_format = "HH"
depends_on = [azurerm_stream_analytics_job.test_saj]
serialization {
type = "Json"
encoding = "UTF8"
format = "LineSeparated"
}
}
resource "azurerm_stream_analytics_stream_input_eventhub" "mpl_saj_ip_eh" {
name = var.saj_joker_event_hub_name
stream_analytics_job_name = "test-stj"
resource_group_name = "myrgname"
eventhub_name = "myehname"
eventhub_consumer_group_name = "myehcgname"
servicebus_namespace = "myehnamespacename"
shared_access_policy_name = "RootManageSharedAccessKey"
shared_access_policy_key = "ehnamespacekey"
serialization {
type = "Json"
encoding = "UTF8"
}
depends_on = [azurerm_stream_analytics_job.test_saj]
}
Following is my tfvars input file:
query=<<EOT
myqueryhere
EOT
saj_jk_blob_output_name="outputtoblob01"
saj_joker_event_hub_name="inputventhub01"
I have no problem with the creation. Now my problem is when I want to create a new input and output for the same stream analytics job, I changed the name values alone in the tfvars file and gave terraform apply (in the same directory where first apply was given. Same state file).
Terraform is replacing the existing i/p and o/p with the new ones which is not my requirement. I want both the old one and the new one. This usecase was satisfied when imported the existing stream analytics using terraform import in a completely different folder and I used the same code. But is there way to do this without terraform import. Can this be done with a single state file itself?
State allows Terraform to know what Azure resources to add, update, or delete. What you want to do can not be done with a single state file itself unless you directly deploy resources with different names in your configuration files.
For example, if you want to create two virtual networks. You can directly create resources like this or use a count parameter on resources level for the loop.
resource "azurerm_virtual_network" "example" {
name = "examplevnet1"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
address_space = ["10.1.0.0/16"]
}
resource "azurerm_virtual_network" "example" {
name = "examplevnet2"
location = azurerm_resource_group.example.location
resource_group_name = azurerm_resource_group.example.name
address_space = ["10.2.0.0/16"]
}
When working with Terraform in a team, you can use remote state to write the state data to a remote data store, which can then be shared between all members of a team. It's recommended to store Terraform state in Azure Storage.
For more information, you could see Terraform workflow in this blog.