Azure AKS - oms agent AND diagnostic settings possible together? - azure

Question

I'm deploying an AKS cluster via Terraform.
I set an oms_agent block within my aks resource block:
resource "azurerm_kubernetes_cluster" "tfdemo-cluster" {
resource_group_name = var.resourcegroup_name
location = var.location
name = "${var.projectname}-aks"
node_resource_group = "${var.resourcegroup_name}-node"
... omitted to shorten ...
oms_agent {
log_analytics_workspace_id = var.log_analytics_workspace_id
}
Like this it works as aspected.
But when I add an additional resource of type diagnostic_settings like so
resource "azurerm_monitor_diagnostic_setting" "aks-diagnostics" {
name = "aks-logs"
storage_account_id = var.storage_account_id
target_resource_id = azurerm_kubernetes_cluster.tfdemo-cluster.id
log {
category = "kube-audit"
enabled = true
}
metric {
category = "AllMetrics"
retention_policy {
days = 30
enabled = true
}
}
}
I run into an error that says:
"diagnosticsettings.DiagnosticSettingsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=nil nil"
When I tried to google that error messages I found issues related to other Azure services where the sku of that service wasn't matching a specified feature or capacity but I'm don't see that here.
Why I want log analytics workspace AND logs dumped into a storage account: My thinking was just that a log anal. ws is really expensive compared to storage in a storage account. So I thought I send say the audit data for long time retention to the cheap storage account (my settings in the given example might not 100% represent that but it's not the point here I'd say) and still have the "expensive" log analytics service to dig into the cluster performance.
Thanks a lot for any input!

Answer 1

I Tried to reproduce the same in my environment to Create an Azure AKS cluster with OMS Agent and Diagnostic Setting using Terraform:
Sending long-term data retention logs to a Azure Storage Account can be more cost-effective than keeping them in a Azure Log Analytics workspace. However, the Azure Log Analytics workspace can still be useful for real-time analysis and performance monitoring.
provider "azurerm" {
features {}
}
resource "azurerm_resource_group" "aksgroup" {
name = "aks-rg"
location = "East US"
}
resource "azurerm_log_analytics_workspace" "oms" {
name = "oms-workspace"
location = azurerm_resource_group.aksgroup.location
resource_group_name = azurerm_resource_group.aksgroup.name
sku = "PerGB2018"
}
resource "azurerm_kubernetes_cluster" "aks" {
name = "cluster-aks1"
location = azurerm_resource_group.aksgroup.location
resource_group_name = azurerm_resource_group.aksgroup.name
dns_prefix = "aks1"
default_node_pool {
name = "default"
node_count = 1
vm_size = "standard_a2_v2"
}
identity {
type = "SystemAssigned"
}
tags = {
Environment = "Production"
}
addon_profile {
oms_agent {
enabled = true
log_analytics_workspace_id = azurerm_log_analytics_workspace.oms.id
}
}
}
output "client_certificate" {
value = azurerm_kubernetes_cluster.aks.kube_config.0.client_certificate
sensitive = true
}
output "kube_config" {
value = azurerm_kubernetes_cluster.aks.kube_config_raw
sensitive = true
}
resource "azurerm_monitor_diagnostic_setting" "aks" {
name = "aks-diagnostic-setting"
target_resource_id = azurerm_kubernetes_cluster.aks.id
storage_account_id = azurerm_storage_account.aks.id
log_analytics_workspace_id = azurerm_log_analytics_workspace.oms.id
log {
category = "kube-audit"
enabled = true
}
metric {
category = "AllMetrics"
retention_policy {
days = 30
enabled = true
}
}
}
resource "azurerm_storage_account" "aks" {
name = "aksdiagnostic"
resource_group_name = azurerm_resource_group.aksgroup.name
location = azurerm_resource_group.aksgroup.location
account_tier = "Standard"
account_replication_type = "LRS"
}
Terraform Apply:
Once ran the code resources are created, like below.
Azure AKS Diagnostic settings created with Log Analytics settings.
Log Analytics settings- created.