I am writing an ARM template that deploys resources forming a part of an application.
Service Bus, including namespace, queue and authorization rules.
Azure Function triggered by the queue above.
The AzureWebJobsServiceBus app setting needs deploying for this Function trigger to work. Of course, the queue connection string contains the shared access key from the authorization rules created within the same ARM template.
I need to somehow build a connection string and refer to the key just made.
"siteConfig": {
"appSettings": [
{
"name": "AzureWebJobsServiceBus",
"value": "[concat('Endpoint=sb://', parameters('serviceBusNamespaceName'), '.servicebus.windows.net/;SharedAccessKeyName=sender-listener;SharedAccessKey=', listKeys(resourceId('Microsoft.ServiceBus/namespaces/queues/authorizationRules', parameters('serviceBusNamespaceName'), '/http-push/sender-listener'),'2015-05-01-preview').key1)]"
},
At the moment, the above is giving me this error.
Deployment template validation failed: 'The template resource 'mycompany-myapp-mycomponent-functionapp' at line '136' and column '9' is not valid: Unable to evaluate template language function 'resourceId': function requires exactly one multi-segmented argument which must be resource type including resource provider namespace.
Update
The error was caused by my dependsOn though its the same syntax I'm using in my value above so it feels like progress. The correct syntax for my situation is:
"dependsOn": [
"[resourceId('Microsoft.ServiceBus/namespaces/queues/authorizationRules', parameters('serviceBusNamespaceName'), 'queue-name', 'auth-rule-name')]"
],
Still working on the other bit.
Update
Using this as the value for the app setting above causes a new error now.
"[concat('Endpoint=sb://', parameters('serviceBusNamespaceName'), '.servicebus.windows.net/;SharedAccessKeyName=auth-rule-name;SharedAccessKey=', listKeys(resourceId('Microsoft.ServiceBus/namespaces/queues/authorizationRules', parameters('serviceBusNamespaceName'), 'queue-name', 'auth-rule-name'),'2015-05-01-preview').key1)]"
No registered resource provider found for location 'uksouth' and API version '2015-05-01-preview' for type 'namespaces/queues/authorizationrules'. The supported api-versions are '2014-09-01, 2015-08-01, 2017-04-01'. The supported locations are ''.
:(
Update
Switched the version of listKeys to 2017-04-01 and now have this very useful error!
'The language expression property 'key1' doesn't exist, available properties are 'primaryConnectionString, secondaryConnectionString, primaryKey, secondaryKey, keyName'.'
Okay that took about 8 hours to sort. #productive
As per the error here:
'The language expression property 'key1' doesn't exist, available properties are 'primaryConnectionString, secondaryConnectionString, primaryKey, secondaryKey, keyName'.'
The connection string is available which sounds like it would negate the need for me to concat my own one, alas a Service Bus Queue trigger does not like the queue name on the end, so I must compose it manually like so.
"value": "[concat('Endpoint=sb://', parameters('serviceBusNamespaceName'), '.servicebus.windows.net/;SharedAccessKeyName=auth-rule-name;SharedAccessKey=', listKeys(resourceId('Microsoft.ServiceBus/namespaces/queues/authorizationRules', parameters('serviceBusNamespaceName'), 'queue-name', 'auth-rule-name'),'2017-04-01').primaryKey)]"
Related
Trying to migrate event monitors according to migration steps
resource "datadog_monitor" "cache_event" {
name = "${var.cache_replication_group} :: Event"
type = "event-v2 alert"
message = join(" ", ["Cache event occured in ${var.cache_replication_group}.", join(" ", var.recipients.general)])
query = "events(\"source:elasticache tags:(replication_group:${var.cache_replication_group}) NOT(Finished)\").rollup(\"count\").last(\"15m\") > 0"
require_full_window = false
tags = concat(var.tags, ["cache:${var.cache_replication_group}", "monitor:stability"])
}
But when trying to apply changes Terraform returns the error Error: error creating monitor: event-v2 alert is not a valid MonitorType, but according to their documentation event-v2 alert is a valid type. DataDog provider version at the moment is 3.10.0, hence it should support this type
My bad. After changing the provider version, I didn't reinitialise the state file.
Edit:
Before monitor migration, I used an older DataDog provider version. After adjusting query to be complient with a new event monitor format, I bumped DataDog provider version to the one that supports new v2 monitors, but instead of deleting .terraform folder and triggering init Terraform command, I just went straight with apply. Instead I should have initialized it first (init).
We have an Azure function v3 running Node, consumption plan, with an input trigger connected to a cosmos database. The function.json looks like this:
{
"disabled": false,
"bindings": [
{
"type": "cosmosDBTrigger",
"name": "productDocuments",
"collectionName": "products",
"direction": "in",
"connectionStringSetting": "DB_CONNECTION_STRING",
"databaseName": "product-management",
"createLeaseCollectionIfNotExists": true,
"maxItemsPerInvocation": 1
},
{
"name": "productDocument",
"type": "cosmosDB",
"databaseName": "product-management",
"collectionName": "products",
"createIfNotExists": true,
"connectionStringSetting": "_DB_CONNECTION_STRING",
"direction": "out"
}
],
"scriptFile": "dist/nameOfFunction.js"
}
But this trigger is working really, really slow and unreliable. If we add an item to the DB it sometimes triggers straight away, sometimes it seems to take hours and sometimes not at all. I am manually monitoring the cosmos db so I can see that items are added.
I am looking at this page, and most of the time nothing happens. I don't know how else to debug this
Should it really take hours for an invocation to show up here? Or is it the trigger that's unreliable?
General guidance is in this doc: https://learn.microsoft.com/azure/cosmos-db/troubleshoot-changefeed-functions#my-changes-take-too-long-to-be-received
What happens on Consumption Plan is that, after a period of inactivity, instances are deprovisioned. When a new instance is provisioned, it hits a cold start.
The key part here is that, when your instances are deprovisioned, they are not checking the Change Feed for events, so how does Functions know when to "wake them up"?
There is a periodic check done by an external component that checks to see if there are new changes, if there are new, then it would provision new instances of your Function to start consuming them.
This external component in your case, could be having an issue or delays in this checks.
If you have no Function logs for an hour even though you are making changes to the monitored collection, I would try to contact Azure Support to understand why is your Function not "waking up".
One of the known issues I've heard about was related to where the Cosmos DB Connection Strings were stored. Apparently this component at some point (maybe it's already fixed) had a problem where it could not access the Connection String if it was saved in "Connection Strings" section of the Function configuration, but was looking for it only on the "App Settings". In this cases, it could not wake up the Function and the Function only woke up if someone opened it on the Azure Portal. My recommendation would be to check where are you storing your connection string and see if you can move it to "App Settings" and see how it behaves.
Our problem with this was that we had two separate functions that both had a CosmosDBTrigger on the same collectionm but used the same lease, and apparently you can't do that. So it was solved by setting two separate leases (we used the leaseCollectionPrefix in the function.json.)
I'm getting an error when I try to deploy an AKS cluster using an ARM template, if the vnetSubnetId in the agentPoolProfiles property is a reference. I've used this exact template before without problems (on October 4th) but now I'm seeing an error with multiple different clusters, and when I do it either through a VSTS pipeline, or manually using PowerShell.
The property is set up like this:
"agentPoolProfiles": [
{
"name": "agentpool",
"count": "[parameters('agentCount')]",
"vmSize": "[parameters('agentVMSize')]",
"osType": "Linux",
"dnsPrefix": "[variables('agentsEndpointDNSNamePrefix')]",
"osDiskSizeGB": "[parameters('agentOsDiskSizeGB')]",
"vnetSubnetID": "[reference(concat('Microsoft.Network/virtualNetworks/', variables('vnetName'))).subnets[0].id]"
}
]
The variable 'vnetName' is based on an input parameter I use for the cluster name, and the vnet itself 100% exists, and is deployed as part of the same template.
If I try to deploy a new cluster I get the following error:
Message: {
"code": "InvalidParameter",
"message": "The value of parameter agentPoolProfile.vnetSubnetID is invalid.",
"target": "agentPoolProfile.vnetSubnetID"
}
If I try to re-deploy a cluster, with no changes to the template or input parameters since it last worked, I get the following error:
Message: {
"code": "PropertyChangeNotAllowed",
"message": "Changing property 'agentPoolProfile.vnetSubnetID' is not allowed.",
"target": "agentPoolProfile.vnetSubnetID"
}
Has something changed that means I can no longer get the vnet ID at runtime? Does it need to be passed in as a parameter now? If something has changed, is there anywhere I can find out the details?
Edit: Just to clarify, for re-deploying a cluster, I have checked and there are no new subnets, and I'm seeing the same behavior on 3 different clusters with different VNets.
Switching from reference() to resourceId() did fix the problem so has been marked as the answer, but I'm still no clearer on why reference() stopped working, will update that here as well if I figure it out.
I think what happened is subnets[0].id returns wrong (DIFFERENT) subnetId. and this is what the error points out. You cannot change the subnetId after deploying the cluster.
Probably somebody created a new subnet in the vnet. But I'd say that overall the approach is flawed. you should build the resourceId() function or just pass it as a parameter
We have started to use the Queue binding in our Azure functions for longer-running tasks such as sending bulk e-mails and "clean-up" tasks for CosmosDB. We develop locally with the Functions emulator then commit into VSTS/Azure DevOps which then auto-deploys into our Function App.
It seems as though pretty quickly we're going to have multiple Functions (two local emulators and one cloud function) all listening to the same queue. We tried disabling locally and renaming locally, but these all seem like awkward workarounds that require too much manual work and have the possibility to push the wrong queue name forward into VSTS.
How do we configure the queue name in the function.json to read an environment variable? The connection setting in the binding takes the name of an environment variable, but the queue setting wants a string.
{
"disabled": false,
"bindings": [
{
"name": "myQueueItem",
"type": "queueTrigger",
"direction": "in",
"queueName": "emailer",
"connection": "STORAGE_CONNECTION_STRING"
}
]
}
Just wrap variable name with % and function can read its value from Application settings on portal and Values in local.settings.json locally.
"queueName": "%myqueue%"
connection property of triggers and bindings is a special case and automatically resolves values as app settings, without percent signs.
See Binding expressions - app settings.
I am trying to declare the following Terraform provider:
provider "mysql" {
endpoint = "${aws_db_instance.main.endpoint}:3306"
username = "root"
password = "root"
}
I get the following error:
Error refreshing state: 1 error(s) occurred:
* dial tcp: lookup ${aws_db_instance.main.endpoint}: invalid domain name
It seems that Terraform is not performing interpolation on my endpoint string, yet I don't see anything in the documentation about this -- what gives?
Yes, it does. There's an example in the docs at https://www.terraform.io/docs/providers/mysql/
# Configure the MySQL provider based on the outcome of
# creating the aws_db_instance.
provider "mysql" {
endpoint = "${aws_db_instance.default.endpoint}"
username = "${aws_db_instance.default.username}"
password = "${aws_db_instance.default.password}"
}
I ran into a similar set of error messages ("connect failed," "invalid domain lookup") and looked into this a bit. I hope this helps you or someone else working across cloud and database providers in Terraform.
This seems to come down to the MySQL provider attempting to establish a database connection as soon as it's initialized, which could be a problem if you're trying to build a database server and configure the database / grants on it as part of the same Terraform run. Providers get initialized based on Terraform finding a resource owned by that provider in your Terraform code, and since this connection attempt happens when the provider gets initialized, you can't work around this with -target=<SPECIFIC RESOURCE>.
The workarounds I can think of would be to have a codebase for setting up the database server and a different codebase for setting up the database grants and suchlike ... or to have Terraform kick off a script that does that work for you (with dynamic parameters, of course!). Either way, you're effectively removing mysql_* resources from your initial Terraform run and that's what fixes this.
There are a couple of code changes that probably need to happen here - the Terraform MySQL provider would need to delay connecting to the database until Terraform tells it to run an operation on a resource, and it may be necessary to look at how Terraform handles dependencies across providers. I tried hacking in deferred connection logic just for the mysql_database resource to see if that solved all my problems and Terraform still complained about a dependency loop in the graph.
You can track the MySQL provider issue here:
https://github.com/terraform-providers/terraform-provider-mysql/issues/2
And the comments from before providers were split into their own releasable codebases:
https://github.com/hashicorp/terraform/issues/5687