I have built a system to store data collected by IoT devices in CosmosDB and display it in a web application.
I am looking for a simple way to duplicate the resource group including the resources.
The system overview and resources included in the resource group are as follows
System Overview
IoT devices collect data and send to IoT Hub
IoT Hub calls Azure Function to store data in CosmosDB
Display stored data on Web Apps (website)
Resources included in resource group
CosmosDB
Web Apps
IoT Hub
Azure Function
Goal
Replicate the entire resource as a separate resource, including resources in the resource group
Do not use ARM templates
Tried
Import resource group exports using ARM templates
Problem
Importing the exported template as is causes errors due to duplicate content names, empty values in parameters, etc., which makes modifying the ARM template costly.
Functions in Azure Function and resources on the website are not duplicated, so separate duplication work is required.
To begin with, exporting/importing a resource group using an ARM template does not include all the information of the resource group from which it was exported, so it cannot be called a duplication.
Thank you in advance for your cooperation with the above.
Related
what is the best way to read service bus connection string from different resource group in arm template without any hardcoded values?
"connectionString": "[listKeys(resourceId('Microsoft.ServiceBus/namespaces/authorizationRules', parameters('servicebusname'), 'RootManageSharedAccessKey'), '2017-04-01').primaryConnectionString]"
Here the apiversion is hard coded , how can I get it programmatically?
API versions are used to distinguish between different APIs. API versioning is the practice of transparently managing changes to your API. It doesn't matter for prod and non-prod environments at your end as it is specific to the service.
For more details on ARM template for service bus you can refer to document here. For different API version for service bus what is added/removed you can refer to this document.
Goal: Move all the necessary, and only the necessary, function apps and their dependencies from one resource group to another. This resource group they are currently in contains an App Configuration, 2 App Services (Function Apps), their Application Insights, a Storage account, and an App Service Plan.
The resource group I want to move them into already has things like Storage accounts, other Function apps, Logic apps, etc.
I have tried consulting the docs, maybe I am consulting the wrong ones. I am still puzzled about the dependencies Function Apps may have, and why this resource group needed to spin up a Storage account and what azure-webjobs-host and what the files populating within it mean or do.
Q1: How can I move them to the new resource group? I am thinking of using their ARM templates, but am vague on details.
Q2: Do I need to bring their Storage account with them?
Q3: Could I instead integrate them with the Storage account in their new resource group, after I moved them? How?
Q4: Can I simply delete Function Apps' Application Insights if I have no further use for them?
Q5: Any good resources or knowledge you want to share about Logic Apps / Function Apps being dependent on other resource types?
Thank you.
Are my Function Apps dependent on Storage Accounts?
Yes, function app on azure needs you to specified a storage account. This is because triggers other than httptrigger needs storage emulator.(This is built-in feature.)
How can I move them to the new resource group?
If you dont have too many function apps to move, just click 'Move' -> 'Move to another resource group' in the overview of your resource group and then select the function app you want to move.
Do I need to bring their Storage account with them?
No. No matter built-in feature or your function app need to deal with some storage accounts, it just needs you to offer connection string to link to.
Could I instead integrate them with the Storage account in their new
resource group, after I moved them? How?
Yes.(If you are talking about built-in feature.) But you need to change this value:
And the value is getting from this place:
Can I simply delete Function Apps' Application Insights if I have no
further use for them?
Yes, you can. It is just a way to monitor your function app. You can delete or re-create it whenever you want.
Any good resources or knowledge you want to share about Logic Apps /
Function Apps being dependent on other resource types?
One thing I want to remind. Please keep your resources in the same region if you dont have some special requirement. This is not only because of some built-in restrictions, but it can also help you save costs. (For resources in Azure, data transmission between resources located in different regions requires additional bandwidth fees.)
I want to provision resources in Azure using ARM where user can select the required input parameters like Vnet or Function app that are already provisioned.
Just like AWS have parameter types such as AWS::EC2::VPC::Id to list down the VPC available in a region. For AWS references https://aws.amazon.com/blogs/devops/using-the-new-cloudformation-parameter-types/
Is there something similar we can do in azure too.
Similar, yes, the same no... You can author a ui definition file that will allow you to restrict input. For some resources there are controls you can leverage but there's also the capability to write a control that uses semi-custom logic (that could call an Azure API to list skus for example).
This is about the ui defintion:
https://learn.microsoft.com/en-us/azure/azure-resource-manager/managed-applications/create-uidefinition-functions
And you can bundle it with a deployment template like this:
https://preview.portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F100-marketplace-sample%2Fazuredeploy.json/createUIDefinitionUri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2F100-marketplace-sample%2FcreateUiDefinition.json
The feature is in preview right now, so you need to use preview.portal.azure.com instead of portal.azure.com but rollout will finish in a few weeks.
No, this is not possible in ARM Templates. If you would be doing managed applications with arm templates you'd have some of the pickers (very limited set), but with regular arm templates you cant do that. you can create a powershell script that would mimic that for you.
This is (now, maybe not when this question was asked?) possible by using the Microsoft.Solutions.ResourceSelector UI element.
I'd like to clarify the following gap regarding Azure ARM templates:
Let's suggest I have a master template with the following inside:
App Service plan creation
Azure SQL server creation
SQL elastic pool creation (using previously created Azure SQL server)
This template will be used for the initial creation of my cloud infrastructure.
Next, I will add a child (nested or linked) template to my master template.
The child template will contain the AppService Web App+SQL creation:
Web App creation (using App Service Plan defined in master template)
Azure SQL database creation (using Azure SQL server defined in master template)
Adding Azure SQL database to elastic pool (defined in master template)
I will omit several details like the initial creation of the Azure Key Vault and creation and store in this vault required credentials like SQL admin username\password or SSL certificates for my Web App.
So, what I want to have at the end of the template deployment execution is:
first template deployment
Creation of basic infrastructure (app service plan for web apps, SQL server added to elastic pool)
A single instance of an app service (web app+SQL) using previously created app service plan and elastic pool (where my SQL database will be placed)
second template deployment
A single (second) instance of an app service (web app+SQL) will be created using the existing infrastructure
N-template deployment
A single (N-instance) of an app service (web app+SQL) will be deployed <...>
The questions are:
Should I use nested or linked templates? What's the exact difference in my case?
Is my overall solution correct or should I modify it\find another approach?
I've already found the following post saying, for example, I can use resource lock (to prevent deletion) or use incremental mode for deployment (to keep existing resources) however, this doesn't answer my question regarding the entire approach.
Nested\Linked template can be used interchangeably. Its the same thing. One might argue that nested templates are inline templates and linked are actually linked, but it doesnt really matter, both are the same thing (they are implemented in the template slightly differently, but the result is the same). Child templates (and this is really how you want to call those).
As for the actual questions:
Why do you want to use child templates at all? I dont see a use case for those.
I dont see anything wrong with the approach, apart from using child templates just for the sake of using them.
If you want your approach to be "modular" (hence child template usage) you could as well use configuration and implementation separation to achieve the result (DRY method).
I know Microsoft Azure API has a way to pull a data slice using a GET request. The api is here
https://management.azure.com/subscriptions/<SubscriptionID>/resourcegroups/<ResourceGroupName>/providers/Microsoft.DataFactory/datafactories/<DataFactoryName>/tables/<TableName>/sliceruns?start=<StartDateTime>&api-version=<Api-Version>
Problem is I have to manually specify the data factory, data set, and start time, what if I want to pull all logs for a start time for a particular resource group. I know I can do it if I list all data factories and sets and then loop through them. But then I'm calling an http request inside a nested for loop which seems like a really bad/expensive idea. I'm working on a logging web app using Kibana that's why I need all logs.
Unfortunately this is not supported. This is due to the way that API routes are designed for Azure Resource Manager (ARM) services, which ADF is one of. The solution you mentioned, while not ideal, is the best available one.
A bit more: API routes for top-level resources (e.g. a data factory) will always contain a subscription ID and resource group name. Similarly, routes for child resources/APIs (e.g. datasets, slices, etc.) must contain the subscription ID, resource group name and the top-level resource name.
If there was such an API that let you list slices from any data factories in a resource group, this would have to be executed as a "fan-out" query. Since the data factories in the resource group can be in any region, ARM would have to send requests to each of the Data Factory resource providers (RPs) that has a data factory in the resource group, then aggregate and return the results to the caller; this is not supported by any of the Azure services.