Best bicep structure with CI/CD pipeline - azure

Question

I'm not sure that there is a right answer for this and it will vary per scenario but I am curious because there isn't much documentation that I can find for a code first azure bicep infrastructure. Most examples you find show how to make a resource within a resource group, or using a module to define scope and deploy to another resource group, but what if you're trying to do more?
Let's do the following scenario: using 2 subscriptions(1 for prod, 1 for dev & qa) with 20 resource groups each containing multiple difference resources and you want to manage this within a CI/CD pipeline, plus the 3 environments: prod, qa, and dev. How would you go about this? I can think of a few scenarios but don't necessarily, but nothing sticks out as the best way to do it, maybe I'm missing something.
CI/CD portion:
Let's assume:
az account set --subscription(set our sub)
az group create --name --location (create resource group if it doesn't exist)
az deployment group create --name --resource-group --template-file --parameters(read from our files to deploy to a resource group)
You could pass an array of resource groups to loop through to create the resource group if it doesn't exist.
You could have the resource group list in a parameters file that you read from and do the same thing as above.
You could create a step for every resource group and the resources inside of it.(seems excessive?)
Bicep Portion:
Bicep restrictions: to specify scope(a resource group in our scenario) we'd have to have use modules dealing with multiple resource groups or have a step for each resource group and have a main.bicep file for the different resource groups/resources.
You could create a folder structure for each resource group and the resources inside of it with a main.bicep but that would mean you have a lot of extra deploy steps(seems excessive?).
You could have 1 main.bicep file and have a folder structure that uses a lot of modules to specify your scope while reading the resource group, resource variables etc using an environment parameters.json file.
You could create a folder for each environment, have folders with each environment then create each resource group and resources inside of it not using a parameters.json but using params in each file instead since they would be specific for each environment.
1 final issue:
Lastly let's say you want to add a step before the deployment of resources to use bicep what-if to check what resources will be updated or deleted(this is pretty important!). Last I checked there was an issue where what-if does not work for bicep modules so you wouldn't get the luxury of knowing what changes would be made prior to a deployment with the what-if. That is a pretty big safety net you'd be losing, so would you want to scratch the module strategy all together?
What would be the best way to tackle something like this while keeping it readable for average non experts to be able to hop in and work on it? I would lean towards making a folder structure using modules and reading from an environment parameters.json but I'm not convinced that's the best way, especially if what-if isn't fully working for bicep modules.

Answer 1

IMO this does depend a lot on the scenario, topology, permissions, etc. The way I would start thinking about this is that you want an "environment" that will vary a bit between dev/test and prod. That env has multiple resourceGroups and a dedicated subscription for each env.
In this case, I would use a single bicep "project" (e.g. main.bicep with modules) and change the deployment using parameter files (for dev/test vs. prod). The project would lay down everything needed for the environment (think greenfield). The main.bicep file is a subscription scoped deployment that will create the RGs and all the resources needed. Oversimplified example:
targetScope = 'subscription'
param sqlAdminUsername string'
param keyVaultResourceGroup
param keyVaultName string
param keyVaultSecretName string
param location string = deployment().location
resource kv 'Microsoft.KeyVault/vaults#2021-06-01-preview' existing = {
scope: resourceGroup(subscription().subscriptionId, keyVaultResourceGroup)
name: keyVaultName
}
resource sqlResourceGroup 'Microsoft.Resources/resourceGroups#2021-04-01' = {
name: 'shared-sql'
location: location
}
resource webResourceGroup 'Microsoft.Resources/resourceGroups#2021-04-01' = {
name: 'shared-web'
location: location
}
module sqlDeployment 'modules/shared-sql.bicep' = {
scope: resourceGroup(sqlResourceGroup.name)
name: 'sqlDeployment'
params: {
sqlAdminUsername: sqlAdminUsername
sqlAdminPassword: kv.getSecret(keyVaultSecretName)
location: location
}
}
module webDeployment 'modules/shared-web.bicep' = {
scope: resourceGroup(webResourceGroup.name)
name: 'webDeployment'
params: {
location: location
}
}
A single template + modules that creates the RGs, creates a SQL Server (via module) and an app service plan with an admin website (also via module). You can then parameterize whatever you want to for each environment.
re: what-if - what if will skip evaluation of a module if that module has a parameter that is an output of another module. If you don't pass outputs between modules then the module will be evaluated by what-if. The sample above does not pass outputs - often you don't need to do this because the information output was known by the parent (i.e. main.bicep) but sometimes you can't avoid it - ymmv.
Once you have the template designed in such a way, the pipeline is really straightforward - just deploy the template to the desired subscription.
That help?