I have a wonderful terraform plan that perfectly describes my infrastructure in Google Cloud Platform, however, I have a problem: since my repository isn't perfectly private, some steps of my plan are encrypted and must be decrypted using Google Key Management Service.
This means my plan must be broken down into two terraform phases:
Setup the Google Cloud Project and create a Key Ring and Key (after this, I encrypt secrets and put them in a variables.tf file)
Apply the entire plan.
Does Terraform support a way to break down my plan into phases? How should I go around implementing this?
Though terraform enables us to automate the resources creation, some preliminary steps need to be done manually, like account creation, billing setup, etc. Similarly for Google cloud setup, the project needs to be created prior running terrform scripts since terraform google provider requires the project details.
The project creation and terraform variables for the keys (as environment variables) can be generated through shell scripts. Then the shell script and the terraform scripts can be sequenced in execution using a make file.
The below link might be helpful for you to create GCP project through shell scripts.
https://medium.com/google-cloud/how-to-automate-project-creation-using-gcloud-4e71d9a70047
Related
We are looking to "reset" a resource group, deleting everything but the necessary infrastructure in it. The problem is we are still immature in our IAC practices and a lot of resources are deployed via the portal. My initial thought is to have the only necessary infra defined in an ARM template and running it in complete mode when we want to reset it. Does Terraform have a complete mode feature? From what I understand, Terraform will only manage stuff in state. Since we wont really respecting the state after initial deployment, the resources deployed via the portal wont be destroyed on a TF destroy. Any thoughts? Thanks!
Does Terraform have a complete mode feature?
AFAIK, No , Terraform doesn't have complete Mode like ARM template has.
From what I understand, Terraform will only manage stuff in state.
Since we wont really respecting the state after initial deployment,
the resources deployed via the portal wont be destroyed on a TF
destroy.
Yes , You are correct the Terraform will only manage the the
resources which are in state file only .
So, by default Terraform will only store the resources deployed through it in the state file but if you want to create some resources from the portal , then also you can use the import resources feature of terraform. Using which Terraform will be able to manage the resources created from Terraform and Portal as well.
Reference:
Import - Terraform by HashiCorp
No, Terraform does not have such a feature.
There is a feature request which mainly covers the "reporting" aspect, but also would allow acting upon it.
You might be able to build something around the import feature of Terraform, as suggested here. However, this would require some effort.
You could also use Terraform to deploy an ARM template in complete mode, but then you might loose most of why you wanted to use Terraform in the first place.
In order to store state remotely terraform has a config block that allows you point to a cloud resource. This article shows how to do it in Azure.
In a nutshell, it instructs the reader to use some AZ powershell commands to provision the resources.
Isn't the idea of Terraform to do the opposite of manual creation of resources? If this is true, what is a better way to do this? If its not, and I've misunderstood something, please clarify what it is that I missed.
Background: working on a greenfield application with nothing provisioned other than a git repo to store the tf files. Attempting to provision an azure static web site.
Its normal. To use a remote backed on azurem you need to have Azure Blob container. So it must exist first, before you can use it for a remote backed.
You don't have to use any powershell commands to create the container. Use other TF code, or Azure console, SDK or whatever you want to create it.
Terraform scaffold for Azure:
https://github.com/whiteducksoftware/terraform-scaffold-for-azure
I am asked to create Terraform Scripts for our Azure Infrastructure setup. For starters I am creating Terraform Scripts for AppService. For me I am confused with whole IaC paradigm and will like to know how it is done in an enterprise environment
1) Do we need to create seperate Terraform Scripts for each App Service. Or do we need to create one script and set the values as run time variables?
2) Do we need to have Terraform pipelines as separate or it should run along with Application deployment pipelines? That is,each time before Deployment of application, do we need to check for configuration drift via Terraform?
Thanks in advance !
This question is basically opinion based. As per my experience,
you can have a single script file and have values injected dynamically
you can have terraform along with your deployment pipeline
Since you have mentioned that you are confused with IaC paradim, this article here might give more clarity. https://thorsten-hans.com/terraform-the-definitive-guide-for-azure-enthusiasts#the-terraform-lifecycle
hope this helps.
I'm working on an asp.net core project and I'm trying to figure out how to keep my source and my pipelines 100% secret free.
I've got a VM running the azure agent and an azure dev ops pipelines for build and release.
If i delete the site on the VM, the release pipeline will auto-magically recreate it for me and deploy the latest build.
Super cool.
Now I read up on best practices for configuring a .Net core app and I found this article: https://www.humankode.com/asp-net-core/asp-net-core-configuration-best-practices-for-keeping-secrets-out-of-source-control
So its a bad idea to keep secrets in code, that makes perfect sense.
But if i apply the same security principals to Yaml, then surely I shouldn't place secrets in my pipelines either.
But I need the pipelines to be able to just recreate the site from scratch and it should just work. Somehow the site needs to know where its default sql connection is, or it needs to have a key to the azure app config service. I shouldn't have to log onto the VM and create an appsettings.json manually after every release!
So whatever the site needs to operate needs to be included in the pipeline, therefore some artifact, or included in the code.
I've googled for days, but I can't seem to find any info on how to fully automate this.
I've considered creating a custom configuration provider that reads from the actual VM registry, but that feels wrong too.
I basically need a config option that is NOT hosted in the site itself. So i set it up once on the VM and never again.
The approach that Lex Li lists in the comments is the Microsoft recommended way of securing "secrets" in pipelines.
Ben Smith's answer in my opinion is just as good, maybe slightly less secure.
I use this approach in our organization. All of our release pipelines do the final configuration transformation with the appropriate settings based on the environment they are being deployed to.
i.e db connections are transformed at the dev, test and UAT and production deployment stages.
I keep the relevant secrets in the pipeline variables as protected secrets. I do this for 2 reasons:
Only a select number of trusted personnel have access to the release pipeline definitions.
Even if someone does have access to those definitions - you cannot see a secured variable. Even you you "undo the padlock" on the variable tab - you cannot see what the setting is.
Our actual secrets are then stored in our enterprise secret vault.
Using the Azure Key Vault is definitely a good approach. However we already have a centralized place to keep our stuff; I don't want it in 3 spots.
I would be remiss to not include Variable Groups as part of the pipeline process. Same concept as the build / release variables - the difference is you can now share them in one spot.
These are all opinions of course. This is just one way of doing this; which I feel is a pretty good balance of security and flexibility.
In addition to the suggestions in your questions comments, you can also store secrets in the pipeline "Variables" section.
In here you can add variables and then mark them as secret by selecting "Keep this value secret". Once you've saved a secret its value is then obfuscated i.e. you can make use of it but you can no long see its original value within Azure Devops (which admittedly can be rather frustrating if you want to revisit the variable to check it!).
You can then reference the secret variable in your pipeline YAML using the syntax:
$(variable-name)
So this approach keeps secrets safe within Azure Devops until they need to be resolved by the pipeline YAML script.
I want to run powershell scripts to create users and usergroups in Azure AD . Is it possible to call ps scripts in ARM Template?
Update:
Now it is possible to do some operations with a new feature (still in Preview)
https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-script-template
Use deployment scripts in templates (Preview)
Learn how to use deployment scripts in Azure Resource templates. With
a new resource type called Microsoft.Resources/deploymentScripts,
users can execute deployment scripts in template deployments and
review execution results. These scripts can be used for performing
custom steps such as:
add users to a directory
create an app registration
perform data plane operations, for example, copy blobs or seed database
look up and validate a license key
create a self-signed certificate
create an object in Azure AD
look up IP Address blocks from custom system
The benefits of deployment script:
Easy to code, use, and debug. You can develop deployment scripts in your favorite development environments. The scripts can be embedded in
templates or in external script files.
You can specify the script language and platform. Currently, only Azure PowerShell deployment scripts on the Linux environment are
supported.
Allow specifying the identities that are used to execute the scripts. Currently, only * * Azure user-assigned managed identity is
supported.
Allow passing command-line arguments to the script.
Can specify script outputs and pass them back to the deployment.
Remember that ARM templates should be idempotent. You should write code that can be executed multiple times, even on environments where your code was already executed.
For example, if you are going to configure a setting, or create a resource, your powershell should probably check if the resource is already in place and properly configured.
No, Azure ARM could not execute scripts directly. Executing scripts need host, Azure template does not provide such host.
One solution, you could select Azure Custom Script Extension.
The Custom Script Extension downloads and executes scripts on Azure virtual machines.