I am stuck with terraform. I want to create dynamically ECS services with terraform.
I have a configuration like this :
module/cluster/cluster.tf
module/service/service.tf
What I want to do is inject the service name from jenkins into the terraform configuration, so if the service doesnt exist, it creates it (update it if it exists)
I tried to set up different backend s3 remote state but I don't manage to build the whole infrastructure in one terraform apple.
Is there any way to specify dynamically the service configuration so its create them on demand ?
terraform supports to use variable TF_VAR_<variable> to do on fly change.
From environment variables
Terraform will read environment variables in the form of TF_VAR_name to find the value for a variable. For example, the TF_VAR_access_key variable can be set to set the access_key variable.
Note: Environment variables can only populate string-type variables. List and map type variables must be populated via one of the other mechanisms.
For example,
TF_VAR_environment=development terraform plan
https://www.terraform.io/intro/getting-started/variables.html#from-environment-variables
Related
I wanted to create or update aws parameter store with specific environment. now I have a terraform code to create or update aws parameter store for all the environments but I wanted to create or updated aws parameter store for specific environment only using terraform code so how to achieve it using terraform code ? I will pass that specific environment at the run time.
I am expecting when I run terraform code it will create or update for specific environment only.
We have a code repo with our IaC in Terraform. This is in Github, and we're going to pull the code, build it, etc. However, we don't want the values of our variables in Github itself. So this may be a dumb question, but where do we store the values we need for our variables? If my Terraform requires an Azure subscription id, where would I store the subscription id? The vars files won't be in source control. The goal is that we'll be pulling the code into an Azure Devops pipeline so the pipeline will have to know where to go to get the input variable values. I hope that makes sense?
You can store your secrets in Azure Key Vault and retrieve them in Terraform using azurerm_key_vault_secret.
data "azurerm_key_vault_secret" "example" {
name = "secret-sauce"
key_vault_id = data.azurerm_key_vault.existing.id
}
output "secret_value" {
value = data.azurerm_key_vault_secret.example.value
}
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/key_vault_secret
There has to be a source of truth eventually.
You can store your values in the pipeline definitions as variables themselves and pass them into the Terraform configuration.
Usually it's a combination of tfvar files (dependent on target environment) and some variables from the pipeline. If you do have vars in your pipelines though, the pipelines should be in code.
If the variables are sensitive then you need to connect to a secret management tool to get those variables.
If you have many environments, say 20 environments and the infra is all the same with exception of a single ID you could have the same pipeline definition (normally JSON or YAML) and reference it for the 20 pipelines you build, each of those 20 would have that unique value baked in for use at execution. That var is passed through to Terraform as the missing piece.
There are other key-value property tracking systems out there but Git definitely works well for this purpose.
You can use Azure DevOps Secure files (pipelines -> library) for storing your credentials for each environment. You can create a tfvar file for each environment with all your credentials, upload it as a secure file in Azure DevOps and then download it in the pipeline with a DownloadSecureFile#1 task.
We have separate pipelines for our infrastructure and our application deployments. For our infrastructure, we are using terraform and i know that you can use terraform outputs as variables in later tasks within the same pipeline but is it possible to save the output as a variable in azure so that it can be used in a different pipeline.
We are looking to use this for S3 bucket names to use in the application code and for VPC subnet and SG ids in serverless.
Is this possible to save variables in the pipeline?
There is a variable group in Azure DevOps to share static values across pipelines.
In your case, if you want to save the terraform output as a variable in the variable group, you need to do something e.g. call the REST API dynamically to set the variable in the variable group, then you can use it in another pipeline.
You could also refer to this similar issue.
I am trying to setup GitHub Actions for execute a terraform template.
My confusion is - how do I provide *.tfvars file which has aws credentials. (I can't check-in these files).
Whats the best practice to share the variable's values expected by terraform commands like plan or apply where they need aws_access_key and aws_secret_key.
Here is my GitHub project - https://github.com/samtiku/terraform-ec2
Any guidance here...
You don't need to provide all variables through *.tfvars file. Apart from -var-file option, terraform command provides also -var parameter, which you can use for passing secrets.
In general, secrets are passed to scripts through environment variables. CI tools give you an option to define environment variables in project configuration. It's a manual step, because as you have already noticed, secrets cannot be stored in the repository.
I haven't used Github Actions in particular, but after setting environment variables, all you need to do is run terraform with secrets read from them:
$ terraform -var-file=some.tfvars -var "aws-secret=${AWS_SECRET_ENVIRONMENT_VARIABLE}
This way no secrets are ever stored in the repository code. If you'd like to run terraform locally, you'll need first to export these variables in your shell :
$ export AWS_SECRET_ENVIRONMENT_VARIABLE="..."
Although Terraform allows providing credentials to some providers via their configuration arguments for flexibility in complex situations, the recommended way to pass credentials to providers is via some method that is standard for the vendor in question.
For AWS in particular, the main standard mechanisms are either a credentials file or via environment variables. If you configure the action to follow what is described in one of those guides then Terraform's AWS provider will automatically find those credentials and use them in the same way that the AWS CLI does.
It sounds like environment variables will be the easier way to go within GitHub actions, in which case you can just set the necessary environment variables directly and the AWS provider should use them automatically. If you are using the S3 state storage backend then it will also automatically use the standard AWS environment variables.
If your system includes multiple AWS accounts then you may wish to review the Terraform documentation guide Multi-account AWS Architecture for some ideas on how to model that. The summary of what that guide recommends is to have a special account set aside only for your AWS users and their associated credentials, and then configure your other accounts to allow cross-account access via roles, and then you can use a single set of credentials to run Terraform but configure each instance of the AWS provider to assume the appropriate role for whatever account that provider instance should interact with.
I heavily use ECS Attributes on our containerized infrastructure. I couldn't find terraform docs to achieve this. Do I need to execute aws cli commands manually to apply those attributes after creating the infrastructure?
I'd recommend having the ECS agent set the ECS attributes if you need these.
You can do this by adding ECS_INSTANCE_ATTRIBUTES to the /etc/ecs/ecs.config file or passing them as an environment variable directly to the ECS agent on startup.
If you have a "base" ECS AMI (either one you rolled your own or the Amazon Linux AMI) then you probably just want to use user data to dynamically set this from Terraform.
You can use "aws_ecs_service" resource and add attributes. For example:
placement_constraints {
type = "memberOf"
expression = "attribute:ecs.availability-zone in [us-west-2a, us-west-2b]"}