In a CI/CD context, I would like to define provider versions outside my terraform configuration using TF_VAR_ environment variables.
I'm trying to use input variable to set the version of helm provider in versions.tf (terraform 0.12) but it seems not allowed :
Error: Invalid provider_requirements syntax
on versions.tf line 3, in terraform:
3: helm = "${var.helm_version}"
provider_requirements entries must be strings or objects.
Error: Variables not allowed
on versions.tf line 3, in terraform:
3: helm = "${var.helm_version}"
Variables may not be used here.
How can I configure this ?
If it's not possible, how I can manage the terraform provider version outside my configuration ?
Cannot be done. I wish it could be done. terraform init resolves and downloads the providers, you won't have access to variables at that point.
Each terraform block can contain a number of settings related to
Terraform's behavior. Within a terraform block, only constant values
can be used; arguments may not refer to named objects such as
resources, input variables, etc, and may not use any of the Terraform
language built-in functions.
https://www.terraform.io/docs/configuration/terraform.html
As #thekbb says, it's not possible to get access to version variable during terraform init at least in 0.12.20. However, I've below workaround to manage providers outside your configuration.
You could use alias with provider configuration to achieve this. Let's assume you want 1.3.0 version of helm. Rather than passing it as a var, you could define it statically with an alias like below.
provider "helm" {
alias = "helm-stable"
version = "1.3.0" (the version you pass via TF_VAR_helm_version)
kubernetes {
host = "https://104.196.242.174"
username = "ClusterMaster"
password = "MindTheGap"
client_certificate = file("~/.kube/client-cert.pem")
client_key = file("~/.kube/client-key.pem")
cluster_ca_certificate = file("~/.kube/cluster-ca-cert.pem")
}
}
Then, in your resource or data providers, you could point to a particular provider like below::
data "some_ds" "example" {
name = "dummy"
provider = helm.helm-stable
}
For more details, refer to the below links::
providers
allow variable in provider field
Related
As an example:
I am deploying Terraform module in us-east-1 which will build the infrastructure + cloudfront distribution. Now the same module will be deployed in us-west-1 as part of the Disaster Recovery region. Now since cloudfront it is a global service, how I can point in Terraform module which will be deployed in us-west-1 to use the existing cloudfront?
If these are two modules within the same configuration (i.e. both modules are applied with the same terraform apply command), then you simply pass the aws_cloudfront_distribution resource out of the module where it's created as an output, and pass it into the other module as an input parameter. E.g.:
module1/main.tf
resource "aws_cloudfront_distribution" "mydist" {
...
}
output "mydist" {
value = aws_cloudfront_distribution.mydist
}
module2/main.tf
variable "mydist" {}
main.tf
module "mod1" {
...
}
module "mod2" {
...
mydist = mod1.mydist
}
And now you can access the CloudFront distribution resource from within module2 by using var.mydist.
If these are two modules in entirely separate Terraform configurations, you can either:
Use the aws_cloudfront_distribution data source to get the details about a distribution that was created in a separate configuration
output the distribution from the configuration where it's created, then use the terraform_remote_state data source to retrieve the output from the remote state file.
I am going through a terraform guide, where the author is spinning up a docker setup using the docker_image and docker_container resources.
In the sample code the main.tf file includes both the required_providers and the provider blocks, as follows:
terraform {
required_providers {
docker = {
source = "kreuzwerker/docker"
}
}
}
provider "docker" {}
Why are they both needed?
Shouldn't terraform be able to understand the need for a docker provider, only by this line?
provider "docker" {}
When considering Terraform providers there are two related notions to think about: the provider itself, and a configuration for the provider.
As an analogy, the provider kreuzwerker/docker here is a bit like a class you're importing from another library, giving it the local name docker. I'll use a pseudo-JavaScript syntax just to make this a bit more concrete:
var docker = require("kreuzwerker/docker");
However, all we have here so far is the class itself. In order to use it we need to create an instance of it, which in Terraform's vernacular is called a "configuration". Again, using pseudo-JavaScript syntax:
var dockerInstance = new docker({});
Terraform's syntax here is decidedly less explicit than this pseudo-JavaScript form, but we can make the distinction more visible by adding a second instance of the provider to the configuration, which in Terraform we do by assigning it a configuration "alias":
provider "docker" {
alias = "example"
host = "ssh://user#remote-host:22"
}
This is like creating a second instance of the provider "class" in our pseudo-JavaScript example:
var dockerInstance2 = new docker({
host: 'ssh://user#remote-host:22'
});
Another variant that shows the distinction is when a module inherits a provider configuration from its calling module. In that case, it's as if the calling module were implicitly passing the provider configuration (instance) into the module, but the child module still needs to import the provider "class" so Terraform can see that we're talking about kreuzwerker/docker as opposed to any other provider that might have the name "docker".
Terraform has some automatic "magic" behaviors that try to make simpler cases implicit, but unfortunately that comes at the cost of making it harder to understand what's going on when things get more complicated. Providers and provider configurations are a particularly hard example of this, because providers have been in the Terraform language for a long time and the current incarnation of the language is trying to stay broadly backward-compatible with the simple uses while still allowing for the newer features like having third-party providers installable from multiple namespaces.
The particularly confusing assumption here is that if you don't declare a particular provider Terraform will create an implicit required_providers declaration assuming that you mean a provider in the hashicorp/ namespace, which makes it seem as though required_providers is only for third-party providers. In fact though, that is largely a backward-compatibility mechanism and so I'd suggest always writing out the required_providers entries, even for the providers in the hashicorp/ namespace, so that less-experienced readers don't need to know about this special backward-compatibility behavior. In your case though, the provider you're using is in a third-party namespace anyway and so the required_providers entry is mandatory.
The source needs to be provided since this isn't one of the "official" HashiCorp providers. There could be multiple providers with the name "docker" in the provider registry, so providing the source is needed in order to tell Terraform exactly which provider to download.
Description:
I am trying to use an Elasticsearch provider for Terraform. Since there is no official one from Elastic or from Hashicorp I am trying to use a community one "https://registry.terraform.io/providers/phillbaker/elasticsearch/latest".
Terraform version: Terraform v0.14.4
Code:
I tried to put everything in 1 .tf file. I also tried to create a separate module for the resources like Hashicorp recommends. Both methods generate the same error message.
terraform {
required_providers {
elk = {
source = "phillbaker/elasticsearch"
version = "1.5.1"
}
}
}
provider "elk" {
url = "https://<my_elk_server>"
}
resource "elasticsearch_index" "index" {
name = var.elasticsearch_index_name
}
Problem:
terraform init isn't able to find the appropriate provider in the Terraform Registry for some reason.
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/elasticsearch...
- Finding phillbaker/elasticsearch versions matching "1.5.1"...
- Installing phillbaker/elasticsearch v1.5.1...
- Installed phillbaker/elasticsearch v1.5.1 (self-signed, key ID 02AD42CD82B6A957)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
Error: Failed to query available provider packages
https://www.terraform.io/docs/plugins/signing.html
Could not retrieve the list of available versions for provider
hashicorp/elasticsearch: provider registry registry.terraform.io does not have
a provider named registry.terraform.io/hashicorp/elasticsearch
If you have just upgraded directly from Terraform v0.12 to Terraform v0.14
then please upgrade to Terraform v0.13 first and follow the upgrade guide for
that release, which might help you address this problem.
No tfstate files are being generated.
How do I use third party providers from the Terraform Registry ?
In your required_providers block you've told Terraform that you intend to refer to this provider as "elk" within this module:
elk = {
source = "phillbaker/elasticsearch"
version = "1.5.1"
}
Typically you'd set the local name of the provider to be the same as the "type" portion of the provider source address, like this:
elasticsearch = {
source = "phillbaker/elasticsearch"
version = "1.5.1"
}
If you change the local name in this way then use references to elasticsearch elsewhere in the module should then refer to the community provider as you intended.
Note that means you'll also need to change the provider block so it has a matching local name:
provider "elasticsearch" {
url = "https://<my_elk_server>"
}
A different approach here would be to continue to use elk as the name and then change the rest of the configuration to properly refer to that non-default name. I don't recommend doing this, because typically I'd expect the local name to only mismatch the type in the unusual case where your module depends on two providers with the same type name, but I'm mentioning this in the hope that it helps to understand how the Terraform language infers provider dependencies when not given explicitly:
terraform {
required_providers {
elk = {
source = "phillbaker/elasticsearch"
version = "1.5.1"
}
}
}
# "elk" here is matched with the local names in the
# required_providers block, so this will work.
provider "elk" {
url = "https://<my_elk_server>"
}
# This "elasticsearch_" prefix causes Terraform to look
# for a provider with the local name "elasticsearch"
# by default...
resource "elasticsearch_index" "index" {
# ...so if you've given the provider a different local
# name then you need to associate the resource with
# the provider configuration explicitly:
provider = elk
name = var.elasticsearch_index_name
}
I expect most Terraform users would find the above approach surprising, so in the interests of using familiar Terraform idiom I'd suggest instead following my first suggestion of renaming the local name to elasticsearch, which will then allow the automatic resource-to-provider association to work.
So, after testing it seems putting the whole code in the same .tf file does the job.
terraform {
required_providers {
elasticsearch = {
source = "phillbaker/elasticsearch"
version = "1.5.1"
}
}
}
provider "elasticsearch" {
url = "http://127.0.0.1:9200"
}
resource "elasticsearch_index" "index" {
name = var.index_name
}
If you want to create a separate module for it you can just source it from another module:
module "elastic" {
index_name = var.index_name
source = "./modules/elastic"
}
Check Martin's answer for more information.
Ok, so I have three .tf-files: main.tf where I state azure as provider, resources.tf where all the my resources are claimed, and variables.tf.
I use variables.tf to store keys used by resources.tf.
However, I want to use variables stored in my variable file to fill in the fields in the backend scope like this:
main.tf:
provider "azurerm" {
version = "=1.5.0"
}
terraform {
backend "azurerm" {
storage_account_name = "${var.sa_name}"
container_name = "${var.c_name}"
key = "${var.key}"
access_key = "${var.access_key}"
}
}
Variables stored in variables.tf like this:
variable "sa_name" {
default = "myStorageAccount"
}
variable "c_name" {
default = "tfstate"
}
variable "key" {
default = "codelab.microsoft.tfstate"
}
variable "access_key" {
default = "weoghwoep489ug40gu ... "
}
I got this when running terraform init:
terraform.backend: configuration cannot contain interpolations
The backend configuration is loaded by Terraform extremely early,
before the core of Terraform can be initialized. This is necessary
because the backend dictates the behavior of that core. The core is
what handles interpolation processing. Because of this, interpolations
cannot be used in backend configuration.
If you'd like to parameterize backend configuration, we recommend
using partial configuration with the "-backend-config" flag to
"terraform init".
Is there a way of solving this? I really want all my keys/secrets in the same file... and not one key in the main which I preferably want to push to git.
Terraform doesn't care much about filenames: it just loads all .tf files in the current directory and processes them. Names like main.tf, variables.tf, and outputs.tf are useful conventions to make it easier for developers to navigate the code, but they won't have much impact on Terraform's behavior.
The reason you're seeing the error is that you're trying to use variables in a backend configuration. Unfortunately, Terraform does not allow any interpolation (any ${...}) in backends. Quoting from the documentation:
Only one backend may be specified and the configuration may not contain interpolations. Terraform will validate this.
So, you have to either hard-code all the values in your backend, or provide a partial configuration and fill in the rest of the configuration via CLI params using an outside tool (e.g., Terragrunt).
There are some important limitations on backend configuration:
A configuration can only provide one backend block.
A backend block cannot refer to named values (like input variables, locals, or data source attributes).
Terraform backends configurations one can see at below link:
https://www.terraform.io/docs/configuration/backend.html
How can I get Terraform 0.10.1 to support two different providers without having to run 'terraform init' every time for each provider?
I am trying to use Terraform to
1) Provision an API server with the 'DigitalOcean' provider
2) Subsequently use the 'Docker' provider to spin up my containers
Any suggestions? Do I need to write an orchestrating script that wraps Terraform?
Terraform's current design struggles with creating "multi-layer" architectures in a single configuration, due to the need to pass dynamic settings from one provider to another:
resource "digitalocean_droplet" "example" {
# (settings for a machine running docker)
}
provider "docker" {
host = "tcp://${digitalocean_droplet.example.ipv4_address_private}:2376/"
}
As you saw in the documentation, passing dynamic values into provider configuration doesn't fully work. It does actually partially work if you use it with care, so one way to get this done is to use a config like the above and then solve the "chicken-and-egg" problem by forcing Terraform to create the droplet first:
$ terraform plan -out=tfplan -target=digitalocean_droplet.example
The above will create a plan that only deals with the droplet and any of its dependencies, ignoring the docker resources. Once the Docker droplet is up and running, you can then re-run Terraform as normal to complete the setup, which should then work as expected because the Droplet's ipv4_address_private attribute will then be known. As long as the droplet is never replaced, Terraform can be used as normal after this.
Using -target is fiddly, and so the current recommendation is to split such systems up into multiple configurations, with one for each conceptual "layer". This does, however, require initializing two separate working directories, which you indicated in your question that you didn't want to do. This -target trick allows you to get it done within a single configuration, at the expense of an unconventional workflow to get it initially bootstrapped.
Maybe you can use a provider instance within your resources/module to set up various resources with various providers.
https://www.terraform.io/docs/configuration/providers.html#multiple-provider-instances
The doc talks about multiple instances of same provider but I believe the same should be doable with distinct providers as well.
A little bit late...
Well, got the same Problem. My workaround is to create modules.
First you need a module for your docker Provider with an ip variable:
# File: ./docker/main.tf
variable "ip" {}
provider "docker" {
host = "tcp://${var.ip}:2375/"
}
resource "docker_container" "www" {
provider = "docker"
name = "www"
}
Next one is to load that modul in your root configuration:
# File: .main.tf
module "docker01" {
source = "./docker"
ip = "192.169.10.12"
}
module "docker02" {
source = "./docker"
ip = "192.169.10.12"
}
True, you will create on every node the same container, but in my case that's what i wanted. I currently haven't found a way to configure the hosts with an individual configuration. Maybe nested modules, but that didn't work in the first tries.