In a base.tf file I have:
data "consul_keys" "project_emails"{
datacenter = "mine1"
key {
name = "notification_list"
path = "project/notification_email_list"
}
}
I would like to use these consul variables in my python code.
The way I'm thinking about this is by outputting this to a file. (so not just another terraform file using the "${project_emails.notification_list.construct}" with either version 11 or 12.).
How would I save all these keys to a file to access the keys?
The general mechanism for exporting data from a Terraform configuration is Output Values.
You can define an output value that passes out the value read from Consul like this:
output "project_emails" {
value = data.consul_keys.project_emails.var.notification_list
}
After you've run terraform apply to perform the operations in your configuration, you can use the terraform output command to retrieve the output values from the root module. Because you want to read it from another program, you'll probably want to retrieve the outputs in JSON format:
terraform output -json
You can either arrange for your program to run that command itself, or redirect the output from that command to a static file on disk first and then have your program read that file.
The above assumes that the Python code you mention will run as part of your provisioning process on the same machine where you run Terraform. If instead you are asking about access to those settings from software running on a virtual machine provisioned by Terraform, you could use the mechanism provided by your cloud platform to pass user data to your instance. The details of that vary by provider.
For long-lived applications that consume data from Consul, a more common solution is to run consul-template on your virtual server and have it access Consul directly. An advantage of this approach is that if the data changes in Consul then consul-template can recognize that and update the template file immediately, restarting your program if necessary. Terraform can only read from Consul at the time you run terraform plan or terraform apply, so it cannot react automatically to changes like consul-template can.
Related
Using terraform and AWS I've created a Postgres RDS DB within a VPC. During creation, the superuser is created, but it seems like a bad idea to use this user inside my app to run queries.
I'd like to create an additional access-limited DB user during the terraform apply phase after the DB has been created. The only solutions I've found expect the DB to be accessible outside the VPC. For security, it also seems like a bad idea to make the DB accessible outside the VPC.
The first thing that comes to mind is to have terraform launch an EC2 instance that creates the user as part of the userdata script and then promptly terminates. This seems like a pretty heavy solution and terraform would create the instance each time terraform apply is run unless the instance is not terminated at the end of the script.
Another option is to pass the superuser and limited user credentials to the server that runs migrations and have that server create the user as necessary. This, however, would mean that the server would have access to the superuser and could do some nefarious things if someone got access to it.
Are there other common patterns for solving this? Do people just use the superuser for everything or open the DB to the outside world?
Consider I am having multiple DAG in Airflow.
Every task in the DAG tries to execute presto queries, I just override the get_conn() method in the airflow. On each call of the get_conn() method, it gets credentials from the AWS secrets manager.
The maximum request to the secrets manager is 5000. In this case, I need to cache my credentials somewhere(Should not use Connections/Variables, DB, S3), so that they can be used across all tasks without calling the secrets manager.
My question here is,
Is there any way we can handle those credentials in our code with Python/Airflow by calling get_conn() at once?
You could write your own custom secret backend https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html#roll-your-own-secrets-backend extending the AWS one and overriding the methods to read the credentials and store it somewhere (for example in local file or a DB as caching mechanism).
If you are using local filesystem however, you have to be aware that your caching reuse/efficiency will depends on how your tasks are run. If you are running a CeleryExecutor, then such local file will be available for all processes running on the same worker (but not to celery processes running on other workers). If you are running KubernetesExecutor, each task runs in it's own Pod, so you'd have to mount/map some persistent or temporary storage to inside your PODS to reuse it. Plus you have to somehow solve the problem of concurrent processes writing there and refreshing such cache periodically or when it changes.
Also you have to be extra careful as it brings some issues regarding the security as such local cache will be available to all DAGs and python code run in tasks even if they are not using the connection (so for example Airflow 2.1+ built-in automated secret masking will not work in this case and you have to be careful not to print the credentials to logs.
Sorry for the noob question... I'm trying to figure out a way to have shared resources between my tf scripts, but I can't find anything, so probably I'm looking for the wrong keywords...
Let's say I have 3 scripts:
base/base.tf
one/one.tf
two/two.tf
base creates an aws vpc and a network load balancer
one and two are two ecs fargate services. they create the task definition and add the mappind to the network load balancer.
My goal is to have something to keep track of the mapped port in the load balancer and read it and update from one and two.
Something like
base sets last_port to 14000
one reads last_port, increases by 1 and updates the value
two reads last_port, increases by 1 and updates the value
Is it possible at all ?
thanks
The general solution to this problem in Terraform is Data Sources, which are special resources that retrieve data from elsewhere rather than creating and managing objects themselves.
In order to use data sources, the data you wish to share must be published somewhere. For sharing between Terraform configurations, you need a data storage location that can be both written to and read from by Terraform.
Since you mentioned ECS Fargate I will assume you're using AWS, in which case a reasonable choice is to store your data in AWS SSM Parameter Store and then have other configurations read it out.
The configuration that creates the data would use the aws_ssm_parameter resource type to create a new parameter:
resource "aws_ssm_parameter" "foo" {
name = "port_number"
type = "String"
value = aws_lb_listener.example.port
}
The configurations that will make use of this data can then read it using the corresponding data source:
data "aws_ssm_parameter" "foo" {
name = "port_number"
}
However, your question talks about the possibility of one configuration reading the value, incrementing it, and writing the new value back into the same place. That is not possible with Terraform because Terraform is a declarative system that works with descriptions of a static desired state. Only one configuration can be managing each object, though many configurations can read an an object.
Instead of dynamically allocating port numbers then, Terraform will require one of two solutions:
Use some other system to manage the port allocations persistently such that once a port number is allocated for a particular caller it will always get the same port number. I don't know of any existing system that is built for this, so this may not be a tenable option in this case, but if such a system did exist then we'd model it in Terraform with a resource type representing a particular port allocation, which Terraform can eventually destroy along with all of the other infrastructure when the port is no longer needed.
Decide on a systematic way to assign consistent port numbers to each system such that each configuration can just know (either by hard-coding or by some deterministic calculation) which port number it should use.
I created an ASA module and deployed it through Set Modules but on reported by device column is assigned NO.
What is the reason and meaning of this status after succesfull deployment and how can I fix it ?
These are my modules
and this is the Route for the Stream Analytics module:
Update : Also this module has aditional input with absolute path reference data.
I tried to deploy another seperate ASA module using simple Job with input, output and it works. So the problem must be in reference data path ?
I have another module which downloads a file in ./file.txt. (This should be in /app/file.txt ?)
I want the ASA module to read already downloaded file as a local reference data. What reference data directory should I put in the job. Now I'm trying with /app/file.txt
"no" in "Reported by device" means this module doesn't has any status to report, usually it indicates that the module isn't created successfully. In your case, there must be error when bind the path of reference data to container during container creation.
what is your host os and container os?
If your host os matches container os, please use the absolute path instead of ./
If you are running a Linux container on windows host, please first create a volume, populate the reference data to the volume and provide the absolute path in volume eg. Vol1/file1.txt
Thanks!
I have a terraform project that sets up a multi-node etcd cluster using etcd discovery url mechanism, with following setup:
main.tf (suppose to dynamically generate etcd discovery url and pass generated url as input variable to module below)
etcd module
cloud-init sh script (receives discovery url variable and start etcd cluster nodes using this url)
The problem: I want to generate new etcd discovery url dynamically every time after running "terraform apply" (based on input variable, say numEtcdNodes) and pass generated url as variable to the cloud-init sh scripts in etcd module.
How can this be achieved?
I tried this option:
It doesn't work. The file interpolation method gets invoked first followed by generation of discovery url. So cloud-init sh scripts get old url used in previous run, so etcd cluster never comes up.