I am new to Terraform and have been trying to understand the constructs of the same. Let's say i have a service which exposes REST API's and i want to call those REST API's as part of my terraform script, what are the steps i need to take ?
My understanding is that i need to write a custom provider but i am unable to connect the dot's on how to add new data source type for the new provider.
Also, assuming that we do have the required provider, whats the protocol that would be used for communicating with my service ? Is it HTTP/s ?
One more point to note is that my service currently is used for configuring storage in the backend.
Recent versions of terraform ( > 0.9 I believe) support external data sources. You don't have to create a custom provider. You can call any arbitrary shell or python script that return values that you can use as data.
data "external" "example" {
program = ["python", "${path.module}/example-data-source.py"]
query = {
# arbitrary map from strings to strings, passed
# to the external program as the data query.
id = "abc123"
}
}
In your case you could use a simple curl in a bash script to call your endpoint and return data to terraform as a map of strings.
Do note the warnings a the top of that page.
This is considerably more difficult then it appears; it is impossible to debug the interaction between what terraform is sending to my script and what the script is expecting. It just fails to parse the arguments and refuses to provide me any feedback as to what is getting into the program
Related
I have 5 different APIs in my Dev environment. This environment was built manually.
However, for the subsequent environments like Test, Pre-Prod, etc.. Terraform is being used.
Since I need to create each of the APIs in the subsequent environments, am extracting each of these APIs as a JSON file, making minor tweaks to the API URLs and importing it into the new environments.
The following is the process that am doing right now.
Went to Resource groups in Azure
Then under API Management service > APIs, clicked on the necessary API
Now, clicked on the three dots next to the API that I need and clicked on Export
Selected OpenAPI v3 (JSON) format
Now, I'm using the extracted JSON file and using the Terraform code below to add it to the APIM
resource "azurerm_api_management_api" "example" {
name = "example-api"
resource_group_name = azurerm_resource_group.example.name
api_management_name = azurerm_api_management.example.name
revision = "1"
display_name = "Example API"
path = "api/path"
protocols = ["https"]
service_url = "https://actualURL-of-the-API"
import {
content_format = "openapi+json"
content_value = file("extracted-filename.json")
}
}
The issue here is:
Even though the API gets added to the APIM, this doesn't create all the data - like Webservice URL, Backend HTTP(s) endpoint
How do I go about doing this?
Are you locked into exporting the Json file and importing it on the other environments through Terraform?
The reason I ask is because I attempted something similar but decided to go another route.
Initially I created the API manually in a Dev environment. I then re-created the same API from the ground up using only Terraform. No Json export & import.
I then used that Terraform script to create my other environments.
That allowed me to bypass the import problem altogether since nothing is imported.
I have found that there are downsides to taking this approach; It is much less intuitive to author the API through the Terraform script than through the Azure GUI. This is therefore more time consuming. Especially since my initial API was discarded for the one generated with the Terraform script.
Additionally, I have had problems with Terraform diffs reporting example changes when there are none (I suspect the same problem is to be had when using the import method).
If you are wondering why I decided to go another route? The reason was twofold; Firstly, similar to you, I had trouble with getting the export/import to generate the API that I wanted. Secondly, I prefer not to rely on auto generated files.
I'm pretty new to puppet and have run into an issue.
We have a proprietary home-grown API-based secrets management platform. We can either query the API directly or configure so that the secrets for that host are mounted to the root filesystem.
My problem is I can't figure out how to get that information within the context of a puppet module and into a variable so that I can use it. It seems you can't get stdout/stderr back from exec (or can you) otherwise this would be cake.
So for simplicity, let's say my secret is /etc/app/example/foo.
$roles.each |$role| {
case downcase($role) {
'foo': {
# SOMEHOW I NEED TO GET TOKEN FROM FILESYSTEM OR API CALL HERE
$token = <GET TOKEN SOMEHOW>
# here I need to do something with my value
exec { "my description":
command => '//bin/foo',
environment => ["TOKEN=${token}"]
}
This is basically what I need to do at a basic level. It doesn't matter if I call curl directly (preferred approach) or read a mounted file.
Thx for any help.
you can't get stdout/stderr back from exec (or can you) otherwise this would be cake.
You cannot capture the standard output or error of an Exec's command for reuse, but Puppet's built-in generate() function serves exactly the purpose of executing a command and capturing its output. Normally that would run the command on the server, during catalog compilation, but if you want it to run on the client instead then you can defer its execution. One of the primary purposes for deferring functions is for interaction with secret stores.
With that said, you might want to consider wrapping the whole thing up in a custom resource type. That's maybe a bit more work (especially if you don't speak Ruby), but it's a lot more flexible, and it should make for cleaner and clearer code on the Puppet DSL side, too.
# Using a single workspace:
terraform {
backend "remote" {
hostname = "app.terraform.io"
organization = "company"
workspaces {
name = "my-app-prod"
}
}
}
For Terraform remote backend, would there be a way to use variable to specify the organization / workspace name instead of the hardcoded values there?
The Terraform documentation
didn't seem to mention anything related either.
The backend configuration documentation goes into this in some detail. The main point to note is this:
Only one backend may be specified and the configuration may not contain interpolations. Terraform will validate this.
If you want to make this easily configurable then you can use partial configuration for the static parts (eg the type of backend such as S3) and then provide config at run time interactively, via environment variables or via command line flags.
I personally wrap Terraform actions in a small shell script that runs terraform init with command line flags that uses an appropriate S3 bucket (eg a different one for each project and AWS account) and makes sure the state file location matches the path to the directory I am working on.
I had the same problems and was very disappointed with the need of additional init/wrapper scripts. Some time ago I started to use Terragrunt.
It's worth taking a look at Terragrunt because it closes the gap between Terraform and the lack of using variables at some points, e.g. for the remote backend configuration:
https://terragrunt.gruntwork.io/docs/getting-started/quick-start/#keep-your-backend-configuration-dry
I am doing some learnings with Terraform and sentinel.
I cant get some of the basic functionality working.
I have a policy here:
import "tfconfig"
default_foo = rule { tfconfig.variables.foo.default is "bar" }
default_number = rule { tfconfig.variables.number.default is 42 }
main = rule { default_foo and default_number }
and a variables file here:
variable "foo" {
default = "bar"
}
variable "number" {
default = 42
}
But when I run:
sentinel apply policy.sentinel
I get the following error:
policy.sentinel:1:1: Import "tfconfig" is not available.
Any ideas as I have been looking for a solution for a number of hours now.
thanks
In order to use the Terraform-specific imports in the Sentinel SDK, you need to use mock data to produce a data structure to test against.
When you run Terraform via Terraform Cloud, a successful plan will produce a Sentinel mocks file that contains the same data that Terraform Cloud would itself use when evaluating policies against that plan, and so you can check that mock data into your repository as part of your test suite for your policies.
You can use speculative plans (run terraform plan on the command line with the remote backend enabled) to create mock data for intentionally-invalid configurations that you want to test your policy against, without having to push those invalid configurations into your version control system.
You can use sentinel test against test cases whose JSON definitions include a mock object referring to those mock files, and then the policies evaluated by those test cases will be able to import tfconfig, tfplan and tfstate and get an equivalent result to if the policies were run against the original plan in Terraform Cloud.
I had my mock data and everything and I was still getting this error.
Runtime error while running the policy:
ensure-policy.sentinel:3:1: Import "tfplan" is not available
I realized I had forgotten the 'sentinel.json' file at the root of my policies dir, which tells sentinel where to look for the mock data
https://www.terraform.io/docs/cloud/sentinel/mock.html
here's how they recommend a default dir setup, but you still need that .json file to tell sentinel where to look
and here's what the sentinel.json file should look like
# Using a single workspace:
terraform {
backend "remote" {
hostname = "app.terraform.io"
organization = "company"
workspaces {
name = "my-app-prod"
}
}
}
For Terraform remote backend, would there be a way to use variable to specify the organization / workspace name instead of the hardcoded values there?
The Terraform documentation
didn't seem to mention anything related either.
The backend configuration documentation goes into this in some detail. The main point to note is this:
Only one backend may be specified and the configuration may not contain interpolations. Terraform will validate this.
If you want to make this easily configurable then you can use partial configuration for the static parts (eg the type of backend such as S3) and then provide config at run time interactively, via environment variables or via command line flags.
I personally wrap Terraform actions in a small shell script that runs terraform init with command line flags that uses an appropriate S3 bucket (eg a different one for each project and AWS account) and makes sure the state file location matches the path to the directory I am working on.
I had the same problems and was very disappointed with the need of additional init/wrapper scripts. Some time ago I started to use Terragrunt.
It's worth taking a look at Terragrunt because it closes the gap between Terraform and the lack of using variables at some points, e.g. for the remote backend configuration:
https://terragrunt.gruntwork.io/docs/getting-started/quick-start/#keep-your-backend-configuration-dry