Is there a way to extract the infrastructure details by using terraform
e.g get a list of Linux server's version, firewall policy, opened ports, software packages installed etc..
My aim is to generate a block of code to describe the current server setup, then I can use a check list if validate against the code. therefor security loopholes can be identified and fixed
Not sure if I completely understand your question. But, there is not such an "automated" way to extract all the details of your not-terraformed infrastructure. Nevertheless, there exists a terraform import command with which you can import your existing resource (here the docs) to your state file.
Btw, if you are using Oracle Cloud, the Resource Discovery could be an option.
Related
I was wondering, why does terraform downloads the modules to the local machine during terraform init command. What is the need to download the modules in our local machine? Why can't terraform directly refers to the file present in the repository?
Example of referring a remote module as follow:
module "signalR" {
source = "git#github.com:fifa-x/terraform-azurerm-signalr.git?ref=v1.1.0"
}
Terraform needs to download the contents of the repository in order to work with it, because otherwise it would have no way to know what is defined in the repository.
However, I think you are asking why Terraform writes what it downloaded into the local filesystem rather than just using it in memory for the duration of the operation. There are a few different reasons for that design tradeoff:
Security: A remote Terraform module is arbitrary code installed over the internet, which you'll presumably eventually be running with access to high-value credentials.
Terraform performs all loading of code from remote systems in terraform init and no other command so that you can review what terraform init installed -- either manually or with some automated security tools -- before you actually execute the module.
Other commands like terraform apply work only with modules already cached on the local system, so that once you've checked what was installed you can be confident that Terraform will be evaluating exactly that code.
Performance: Fetching code over the network can potentially be slow and/or expensive, particularly from Git repositories where the protocol requires always retrieving at least the entire content of the current commit.
By downloading remote code only once during terraform init and then reusing it for subsequent commands Terraform can amortize the cost of downloading the dependencies across multiple commands.
Debugging: If something goes wrong when you run terraform apply, Terraform might report an error that refers to source code in an external module. Although Terraform does include a minimal snippet of the part of the module which was in error, you may need to review other parts of that module's source code to fully understand what went wrong.
Keeping a copy of the source code on your local system means that you can more easily inspect the rest of the module source code and be sure that you're looking at the module code that Terraform was really reading. If Terraform instead only kept that source code briefly in memory while it was running, you'd need to look up the rest of the source code in the original repository and you might inadvertently be looking at the wrong commit and therefore get a misleading answer as to how the rest of the module behaves.
Creating a local cache of remote dependencies is a typical design choice for computer languages that support direct installation of remote libraries, and it's often for similar reasons as with Terraform. Terraform's details are not exactly the same, but the same general principles apply as with other programming languages.
From the documentation: "so that other Terraform commands can use it."
That said,
Local paths are special in that they are not "installed" in the same sense that other sources are: the files are already present on local disk (possibly as a result of installing a parent module) and so can just be used directly. Their source code is automatically updated if the parent module is upgraded.
Hope that helps.
I am trying to use Terraformer ( https://github.com/GoogleCloudPlatform/terraformer ) to import some of our legacy into Terraform for ease of maintainability going forward. I keep running into a few issues though.
Current State - I am on Terraform version 1.1.7 with a remote backend configuration on S3. The desired end state is to use Terraformer to generate configurations for the legacy infrastructure, and include them as a separate module in the backend remote state.
Problem(s) -
Terraformer only seems to work with Terraform version 0.21.31, and the state files it generates also confides in this. I did some lookup and the standard suggestion here was to use https://github.com/tfutils/tfenv and have multiple Terraform versions. However it seems that TF 0.21.31 does not support provider plugin sources, which means I can't install the Hashicorp AWS provider plugin, without which, Terraformer can't work. So I am stuck in a deadlock here. Curious to know how others solved it?
Assuming I do somehow manage to solve the version issue, Terraformer seems to generate a local state file. Now, since my remote backend is on S3, how do I merge this local state file to the remote backend state file, so that Terraform realises that I am just planning to import some infra, and not re-create them?
Any help here would be much appreciated. Thanks in advance!
Please let me know if we need some more additional info for a better prognosis!
I want to run a script using terraform inside an existing instance on any cloud which is pre-created .The instance was created manually , is there any way to push my script to this instance and run it using terraform ?
if yes ,then How can i connect to the instance using terraform and push my script and run it ?
I believe ansible is a better option to achieve this easily.
Refer the example give here -
https://docs.ansible.com/ansible/latest/modules/script_module.html
Create a .tf file and describe your already existing resource (e.g. VM) there
Import existing thing using terraform import
If this is a VM then add your script to remote machine using file provisioner and run it using remote-exec - both steps are described in Terraform file, no manual changes needed
Run terraform plan to see if expected changes are ok, then terraform apply if plan was fine
Terraform's core mission is to create, update, and destroy long-lived infrastructure objects. It is not generally concerned with the software running in the compute instances it deploys. Instead, it generally expects each object it is deploying to behave as a sort of specialized "appliance", either by being a managed service provided by your cloud vendor or because you've prepared your own machine image outside of Terraform that is designed to launch the relevant workload immediately when the system boots. Terraform then just provides the system with any configuration information required to find and interact with the surrounding infrastructure.
A less-ideal way to work with Terraform is to use its provisioners feature to do late customization of an image just after it's created, but that's considered to be a last resort because Terraform's lifecycle is not designed to include strong support for such a workflow, and it will tend to require a lot more coupling between your main system and its orchestration layer.
Terraform has no mechanism intended for pushing arbitrary files into existing virtual machines. If your virtual machines need ongoing configuration maintenence after they've been created (by Terraform or otherwise) then that's a use-case for traditional configuration management software such as Ansible, Chef, Puppet, etc, rather than for Terraform.
I've created some (remote-exec and file) provisioners to bootstrap (GCP) VMs that I'm creating that I want to apply to all my VMs, but I can't seem to figure out how to reuse them...?
Modules seem like the obvious answer, but creating a module to create the VMs means I'd need to make input vars for everything that I'd want to configure on each of the VMs specifically...
Reusing the snippets with the provisioners doesn't seem possible though?
Terraform's Provisioner feature is intended as a sort of "last resort" for situations where there is no alternative but to SSH into a machine and run commands on it remotely, but generally we should explore other options first.
The ideal case is to design your machine images so that they are already correctly configured for what they need to do and so they can immediately start doing that work on boot. If you use HashiCorp Packer then you can potentially run very similar steps at image build time to what you might've otherwise run at Terraform create time with provisioners, perhaps allowing you to easily adapt the work you already did.
If they need some configuration parameters from Terraform in order to start their work, you can use features like the GCP instance metadata argument to pass in those values so that the software in the image can access it as soon as the system boots.
A second-best sort of option is to use features like GCP startup scripts to pass the script to run via metadata so that again it's available immediately on boot, without the need to wait for the SSH server to start up and become available.
In both of these cases, the idea is to rely on features provided by the compute platform to treat the compute instances as a sort of "appliance", so Terraform (and you) can think of them as being similar to a resource modelling a hosted service. Terraform is concerned only with starting and stopping this black box infrastructure and wiring it in with other infrastructure, and the instance handles its implementation details itself. For use-cases where horizontal scaling is appropriate, this also plays nicely with managed autoscaling functionality like google_compute_instance_group, since new instances can be started by that system rather than directly by Terraform.
Because Provisioners are designed as a last-resort for when approaches like the above are not available, their design does not include any means for general reuse. It's expected that each provisioner will be a tailored solution to a specific problem inline in the resource it relates to, not something you use systematically across many separate callers.
With that said, if you are using file and remote-exec in particular you can get partway there by factoring out the specific file to be uploaded and the remote command to execute, in which case your resource blocks will contain just the declaration boilerplate while avoiding repetition of the implementation details. For example, if you had a module that exported outputs local_file_path, remote_file_path, and remote_commands you could write something like this:
module "provisioner_info" {
source = "./modules/provisioner-info"
}
resource "any" "example" {
# ...
provisioner "file" {
source = module.provisioner_info.local_file_path
destination = module.provisioner_info.remote_file_path
}
provisioner "remote-exec" {
inline = module.provisioner_info.remote_commands
}
}
That is the limit for factoring out provisioner details in current versions of Terraform.
I usually run all my Terraform scripts through Bastion server and all my code including the tf statefile resides on the same server. There happened this incident where my machine accidentally went down (hard reboot) and somehow the root filesystem got corrupted. Now my statefile is gone but my resources still exist and are running. I don't want to again run terraform apply to recreate the whole environment with a downtime. What's the best way to recover from this mess and what can be done so that this doesn't get repeated in future.
I have already taken a look at terraform refresh and terraform import. But are there any better ways to do this ?
and all my code including the tf statefile resides on the same server.
As you don't have .backup file, I'm not sure if you can recover the statefile smoothly in terraform way, do let me know if you find a way :) . However you can take few step which will help you come out from situation like this.
The best practice is keep all your statefiles in some remote storage like S3 or Blob and configure your backend accordingly so that each time you destroy or create a new stack, it will always contact the statefile remotely.
On top of it, you can take the advantage of terraform workspace to avoid the mess of statefile in multi environment scenario. Also consider creating a plan for backtracking and versioning of previous deployments.
terraform plan -var-file "" -out "" -target=module.<blue/green>
what can be done so that this doesn't get repeated in future.
Terraform blue-green deployment is the answer to your question. We implemented this model quite a while and it's running smoothly. The whole idea is modularity and reusability, same templates is working for 5 different component with different architecture without any downtime(The core template remains same and variable files is different).
We are taking advantage of Terraform module. We have two module called blue and green, you can name anything. At any given point of time either blue or green will be taking traffic. If we have some changes to deploy we will bring the alternative stack based on state output( targeted module based on terraform state), auto validate it then move the traffic to the new stack and destroy the old one.
Here is an article you can keep as reference but this exactly doesn't reflect what we do nevertheless good to start with.
Please see this blog post, which, unfortunately, illustrates import being the only solution.
If you are still unable to recover the terraform state. You can create a blueprint of terraform configuration as well as state for a specific aws resources using terraforming But it requires some manual effort to edit the state for managing the resources back. You can have this state file, run terraform plan and compare its output with your infrastructure. It is good to have remote state especially using any object stores like aws s3 or key value store like consul. It has support for locking the state when multiple transactions happened at a same time. Backing up process is also quite simple.