I need to copy files from puppet master to agent using tasks ,whenever I need to copy on random linux servers. Please help me on this.
A little light on detail, so I have to make some assumptions.
You state you use a task, but you mention you want to get rid of corrective changes, which indicates a normal Puppet run, not a task.
My first assumption then is that you are using Puppet code, not executing a task via Bolt or the GUI.
Using the file resource, you can state the endpoint for the file in the resource itself: doco
Alternatively, you can use an exec resource with a 'creates' tag to ensure it only pulls it once.
The easiest might be to just use the file resource and place the file in the module.
But, without more info, it will be hard to help properly.
Related
I have a Terraform script that, after creating several nodes, installs a piece of software on them by telling one node about the other ones using cloud-init. I want this cloud-init piece to only run when this one node is initially created, and not if it's altered or re-created. Now thanks to Terraform plan, Terraform has the information needed, it's telling the user clearly if the node has to be re-created or if it's created the first time at all. But how do I get this kinda information in my script?
The (boring) and manual way is of course to make it a variable that a human enters after reviewing the plan. But this is hardly scalable, secure or sophisticated.
Maybe Terraform is entirely the wrong tool for this kinda job?
Terraform does not expose the information about what action is planned for an object for use in the configuration itself, because Terraform is a desired state system and so the actions are derived from the configuration, rather than the configuration being derived from the actions.
To achieve what you described I think you'll need to arrange for the initialization process to itself remember that it already run in some persistent location. Cloud-init itself remembers when it has run to completion so that rebooting the system won't re-run initialization tasks, but of course that information cannot survive replacing the VM entirely and so you'd need to create a similar marker yourself in a data store that will outlive that particular VM.
One unanswered question down that path is how that external state would get cleaned up if you were to destroy the VM entirely, since the software running in the VM can't tell whether the system is being shut down in preparation for replacement or being shut down in response to just destroying.
I'm trying to decide whether to use Ansible or Terraform for a project. This project is doing scientific computations on cloud servers. I want to be able to say "here's an input file, go grab a computer with 16vCPUs. When everything's done, copy the output file back and destroy the resource". Is there a way for me to tell terraform "destroy/take action on this resource when you detect a command is done or some api is hit"?
Is there a way for me to tell terraform "destroy/take action on this
resource when you detect a command is done or some api is hit"?
If it's detectable via API hit then you can use local-exec Provisioner and use curl to hit the api endpoint in loop (in a wrapping shell script), polling the status of the API to deduce if the task is done or not. If you do so, make sure to give appropriate timeout value to this provisioner.
For the cleanup sequence (post API status is affirmative), declare the dependency explicitly to ensure Terraform is able to follow the execution accordingly.
Having explained all this, your end to end objective is best suited with a combination of IaC + Config mgmt tool. If I had to do it, I'd have preferred to use Terraform + Ansible combination in following outline
A shell script invoking Terraform to provision the 16vCPUs resource with all necessary depdendency
From within Terraform, invoke local-exec provisioner to launch ansible for appropriately configuring the instance, copying the file and launching remote execution etc.
Let the control come back to shell script which will start polling the status of the task based on API polling using curl
As and when shell script detects the task is done, it'd have copied the remote file locally (this step can be made better but that's a topic for another discussion)
Invoke Terraform again, with destroy this time, to tear down entire environment.
I'd further recommend to have it tied up in a jenkins workflow (if you are using jenkins already) giving control to other user's and still having full visibility and auditing in place.
I usually run all my Terraform scripts through Bastion server and all my code including the tf statefile resides on the same server. There happened this incident where my machine accidentally went down (hard reboot) and somehow the root filesystem got corrupted. Now my statefile is gone but my resources still exist and are running. I don't want to again run terraform apply to recreate the whole environment with a downtime. What's the best way to recover from this mess and what can be done so that this doesn't get repeated in future.
I have already taken a look at terraform refresh and terraform import. But are there any better ways to do this ?
and all my code including the tf statefile resides on the same server.
As you don't have .backup file, I'm not sure if you can recover the statefile smoothly in terraform way, do let me know if you find a way :) . However you can take few step which will help you come out from situation like this.
The best practice is keep all your statefiles in some remote storage like S3 or Blob and configure your backend accordingly so that each time you destroy or create a new stack, it will always contact the statefile remotely.
On top of it, you can take the advantage of terraform workspace to avoid the mess of statefile in multi environment scenario. Also consider creating a plan for backtracking and versioning of previous deployments.
terraform plan -var-file "" -out "" -target=module.<blue/green>
what can be done so that this doesn't get repeated in future.
Terraform blue-green deployment is the answer to your question. We implemented this model quite a while and it's running smoothly. The whole idea is modularity and reusability, same templates is working for 5 different component with different architecture without any downtime(The core template remains same and variable files is different).
We are taking advantage of Terraform module. We have two module called blue and green, you can name anything. At any given point of time either blue or green will be taking traffic. If we have some changes to deploy we will bring the alternative stack based on state output( targeted module based on terraform state), auto validate it then move the traffic to the new stack and destroy the old one.
Here is an article you can keep as reference but this exactly doesn't reflect what we do nevertheless good to start with.
Please see this blog post, which, unfortunately, illustrates import being the only solution.
If you are still unable to recover the terraform state. You can create a blueprint of terraform configuration as well as state for a specific aws resources using terraforming But it requires some manual effort to edit the state for managing the resources back. You can have this state file, run terraform plan and compare its output with your infrastructure. It is good to have remote state especially using any object stores like aws s3 or key value store like consul. It has support for locking the state when multiple transactions happened at a same time. Backing up process is also quite simple.
I'm using the ibm_installation_manager module from the puppet forge and it is a bit basic because IBM wrote Installation Manager in a time where idempotency was done much.
ref: https://forge.puppet.com/puppetlabs/ibm_installation_manager
As such it does not cater nicely for upgrades - so the module will not detect if an upgrade is needed, stop existing processes, do the upgrade and then start the processes again. It will just detect if an upgrade is needed and try to install the desired version and if that constitutes an upgrade that's great, but it will probably fail due to running instances.
So I need to implement some "stop processes" pre-upgrade functionality.
I need to mention at this point I'm new to ruby and fairly new to puppet.
The provider that the module uses (imcl.rb) has an exists method.
The ideal way for me to detect if an upgrade is going to happen (and stop the instances if it is) would be for my puppet manifest to be able to somehow call the exists method. Is this possible?
Or how would you approach this problem?
Something like imcl.exists(ibm_pkg["my_imcl_pkg_resource"])
The ideal way for me to detect if an upgrade is going to happen (and stop the instances if it is) would be for my puppet manifest to be able to somehow call the exists method. Is this possible?
No, it is not possible, at least not in any useful way. Your manifests describe how to build a catalog of resources describing the target state of the machine. In a master / agent setup, this happens on the master. The catalog is then used as input to a separate step, in which it is transferred to the target machine and applied there. It is in this second step that providers are engaged.
To the extent that you want the contents of your catalogs to be influenced by the current state of the target machine, the Puppet mechanism for that is to convey the needed state details to the catalog builder in the form of facts. It is relatively straightforward to add your own facts. Indeed, there are at least two distinct, non-exclusive mechanisms, going under the names "external facts" and "custom facts".
I'm working on a tool which manages WordPress instances using puppet. The flow is the following: the user adds the data of the new WordPress installation in the web interface and then that web interface is supposed to send a message to the puppet master to tell it to deploy it to the selected machine.
Currently the setup is done via a manifest file which contains the declaration of all WordPress instances, and that is applied manually via puppet apply on the puppet agent. This brings me to my 2 questions:
Are manifests the correct way of doing this? If so, is it possible to apply them from the puppet master to a specific node instead of going to the agent?
Is it possible to automatically have a puppet run triggered once the list of instances is altered?
To answer your first question, yes there's absolutely a way of doing this via a puppetmaster, what you have at the moment is a masterless setup which assumes you're distributing your configuration with some kind of version control (like git) or manual process. This is a totally legitimate way of doing things if you don't want a centralized master.
If you want to use a master, you'll need to drop your manifest in the $modulepath of your master (it varies depending on your version, you can find it using puppet config print modulepath on your master) and then point the puppet agent at the master.
If you want to go down the master route, I'd suggest following the puppet documentation which will help you get started.
The second question brings me on to a philosphical argument of 'is this really want you want to do?'
Puppet traditionally (in my opinion) is a declarative config management tool that is designed to make your systems look a certain way. You write code to determine 'this is how I want it to look' and Puppet will converge to make it look that way. What you're looking to do is more of an orchestration task (ie when X do Y). There are ways of doing this with Puppet like using mcollective (to trigger a puppet run) which is managed by a webhook, but I think there are better tools for the job.
I'd suggest looking at ansible, saltstack or Chef's knife tool to do deploys like this.