I did search, but didn't find exactly what I'm looking for. In the past, I've had multiple Puppet masters, one for each environment, then made scripts to promote code from one environment to the next on a monthly basis, but that process is a pain. I'm currently working on a single Puppet Master and am trying to take advantage of the environments feature but don't want the default environment set to production. This should be simple, but I've not yet worked out the solution. Without needing to set environments in the agents, how do I set the default environment on the master to "quarantine"? I'm running puppetserver 7.9.2.1 on Oracle Linux 8.
Without needing to set environments in the agents, how do I set the default environment on the master to "quarantine"?
This is the environment setting in the main puppet.conf configuration file. For [agent] configuration it specifies which environment the agent should request (which is not necessarily the one it gets), but for [server] configuration it specifies the default environment assigned when the agent does not request one and none is provided by the node terminus.
Example:
# ...
[server]
environment = quarantine
# ...
Because it has different significance to different Puppet components, you should avoid specifying this setting in the [main] section. You can meaningfully specify different values for this setting in different sections.
Although that answers what you actually asked, it might not be what you really want. Note well that the default environment set at the server will be overridden if a client requests a specific environment. If you want to prevent your Puppet clients from (successfully) specifying their own environments then you will want to set up an external node classifier that specifies at least the environment for each node. And if you have that, then whatever default you set in puppet.conf will be moot.
Related
I have two different AWS configurations. On a dev laptop, the developer uses a mfa-secured profile inside a shared_credentials_file.
On jenkins, we export environment variables and then assume a role.
This means that the provider blocks look really different. At the root level, they share the same backend.tf.
I know I can have two different roots with different providers, but is there a way so I don't have to duplicate backend.tf and other root files?
I understood your point, but it is not recommended. Make aws configuration with system environment variables ready before you run terraform commands.
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN (optional)
AWS_DEFAULT_REGION
AWS_DEFAULT_PROFILE (optional)
https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html
The solution that I think makes most sense is to place local developers and jenkins automations into two separate environment directories, each with their own aws.tf and backend workspace.
This made sense because developers should not be messing with resources created by automation, and any operations to the jenkins backend should be done by jenkins, otherwise devs could overwrite resources that jenkins put up, and vice versa.
As per apache doc "http://spark.apache.org/docs/latest/monitoring.html"
spark.history.retainedApplications points to "The number of application UIs to retain. If this cap is exceeded, then the oldest applications will be removed"
But I see more than configured apps into the UI. Is it correct or it stores those many apps into memory only or load again into memory when needed. Please clarify. Thx
That setting specifically applies to the history server. If you don't have one started (it's typically used with YARN and Mesos I believe), then the setting you're after is spark.ui.retainedJobs. Check the Spark UI configuration parameters for more details.
These settings only apply to jobs, so in order to pass them to the master itself, check the spark.deploy options in the stand-alone deployment section. You can set them via the SPARK_MASTER_OPTS environment variable.
If you want to clean the data files produced by workers, check the spark.worker.cleanup options in the same section. You can set them via the SPARK_WORKER_OPTS environment variable on your workers.
There is the need that one puppet agent contacts some different puppet masters.
Reason: there are different groups that create different and independent sets of manifests.
Possible groups and their tasks
Application Vendor: configuration of application
Security: hardening
Operations: routing tables, monitoring tools
Each of these groups should run it's own puppet master - the data (manifests and appropriate data) should be strictly separated. If it is possible, one group should even not see / have access to the manifests of the others (we are using MAC on the puppet agent OSes).
Thoughts and ideas that all failed:
using (only) hira is not flexible as needed - there is the need to have different manifests.
r10k: supports more than one environment, but in each environment can only access one set of manifests.
multi but same puppet server using e.g. DNS round robin: this is the other way round. We need different puppet masters.
Some ways that might be possible but...
running multiple instances of puppet agents. That 'feels' strange. Advantage: the access rights can be limited in the way as needed (e.g. the application puppet agent can run under the application user).
patching puppet that it can handle more than one puppet master. Disadvantage: might be some work.
using other mechanisms to split responsibility. Example: use different git-repositories. Create one puppet master. The puppet master pulls all the different repositories and serves the manifests.
My questions:
Is there a straight forward way implementing this requirement with puppet?
If not, is there some best practice how to do this?
While I think what you are trying to do here is better tackled by incorporating all of your modules and data onto a single master, and that utilizing environments will be effectively the exact same situation (different masters will provide a different set of modules/data) this can be achieved by implementing a standard multi-master infrastructure (one CA master for cert signing, multiple compile masters with certs signed by the same CA master, configured to forward cert traffic elsewhere) and configure each master to have whatever you need. You then end up having to specify which master you want to check in to on each run (a cronjob or some other approach), and have the potential for one checkin to change settings set by another (kinda eliminating the hardening/security concept).
I would urge you to think deeper on how to collaborate your varied aspects (git repos for each division's hiera data and modules that have access control) so that a central master can serve your needs (and access to that master would be the only way to get data/modules from everywhere).
This type of setup will be complex to implement, but the end result will be more reliable and maintainable. Puppet inc. may even be able to do consultation to help you get it right.
There are likely other approaches too, just fyi.
I've often found it convenient to multi-home a puppet agent for development purposes, because with a local puppet server you can instantly test manifest changes - there's no requirement to commit, push and r10k deploy environment like there is if you're just using directory environments and a single (remote) puppet server.
I've found the best way to do that is to just vary the path configuration (otherwise you run into problems with e.g. the CA certs failing to verify against the other server) - a form of your "running multiple instances of puppet agents" suggestion. (I still run them all privileged, so they can all use apt package {} etc.)
For Puppet 3, I'd do this by varying the libdir with --libdir (because the ssldir was under the libdir), but now (Puppet 4+) it looks more sensible to vary the --confdir. So, for example:
$ sudo puppet agent -t # Runs against main puppet server
$ sudo puppet agent -t \
--server=puppet.dev.example.com \
--confdir=/etc/puppetlabs/puppet-dev # Runs against dev puppet server
I'm working on a tool which manages WordPress instances using puppet. The flow is the following: the user adds the data of the new WordPress installation in the web interface and then that web interface is supposed to send a message to the puppet master to tell it to deploy it to the selected machine.
Currently the setup is done via a manifest file which contains the declaration of all WordPress instances, and that is applied manually via puppet apply on the puppet agent. This brings me to my 2 questions:
Are manifests the correct way of doing this? If so, is it possible to apply them from the puppet master to a specific node instead of going to the agent?
Is it possible to automatically have a puppet run triggered once the list of instances is altered?
To answer your first question, yes there's absolutely a way of doing this via a puppetmaster, what you have at the moment is a masterless setup which assumes you're distributing your configuration with some kind of version control (like git) or manual process. This is a totally legitimate way of doing things if you don't want a centralized master.
If you want to use a master, you'll need to drop your manifest in the $modulepath of your master (it varies depending on your version, you can find it using puppet config print modulepath on your master) and then point the puppet agent at the master.
If you want to go down the master route, I'd suggest following the puppet documentation which will help you get started.
The second question brings me on to a philosphical argument of 'is this really want you want to do?'
Puppet traditionally (in my opinion) is a declarative config management tool that is designed to make your systems look a certain way. You write code to determine 'this is how I want it to look' and Puppet will converge to make it look that way. What you're looking to do is more of an orchestration task (ie when X do Y). There are ways of doing this with Puppet like using mcollective (to trigger a puppet run) which is managed by a webhook, but I think there are better tools for the job.
I'd suggest looking at ansible, saltstack or Chef's knife tool to do deploys like this.
When using the Ansible provisioner, the delegate_to: ip_address can be used to execute actions on the machine that originally invoked ansible (the host) instead of the guest.
When using Puppet, what would be a similar equivelent?
Functions in a manifest are executed on the puppet master (if using agent). Resources are evaluated by the agent on the node. Note that this happens in stages, so functions are all called when the manifest is compiled and resources happen later after the compiled catalog is sent to the agent. Catalog caching can also prevent functions from being called on ever invocation of the puppet master.
Puppet implements a client / server paradigm (agent / master in Puppet jargon). I'm not sure whether that maps cleanly to Ansible's guest / host.
Nevertheless, Puppet DSL functions run on the master during catalog building. You can write custom DSL functions relatively easily, and you can run arbitrary commands (within the capabilities of the relevant user) via the built-in generate() function.
Additionally, if the master manages itself (which is common) then you can use exported resources to cause resources to be defined during building of any node's catalog that can later be collected and applied to the master.
Puppet does not provide any means, however, to cause code to run on the master as part of the process of the agent applying a catalog to a different node.