Primary goal is to add all puppet modules automatically, so that all dev-env's and prod-env could be started with one command. How can I install puppet modules through puppet manifest?
We've been happily using librarian-puppet to sync all 3rd party modules, it supports setting the modules' locations and versions. So production and dev run the exact same code.
The usage is one liner
librarian-puppet install
In other cases we have a shell script that runs puppet two times, one time a minimal module that is only responsible for fetching the required modules, and then the full blown puppet flow when all modules are available.
The "Puppet module" type and provider does exactly that:
module { 'author/mymodule':
ensure => present,
}
module { 'puppetlabs/stdlib':
ensure => '2.6.0',
}
module { 'puppetlabs/stdlib':
ensure => present,
modulepath => '/etc/puppet/modules',
}
Recent versions of Puppet also have a puppet module command that allows you to install arbitrary Puppet modules from the Puppet forge, example:
$ puppet module install rcoleman/puppet_module
Notice: Preparing to install into /home/anarcat/.puppet/modules ...
Notice: Created target directory /home/anarcat/.puppet/modules
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/home/anarcat/.puppet/modules
└── rcoleman-puppet_module (v0.0.3)
Other alternatives include:
librarian
r10k (cited as a replacement for librarian), supports dynamic environments and much more
r10k is now part of Puppet Entreprise 2015.03, so it can certainly be considered best practice.
Here is an example of mine:
$module_stdlib = 'puppetlabs-stdlib'
exec { 'puppet_module_stdlib':
command => "puppet module install ${module_stdlib}",
unless => "puppet module list | grep ${module_stdlib}",
path => ['/bin', '/opt/puppetlabs/bin']
}
Where $module_stdlib is a module I whant to install.
The /bin path is a path where the grep comes from.
And the /opt/puppetlabs/bin is a path where for the puppet binary exists in my installation.
Seems that writing module for installing puppet modules is possible - it'll be just wrapper for puppet module tool. However, I didn't hear about such module yet.
I suppose this mechanism of installation is not popular because often you need to modify installed module, do some customizations. Practical tool for management of such modifications is version control system. For example, in our team we keep /etc/puppetlabs/puppet directory in git repostitory. After installation of any module from Puppet Forge we add its files to version control and push it to remote git server. Our internally developed modules are also kept in this repository. This way, several puppet masters (dev and prod environments) are synchronized with this central repository and always have up-to-date versions of all modules.
I did this and it seemed to work
exec { 'puppet-fstab':
path => '/bin:/usr/bin',
command => 'puppet module install -i /usr/share/puppet/modules/AlexCline-fstab >>/tmp/err.log',
}
Related
I use puppet to install apache with the following code in the manifest.
class{ 'apache':
docroot => '/var/www', # ubu default, ignored
default_vhost => false,
default_ssl_vhost => false,
service_enable => false, # Do not start at boot
service_ensure => stopped, # Apache should be stopped if puppet runs
}
In my puppet.conf I have mentioned like below.
mod "apache",
:git => 'ssh://git.*.*.com:7999/xyz/jira-apache-puppet-module.git',
:ref => 'master'
when i checked apache is getting installed with latest version as in my ubuntu repo.So is puppet using ubuntu repo for installing the package or the module as defined in puppet.conf
So is puppet using ubuntu repo for installing the package or the module as defined in puppet.conf[?]
Both.
The declaration in your manifest simply tells Puppet to include a class named 'apache' in the target node's manifest, with the specified parameter values. Puppet itself knows nothing about such a class, or any associated other classes, defined types, files, templates, data, etc. that belong to its module and support it. That's where your puppetfile entry comes in: tells Puppet which module you mean, and where to find it.
The Puppet module contains instructions for how to install and configure Apache, but it will not contain Apache itself. The approach to installation is sure to be to obtain the software from a package repository appropriate for the target system, as determined by the target system and its configuration. Puppet will use the same command-line interface for that purpose that you could use manually.
All the modules in Puppetfile will get installed during r10k run.
r10k deploy environment -pv
Have a basic doubt about puppet package resource.If I have a package resource declared in the the manifest file for eg:to install apache using apt-get.
1.During the first run of puppet agent,apache will get installed.
2.If I run the agent(using the existing code for package resource) again after ubuntu repo is refreshed with latest version of apache.
Will puppet update/refresh the apache in agent server?
The Package's ensure attribute determines the state.
What state the package should be in. On packaging systems that can retrieve new packages on their own, you can choose which package to retrieve by specifying a version number or latest as the ensure value. On packaging systems that manage configuration files separately from “normal” system files, you can uninstall config files by specifying purged as the ensure value. This defaults to installed.
Version numbers must match the full version to install, including release if the provider uses a release moniker. Ranges or semver patterns are not accepted except for the gem package provider. For example, to install the bash package from the rpm bash-4.1.2-29.el6.x86_64.rpm, use the string '4.1.2-29.el6'.
Valid values are present (also called installed), absent, purged, held, latest. Values can match /./.
Source: https://puppet.com/docs/puppet/5.3/types/package.html#package-attribute-ensure
We are building Node.js microservices. For some reusable components we have created a utils folder. This folder is outside the actual microservices package. When we run the microservices, we can refer to that code using require(../../utils/logger) and it works like a charm.
However when trying to create the docker image for my microservices
project the container gives me an error saying:
Error: Cannot find module '../../Utils/logger
which makes a lot of sense as we are building the docker image inside the microservice project.
There are few architectural decisions which needs to be taken here:
We move the utils code into each microservice as required.
Pro: Microservice remains self sustained completely and no code level dependency on any other package.
Cons: Maintenance of cross cutting concerns and the changes would be cumbersome.
2.Create a private npm module and inject dependency into the microservice package.json file. Not sure if that would work.
Any suggestions on this are highly appreciated.
Best,
- Vaibhav
Don't use require(../../utils/logger), use npm packages
You should avoid using same files for microservice with symlink or requiring from one folder, because it destroys Loose coupling.
Loose coupling is a design goal that seeks to reduce the
inter-dependencies between components of a system with the goal of
reducing the risk that changes in one component will require changes
in any other component. Loose coupling is a much more generic concept
intended to increase the flexibility of a system, make it more
maintainable, and makes the entire framework more "stable".
Simply put, you can't have different version of your logger file, but you can have different version of your logger npm package.
Implementation details for using npm modules as reusable components for Node.js microservices:
Chose naming convention for packages. My advice is scoped packages. Example: #vaibhav/logger
Chose npm registry. There are such options:
2.1. npmjs.com and public packages. It's free, but your packages should have only universal code without any business-valuable details.
2.2 npmjs.com with private packages. It's fast, but not free.
2.3 verdaccio your own npm registry server. It's free simple Node.js solution, which should be install as server in your infrastructure.
2.4 nexus. Universal private registry with npm and docker support.
If you use 2.3 or 2.4 solution, then you need to choose ip or link for your server. My advice is use link. Example https://your-registry.com
If you use 2.3 or 2.4 solution, the you need to chose install approach in your .npmrc file inside your microservice. There are two options:
install all required packages from your registry. The .npmrc file will looks like registry=https://your-registry.com. Your registry should be able to cache public packages.
install only your package from your registry, install other packages from public registry. The .npmrc file will looks like #vaibhav:registry=https://mycustomregistry.example.org
Define processes for package development, publishing and updating the package version in microservice package-lock.json file. In our projects we define processes in this way:
We use GitHub flow for package development. There are only master branch for publishing and feature branches for development. Master branch can be updated only with pull requests from developers or commits from CI server.
We use Jenkins as continues integration server for autoupdating version and publish after merging pull request. Jenkins runs npm version command for update version, then publishes new commit to master branch and then publishes to npm registry. Jenkins checks some our rules and use npm version with patch or minor param. Updating major version is breaking change, which we do manualy.
We don't have 100% automated process for updating the package version in microservices. We automate only opening pull requests with new package version in package-lock.json file. Developers should check build status and press merge button manually.
Its not part of your question to elaborate how to deal with shared libraries in Microservice-Ecosystems and what to avoid there, but if you like, you should read this up to get you at least a list for pros and cons of "sharing".
Beside that, you can create a library container which only offers this library to be mounted.
version: "2"
services:
shared:
image: me/mysharelib
m1:
volume_from:
- shared:ro
m2:
volume_from:
- shared:ro
while your mysharedlib image looks more or less like this
FROM busybox
COPY bin/busyscript.sh /usr/local/bin/busyscript
WORKDIR /your/lib/folder
VOLUME /your/lib/folder
CMD ["busyscript"]
and your busyscript is just a dummy like this
#!/bin/sh
#set -x
pid=0
# SIGTERM-handler
term_handler() {
if [ $pid -ne 0 ]; then
kill -SIGTERM "$pid"
wait "$pid"
fi
exit 143; # 128 + 15 -- SIGTERM
}
# setup handlers
# on callback, kill the last background process, which is `tail -f /dev/null` and execute the specified handler
trap 'kill ${!}; term_handler' SIGTERM
echo "Started DW php code"
# wait forever
while true
do
tail -f /dev/null & wait ${!}
done
As you see, m1/m2 ... m10 mount the library which and it is truly shared across all microservices.
Alternatives:
You can for sure use an private NPM packages or simply package the shared lib into the microservice m1..m10 during image build time.
What describe above especially suits you well when you want to replace the shared library in the stack with very little overhead and want to ensure the library is in-sync for all container instancs
I have a problem while configuring Puppet 4 master to work with HTTP requests so I can use CouchDB for hiera.
These are the steps I did so far:
created new CouchDB with test database
created new document called common
gem install hiera-http-1.0.0
put the http_backend.rb file in /opt/puppetlabs/code/environments/production/mpdules/hiera_http and /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/hiera-http
When I run gem list I can see:
Hiera (3.2.0)
Hiera-http (1.0.0)
Now, when I try running hiera common or anything else I get ERROR :
'require' : cannot load such file -- lookup_http (LoadError)
My hiera.yaml looks like :
:backends:
- http
And, of course, all the required settings (host,port..)
When i run puppet agent -t on the agent I get
cannot load backend http: no such file to load -- hiera/backend/http_backend at site.pp
Your steps 3 and 4:
gem install hiera-http-1.0.0
put the http_backend.rb file in /opt/puppetlabs/code/environments/production/mpdules/hiera_http and /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/hiera-http
need to be replaced by:
/opt/puppetlabs/puppet/bin/gem install hiera-http
That will ensure the hiera-http hiera backend gem is automatically properly installed and configured for the ruby that puppet uses.
If you want to use the system ruby for puppet so that it recognizes the hiera-http installed from the system gem, then you need to install puppet, facter, and hiera with gem and not your OS package manager.
I am struggling around a wrong usage of composer, for sure.
I set up this repository: https://github.com/alle/assets-merger
I forked the project and was just trying to make it a kohana-module, including all the dependencies.
As for it would need the YUI comporess JAR, I was trying to make just that JARfile as a dependency, and I ended to declare it in the composer.json file (please, look at this).
Once I need to add my new package to a project I add it in the require section as follows:
...
"alle/assets-merger": "dev-master",
...
But the (latest) composer update command says:
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.
Problem 1
- Installation request for alle/assets-merger dev-develop -> satisfiable by alle/assets-merger[dev-develop].
- alle/assets-merger dev-develop requires yui/yuicompressor 2.4.8 -> no matching package found.
Potential causes:
- A typo in the package name
- The package is not available in a stable-enough version according to your minimum-stability setting see <https://groups.google.com/d/topic/composer-dev/_g3ASeIFlrc/discussion> for more details.
And my story ends here.
How should I configure my composer.json in the https://github.com/alle/assets-merger repository, in order to include it as a fully satisfied kohana-module in other projects?
Some things I notice in your composer.json.
There is a version of that CSS minify available on Packagist which says it is just a copy of the original Goole-Code hosted files, but with Composer: natxet/cssmin. It is version 3.0.2, but I think that shouldn't make a difference.
mrclay/minify is included twice in the packages with the same version. It also is available on Packagist. You will probably already use that (version 2.2.0 is registered, and because you didn't turn of Packagist access, it will be generally available for install unless a version requirement or conflict prevents it).
You are trying to download a JAR file (which is a java executable without and PHP), but try to get PHP classmaps out of it. That will fail for sure.
You did miss the big note in the Composer documentation saying that Composer cannot resolve repositories mentioned in sub packages, only in the root package. That means that whatever you mention in your alle/asset-merger package will not be used if you use that package anywhere else. You'd have to duplicate these repositories in every package in addition to adding the package name itself as "required".
What this means is that you probably avoided missing mrclay/minify because it is available on Packagist, you might as well have added the cssmin by accident, but you definitly did not add YUICompressor.
But you shouldn't add this in the first place, because it is no PHP software. You can however add post-install commands to your projects. All your Composer integration does is download the JAR file. You can do that with a post-install or post-update command. See the documentation here.