I am learning Puppet and just read about Hiera.
Before coming to the issue, i'm providing some config settings below:
$ cat /etc/puppetlabs/puppet/puppet.conf
[master]
codedir = /etc/puppetlabs/code
[agent]
server = puppet.example.com
[master]
certname = puppet.example.com
[master]
vardir = /var/opt/puppetlabs/puppetserver
ssldir = $vardir/ssl
[main]
environmentpath = /etc/puppetlabs/code/environments
basemodulepath = /etc/puppetlabs/code/modules
$ cat /etc/puppetlabs/puppet/hiera.yaml
---
:backends:
- yaml
:hierarchy:
- "nodes/%{::trusted.certname}"
- common
:yaml:
# datadir is empty here, so hiera uses its defaults:
# - /etc/puppetlabs/code/environments/%{environment}/hieradata on *nix
# - %CommonAppData%\PuppetLabs\code\environments\%{environment}\hieradata on Windows
# When specifying a datadir, make sure the directory exists.
:datadir:
UPDATE: After going through #um-FelixFrank answer, i changed my default Hiera config file location. The config file has the following content:
$ cat /etc/puppetlabs/code/hiera.yaml
---
:backends:
- yaml
:hierarchy:
- "hostname/%{facts.hostname}"
- "os/%{facts.osfamily}"
- common
:yaml:
:datadir: /etc/puppetlabs/code/hieradata
I created hieradata dir under /etc/puppetlabs/code
$ ll /etc/puppetlabs/code/
total 4
drwxr-xr-x. 4 root root 32 Dec 20 11:17 environments
drwxr-xr-x. 3 root root 39 Dec 21 12:22 hieradata
-rw-r--r--. 1 root root 153 Dec 20 16:51 hiera.yaml
drwxr-xr-x. 2 root root 6 Dec 21 11:02 modules
$ cat /etc/puppetlabs/code/hieradata/common.yaml
---
puppet::status: 'running'
puppet::enabled: true
I tried overwriting the above values in my hostname yaml file as stated under:
$ cat /etc/puppetlabs/code/hieradata/hostname/delvmplgc1.yaml
---
puppet::status: 'stopped'
puppet::enabled: false
$ facter hostname
delvmplgc1
I created a sample manifest on Puppet server to see whether i am able to perform Hiera lookup in manifest.
$ cat /etc/puppetlabs/code/environments/qa/manifests/hierasample.pp
notify { 'welcome':
message => "Hello!",
}
$status = lookup({ name => 'puppet::status', default_value => 'running' })
$enabled = lookup({ name => 'puppet::enabled', default_value => true })
service { 'puppet':
ensure => $status,
enable => $enabled,
}
Now the problem is that when i try executing the manifest, i see no messages related to Hiera.
$ puppet apply /etc/puppetlabs/code/environments/qa/manifests/hierasample.pp
Notice: Compiled catalog for delvmplgc1.sapient.com in environment production in 0.99 seconds
Notice: Hello!
Notice: /Stage[main]/Main/Notify[welcome]/message: defined 'message' as 'Hello!'
Notice: Applied catalog in 0.10 seconds
Any help will be appreciated.
As the comment in the standard hiera.yaml states, the datadir is located in
/etc/puppetlabs/code/environments/%{environment}/hieradata
So instead of creating hieradata in /etc/puppetlabs/code directly, move it down two levels into /etc/puppetlabs/code/environments/qa and other environments.
Related
I have configured Metricbeat to capture system level metrics, Metricbeat is intsalled as a service in Redhat linux , i installed metricbeat using rpm package.
Bellow is my metricbeat configuration .
> #============================ Config Reloading ==============================
> metricbeat.config.modules:
>
> # Glob pattern for configuration reloading
> path: ${path.config}/conf.d/*.yml
>
> # Period on which files under path should be checked for changes
> reload.period: 10s
>
> # Set to true to enable config reloading
> reload.enabled: false
>
> # Maximum amount of time to randomly delay the start of a metricset. Use 0 to
> # disable startup delay.
> metricbeat.max_start_delay: 10s
>
> #------------------------------- System Module -------------------------------
> - module: system
> metricsets:
> - cpu # CPU usage
> - load # CPU load averages
> - memory # Memory usage
> - network # Network IO
> - process # Per process metrics
> - process_summary # Process summary
> - uptime # System Uptime
> #- core # Per CPU core usage
> #- diskio # Disk IO
> #- filesystem # File system usage for each mountpoint
> #- fsstat # File system summary metrics
> #- raid # Raid
> #- socket # Sockets and connection info (linux only)
> enabled: true
> period: 10s
> processes: ['.*']
>
> # Configure the metric types that are included by these metricsets.
> cpu.metrics: ["percentages"] # The other available options are normalized_percentages and ticks.
> core.metrics: ["percentages"] # The other available option is ticks.
>
>
> #----------------------------- Logstash output ---------------------------------
> output.logstash:
> # Boolean flag to enable or disable the output module.
> #enabled: true
>
> # The Logstash hosts
> hosts: ["localhost:5044"]
And logstash is also installed as service in Redhat linux using rpm package.
bellow is my logstash configuration where i am trying to read inputs from
beat.input {
beats {
port => 5044
}
}
output {
elasticsearch {
host => "localhost:9200"
index => "metricbeat_logs"
}
}
bellow is my user and group permissions for both logstash and metricbeat
cd /etc/logstash
ls -lrt
-rw-------. 1 apelkdev apa 1696 Aug 18 02:29 startup.options
> -rw-r--r--. 1 apelkdev apa 285 Aug 18 02:29 pipelines.yml
> -rw-r--r--. 1 apelkdev apa 342 Aug 18 02:29 logstash-sample.conf
> -rw-r--r--. 1 apelkdev apa 4466 Aug 18 02:29 log4j2.properties
> -rw-r--r--. 1 apelkdev apa 1846 Aug 18 02:29 jvm.options
> -rwxr-xr-x. 1 apelkdev apa 8154 Feb 8 07:41 logstash.yml
> drwxrwxr-x. 2 apelkdev apa 4096 Feb 8 10:29 conf.d
cd /etc/metricbeat
ls -lrt
-rw-r--r--. 1 apelkdev apa 57809 Aug 18 00:28 metricbeat.reference.yml
-rw-r--r--. 1 apelkdev apa 76697 Aug 18 00:28 fields.yml
-rw-r--r--. 1 apelkdev apa 0 Feb 7 06:24 metricbeat.
-rw-------. 1 apelkdev apa 7892 Feb 8 08:14 metricbeat.yml
and I am running both the services using bellow commands
sudo service metricbeat start
sudo service logstash start
I am really not getting what went wrong here, when we check metricbeat logs it is having captured metrics details for current timestamp, not very sure what happens to logstash and why it is not reading metricbeat events.
You haven't enabled output to logstash in code.
> output.logstash:
> # Boolean flag to enable or disable the output module.
> #enabled: true
On above please change to
enabled: true
Using logstash 2.4 (I have my reasons) on Ubuntu 16.04
root#logbox:/etc/logstash/conf.d# ls -al
total 16
drwxrwxr-x 2 root root 4096 Nov 2 19:53 .
drwxrwxr-x 3 root root 4096 Nov 2 15:46 ..
-rwxrwxrwx 1 root root 277 Nov 2 19:52 01_01_input.conf
-rwxrwxrwx 1 root root 604 Nov 2 19:48 03_02_output_pa_http.conf
root#logbox:/etc/logstash/conf.d# cat *.conf
input {
stdin {}
file {
path => "/usr/share/logstash/files/production_input.txt"
start_position => "beginning"
# codec => plain { charset => "ISO-8859-1" }
codec => json
# add_field => [ 'redis_db', '10' ]
}
}
output {
stdout {}
}
root#logbox:/etc/logstash/conf.d# ls -al /usr/share/logstash/files/production_input.txt
-rwxrwxrwx 1 root root 11910 Nov 2 16:09 /usr/share/logstash/files/production_input.txt
However...
vagrant#logbox:/etc/logstash/conf.d$ sudo tail -f /var/log/logstash/logstash*
Sending logstash logs to /var/log/logstash/logstash.log.
==> /var/log/logstash/logstash.log <==
{:timestamp=>"2018-11-02T19:59:52.947000+0000", :message=>"Pipeline main started"}
==> /var/log/logstash/logstash.stdout <==
{:timestamp=>"2018-11-02T19:59:52.947000+0000", :message=>"Pipeline main started"}
The file is not printed in stdout...I have tried both codecs (json and plain)
edit: actually it might be the case the file has been read only once; does this have to do with sincedb?
how do I force logstash to read it again? isn't start_position => beginning enough?
start_position is used the first time the file is read.
If you want to force logstash to read it again, you need to add in your file input plugin conf this parameter :
sincedb_path => "/dev/null"
Please read this for further explanation
How to force Logstash to reparse a file?
I recently installed Cygwin64 after using Cygwin32 for quite some time and I'm now having problems with one of our production scripts. Node.js's 'readFileSync' seems to be prepending the windows drive letter to the path and then failing to resolve - e.g. /cygdrive/c/foo becomes c:/cygdrive/c/foo.
I've found various mentions of similar issues online but so far I've been unable to resolve this problem. My co-worker has a seemingly identical setup and does not experience the problem.
Here it is in a nutshell -
$ cat filetest.js
var fs = require('fs');
function main(argv) {
console.log("fileName => ", argv[2]);
var data = fs.readFileSync(argv[2], 'utf8');
console.log("success");
}
main(process.argv);
$ s/node filetest.js filetest.js
fileName => filetest.js
success
$ ls -l /cygdrive/c/temp/Test.txt
----rwx---+ 1 bdodd Domain Users 14 Jun 3 14:12 /cygdrive/c/temp/Test.txt
$ s/node filetest.js /cygdrive/c/temp/Test.txt
fileName => /cygdrive/c/temp/Test.txt
fs.js:338
return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
^
Error: ENOENT, no such file or directory 'C:\cygdrive\c\temp\Test.txt'
and just for completeness...
$ cat /etc/fstab
# /etc/fstab
#
# This file is read once by the first process in a Cygwin process tree.
# To pick up changes, restart all Cygwin processes. For a description
# see https://cygwin.com/cygwin-ug-net/using.html#mount-table
# This is default anyway:
none /cygdrive cygdrive binary,posix=0,user 0 0
$ cd /cygdrive/
$ ls -l
total 40
d---rwx---+ 1 NT SERVICE+TrustedInstaller NT SERVICE+TrustedInstaller 0 Jun 3 14:09 c
dr-xrwxr-x 1 Unknown+User Unix_Group+33 0 Apr 29 2014 u
Insight would be greatly appreciated. Thanks!
[root#puppet puppet]# cat /etc/hiera.yaml
---
:backends:
- yaml
:yaml:
:datadir: '/etc/puppet/hieradata'
:hierarchy:
- env/%{::environment}/fqdn/%{::fqdn}
- hostgroup/%{::hostgroup1}
- global
[root#puppet puppet]# cat hieradata/env/dev/fqdn/client00.itw.local.yaml
fruit::a:
- 'DevFQDN-kiwi'
[root#puppet puppet]# cat environments/dev/modules/fruit/manifests/init.pp
class fruit(
$a = hiera('fruit::a' ),
$b = hiera('fruit::b'),
$c = hiera('fruit::c')
) {
notify { 'foo':
message => "a is: ${a}, b is: ${b}, c is : ${c}",
}
}
fruit::a seems to resolve fine on client00.itw.local
[root#client00 ~]# puppet agent -t
Warning: Local environment: "production" doesn't match server specified node environment "dev", switching agent to "dev".
Info: Retrieving plugin
Info: Caching catalog for client00.itw.local
Info: Applying configuration version '1411407772'
Notice: a is: DevFQDN-kiwi, b is: HostgroupAll-orange, c is : global-lime-C
But CLI hiera does not return the correct value on the puppet master
[root#puppet puppet]# hiera -d fruit::a ::fqdn=client00.itw.local ::hostgroup1=all
DEBUG: Mon Sep 22 13:57:16 -0400 2014: Hiera YAML backend starting
DEBUG: Mon Sep 22 13:57:16 -0400 2014: Looking up fruit::a in YAML backend
DEBUG: Mon Sep 22 13:57:16 -0400 2014: Looking for data source hostgroup/all
DEBUG: Mon Sep 22 13:57:16 -0400 2014: Looking for data source global
DEBUG: Mon Sep 22 13:57:16 -0400 2014: Found fruit::a in global
["global-lime-A"]
With mcollective, hiera -d fruit::a -m client00.itw.local, I got the same result.
Thanks for your help.
environment is a Puppet specific fact, included in Puppet libraries
When using hiera in the command line you have to pass those facts.
I got this file1.csv file which has lots of network interface data (approx 1000) and I have to create the network interface file , as in ifcfg-lo:x files.
the file1.csv file contents as follows:
Hostname Loop_back_ip netmask interface
localhost1 192.168.1.10 255.255.255.255 lo:116
So the script should read the contents from the file1.csv file and create interface file as :
file name = ifcfg-lo:116
File contents :
DEVICE=lo:116
IPADDR=192.168.1.10
NETMASK=255.255.255.255
NETWORK=192.168.1.0
BROADCAST=255.255.255.255
ONBOOT=yes
NAME=loopback
I tried yesterday and I am very close to the solution. Researching the internet , I made a perl script and was able to create multiple interface by extracting data from csv file.
But there is small issue I see and not able to figure out why an extra character is coming in the file.
Following is the code:
#!/usr/bin/perl
use strict;
use warnings;
sub main
{
# Note: this could be a full file path
my $filename = "file1.csv";
open(INPUT, $filename) or die "Cannot open $filename";
# Read the header line.
#my $line = <INPUT>;
my $line;
# Read the lines one by one.
while($line = <INPUT>)
{
chomp($line);
# Display the header, just to check things are working.
my ($hostname, $ip, $netmask, $interface) = split(',', $line);
print "$hostname $ip $netmask $interface\n";
{
if( -d "/var/tmp/$hostname")
{
open(EF, ">/var/tmp/$hostname/ifcfg-$interface") or die "writing /var/tmp/$hostname/ifcfg-$interface\n";
print EF "DEVICE=$interface\n";
print EF "IPADDR=$ip\n";
print EF "NETMASK=$netmask\n";
print EF "NAME=loopback\n";
print EF "BOOTPROTO=none\n";
print EF "TYPE=Ethernet\n";
print EF "ONBOOT=yes\n";
close EF;
}
else
{
system ("mkdir /var/tmp/$hostname");
open(EF, ">/var/tmp/$hostname/ifcfg-$interface") or die "writing /var/tmp/$hostname/ifcfg-$interface\n";
print EF "DEVICE=$interface\n";
print EF "IPADDR=$ip\n";
print EF "NETMASK=$netmask\n";
print EF "NAME=loopback\n";
print EF "BOOTPROTO=none\n";
print EF "TYPE=Ethernet\n";
print EF "ONBOOT=yes\n";
close EF;
}
}
}
close(INPUT);
}
main();
So after executing the script it creates file as :
-rw-r--r-- 1 root root 112 Jul 24 10:09 ifcfg-lo:20
-rw-r--r-- 1 root root 112 Jul 24 10:09 ifcfg-lo:21
-rw-r--r-- 1 root root 112 Jul 24 10:09 ifcfg-lo:22
-rw-r--r-- 1 root root 112 Jul 24 10:09 ifcfg-lo:23
When I "cat" the file for display it shows no issues:
$> cat ifcfg-lo:20
DEVICE=lo:20
IPADDR=A.B.C.D
NETMASK=255.255.255.255
NAME=loopback
BOOTPROTO=none
TYPE=Ethernet
ONBOOT=yes
But when I "vi" into the file then I see an extra character ( ^M) and I am able to figure out where it came from and its in every file I created:
$> vi ifcfg-lo:20
DEVICE=lo:20^M
IPADDR=A.B.C.D
NETMASK=255.255.255.255
NAME=loopback
BOOTPROTO=none
TYPE=Ethernet
ONBOOT=yes