I am managing users with puppet with managehome set to true. This home directory is then populated with a few files (2 dot files in my case).
user { 'guillaume':
ensure => present,
managehome => true,
}
file {'/home/guillaume':
ensure => present,
purge => false,
recurse => true,
source => "puppet:///modules/${module_name}/home/${title}",
}
It is all fine and dandy, but I ended up putting a 25GB file in my home dir, which puppet was computing a hash of (at least it is my understanding. I could see from strace that the file was indeed fully read by puppet). It took about 20 minutes, for a full puppet run which should be done in less than a minute in theory. Removing the file made puppet run fast again, confirming my guess.
Why would puppet compute a hash of an unmanaged file, and how can I prevent sabotaging puppet by just putting such a (legit) file in a managed directory?
The reason Puppet is computing the checksum of the file in the home directory is because you are managing the contents of an entire directory recursively and that file is part of the directory's contents. There are a couple ways to improve your Puppet resources to avoid computing this checksum.
The first is to just manage the two hidden files directly:
user { 'guillaume':
ensure => present,
managehome => true,
}
file {'/home/guillaume/.file_one':
ensure => file,
source => "puppet:///modules/${module_name}/home/.file_one",
require => User['guillaume'],
}
file {'/home/guillaume/.file_two':
ensure => file,
source => "puppet:///modules/${module_name}/home/.file_two",
require => User['guillaume'],
}
Note that above I also fixed the unspecified ensure value on the file resources and the missing dependency metaparameter of the file resources on the user resource.
The second solution is to not recursively manage the contents of the directory and therefore ignore the files in the directory contents that are not being managed with the source attribute. You achieve this by setting the recurse attribute to remote:
user { 'guillaume':
ensure => present,
managehome => true,
}
file {'/home/guillaume':
ensure => directory,
recurse => remote,
source => "puppet:///modules/${module_name}/home/guillaume",
require => User['guillaume'],
}
Note that this makes the same fixes as the above solution.
Some helpful documentation:
https://puppet.com/docs/puppet/5.3/types/file.html
https://puppet.com/docs/puppet/5.3/metaparameter.html#require
Related
I try using fs.open(). But when the file not exist, the data exist.
Below is my code:
fs.open('person.json', 'w', function (err, data) {
if (err) throw err;
console.log(data)
});
console.log(data) result is
3
Why is that? Where the 3 come from?
My purpose is to read the file if exist and create new file if doesn't exist. How to do it in node.js?
It sounds like the FIRST thing you want to do is call fs.statSync(), to check if the file exists.
If it exists, then call fs.open("r"), to read it.
Otherwise, it sounds like you want to create it. fs.open("w"), as you've done above, should work fine.
fs.open returns a file descriptor. I suspect that's probably the "3" you're asking about.
Addendum 4/24/19
Historically speaking (other languages, other times), the idea of using "exceptions" to handle "control flow" is frankly HORRIFYING.
But repeatdomiau makes a valid point. The documentation does seem to suggest simply opening the file, and handling any exception that might arise:
https://nodejs.org/api/fs.html
// Check if the file exists in the current directory, and if it is writable.
fs.access(file, fs.constants.F_OK | fs.constants.W_OK, (err) => {
if (err) {
console.error(
`${file} ${err.code === 'ENOENT' ? 'does not exist' : 'is read-only'}`);
} else {
console.log(`${file} exists, and it is writable`);
}
});
Using fs.access() to check for the accessibility of a file before
calling fs.open(), fs.readFile() or fs.writeFile() is not recommended.
Doing so introduces a race condition, since other processes may change
the file's state between the two calls. Instead, user code should
open/read/write the file directly and handle the error raised if the
file is not accessible.
'3' - is a file descriptor.
https://nodejs.org/api/fs.html#fs_file_descriptors
'w' - Open file for writing. The file is created (if it does not exist) or truncated (if it exists).
Use 'r+' - Open file for reading and writing. An exception occurs if the file does not exist.
https://nodejs.org/api/fs.html#fs_file_system_flags
you can use "ax" mode means Open for appending. If the file does not exist, it is created but the file is opened in exclusive mode.
or can use "a+" mode means Open for reading and appending. If the file does not exist, it is created
and '3' you are getting is a file descriptor ("On POSIX systems, for every process, the kernel maintains a table of currently open files and resources. Each open file is assigned a simple numeric identifier called a file descriptor."
At the system-level, all file system operations use these file descriptors to identify and track each specific file. Windows systems use a different but conceptually similar mechanism for tracking resources. To simplify things for users, Node.js abstracts away the specific differences between operating systems and assigns all open files a numeric file descriptor.
The fs.open() method is used to allocate a new file descriptor. Once allocated, the file descriptor may be used to read data from, write data to, or request information about the file.)
Try to check this code, It might work for you
const fs = require('fs');
const readline = require('readline-sync');
// Check if the file exists in the current directory, and if it is writable.
var path = ("file.txt"); // Path to your file
var data = output; // this variable contains some data to insert
fs.access(path, fs.constants.F_OK | fs.constants.W_OK, (err) => {
if (err) {
fs.writeFileSync('file.txt', output, 'utf-8'); //create a new file
} else {
try {
fs.writeFileSync(path, data, {flag:'a+'}); //append mode: adds text to actually existing file
} catch(err) {
console.error(err); //shows error if script can't add data
}
}
});
Also you must install readline-sync module from npm, also change 'path' and 'output' variables to make this work. After this if you will don't have file, script make a new one, if will exist, will insert actual data to your file without reseting it.
P.S: Sorry for my English language.
I am looking for a manifest to start a service on windows server if the service exists.
I am looking to check a service with a condition if the file exists.
service { 'test service':
ensure => 'running',
enable => true,
unless => 'if((test-path "C:\Program Files\test1") -or (test-path "C:/Program Files/test2")){ exit 0 } else { exit 1 }',
provider => powershell,}
Above code returns an error that unless is an invalid parameter. Any other way to check?
Thanks a lot!!
it doesn't look like the service resource has a parameter to support conditional inclusion. Puppet also does not have an easy way to check for the existence of a file on the client that I'm aware of. I have gotten around this by creating a custom fact that checked the condition I needed and then wrapping the resource in an unless block:
unless $facts['test_file_exists'] {
service { 'test service':
ensure => 'running',
enable => true,
}
since a custom fact can be written in any language that can output "key=value" pairs to stdout you could use your existing powershell code as a fact with a small change:
if((test-path "C:\Program Files\test1") -or (test-path "C:\program Files\test2")){
write-host "test_file_exists=true
}
else{
write-host "test_file_exists=false"
}
there are a few different ways to distribute the fact out to your client systems. I prefer to put the fact into a module and let pluginsync distribute it out for me. If you don't want to include it in a module then you would have to find some way to get the powershell code into C:\ProgramData\PuppetLabs\facter\facts.d\ with the proper .ps1 extension on the end.
I needed to get all mustache templates directory named templates/, and compile them with hogan.
In theory, assume their names are,
file1.mustache
file2.mustache
file3.mustache
Then we get a view on each one, and save the result to an output directory named build/.
In theory the resulting names would be,
name.file1
name.file2
name.file3
Obviously async is preferable, but I am most interested in how you'd do this efficiently? I can't believe that the only way is doing per file objects and anonymous functions.
You could use the fs-promise module along with Promise.all to easily read, process, and write your files in parallel:
const fsp = require('fs-promise');
function processTemplate(filename) {
return fsp.readFile(filename, 'utf8')
.then((template) => hogan.compile(template))
.then((compiledTemplate) => fsp.writeFile('path/to/compiled', compiledTemplate));
}
fsp.readdir('./templates')
.then((files) => Promise.all(files.map(processTemplate)))
.catch((error) => console.log(error));
Although I'm not sure I understand what you mean by "per file objects and anonymous functions".
In my Puppet module, I have something like this:
file {'myfile':
ensure => 'file',
path => '/whatever/myfile',
content => inline_template(file(
"puppet:///modules/my_module/templates/${domain}/${::hostname}_myfile.erb",
"puppet:///modules/my_module/templates/${domain}/myfile.erb"
))
}
And my spec looks like:
require 'spec_helper'
describe 'my_module' do
context 'with defaults for all parameters' do
it { should compile }
end
end
If try to run it, I get:
Failure/Error: it { should compile }
error during compilation: Evaluation Error: Error while evaluating a Function Call, Could not find any files from puppet:///modules/my_module/templates/dev.contaazul.local/myfile.erb, puppet:///modules/my_module/templates/dev.contaazul.local/myfile.erb at /home/carlos/Code/Puppet/modules/my_module/spec/fixtures/modules/my_module/manifests/init.pp:48:33 on node becker.local
Obviously, it cannot find any of the ERB templates. If I replace the puppet:// part for /home/carlos/Code/Puppet/ (where the code actually lives), it passes, but in production it is /etc/puppet/, so, it will not work.
How can I make this work?
RI Pienaar has released a code snippet for a function with this behavior. You will need to copy this code into a file in one of your modules at the path lib/puppet/parser/functions/multi_source_template.rb.
Once you do that, you should be able to use it like so:
file {'myfile':
ensure => 'file',
path => '/whatever/myfile',
content => multi_source_template(
"my_module/${domain}/${::hostname}_myfile.erb",
"my_module/${domain}/myfile.erb"
)
}
As to why the original approach doesn't work: URLs are usually used with the source property only and transferred to the agent as is. The agent consumes the URL and makes an according request to a Puppet fileserver (which is just another master).
The file function on the other hand (as used here with the content property in conjunction with inline_template) will not process URLs and expects local paths instead.
All that being said, the issue could possibly have been side-stepped by specifying both the paths for the test box and the production system, with the latter just acknowledging that /home/carlos is missing. But that would have been far from a clean solution.
Logstash seems to hang when processing a local file. The logstash process is still alive and everything looks fine, but no data get written to the output (elasticsearch). The index gets written, though.
Logstash seems to "hang" and not process any of the input data for the following reason:
Logstash keeps track of what has previously been processed, so when you run it again on the same input data (as will be the case during testing), Logstash will think it has already seen and processed this data the previous time and will not read it again. To bypass this during testing, specify explicitly the location of the sincedb file where Logstash should keep track of what it has read or not and manually delete this sincedb file before each test run.
Here is an example:
input {
file {
path => "~/logstash/data/input_file"
start_position => "beginning"
sincedb_path => "~/logstash/data/sincedb.db"
}
}
or maybe even better (added based on comment below):
input {
file {
path => "~/logstash/data/input_file"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}