Actually I am downloading list of files from ftp and from the downloaded path I am reading all the list of filenames for processing.
In exec{"download from ftp ${value}" I am downloading directories and sub directories with files from ftp to local. From that path am getting the list using custom facts $facts['listdirectory']
My problem is that Facts['listdirectory'] is executed before being downloaded from ftp.
How to add dependency to $datadir=$facts['listdirectory'] or how to make this facts get executed after download?
class classname{
exec{"download from ftp ${value}":
command => "wget -r --user=${ftp_username} --
password=${ftp_password} ${value}/* -P ${patch_download_path}",
path => ['/usr/bin', '/usr/sbin',],
timeout => 1800,
user =>'root',
}
$datadir=$facts['listdirectory']
}
My problem is that Facts['listdirectory'] is executed before being downloaded from ftp.
It looks like you mean that the fact's value is determined before the directory contents (not the fact implementation) is downloaded. Certainly that's what will happen, in any case.
All facts that will inform a given catalog-building run are evaluated first, then delivered as a group to the catalog builder (which typically runs remotely on a puppet master). This gives the catalog builder a consistent snapshot of machine state to work from as it computes the desired target state by evaluating your manifests in light of the facts presented. The result is delivered in the form of a catalog of classes and resources, which the local Puppet then applies.
Only at the catalog-application stage will the command specified by your Exec resource run. This is after the whole catalog has been built, and long after fact evaluation. If you want to adapt dynamically to what has been downloaded then you must either do so on the next Puppet run, or script it and run the script via the same or another Exec resource, or write a custom type and provider that encompass the whole process (probably including the download, too).
Related
Using the functions in my earlier queries (see reference below), I am able to pull the file from the agent and perform the necessary tasks. However, this is affecting all the users on the system as it throws an exception stating that the file is not found. Is there anyway I can add some logic like unless file_exists .... to this ruby function ?
My hierarchy is shown below. I am not following why it affects other users who are not even in "mymodules".
Root
modules
mymodules
lib
facter
ruby_function1.rb
ruby_function2.rb
modules_by_userxx1
modules_by_userxx2
modules_by_userxx3
....
Reference :
Puppet : How to load file from agent
Puppet : How to load file from agent - Part 2
As requested by Dominic, adding reference code :
# module_name/lib/facter/master_hash.rb
require 'json'
Facter.add(:master_hash) do
setcode do
# return content of foo as a string
f = File.read('/path/to/some_file.json')
master_hash = JSON.parse(f)
master_hash
end
end
I'll assume you're talking about the custom fact from the previous answers rather than a Ruby function, in which case, add a File.exist? conditional:
# module_name/lib/facter/master_hash.rb
require 'json'
Facter.add(:master_hash) do
setcode do
if File.exist?('/path/to/some_file.json')
# return content of foo as a string
f = File.read('/path/to/some_file.json')
master_hash = JSON.parse(f)
master_hash
end
end
end
Please include the full error message (with the filename/line number it provides) and the source code when asking questions.
Custom facts are shipped within modules, but are all synced to agents as they're used to discover data.
Facts are synced and run on agents before the node is classified because they can be used to make classification and catalog decisions, e.g. based on hostname or OS. Because facts run before classification, it isn't yet possible to know which classes (and perhaps modules) are going to be applied, so should be safe to run on every node in the environment.
I'm trying to set up a build system involving a code generator. The exact files generated are unknown until after the generator is run, but I'd like to be able to run further build steps by pattern matching (run some program on all files with some extension). Is this possible?
Some of the answers here involving code generation seem to assume that the output is known or a listing of generated files is created. This isn't impossible in my case, but I'd like to avoid it since it makes things more complicated.
https://bitbucket.org/scons/scons/wiki/DynamicSourceGenerator seems to indicate that it's possible to add additional targets during Builder actions, but while I could get the build to run and list the generated files, any build steps introduced don't run.
https://bitbucket.org/scons/scons/wiki/NonDeterministicDependencies uses Scanners to add build steps. I put a glob(...) in a scanner, and it succeeds in detecting the generated files, but the files are inexplicably deleted before it actually runs the dependent step.
Is this use case possible? And why is SCons deleting my generated files?
A toy example
source (the file referenced in SConscript)
An example generator, constructs 3 files (not easily known to the build system) and puts them in the argument folder
echo "echo 1" > $1/gen1.txt
echo "echo 2" > $1/gen2.txt
echo "echo 3" > $1/gen3.txt
SConstruct
Just sets up a variant_dir
SConscript('SConscript', variant_dir='build')
SConscript
The goal is for it to:
"Compile" the generator (in this toy example, just copies a file called 'source' and adds execute permissions
Run the "compiled" generator ('source' is a script that generates files)
Perform some operation on each of those generated files by extension. This example just runs the "compile" copy operation on them (for simplicity).
env = Environment()
env.Append(BUILDERS = {'ExampleCompiler' :
Builder(action=[Copy('$TARGET', '$SOURCE'),
Chmod('$TARGET', 0755)])})
generator = env.ExampleCompiler('generator', 'source')
env.Append(BUILDERS = {'GeneratorRun' :
Builder(action=[Mkdir('$TARGET'),
'$SOURCE $TARGET'])})
generated_dir = env.GeneratorRun(Dir('generated'), generator)
Everything's fine up to here, where all the targets are explicitly known to the build system ahead of time.
Attempting to use this block of code to glob over the generated files causes SCons to delete (!!) the generated files:
for generated in generated_dir[0].glob('*.txt'):
generated_run = env.ExampleCompiler(generated.abspath + '.sh', generated)
Attempting to use an action to update the build tree results in additional actions not being run:
def generated_scanner(target, source, env):
for generated in source[0].glob('*.txt'):
print "scanned " + generated.abspath
generated_target = env.ExampleCompiler(generated.abspath + '.sh', generated)
Alias('TopLevelAlias', generated_target)
env.Append(BUILDERS = {'GeneratedOperation' :
Builder(action=[generated_scanner])})
dummy = env.GeneratedOperation(generated_dir[0].File('#dummy'), generated_dir)
Alias('TopLevelAlias', dummy)
The Alias operations are suggested in above dynamic source generator guide, but don't seem to do anything. The prints do execute and indicate that the action gets run.
Running some build pattern on special file extensions is possible with SCons. For C/CPP files this is the preferred scheme, for example:
env = Environment()
env.Program('main', Glob('*.cpp'))
The main task of SCons, as a build system, is to do the minimum amount of work such that all your targets are up-to-date. This makes things complicated for the use case you've described above, because it's not clear how you can reach a "stable" situation where no generated files are added and all targets are built.
You're probably better off by using a simple Python script directly...I really don't see how using SCons (or any other build system for that matter) is mission-critical in this case.
Edit:
At some point you have to tell SCons about the created files (*.txt in your example above), and for tracking all dependencies properly, the list of *.txt files has to be complete. This the task of the Emitter within SCons, which is responsible for returning the list of resulting target and source files for a Builder call. Note, that these files don't have to exist physically during the "parse" phase of SCons. Please also have a look at my answer to Scons: create late targets , which goes into some more detail.
Once you have a proper Emitter in place (see also https://bitbucket.org/scons/scons/wiki/ToolsForFools , "Using Emitters") you should be able to use the Glob('*.txt') call, which will detect and track your created files automatically.
Finally, on our page "Talks and Slides" ( https://bitbucket.org/scons/scons/wiki/TalksAndSlides ) you can find my talk from the PyCon FR.2014, "Why SCons is Not Slow", which explains shortly how SCons works internally. This might be helpful in understanding this problem better and coming up with a full solution.
This is probably a basic question but I've been Googling for a while on it... I have a Cabal-ized Haskell project and I'm in the process of writing integration tests for it. I want to be able to include test resources for my project in the same repo and access them in tests. For example, here are a couple things I want to accomplish:
1) Check a dummy database instance into my repo, including a shell script that spins up a database process. I want to write an Hspec integration test that spins up the database process, makes some calls to it, and then shuts it down. So I need to be able to find the shell script so I can use System.Process.createProcess on it.
2) Check in paired "input" and "output" files. My test should process each of the input files and compare them to a corresponding output file to make sure they match. (I've read about "golden" but it doesn't seem to solve the problem of finding/reading the input files in the first place?)
In short, how can I go about creating a "resources" folder in the root folder of my Haskell project and find the path to it inside tests?
Have a look at an existing project that uses input and output file.
For example, take haddock, the source code is at https://github.com/haskell/haddock. They have the test files under a folder (https://github.com/haskell/haddock/tree/master/html-test/ref) and they are referenced as extra-source-files in the cabal file (https://github.com/haskell/haddock/blob/master/haddock.cabal). Then the test code (https://github.com/haskell/haddock/blob/master/html-test/run.lhs) uses some CPP macro (__FILE__) to get the current directory, and can then resolve the files relative to that folder.
I have a Vagrantfile that provisions a VM by running a chef recipe. The first resource in the chef recipe is:
directory "/downloads" do
owner "root"
group "root"
mode "0755"
action :create
end
# check that it worked:
raise "/downloads doesn't exist!" unless File.exists? "/downloads"
When I run this at work, it works fine.
When I run it at home, it fails, the exception is raised when I check to see if /downloads exists.
I'm not sure why this is happening. I would expect it to behave identically, since the underlying Vagrant box is the same both at work and at home. I am a chef newb so perhaps there is something I am not understanding about the order in which the resources are run within my recipe? I would expect them to run in sequential order...
I also tried putting a notifies call inside the directory block, where I call another execute block :immediately. That works, but inside the second execute block I test to see whether /downloads has been created and it hasn't.
Clearly I'm missing something very basic.
Chef has two phases of execution: a compile phase and a converge phase.
In the compile phase, any resource declarations you write (like directory) are compiled but not executed. Any bare Ruby code you write is also executed at this time. In the converge phase, any compiled resources are then converged in a test-and-set operation.
I'm going to assume that at work, you already have a /downloads directory pre-existing, so the resource is a no-op during converge, and the raise doesn't happen during compile.
If you want arbitrary Ruby code to execute at converge time, put it in a ruby_block resource.
My goal is to run a bash script automatically whenever any new file is added to a particular directory or any subdirectory of that particular directory.
Detail Scenario:
I am creating an automated process for file submission from teachers to students and vice versa. Sender will upload file and it will be stored inside the Uploads directory in the LAMP server in the format, ex. "name_course-name_filename.pdf". I want some method so that when any file stored inside the Uploads folder, the same time a script will be called and send that file to the list of receives.
From the database I can find the list of receiver for that particular course and student.
The only concern of mine is, how to call a script automatically and make it work on individual file whenever the content of the directory changes. Cron will do in intervals but not a real time work.
Linux provides a nice mechanism for that purpose which is called inotify. inotify is mostly available as a C API. But there have been developed shell utilities as well. You should use inotifywait from inotifytools (pkg name in debian) for this. Here comes a basic example:
#!/bin/bash
directory="/tmp" # or whatever you are interested in
inotifywait -m -e create "$directory" |
while read folder eventlist eventfile
do
echo "the following events happened in folder $folder:"
echo "$eventlist $eventfile"
done
Update:
If the problem goes complicated, for example you'll have to monitor recursive, dynamic directory structures, you should have a look at incron It's a cron like daemon which executes scripts on certain events. But the events are file system events rather than timer events.
There is another option to 'inotifywait':
-d --daemon
Same as --monitor, except run in the background logging events to a file
that must be specified by --outfile. Implies --syslog.
For completeness:
-m --monitor
Instead of exiting after receiving a single event, execute indefinitely.
The default behaviour is to exit after the first event occurs.
Within the do-done block of your 'while' statement, you might parse each event report for interesting details then use 'case-esac' to take action based on each event that you care about.
For something that you plan to rely on for your operations, you might also consider replacing the hard-coded '$directory' with some sort of configuration file. Such a file might include the path and filename, the interesting events for that path and file, and a script to run when those events happened.
The script might take the list of events as parameters and then 'case-esac' again.
Just one man's ramblins,
~~~ 8d;-Dan