I'm using puppet 5 and writing a module which refers to some hiera which has some duplication in it (example below - see gpgkey):
profile::example1:
repo1:
descr: Centos repo
gpgkey: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
repo2:
descr: Centos repo
gpgkey: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
repo3:
descr: puppet repo
gpgkey: http://puppet.repo/GPG-KEY
I've successfully managed to retrieve the repo names (repo1, repo2 and repo3 in this example). What I'd like to do next is run an exec to import the gpgkey, however this is failing with a duplicate declaration error (I assume because the gpgkey is duplicated in the hiera). Any ideas or help on how to do this? Here's the lookup:
$repo_name = lookup('profile::example1', Hash, 'deep')
$repo_name.each | $name, Hash $config_hash | {
notify { "${name}": }
}
I've looked at embedding another loop to pull values from $config_hash but I usually get the same result regardless.
Puppet's built-in unique() function seems natural for this job. For example:
$repos = lookup('profile::example1', Hash, 'deep')
$unique_keys = $repos.map |$_name, $props| { $props['gpgkey'] } .unique
$unique_keys.each |$key| {
# ...
}
Related
I have this issue
*********** codeception.yml ***************
paths:
tests: tests
output: tests/_output
data: tests/_data
support: tests/_support
envs: tests/_envs
actor_suffix: Tester
#amount customer per product
amountPerProduct: 1
wishCountry: 1
wishProduct: 0
I am using the param like this:
$countryIndex = Configuration::config()['wishCountry'];
but on the console I calling the test like this:
codecept run tests/acceptance/ChangeCest.php --env chrome --xml --html -o "wishProduct:55"
I get this error:
enter image description here
QUESTION: how can I override this config in Gitlab?
Thank you
Not entirely sure if this solves your problem but for Codeception v5 (also my current local version v4.2.1) the option is exploded with : , i.e. with an appended space. I notice in your screenshot the exception does show the key/value without the space in between, but you didn't specify which version of Codeception you're using.
From Config.php:
foreach ($configOptions as $option) {
$keys = explode(': ', $option);
// ^ notice the space here
if (count($keys) < 2) {
throw new \InvalidArgumentException('--override should have config passed as "key: value"');
}
https://github.com/Codeception/Codeception/blob/5.0/src/Codeception/Command/Shared/ConfigTrait.php
i find the answer to my problem.
I store the parameters in a new file params.yml
params.yml
parameters:
WISH_PRODUCT: 55
codeception.yml
params:
- params.yml
acceptance.suite.yml
config:
custom_params:
wishProduct: %WISH_PRODUCT%
AcceptanceTester.php
so in AcceptanceTester I can read the values like this
$custom_params = Configuration::suiteSettings('acceptance', Configuration::config())['modules']['custom_params'];
Terraform v0.10.7
AWS provider version = "~> 1.54.0"
Are there any examples how to translate a string or list into a map in Terraform?
We are setting up Consul key/value store like this:
consul kv put common/rules/alb/service1 name=service1,port=80,hcproto=http,hcport=80
I can access keys and values properly, and now I am trying to use values as a map in Terraform:
data "consul_key_prefix" "common" {
path_prefix = "common/rules"
}
output "common"{
value = "${jsonencode(lookup(var.CommonRules,element(keys(var.CommonRules),1))) }"
}
$ terraform output
common = "{name=service1,port=80,hcproto=http,hcport=80}"
But when I try to access it as a map, it doesn't work:
output "common"{
value = "${lookup(jsonencode(lookup(var.CommonRules,element(keys(var.CommonRules),1))),"name") }"
}
$ terraform output
(no response)
I tried few things here - e.g. splitting these values and joining them again into a list, and then running "map" function but it doesn't work either:
$ terraform output
common = [
name,
service1,
port,
80,
hcproto,
http,
hcport,
80
]
and then trying to create map of that list:
output "common2" {
value = "${map(split(",",join(",",split("=",lookup(var.CommonRules,element(keys(var.CommonRules),1))))))}"
}
but it doesn't work either.
So my question would be - does anyone has working example where he did translated string (or list) into a map?
Thanks in advance.
jsondecode function in upcoming Terraform v0.12 would be the tool to solve this problem.
jsondecode function github issue
I was reading the Origen documentation on remotes and had a question. When do the remote files get retrieved relative to the Origen callbacks? The reason i ask is that the files we want to retrieve would be used to construct our DUT model and there are some order dependencies.
I have tried all of the existing callbacks, in an attempt to configure the remotes, with no success.
def pre_initialize
Origen.app.config.remotes = [
{
dir: 'product',
vault: 'ssh://git#stash.com/myproduct/origen.git',
version: '0.1.0'
}
]
end
If I add the remotes config to the application file it works:
config.remotes = [
{
dir: 'product',
vault: 'ssh://git#stash.com/myproduct/origen.git',
version: '0.1.0'
}
]
The problem with using the config/application.rb file is that we can't keep product specific information anywhere in our application space. We use symbolic links to map to source files that are stored in test program stash repositories. I think we may need a new callback, please advise.
thx
** EDIT **
So I defined the remotes in another file and call the method to actually do it in the boot.rb file. Then I placed the remote manager require method in the on_create callback but got no remotes fetched.
284: def on_create
285: binding.pry
=> 286: Origen.remote_manager.require!
287: end
[1] pry(#<PPEKit::Product>)> Origen.config.remotes
=> [{:dir=>"remote_checkout", :rc_url=>"ssh://git#stash.us.com:7999/osdk/us-calendar.git", :version=>"v0.1.0"}]
[2] pry(#<PPEKit::Product>)>
origen(main):001:0>
It seems like the Origen.remote_manager.require! is not working. So I checked the remotes manager file and don't see how the require! method could ever work with a callback because it seems to be checking the remotes are dirty, which can never happen for a remote definition that was set after the application.rb file was loaded. So I created a resolve_remotes! method that seems to work:
def resolve_remotes!
resolve_remotes
remotes.each do |_name, remote|
dir = workspace_of(remote)
rc_url = remote[:rc_url] || remote[:vault]
tag = Origen::VersionString.new(remote[:version])
version_file = dir.to_s + '/.current_version'
begin
if File.exist?("#{dir}/.initial_populate_successful")
FileUtils.rm_f(version_file) if File.exist?(version_file)
rc = RevisionControl.new remote: rc_url, local: dir
rc.checkout version: prefix_tag(tag), force: true
File.open(version_file, 'w') do |f|
f.write tag
end
else
rc = RevisionControl.new remote: rc_url, local: dir
rc.checkout version: prefix_tag(tag), force: true
FileUtils.touch "#{dir}/.initial_populate_successful"
File.open(version_file, 'w') do |f|
f.write tag
end
end
rescue Origen::GitError, Origen::DesignSyncError => e
# If Git failed in the remote, its usually easy to see what the problem is, but now *where* it is.
# This will prepend the failing remote along with the error from the revision control system,
# then rethrow the error
e.message.prepend "When updating remotes for #{remote[:importer].name}: "
raise e
end
end
end
The resolve_remotes! method just forces all known remotes to be fetched. Would a PR be accepted for this solution?
thx
Remotes are currently required at application load time, which means it occurs before any of the application call back points.
The content of config.remotes can still be made dynamic by assigning it in a block:
config.remotes do
r = [{
dir: 'product',
vault: 'ssh://git#stash.com/myproduct/origen.git',
version: '0.1.0'
}]
if some_condition
r << { dir: ... }
end
r
end
The config.remotes attribute will be evaluated before the target is loaded however, so you won't be able to reference dut for example, though maybe that is good enough.
Alternatively, you could implement a post target require of the remotes within your application pretty easily.
Make the remotes return en empty array if the dut is not available yet, that will make it work ok when it is called by Origen during application load:
config.remotes do
if dut
# As above example
else
[]
end
end
Then within the callback handler of your choice:
Origen.remote_manager.require!
That should cause it to re-evaluate config.remotes and fetch any remotes that are missing.
How can I get all the keys set in my memcached instance(s)?
I tried googling, but didn't find much except that PHP supports a getAllKeys method, which means it is actually possible to do this somehow. How can I get the same within a telnet session?
I have tried out all the retrieval related options mentioned in memcached cheat sheet and Memcached telnet command summary, but none of them work and I am at a loss to find the correct way to do this.
Note: I am currently doing this in development, so it can be assumed that there will be no issues due to new keys being set or other such race conditions happening, and the number of keys will also be limited.
Found a way, thanks to the link here (with the original google group discussion here)
First, Telnet to your server:
telnet 127.0.0.1 11211
Next, list the items to get the slab ids:
stats items
STAT items:3:number 1
STAT items:3:age 498
STAT items:22:number 1
STAT items:22:age 498
END
The first number after ‘items’ is the slab id. Request a cache dump for each slab id, with a limit for the max number of keys to dump:
stats cachedump 3 100
ITEM views.decorators.cache.cache_header..cc7d9 [6 b; 1256056128 s]
END
stats cachedump 22 100
ITEM views.decorators.cache.cache_page..8427e [7736 b; 1256056128 s]
END
memdump
There is a memcdump (sometimes memdump) command for that (part of libmemcached-tools), e.g.:
memcdump --servers=localhost
which will return all the keys.
memcached-tool
In the recent version of memcached there is also memcached-tool command, e.g.
memcached-tool localhost:11211 dump | less
which dumps all keys and values.
See also:
What's the simplest way to get a dump of all memcached keys into a file?
How do I view the data in memcache?
Base on #mu 無 answer here. I've written a cache dump script.
The script dumps all the content of a memcached server. It's tested with Ubuntu 12.04 and a localhost memcached, so your milage may vary.
#!/usr/bin/env bash
echo 'stats items' \
| nc localhost 11211 \
| grep -oe ':[0-9]*:' \
| grep -oe '[0-9]*' \
| sort \
| uniq \
| xargs -L1 -I{} bash -c 'echo "stats cachedump {} 1000" | nc localhost 11211'
What it does, it goes through all the cache slabs and print 1000 entries of each.
Please be aware of certain limits of this script i.e. it may not scale for a 5GB cache server for example. But it's useful for debugging purposes on a local machine.
If you have PHP & PHP-memcached installed, you can run
$ php -r '$c = new Memcached(); $c->addServer("localhost", 11211); var_dump( $c->getAllKeys() );'
Bash
To get list of keys in Bash, follow the these steps.
First, define the following wrapper function to make it simple to use (copy and paste into shell):
function memcmd() {
exec {memcache}<>/dev/tcp/localhost/11211
printf "%s\n%s\n" "$*" quit >&${memcache}
cat <&${memcache}
}
Memcached 1.4.31 and above
You can use lru_crawler metadump all command to dump (most of) the metadata for (all of) the items in the cache.
As opposed to cachedump, it does not cause severe performance problems and has no limits on the amount of keys that can be dumped.
Example command by using the previously defined function:
memcmd lru_crawler metadump all
See: ReleaseNotes1431.
Memcached 1.4.30 and below
Get list of slabs by using items statistics command, e.g.:
memcmd stats items
For each slub class, you can get list of items by specifying slub id along with limit number (0 - unlimited):
memcmd stats cachedump 1 0
memcmd stats cachedump 2 0
memcmd stats cachedump 3 0
memcmd stats cachedump 4 0
...
Note: You need to do this for each memcached server.
To list all the keys from all stubs, here is the one-liner (per one server):
for id in $(memcmd stats items | grep -o ":[0-9]\+:" | tr -d : | sort -nu); do
memcmd stats cachedump $id 0
done
Note: The above command could cause severe performance problems while accessing the items, so it's not advised to run on live.
Notes:
stats cachedump only dumps the HOT_LRU (IIRC?), which is managed by a background thread as activity happens. This means under a new enough version which the 2Q algo enabled, you'll get snapshot views of what's in just one of the LRU's.
If you want to view everything, lru_crawler metadump 1 (or lru_crawler metadump all) is the new mostly-officially-supported method that will asynchronously dump as many keys as you want. you'll get them out of order but it hits all LRU's, and unless you're deleting/replacing items multiple runs should yield the same results.
Source: GH-405.
Related:
List all objects in memcached
Writing a Redis client in pure bash (it's Redis, but very similar approach)
Check other available commands at https://memcached.org/wiki
Check out the protocol.txt docs file.
The easiest way is to use python-memcached-stats package, https://github.com/abstatic/python-memcached-stats
The keys() method should get you going.
Example -
from memcached_stats import MemcachedStats
mem = MemcachedStats()
mem.keys()
['key-1',
'key-2',
'key-3',
... ]
I was using Java's spyMemcached, and used this code. It is based on Anshul Goyal's answer
#Autowired
#Qualifier("initMemcachedClient")
private MemcachedClient memcachedClient;
public List<String> getCachedKeys(){
Set<Integer> slabIds = new HashSet<>();
Map<SocketAddress, Map<String, String>> stats;
List<String> keyNames = new ArrayList<>();
// Gets all the slab IDs
stats = memcachedClient.getStats("items");
stats.forEach((socketAddress, value) -> {
System.out.println("Socket address: "+socketAddress.toString());
value.forEach((propertyName, propertyValue) -> {
slabIds.add(Integer.parseInt(propertyName.split(":")[1]));
});
});
// Gets all keys in each slab ID and adds in List keyNames
slabIds.forEach(slabId -> {
Map<SocketAddress, Map<String, String>> keyStats = memcachedClient.getStats("cachedump "+slabId+" 0");
keyStats.forEach((socketAddress, value) -> {
value.forEach((propertyName, propertyValue) -> {
keyNames.add(propertyName);
});
});
});
System.out.println("number of keys: "+keyNames.size());
return keyNames;
}
Java Solution:
Thanks! #Satvik Nema
Your solution helped me to find the approach, but it doesn't work for memcached 2.4.6 version. (implementation 'com.googlecode.xmemcached:xmemcached:2.4.6')
Not sure when did new method getStatsByItem included.
I figured out required changes using documentation and below code worked for me.
// Gets all the slab IDs
Set<Integer> slabIds = new HashSet<>();
Map<InetSocketAddress, Map<String, String>> itemsMap = null;
try {
itemsMap = this.memcachedClient.getStatsByItem("items");
} catch (Exception e) {
log.error("Failed while pulling 'items'. ERROR", e);
}
if (Objects.nonNull(itemsMap)) {
itemsMap.forEach((key, value) -> {
log.info("itemsMap {} : {}", key, value);
value.forEach((k, v) -> {
slabIds.add(Integer.parseInt(k.split(":")[1]));
});
});
}
// Gets all keys in each slab ID and adds in List keyNames
slabIds.forEach(slabId -> {
Map<InetSocketAddress, Map<String, String>> keyStats = null;
try {
keyStats = this.memcachedClient.getStatsByItem("cachedump " + slabId + " 0");
} catch (Exception e) {
log.error("Failed while pulling 'cachedump' for slabId: {}. ERROR", slabId, e);
}
if (Objects.nonNull(keyStats)) {
keyStats.forEach((socketAddress, value) -> {
value.forEach((propertyName, propertyValue) -> {
//keyNames.add(propertyName);
log.info("keyName: {} Value: {}", propertyName, propertyValue);
});
});
}
});
Can you think of a way to solve this problem in Puppet?
I have a custom fact with generates a string of IP addresses depending on the domain it is run on, it can resolve to have 1 to n addresses.
"10.1.29.1"
"10.1.29.1,10.1.29.5"
"10.1.29.1,10.1.29.5,10.1.29.7"
etc
I want to add these to the host file with a generated server names of servernameX for example;
10.1.29.1 myservername1
10.1.29.5 myservername2
10.1.29.7 myservername3
So how can you do this as puppet doesn't have an array iterator like "for each"?
Sadly, even if you go about and use a custom "define" to iterate over an array upon splitting your custom fact on a comma, the result will be rather not what you expect and not even close to a "for each" loop -- aside of causing you a headache, probably.
Said that, I am not sure if this is what you want to achieve, but have a look at this approach:
$fact = '1.1.1.1,2.2.2.2,3.3.3.3'
$servers = split($::fact, ',')
$count = size($servers)
$names = bracket_expansion("host[01-${count}].address")
file { '/tmp/test.txt':
content => inline_template('<%= #servers.each_with_index.map {|v,i| "#{v}\t\t#{#names[i]}\n" } %>'),
ensure => present
}
What we have there are two custom functions: size() and bracket_expansion(); which we then use values that they provide inside a hack that leverages the inline_template() function to render content of the file utilising parallel access to two arrays -- one with IP addresses from your fact and one with host names that should follow.
The result is a follows:
matti#acrux ~ $ cat | puppet apply
$fact = '1.1.1.1,2.2.2.2,3.3.3.3'
$servers = split($::fact, ',')
$count = size($servers)
$names = bracket_expansion("host[01-${count}].address")
file { '/tmp/test.txt':
content => inline_template('<%= #servers.each_with_index.map {|v,i| "#{v}\t\t#{#names[i]}\n" } %>'),
ensure => present
}
notice: /Stage[main]//File[/tmp/test.txt]/ensure: created
notice: Finished catalog run in 0.07 seconds
matti#acrux ~ $ cat /tmp/test.txt
1.1.1.1 host01.address
2.2.2.2 host02.address
3.3.3.3 host03.address
matti#acrux ~ $
Both size() and bracket_expansion() functions can be found here:
https://github.com/kwilczynski/puppet-functions/tree/master/lib/puppet/parser/functions/
I hope this helps a little :-)