Case insensitive hash keys perl - linux

Problem
I have a hash/array structure, some of the hash keys are not in the same case though.
I would like to know if there is a way to handle this case without manually checking the keys of every hash.
In the example below i would like all ID/iD/id/Id fields to be printed.
Example code
use warnings;
use strict;
my $Hash = {
Server => [
{
Id=>123
},
{
iD=>456
},
{
ID=>789
}
]
};
for (#{$Hash->{Server}}){
print "$_->{ID}\n"
#This is the problematic part
}
Other
perl version: v5.10.0
This data is recieved from elsewhere and must remain the same case, the example above is minimal and i cannot just simply change them all to the same case.
Any more info needed let me know.

Well, it depends a little bit on your source of information. This looks like you've parsed something, so there may be a better solution.
However, with what we've got here, I'd do it like this:
for my $entry (#{$Hash->{Server}}){
#grep, find first match. Dupes discarded.
my ( $key ) = grep { /^id$/i } keys %$entry;
print "$key => ",$entry -> {$key},"\n";
}
This works by using grep with an i regex for case insensitive on keys, and grabbing whatever comes out first. So if you have multiple matches for /id/i then it'll be random which one you get. (sort could help with that though)
Given you're working with XML though, I'd probably backtrack a bit, throw out XML::Simple and do it like this instead:
#!/usr/bin/perl
use strict;
use warnings;
use XML::Twig;
my $twig = XML::Twig -> new ( twig_handlers => { '_all_' => sub { $_ -> lc_attnames }} );
$twig -> parse ( \*DATA );
print "XML looks like:\n";
$twig -> set_pretty_print ( 'indented_a');
$twig -> print;
print "Output:\n";
foreach my $server ( $twig -> get_xpath('//Server') ) {
print $server -> att('id'),"\n";
}
__DATA__
<XML>
<Server ID="123" />
<Server Id="456" />
<Server id="789" />
</XML>
Or you can just:
foreach my $server ( $twig -> get_xpath('//Server') ) {
$server -> lc_attnames;
print $server -> att('id'),"\n";
}
in lieu of doing it in the twig handlers. The first answer will 'fix' all of your XML to having lower case attributes, which might not be what you want. But then, it might be useful for other scenarios, which is why I've given two examples.

There is no built-in way to do that. What you could do is use List::Util's first to at least get less checks, and then still try until one fits for each of the keys.
use strict;
use warnings;
use feature 'say';
use List::Util 'first';
my $Hash = {
Server => [
{
Id => 123
},
{
iD => 456
},
{
ID => 789
}
]
};
foreach my $thing ( #{ $Hash->{Server} } ) {
# this returns the first match in the list, like grep
# so we need to use it here to return the actual value
say $thing->{ first { $thing->{$_} } qw/id ID iD Id/ };
}
If there are a lot of other keys in the data structure, this is cheaper than looking at all the keys, because you at max look up all possible id keys plus one, and at best two.
If you want the list of possible keys to auto-generate and the uppercase and lowercase letters can be arbitrarily mixed, take a look at this answer.

I would suggest you to use regex to ignore case of keys using i flag.
for my $item ( # { $Hash->{Server} }) {
for(keys %{$item}) {
print $item -> {$_},"\n" if /^ID$/i;
}
}

Related

Construct mutable string from hash iteration

I'm stuck on the following puppet code from days, can you please give me an hand?
In hiera I have the following structure:
my_list::servers:
server1.subnet.env.com:
id: 0
server2.subnet.env.com:
id: 1
server3.subnet.env.com:
id: 2
server4.subnet.env.com:
id: 3
Where the various server1..n (n=>2) are the FQDN of the servers in the specific environment. The ID is always in order, but starting from 0.
I need to create a string that contains a comma separated list of string as broker-${id}-check, where id is different from the FQDN of the server where I'm running puppet, so for example if I'm running the script on server2.subnet.env.com the string should be broker-0-check,broker-2-check,broker-3-check. If I'm running on server1.subnet.env.com it will be broker-1-check,broker-2-check,broker-3-check, etc..
My last tentative is:
$servers_list = hiera('my_list::servers', {"${::fqdn}" => {'id' => 0 } })
$list_broker=''
$servers_list.each |$key, $value| {
if $key != $::fqdn {
$list_broker="${list_broker},broker-${value['id']}-check"
}
}
notify {"******* ${list_broker}": }
but list_broker is still empty and then I will have to fix the leading comma.
Is there a better way to do that?
I'm using Puppet 4.
The problem is that although Puppet has an iteration feature, it doesn't allow the reassignment of variables (ref).
For this reason, Puppet has a lot of functional programming features that allow you to solve problems like this without needing to reassign variables.
This works (Puppet < 5.5), where join() comes from stdlib:
$list_broker = join(
$servers_list
.filter |$key, $value| { $key != $::fqdn }
.map |$key, $value| { "broker-${value['id']}-check" },
','
)
Or in Puppet >= 5.5 (as suggested in comments), where the join command is built-in, the join can be chained too:
$list_broker = $servers_list
.filter |$key, $value| { $key != $::fqdn }
.map |$key, $value| { "broker-${value['id']}-check" }
.join(',')
If you'd prefer it in more steps:
$filtered = $servers_list.filter |$key, $value| { $key != $::fqdn }
$mapped = $filtered.map |$key, $value| { "broker-${value['id']}-check" }
$list_broker = join($mapped, ',')
Explanation:
The filter function selects elements from an Array or Hash on the basis of some criteria.
The map function performs a transformation on all elements of an Array or Hash and returns an Array of transformed data.
The join function (which prior to Puppet 5.5 comes from stdlib) joins an array of strings.

Perl: share complex data-structure between threads

I like to share a complex-datastructure between threads.
As far I know that is not possible with threads:shared (only basic types are shareable).
So I think about serialize/deserialize the structure with JSON or Storable so it is just a string that I can share perfectly. But I need to unpack it before use and pack it after a change.
Is that a common way to work on that problem?
Are there better ways?
Whould you prefer JSON or Storable or something else?
Thanks for help!
EDIT
I just did some tests with Storable and JSON. JSON is quicker and produces smaller serialized strings. I did not expect that.
When dealing with this problem, I use Thread::Queue to pass my objects around, and usually use Storable to serialise.
I haven't bothered doing performance comparisons, because usually my data-passing overhead isn't the limiting factor.
Note - the key advantage of Storable is that it allows some limited object support (Not - be careful - it only works if your object is self contained):
#!/usr/bin/env perl
use strict;
use warnings;
package MyObject;
sub new {
my ( $class, $id ) = #_;
my $self = {};
$self -> {id} = $id;
$self -> {access_count} = 0;
bless $self, $class;
return $self;
}
sub access_thing {
my ( $self ) = #_;
return $self -> {access_count}++;
}
sub get_id {
my ( $self ) = #_;
return $self -> {id};
}
package main;
use threads;
use Thread::Queue;
use Storable qw ( freeze thaw );
my $thread_count = 10;
my $work_q = Thread::Queue -> new;
sub worker {
while ( my $item = $work_q -> dequeue ) {
my $obj = thaw ( $item );
print $obj -> get_id, ": ", $obj -> access_thing,"\n";
}
}
for (1..$thread_count) {
threads -> create (\&worker);
}
for my $id ( 0..1000 ) {
my $obj = MyObject -> new ( $id );
$work_q -> enqueue ( freeze ( $obj ) );
}
$work_q -> end;
$_ -> join for threads -> list;
If JSON would limit you to array/hash data structures - which may be fine for your use case.
Complex data structures can be shared using shared_clone. The components of the data structure need be cloned before being added to it.
use strict;
use feature 'say';
use Data::Dump qw(dd);
use threads;
use threads::shared;
my $cds = {
k1 => shared_clone( { k1_l2 => [ 1..2 ] } ),
k2 => shared_clone( { k2_l2 => [10..11] } )
};
my #threads = map { async(\&proc_ds, $cds->{$_}) } keys %$cds;
$_->join() for #threads;
dd $cds;
sub proc_ds {
my ($ds) = #_;
lock $ds;
push #{$ds->{$_}}, 10+threads->tid for keys %$ds;
}
Note that you don't want to allow autovivification when working with shared values, as it would create unshared (and empty) components in the structure. Check explicitly for existence.
A ready data structure needs to be cloned-and-shared
my $cds = { k => [ 5..7 ] }; # already built, need be shared
my $cds_share = shared_clone( $cds );
my #threads = map { async(\&proc_ds, $cds_share) } 1..3;
$_->join() for #threads;
With the same proc_ds() as above this prints the structure (condensed output)
{ 'k' => [ '5', '6', '7', '11', '12', '13' ] };
When data structure is populated for sharing, as in the first example, then there is less overhead to pay. Otherwise there is a data copy involved, as in the second example, and whether that is OK depends on details (data size, how often a copy is made, etc).
The idea of serializing data is workable as well, but how suitable it is again depends on details since in that case you'd not only copy data but would go to disks as well.
In that case JSON is certainly one good way to go, being a data format that is simple and readable, and can also be shared between tools. The Storable is binary, works directly with Perl data structures, and is supposed to be fast (what should show with larger data).
One other option is to use a worker model and pass data over a message queue. Then you'd use Thread::Queue, or perhaps make use of Thread::Queue::Any, for communication channels.

How do I `say` and `print` into a buffer?

In Perl 6 the Str type is immutable, so it seems reasonable to use a mutable buffer instead of concatenating a lot of strings. Next, I like being able to use the same API regardless if my function is writing to stdout, file or to an in-memory buffer.
In Perl, I can create an in-memory file like so
my $var = "";
open my $fh, '>', \$var;
print $fh "asdf";
close $fh;
print $var; # asdf
How do I achieve the same thing in Perl 6?
There's a minimal IO::String in the ecosystem backed by an array.
For a one-off solution, you could also do someting like
my $string;
my $handle = IO::Handle.new but role {
method print(*#stuff) { $string ~= #stuff.join };
method print-nl { $string ~= "\n" }
};
$handle.say("The answer you're looking for is 42.");
dd $string;
What I currently do is that I wrapped string concatenation in a class as a temporary solution.
class Buffer {
has $!buf = "";
multi method print($string) {
$!buf ~= $string;
}
multi method say($string) {
$!buf ~= $string ~ "\n";
}
multi method Str() {
return $!buf;
}
}
With that, I can do
my $buf = Buffer.new();
say $buf: "asdf";
print $buf.Str;

update Conf File using perl

I am using perl script to update sections in conf file . my script i working properly but when i execute script the ordered on sections changed automatically while i dont want to change the order.script is following .
#!/usr/bin/perl
use Config::Tiny;
$file = 'sss.conf';
my $Config = Config::Tiny->new;
$Config = Config::Tiny->read( $file );
my $rootproperty = $Config->{_}->{rootproperty};
$Config->{amrit}->{host} = 'Not Bar!';
$Config->write( $file );
and file contents following line;
[amrit]
type=friend
host=111.118.253.145
port=2776
username=amrit
secret=password
disallow=all
allow=gsm
context=sip-calling
qualify=yes
call-limit=22
Please help me here i dont want to change order of fields
From the documentation for Config::Tiny:
...Config::Tiny does not preserve your comments, whitespace, or the order of your config file.
See Config::Tiny::Ordered (and possibly others) for the preservation of the order of the entries in the file.
So, do as the documentation suggests, and see Config::Tiny::Ordered.
Update: Based on the comments I'll try to provide additional help here.
First, your existing script is mostly just a copy-paste from the Config::Tiny synopsis, without any deep understanding of what most of it does. The relevant parts... or at least those parts you should keep are:
use Config::Tiny;
$file = 'sss.conf';
$Config = Config::Tiny->read( $file );
$Config->{amrit}->{host} = 'Not Bar!';
$Config->write( $file );
If you add use Data::Dumper; at the top of the script, and immediately after reading the config file you add print Dumper $Config;, the structure would look like this:
{
'amrit' => {
'call-limit' => '22',
'host' => '111.118.253.145',
'secret' => 'password',
'context' => 'sip-calling',
'port' => '2776',
'username' => 'amrit',
'allow' => 'gsm',
'qualify' => 'yes',
'type' => 'friend',
'disallow' => 'all'
}
}
Modifying the structure with the script I've posted above would work if you don't mind the key/value pairs to have their orders rearranged. But you need to preserve order. So the suggestion was to switch to Config::Tiny::Ordered. That module preserves order by rearranging the structure differently. If you change the script to look like this:
use Data::Dumper;
use Config::Tiny::Ordered;
$file = 'conf.conf';
$Config = Config::Tiny->read( $file );
print Dumper $Config;
You will see that the structure now looks like this:
{
'amrit' =>
[
{
'value' => 'friend',
'key' => 'type'
},
{
'key' => 'host',
'value' => '111.118.253.145'
},
{
'value' => '2776',
'key' => 'port'
},
{
'value' => 'amrit',
'key' => 'username'
},
{
'value' => 'password',
'key' => 'secret'
},
{
'value' => 'all',
'key' => 'disallow'
},
{
'key' => 'allow',
'value' => 'gsm'
},
{
'key' => 'context',
'value' => 'sip-calling'
},
{
'key' => 'qualify',
'value' => 'yes'
},
{
'value' => '22',
'key' => 'call-limit'
}
]
}
Or in other words, the internal structure of the Config::Tiny::* object has changed from a hash of hashes, to a hash of arrays of hashes. ("All problems in computer science can be solved by another level of indirection" -- David Wheeler) This change in the shape of the datastructure exists to move away from the problem of hashes being unordered containers.
So now instead of convenient hash lookups for the key named "host", you have to iterate through your structure to find the array element that has an anonymous hash with a key field named host. More work:
use List::Util qw(first);
use Config::Tiny::Ordered;
$file = 'sss.conf';
$Config = Config::Tiny::Ordered->read( $file );
my $want_ix = first {
$Config->{amrit}[$_]{key} eq 'host'
} 0 .. $#{$Config->{amrit}};
$Config->{amrit}[$want_ix]{value} = 'Not Bar!';
$Config->write( $file );
This works by running through the amrit section of the config file looking for the element in the structure's anonymous array that has an anonymous hash with a field named 'key', and once that array element is found, modifying the 'value' hash element to 'Not Bar!'
If this is a one time thing for you, you're done. If you want to be able to do it yourself next time, read perldoc perlreftut, perldoc perldsc, as well as documentation for any of the functions used herein that aren't immediately clear.
From the CPAN page:
Lastly, Config::Tiny does not preserve your comments, whitespace, or the order of your config file.
Use COnfig::Tiny::Ordered instead
See Config::Tiny::Ordered (and possibly others) for the preservation of the order of the entries in the file.

Why does a shared perl hash not share updates with other threads?

I'm trying to make a web server whose requests are farmed out to a set of interpreters hidden behind open2(), based on which 'device' is indicated in the cgi parameters.
The trouble is, I want it multi-threaded but the hash I'm using to try to keep track of the event queue relating to each device doesn't remember the new device created for each request: the server below only prints this sort of thing:
Did not find default-device in (alreadyThere)...
Added default-device with Sun Oct 27 20:43:35 2013 to alreadyThere, default-device
Now... does (alreadyThere, default-device) persist for the next request?
Here is the script:
#!/usr/bin/perl -w
use strict;
use threads;
use threads::shared;
use base qw(Net::Server::HTTP);
our $monkeys = shared_clone({ alreadyThere => { 'a' => 'b' } });
sub process_http_request {
require CGI;
my $cgi = CGI->new;
my $device = $cgi->param('device') || 'default-device';
print "HTTP/1.0 200 OK\r\nContent-type: text/html\r\n\r\n<pre>";
unless (exists $monkeys->{$device}) {
print "Did not find $device in (".join(", ", sort keys %$monkeys).")...\n";
lock $monkeys;
unless (exists $monkeys->{$device}) {
my $t = localtime;
$monkeys->{$device} = $t;
print "\nAdded $device with ".$t." to ".join(", ", sort keys %$monkeys);
} else {
print "\nSurprise device... ".$device;
}
} else {
print "\nFound device... ".$device;
}
print "\nNow... does (".join(", ", sort keys %$monkeys).") persist for the next request?</pre>";
}
__PACKAGE__->run(port => 8080);
It's not the $t bit - that was previously shared_clone({ id => $t }), but I'm darned if I can see why $monkeys never seems to update.
The different requests are served by different processes, not threads.
Net::Server doesn't have a multi-threaded "personality"[1], so you're going to have to use a different sharing mechanism.
Notes:
"in the near future, we would like to add a 'Thread' personality"
Building on Ikegami's answer, I'm trying with this additional code to fake a 'threaded' personality with some success (and some problems with 'open3' misbehaving):
sub default_server_type { 'Single' }
sub loop {
my $self = shift;
while( $self->accept ){
async {
$self->run_client_connection;
};
last if $self->done;
}
}
a) Is there any reason to use Net::Server::HTTP instead of the higher level and easier to use Plack?
b) I've had to solve a problem not unlike this one recently, and settled on using event-based httpd with AnyEvent (or higher abstraction, Coro). There's Net::Server::Coro if you need a drop-in replacement for your code, or even a plethora of canned AnyEvent-based httpds like Twiggy, Feersum, etc.

Resources