Related
In this code two keys "39" are same name but different values , I want to print both keys
use strict;
use warnings;
my %studentnames = (
14 => Martha,
27 =>Vivek,
31 =>Era,
16 =>Marty,
25 =>Jason,
29 =>Socrates,
19 =>Uri,
39 =>Nitin ,
39 =>Plato,
);
foreach my $name (sort keys %studentnames)
{
printf "%-8s %s\n", $name, $studentnames{$name};
}
I am getting error.
Bareword "Martha" not allowed while "strict subs" in use at /home/9945b48d30946ed2641d9778b42cb182.pl line 10.
Bareword "Vivek" not allowed while "strict subs" in use at /home/9945b48d30946ed2641d9778b42cb182.pl line 10.
Bareword "Era" not allowed while "strict subs" in use at /home/9945b48d30946ed2641d9778b42cb182.pl line 10.
Bareword "Marty" not allowed while "strict subs" in use at /home/9945b48d30946ed2641d9778b42cb182.pl line 10.
Bareword "Jason" not allowed while "strict subs" in use at /home/9945b48d30946ed2641d9778b42cb182.pl line 10.
Bareword "Socrates" not allowed while "strict subs" in use at /home/9945b48d30946ed2641d9778b42cb182.pl line 10.
Bareword "Uri" not allowed while "strict subs" in use at /home/9945b48d30946ed2641d9778b42cb182.pl line 10.
Bareword "Nitin" not allowed while "strict subs" in use at /home/9945b48d30946ed2641d9778b42cb182.pl line 10.
Bareword "Plato" not allowed while "strict subs" in use at /home/9945b48d30946ed2641d9778b42cb182.pl line 10.
Expected Output
14 Martha
27 Vivek
31 Era
16 Marty
25 Jason
29 Socrates
19 Uri
39 Nitin
39 Plato
Can anyone tell me how to do it?
Two keys cannot be the same. One will overwrite the other. If you want to have multiple values for one key then you need to design your data structure to support that (e.g. by having the value be an arrayref).
Your error messages are unrelated to that problem (you forgot to put quotes around your string values).
This is kinda close:
use strict;
use warnings;
use Tie::Hash::MultiValueOrdered;
tie my %studentnames, 'Tie::Hash::MultiValueOrdered';
%studentnames = (
14 => 'Martha',
27 => 'Vivek',
31 => 'Era',
16 => 'Marty',
25 => 'Jason',
29 => 'Socrates',
19 => 'Uri',
39 => 'Nitin',
39 => 'Plato',
);
tied(%studentnames)->fetch_list;
while ( my ( $key, $value ) = each %studentnames ) {
print "$key => #$value\n";
}
But really you want to use a different data structure. Perhaps an array of arrayrefs?
use strict;
use warnings;
my #students = (
[ 14 => 'Martha' ],
[ 27 => 'Vivek' ],
[ 31 => 'Era' ],
[ 16 => 'Marty' ],
[ 25 => 'Jason' ],
[ 29 => 'Socrates' ],
[ 19 => 'Uri' ],
[ 39 => 'Nitin' ],
[ 39 => 'Plato' ],
);
for my $student ( #students ) {
my ( $num, $name ) = #$student;
print "$num => $name\n";
}
Or an array of hashrefs:
use strict;
use warnings;
my #students = (
{ num => 14 , name => 'Martha' },
{ num => 27 , name => 'Vivek' },
{ num => 31 , name => 'Era' },
{ num => 16 , name => 'Marty' },
{ num => 25 , name => 'Jason' },
{ num => 29 , name => 'Socrates' },
{ num => 19 , name => 'Uri' },
{ num => 39 , name => 'Nitin' },
{ num => 39 , name => 'Plato' },
);
for my $student ( #students ) {
print "$student->{num} => $student->{name}\n";
}
Or a hash of arrayrefs:
use strict;
use warnings;
my %students = (
14 => [ 'Martha' ],
27 => [ 'Vivek' ],
31 => [ 'Era' ],
16 => [ 'Marty' ],
25 => [ 'Jason' ],
29 => [ 'Socrates' ],
19 => [ 'Uri' ],
39 => [ 'Nitin', 'Plato' ],
);
for my $key ( sort keys %students ) {
for my $name ( #{$students{$key}} ) {
print "$key => $name\n";
}
}
Or you could even create a lightweight "person" class.
use Z;
my $Person = class sub {
has num => ( type => PositiveInt );
has name => ( type => NonEmptyStr );
};
my #students = (
$Person->new( num => 14, name => 'Marta' ),
$Person->new( num => 27, name => 'Vivek' ),
$Person->new( num => 31, name => 'Era' ),
$Person->new( num => 16, name => 'Marty' ),
$Person->new( num => 25, name => 'Jason' ),
$Person->new( num => 29, name => 'Socrates' ),
$Person->new( num => 19, name => 'Uri' ),
$Person->new( num => 39, name => 'Nitin' ),
$Person->new( num => 39, name => 'Plato' ),
);
for my $student ( #students ) {
printf "%s => %s\n", $student->num, $student->name;
}
There's a lot of ways to go about solving this, but a single flat hash of strings is probably not one of them.
As a starter: the hash values are strings, so they need to be quoted. This is why you are getting a syntax error:
my %studentnames = (
14 => 'Martha',
27 => 'Vivek',
31 => 'Era',
...
);
Then: there is a misconception of what a Perl hash is. Each key in the hash must be unique. Perl tolerates declaring a hash with duplicate keys, but under the hood, only the last value of each key is retained.
So this:
my %studentnames = (
14 => 'Martha',
39 => 'Nitin',
39 => 'Plato'
);
Is equivalent to:
my %studentnames = (
14 => 'Martha',
39 => 'Plato'
);
Another way to see it is to put the assignments in separate instructions:
my %studentnames;
$studentnames{14} = 'Martha';
$studentnames{39} = 'Nitin';
$studentnames{39} = 'Plato';
print $studentnames{39}, "\n";
# Plato
In addition to excellent answers provided by others, and explanations of how to make the code work (e.g., quote the names, etc), here is another simple solution. I assume that the student names are unique (as they indeed appear in your example). In this case, use the reverse of your hash. That is, change it from the mapping of number to name to the mapping of name to number. Sort the hash by values (student numbers) numerically, then by keys (student names) ASCIIbetically. This is my guess as to one possible way of sorting that makes intuitive sense to the user.
#!/usr/bin/env perl
use strict;
use warnings;
my %student_name_to_num =
reverse (
14 => 'Martha',
27 => 'Vivek',
31 => 'Era',
16 => 'Marty',
25 => 'Jason',
29 => 'Socrates',
19 => 'Uri',
39 => 'Nitin',
39 => 'Plato',
);
foreach my $name ( sort {
$student_name_to_num{$a} <=> $student_name_to_num{$b} ||
$a cmp $b
} keys %student_name_to_num ) {
printf "%-8s %s\n", $student_name_to_num{$name}, $name;
}
Output:
14 Martha
16 Marty
19 Uri
25 Jason
27 Vivek
29 Socrates
31 Era
39 Nitin
39 Plato
Note that the order of the records is different from the sorted order you showed. But it is not clear how you want the records to be sorted (see also my comment under the question).
I have written the logstash config file to upload a csv, csv has multiple applicant informations, I need to upload as array of dictionary in the kibana index instead of being a dictionary of dict with index.
filter {
csv {
separator => ","
skip_header => true
columns => [LoanID,Applicant_Income1,Occupation1,Time_At_Work1,Date_Of_Join1,Gender,LoanAmount,Marital_Status,Dependents,Education,Self_Employed,Applicant_Income2,Occupation2,Time_At_Work2,Date_Of_Join2,Applicant_Income3,Occupation3,Time_At_Work3,Date_Of_Join3]
}
mutate {
convert => {
"Applicant_Income1" => "float"
"Time_At_Work1" => "float"
"LoanAmount" => "float"
"Applicant_Income2" => "float"
"Time_At_Work2" => "float"
"Applicant_Income3" => "float"
"Time_At_Work3" => "float"
}
}
mutate{
rename => {
"Applicant_Income1" => "[Applicant][0][Applicant_Income]"
"Occupation1" => "[Applicant][0][Occupation]"
"Time_At_Work1" => "[Applicant][0][Time_At_Work]"
"Date_Of_Join1" => "[Applicant][0][Date_Of_Join]"
"Applicant_Income2" => "[Applicant][1][Applicant_Income]"
"Occupation2" => "[Applicant][1][Occupation]"
"Time_At_Work2" => "[Applicant][1][Time_At_Work]"
"Date_Of_Join2" => "[Applicant][1][Date_Of_Join]"
"Applicant_Income3" => "[Applicant][2][Applicant_Income]"
"Occupation3" => "[Applicant][2][Occupation]"
"Time_At_Work3" => "[Applicant][2][Time_At_Work]"
"Date_Of_Join3" => "[Applicant][2][Date_Of_Join]"
}
}
date {
match => [ "Date_Of_Join1", "yyyy-MM-dd'T'HH:mm:ss.SSZZ" ]
}
date {
match => [ "Date_Of_Join2", "yyyy-MM-dd'T'HH:mm:ss.SSZZ" ]
}
date {
match => [ "Date_Of_Join3", "yyyy-MM-dd'T'HH:mm:ss.SSZZ" ]
}
}
I got the Applicant field as
But I need the Applicant field to be an array of dictionaries, like
I tried add_field, but not working
mutate{
add_field => { "[Applicant][Applicant_Income1]" => "Applicant_Income1",
"[Applicant][Occupation1]" => "Occupation1",
"[Applicant][Time_At_Work1]" => "Time_At_Work1",
"[Applicant][Date_Of_Join1]" => "Date_Of_Join1"
}
}
The square brackets in Logstash Filters do not behave like array elements/entries as in other programming languages, e.g. Java.
[Applicant][0][Applicant_Income]
is not the right syntax to set the value of field Applicant_Income of the first element (zero-based index) in the Applicant-Array. Instead, you create sub-elements 0, 1, 2 underneath the Applicant-element as shown in Figure 1.
To create an array of objects, you should use the ruby filter plugin (https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html). Since you can execute arbitrary ruby code with that filter, it gives you more control/freedom:
filter {
csv {
separator => ","
skip_header => true
columns => [LoanID,Applicant_Income1,Occupation1,Time_At_Work1,Date_Of_Join1,Gender,LoanAmount,Marital_Status,Dependents,Education,Self_Employed,Applicant_Income2,Occupation2,Time_At_Work2,Date_Of_Join2,Applicant_Income3,Occupation3,Time_At_Work3,Date_Of_Join3]
}
mutate {
convert => {
"Applicant_Income1" => "float"
"Time_At_Work1" => "float"
"LoanAmount" => "float"
"Applicant_Income2" => "float"
"Time_At_Work2" => "float"
"Applicant_Income3" => "float"
"Time_At_Work3" => "float"
}
}
ruby{
code => '
event.set("Applicant",
[
{
"Applicant_Income" => event.get("Applicant_Income1"),
"Occupation" => event.get("Occupation1"),
"Time_At_Work" => event.get("Time_At_Work1"),
"Date_Of_Join" => event.get("Date_Of_Join1")
},
{
# next object...
}
]
'
}
date {
match => [ "Date_Of_Join1", "yyyy-MM-dd'T'HH:mm:ss.SSZZ" ]
}
date {
match => [ "Date_Of_Join2", "yyyy-MM-dd'T'HH:mm:ss.SSZZ" ]
}
date {
match => [ "Date_Of_Join3", "yyyy-MM-dd'T'HH:mm:ss.SSZZ" ]
}
mutate{
remove_field => [
"Applicant_Income1",
"Occupation1",
"Time_At_Work1",
"Date_Of_Join1",
"Applicant_Income2",
"Occupation2",
"Time_At_Work2",
"Date_Of_Join2",
"Applicant_Income3",
"Occupation3",
"Time_At_Work3",
"Date_Of_Join3"
]
}
}
With event.set you add a field to the document. The first argument is the fieldname, the second one its value. In this case, you add the field "Applicants" to the document with an array of objects as its value.
event.get is used to get the value of a certain field in the document. You retrieve the value by passing the fieldname to the method.
Please refer to this guide
https://www.elastic.co/guide/en/logstash/current/event-api.html to get more insights of the event API.
I hope I could help you.
I'm having trouble configuring Logstash properly. There are two lines in the postfix logs that I care about:
Jun 14 09:06:22 devmailforwarder postfix/smtp[1994]: A03CA9F532: to=<person#gmail.com>, relay=server[0.0.0.0]:25, delay=0.02, delays=0.01/0.01/0/0.01, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as A0B4D5C49)
Jun 14 09:15:04 devmailforwarder postfix/cleanup[2023]: 0E1969F533: warning: header Subject: subjectline from server[0.0.0.0]; from=<from#gmail.com> to=<to#gmail.com> proto=SMTP helo=<server>
My grok filter patterns are:
POSTFIX_QUEUEID ([0-9A-F]{6,}|[0-9a-zA-Z]{15,})
POSTFIX_STATUS (?<=status=)(.*)(?= \()
POSTFIX_PROCESS (?=postfix\/)(.*?\[)(.*?)(?=: )
POSTFIX_TO (?<=to=<)(.*?)(?=>,)
POSTFIX_RELAY (?<=relay=)(.*?)(?=,)
POSTFIX_SUBJECT (?<=Subject: )(.*)(?= from )
SMTP ^%{SYSLOGTIMESTAMP:timestamp}%{SPACE}%{DATA:hostname}%{SPACE}%{POSTFIX_PROCESS:process}%{GREEDYDATA}%{POSTFIX_QUEUEID:queueid}%{GREEDYDATA}%{POSTFIX_TO:to}%{GREEDYDATA}%{POSTFIX_RELAY:relay}%{GREEDYDATA}%{POSTFIX_STATUS:status}%{SPACE}%{GREEDYDATA:response}
CLEANUP ^%{SYSLOGTIMESTAMP:timestamp}%{SPACE}%{DATA:hostname}%{SPACE}%{POSTFIX_PROCESS:process}:%{SPACE}%{POSTFIX_QUEUEID:queueid}%{GREEDYDATA}%{POSTFIX_SUBJECT:subject}%{GREEDYDATA:something2}
(non-working) Logstash Config is:
input {
file {
path => "/var/log/mail.log*"
exclude => "*.gz"
start_position => "beginning"
type => "postfix"
}
}
filter {
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => { "message" => ["%{SMTP}", "%{SUBJECT}"] }
}
if "_grokparsefailure" in [tags] {
drop {}
}
mutate {
add_field => { "logstashSource" => "source-server" }
}
aggregate {
task_id => "%{POSTFIX_QUEUEID}"
code => "
map['to'] ||= event.get('to')
map['from'] ||= event.get('from')
map['relay'] ||= event.get('relay')
map['status'] ||= event.get('status')
map['response'] ||= event.get('response')
map['from'] ||= event.get('timestamp')
map['relay'] ||= event.get('hostname')
map['status'] ||= event.get('process')
map['response'] ||= event.get('queueid')
map['subject'] ||= event.get('subject')
"
map_action => "create_or_update"
push_previous_map_as_event => true
timeout => 2
timeout_tags => ['aggregated']
}
}
output {
if [type] == "postfix" {
file {
path => "/var/log/logstash/postfix.log"
}
}
}
My goal here would be to have one elasticsearch document where each field is populated. The cleanup messages always come in the logs first. The logs are matched by a unique queue ID. I'm struggling with getting the aggregate piece working.
Solved it. Config below. Also needed to update the logstash.yml to add
pipeline.workers: 1
filter {
grok {
patterns_dir => ["/etc/logstash/conf.d/patterns"]
match => { "message" => ["%{SMTP}", "%{SUBJECT}", "%{CONNECTION}"] }
}
if "_grokparsefailure" in [tags] {
drop {}
}
mutate {
add_field => { "logstashSource" => "logstash-server-name" }
}
if ("" in [queueid]) {
aggregate {
task_id => "%{queueid}"
code => "
map['to'] ||= event.get('to')
map['from'] ||= event.get('from')
map['relay'] ||= event.get('relay')
map['status'] ||= event.get('status')
map['response'] ||= event.get('response')
map['from'] ||= event.get('timestamp')
map['relay'] ||= event.get('hostname')
map['status'] ||= event.get('status')
map['subject'] ||= event.get('subject')
map['queueid'] ||= event.get('queueid')
"
timeout => 2
timeout_tags => ['aggregated']
map_action => 'create_or_update'
push_map_as_event_on_timeout => true
}
}
}
output {
if ("aggregated" in [tags] or "" in [connection])
{
elasticsearch {
index => "postfix-%{+YYYY.MM.dd}"
hosts => "your-es-host-here"
}
file {
path => "/var/log/logstash/postfix.log"
}
}
}
I've got a similar problem. In this case I see a PatternError:
Pipeline error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{SUBJECT} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in `block in compile'
Can you update again your code? yml and pattern please
I'm sure this function is likely common ( or possibly achievable other ways ) but I'm not sure of what it'd be called. I'm thinking of a sliding window of a certain size :-
let slidingMap = (arr,size, f) => {
r = []
arr.reduce((acc, n) => {
let b = acc.concat(n);
if(b.length > size) {
b.shift();
}
if(b.length == size) {
r.push(f(b))
}
return b;
},[])
return r;
}
so given slidingMap([1,2,3,4,5,6], 2, b => b)
you'd get [ [ 1, 2 ], [ 2, 3 ], [ 3, 4 ], [ 4, 5 ], [ 5, 6 ] ]
and slidingMap([1,2,3,4,5,6], 3, b => b)
you'd get [ [ 1, 2, 3 ], [ 2, 3, 4 ], [ 3, 4, 5 ], [ 4, 5, 6 ] ]
or for calculating differences :-
slidingMap([1,2,3,7,5,6],2, b => b.reduceRight((a, n) => a? a-n : n))
you'd get [ 1, 1, 4, -2, 1 ]
or moving average :-
slidingMap([1,2,3,7,5,6],3, b => b.reduce((a, n) => a+n,0)/b.length)
you'd get [ 2, 4, 5, 6 ]
so, is there a commonly implemented function(s) that achieves this?
Update
Probablly better implemented as
let sliding = (arr,size) => {
r = []
arr.reduce((acc, n) => {
let b = acc.concat(n);
if(b.length > size) {
b.shift();
}
if(b.length == size) {
r.push(b)
}
return b;
},[])
return r;
}
then just use map
sliding([1,2,3,4,5],2).map(somefunc);
Or perhaps using zip and skip ( using lodash in this case )
let sliding = (arr, size) =>
_.zip(..._.range(size).map(i => arr.slice(i)))
.filter(a => !a.some(v => v == undefined))
only trick here is the zip will insert undefined when it has no match so they need to be filtered out.
I have the following code:
my $workbook = Excel::Writer::XLSX->new( 'a_simple.xlsx' );
my $worksheet = $workbook->add_worksheet();
my #chart_performance1 = $workbook->add_chart( type => 'column', embedded => 1 );
my $no_of_titles = 3;
for ( my $no_of = 0; $no_of < $no_of_titles; $no_of++ ) {
$chart_performance1[ $no_of ]->add_series(
name => $chart_heading[ 0 ],
categories => [ 'Sheet1', $array_game_titles[ $no_of ] , $row_range_max , 0, 0 ],
values => [ 'Sheet1', $array_game_titles[ $no_of ] , $row_range_max , 1, 1 ],
);
}
When I run it, I get the error:
Can't call method "add_series" on an undefined value
Why?
This line
my #chart_performance1 = $workbook->add_chart( type => 'column', embedded => 1 );
looks wrong. The add_chart method returns a single Excel::Writer::XLSX::Chart object, so the result is normally assigned to a scalar, not an array. It's unclear what you're asking, but if you are trying to create one chart that plots three series of data, then you want something more like:
my $chart = $workbook->add_chart( type => 'column', embedded => 1 );
...
for ( my $no_of = 0; $no_of < $no_of_titles; $no_of++ ) {
$chart->add_series( ... );
}
Got my Answer, Need to only declare the array instead of declaring and defining the array "chart_performance" with add_chart method at the same instance. As the add_chart method returns a single object i was getting the error. Thanks for the help BTW.
my $workbook = Excel::Writer::XLSX->new( 'a_simple.xlsx' );
my $worksheet = $workbook->add_worksheet();
my #chart_performance;
my $no_of_titles = 3;
for ( my $no_of = 0; $no_of < $no_of_titles; $no_of++ ) {
$chart_performance[ $no_of ] = $workbook->add_chart( type => 'column', embedded => 1);
$chart_performance[ $no_of ]->add_series(
name => $chart_heading[ 0 ],
categories => [ 'Sheet1', $array_row[ $no_of ], $array_col[ $no_of ], 0, 0 ],
values => [ 'Sheet1', $array_row[ $no_of ], $array_col[ $no_of ], 1, 1 ],
);
}