I am trying to parse a XML file and want to split. But the split field is not always array.
XML structure:
<A a="1">
<B b="1" q="10">
<C c="1" r="20"></C>
</B>
<B b="2" q="11">
<C c="2" r="21"></C>
<C c="3" r="22">
<M m="1" />
</C>
<B>
</A>
When I try to split as
xml {
source => "message"
target => "doc"
force_array => false
}
split {
field => "doc[B][C]"
}
I get error as Split can be done only in strings or arrays.
How can I mutate the Strings into Array or is there another way to solve this?
Expected output:
{
C => {
c => 1,
r => 20,
}
B => {
b => 1,
q => 10,
}
}
{
C => {
c => 2,
r => 20,
}
B => {
b => 2,
q => 11,
}
}
{
C => {
c => 3,
r => 22,
M => {
m => 1
}
}
B => {
b => 2,
q => 11,
}
}
Each nested acts as a separate event and stored as a different event (row) in elastic search.
Related
This is the filter part in my logstash config file.
filter {
mutate {
split => ["message", "|"]
add_field => {
"start_time" => "%{[message][1]}"
"end_time" => "%{[message][2]}"
"channel" => "%{[message][5]}"
"[range_time][gte]" => "%{[message][1]}"
"[range_time][lte]" => "%{[message][2]}"
# "duration" => "%{[end_time]-[start_time]}"
}
# remove_field => ["message"]
}
date {
match => ["start_time", "yyyyMMddHHmmss"]
target => "start_time"
}
date {
match => ["end_time", "yyyyMMddHHmmss"]
target => "end_time"
}
ruby {
code =>
"
event.set('start_time', event.get('start_time').to_i)
event.set('end_time', event.get('end_time').to_i)
"
}
mutate {
remove_field => ["message", "#timestamp"]
}
ruby {
init => "require 'time'"
code => "event['duration'] = event['end_time'] - event['start_time'];"
}
In the end, I wanna create a new field named duration to represent the difference between end_time and start_time.
Obviously, the last ruby part was wrong. How could I write for this part?
To start you must make sure that the field you want to put the duration in exists prior to your setting of the value.
Make sure you add the field up front.
As it will be numeric you could do it like this
mutate {
add_field {
"duration" => 0
}
}
After this you can calculate the value and set it using ruby
ruby {
code => "event.set('duration', event.get('end_time').to_i - event.get('start_time').to_i)"
}
Given data that represents an enum, such as:
my %enums := {
Color => { red => 0, black => 1, green => 2 },
Status => { fail => 0, pass => 1 }
};
How can I use Metamodel::ClassHOW to create enums equivalent to:
enum Color ( red => 0, black => 1, green => 2 );
enum Status ( fail => 0, pass => 1 );
Timo's ADT library gives an example of how to create a class with ClassHOW, but it doesn't cover enums: https://github.com/timo/ADT/blob/master/lib/ADT.pm6
This seems to do the trick, but it's mostly untested:
my %enums := {
Color => { red => 0, black => 1, green => 2 },
Status => { fail => 0, pass => 1 }
};
my #types = gather {
for %enums.kv -> $name, %values {
my $type = Metamodel::EnumHOW.new_type(:$name, base_type => Int);
for %values -> $pair {
$type.^add_enum_value($pair);
}
$type.^add_role(Enumeration);
$type.^add_role(NumericEnumeration);
$type.^compose;
take $type;
}
}.list;
say #types; # Output: [(Status) (Color)]
Note that this puts the types into a data structure, because lexical scopes are immutable at run time, so you can't declare them just as you would with enum Color ....
I have one custom field in Logstash event defined as expression:
{ "customIndex" => "my-service-%{+YYYY.MM}" }
And filter that calculates index name for elasticsearch output plugin:
filter {
if [customIndex] {
mutate {
add_field => { "indexName" => "custom-%{customIndex}" }
}
} else {
mutate {
add_field => { "indexName" => "common-%{+YYYY.MM.dd}" }
}
}
}
But for custom index it creates invalid name custom-my-service-%{+YYYY.MM} and does not evaluate %{+YYYY.MM} expression.
Is it possible to evaluate field and get custom-my-service-2016.11?
If you can reformat your created field to this:
{ "customIndex" => "my-service-%Y.%m" }
Then this Ruby filter will do the trick:
ruby {
init => "require 'date'"
code => "event['indexName'] = 'custom-' + Date.today.strftime(event['customIndex'])"
}
Here is a documentation on placeholders you can use.
I'm trying to create the menu link programmatically. But its not working where source language is other than english. Here is my code.
$language_list = language_list();
foreach ($language_list as $language_code => $language_object) {
$menu_item = array(
'link_title' => t('Fruit'),
'menu_name' => 'menu-main-footer',
'customized' => 1,
'link_path' => $custom_path,
'language' => $language_code,
'weight' => 30,
);
menu_link_save($menu_item);
}
Any one have some idea on this?
I changed my code. And it work for me.
// Create menu translation set.
$menu_translation_set = i18n_translation_set_create('menu_link');
// Create translated menu link for all site enable language.
$language_list = language_list();
foreach ($language_list as $language_code => $language_object) {
// Add Fruit link in menu-main-footer.
// 'change-fruit' is node title.
$fruit_path = drupal_get_normal_path('change-fruit', $language_code);
if (!menu_link_get_preferred($fruit_path, 'menu-main-footer')) {
$menu_item = array(
'link_title' => t('fruit'),
'menu_name' => 'menu-main-footer',
'customized' => 1,
'link_path' => $fruit_path,
'language' => $language_code,
'weight' => 30,
'i18n_tsid' => $menu_translation_set->tsid,
);
menu_link_save($menu_item);
$menu_translation_set->add_item($menu_item, $language_code);
$menu_translation_set->save();
}
}
May be helpful to other.
I had to migrate an old menu to a new one with its localized translations so here is what I did :
$old_name = 'menu-old';
$new_name = 'menu-new';
$old_menu = menu_load($old_name);
if(isset($old_menu)){
$old_mlids = db_query("SELECT mlid from {menu_links} WHERE menu_name=:menu_name", array(':menu_name' => $old_name))->fetchAll();
if(!empty($old_mlids)){
// Clean existing items in new menu.
$new_mlids = db_query("SELECT mlid from {menu_links} WHERE menu_name=:menu_name", array(':menu_name' => $new_name))->fetchAll();
if(!empty($new_mlids)){
foreach($new_mlids as $record){
menu_link_delete($record->mlid);
}
}
// Copy old to new menu.
foreach($old_mlids as $record){
$old_menu_item = menu_link_load($record->mlid);
$new_menu_item_config = array(
'link_title' => $old_menu_item['link_title'],
'link_path' => $old_menu_item['link_path'],
'menu_name' => $new_name,
'customized' => 1,
'weight' => $old_menu_item['weight'],
'expanded' => $old_menu_item['expanded'],
'options' => $old_menu_item['options'],
);
$new_menu_item = $new_menu_item_config;
menu_link_save($new_menu_item);
// Migrate translations.
$languages = language_list('enabled')[1];
foreach($languages as $lang_code => $language_object){
if ($lang_code == language_default('language')) {
continue;
}
$translation_value = i18n_string_translate('menu:item:'.$old_menu_item['mlid'].':title', $old_menu_item['link_title'], array('langcode' => $lang_code));
if($translation_value != $old_menu_item['link_title']){
i18n_string_translation_update('menu:item:'.$new_menu_item['mlid'].':title', $translation_value, $lang_code, $old_menu_item['link_title']);
}
}
}
}
// Delete old menu.
menu_delete(array('menu_name' => $old_name));
}
I have a running manifest, where I create a folder and a file from a setting (exerpt):
define ffnord::mesh(
$mesh_if_id = "low",
$mesh_mtu_low = 1280,
$fastd_low_port = 11280, # fastd port
) {
ffnord::fastd { "fastd_${mesh_code}":
mesh_if_id => $mesh_if_id,
mesh_mtu_low => $mesh_mtu_low,
fastd_low_port => $fastd_low_port,
}
}
and
define ffnord::fastd( $mesh_if_id
, $mesh_code
, $mesh_mtu_low = 1280
, $fastd_low_port
) {
file {
"/etc/fastd/${mesh_code}-mesh-low-vpn/":
ensure =>directory,
require => Package[ffnord::resources::fastd];
"/etc/fastd/${mesh_code}-mesh-low-vpn/fastd.conf":
ensure => file,
notify => Service[ffnord::resources::fastd],
content => template('ffnord/etc/fastd/fastd-low.conf.erb');
}
}
How can I define a variable amount of those configs:
$mesh_if_id = "low",
$mesh_mtu_low = 1280,
$fastd_low_port = 11280, # fastd port
$mesh_if_id = "something",
$mesh_mtu_low = 12345,
$fastd_low_port = 112345, # fastd port
...
and loop through those blocks to create a folder and file inside ffnord/etc/fastd/ for each block automatically?
(I want to solve this problem: https://github.com/ffnord/ffnord-puppet-gateway/pull/116#issuecomment-100619610 )
In Puppet 3.x there is no "looping", but there are a few tricks. You can pass a Hash of data that represents N number of ffnord::fastd instances:
define define ffnord::mesh($fastd_hash) {
create_resources('ffnord::fastd', $fastd_hash)
}
define ffnord::fastd($mesh_code, $fastd_low_port, $mesh_mtu_low = 1280) {
file {
"/etc/fastd/${mesh_code}-mesh-low-vpn/":
ensure =>directory,
require => Package[ffnord::resources::fastd];
"/etc/fastd/${mesh_code}-mesh-low-vpn/fastd.conf":
ensure => file,
notify => Service[ffnord::resources::fastd],
content => template('ffnord/etc/fastd/fastd-low.conf.erb');
}
}
$hash_of_fastds = {
"low_id" => {
mesh_code => 'low,
mesh_mtu_low => 1280,
fastd_low_port => 11280,
},
"some_id" => {
mesh_code => 'something',
mesh_mtu_low => 12345,
fastd_low_port => 112345,
},
}
ffnord::mesh { 'foo': fastd_hash => $hash_of_fastds, }
Note I've modified define ffnord::fastd slightly, where you had a $mesh_if_id parameter I've turned this into the $namevar of ffnord::fastd.
The first level of $hash_of_fastds translates to the names of the ffnord::fastd instances, the second level of the hash are the parameters for each ffnord::fastd.
See the documentation on the create_resources function for more information.
In Puppet 4 you could use the each function to achieve a similar result.