We are using syslog-ng to send access-log file to remote servers via tcp. And I already know that multiple destination can be configured to do this job, just like:
source s_xxx { file("/xxx/access.log"); };
destination d_one {tcp("1.2.3.4", port(1234));};
destination d_two {tcp("1.2.3.5", port(1234));};
log {source(s_xxx); destination(d_one); destination(d_two);};
What I am going to figure out is that how to poll my content to these two destinations(such as round-robin). In other words, my content is either sent to d_one or d_two, not both of them.
thanks very much.
My scenario is very similar: I have a syslog-ng collector that forwards messages to an analytic application. It became overloaded and I needed to split the load. I have no requirement for traffic on which to filter and I did not want to maintain a list of types. I simply wanted message by message to round-robin as you are seeking. I decided to use mod(%) to achieve this.
Syslog-ng OSE v3.7.2:
destination d_net_qr1 { network("ip1"); };
destination d_net_qr2 { network("ip2"); };
filter f_qr1 { "$(% ${RCPTID} 2)" eq "0" };
filter f_qr2 { "$(% ${RCPTID} 2)" eq "1" };
log { source(s_net); filter(f_qr1); destination(d_net_qr1); };
log { source(s_net); filter(f_qr2); destination(d_net_qr2); };
syslog-ng Open Source Edition does not currently have a straightforward way to send the messages in round-robin fashion. If you want to do this for load-balancing, you can probably come up with a filter that switches between the destinations every few second using the $SEC macro and comparing macro values, see http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-v3.6-guide-admin/html/filters-comparing.html
HTH,
Regards,
Robert
Related
Hello all!
I am trying to use the Aggregate filter plugin of Logstash v7.7 to correlate and combine data from two different CSV file inputs which represent API data calls. The idea is to produce a record showing a combined picture. As you can expect the data may or may not arrive in the right sequence.
Here is as an example:
/data/incoming/source_1/*.csv
StartTime, AckTime, Operation, RefData1, RefData2, OpSpecificData1
231313232,44343545,Register,ref-data-1a,ref-data-2a,op-specific-data-1
979898999,75758383,Register,ref-data-1b,ref-data-2b,op-specific-data-2
354656466,98554321,Cancel,ref-data-1c,ref-data-2c,op-specific-data-2
/data/incoming/source_1/*.csv
FinishTime,Operation,RefData1, RefData2, FinishSpecificData
67657657575,Cancel,ref-data-1c,ref-data-2c,FinishSpecific-Data-1
68445590877,Register,ref-data-1a,ref-data-2a,FinishSpecific-Data-2
55443444313,Register,ref-data-1a,ref-data-2a,FinishSpecific-Data-2
I have a single pipeline that is receiving both these CSVs and I am able to process and write them as individual records to a single Index. However, the idea is to combine records from the two sources into one record each representing a superset. of Operation related information
Unfortunately, despite several attempts I have been unable to figure out how to achieve this via Aggregate filter plugin. My primary question is whether this is a suitable use of the specific plugin? And if so, any suggestions would be welcome!
At the moment, I have this
input {
file {
path => ['/data/incoming/source_1/*.csv']
tags => ["source1"]
}
file {
path => ['/data/incoming/source_2/*.csv']
tags => ["source2"]
}
# use the tags to do some source 1 and 2 related massaging, calculations, etc
aggregate {
task_id = "%{Operation}_%{RefData1}_%{RefData1}"
code => "
map['source_files'] ||= []
map['source_files'] << {'source_file', event.get('path') }
"
push_map_as_event_on_timeout => true
timeout => 600 #assuming this is the most far apart they will arrive
}
...
}
output {
elastic { ...}
}
And other such variations. However, I keep getting individual records being written to the Index and am unable to get one combined. Yet again, as you can see from the data set there's no guarantee of the sequencing of records - so I am wondering if the filter is the right tool for the job, to begin with? :-\
Or is it just me not being able to use it right! ;-)
In either case, any inputs/ comments/ suggestions welcome. Thanks!
PS: This message is being cross-posted over from Elastic forums. I am providing a link there just in case some answers pop up there too.
The answer is to use Elastic search in upsert mode. Please see the specifics here..
I recommend first that the information reaches you in order so that the filter can take it better, secondly, you could set the options in your pipeline.yml: pipeline.workers: 1 and pipeline.ordered: true, thus guaranteeing the order of processing.
So I am doing a data visualization of netflow traffic, and I am running packetbeat in "af mode" to gather all of the netflow data.
The problem is that the IP that I am connecting to the box with packetbeat on it, is something I want to ignore. Since I know what it is and it is just cluttering things up in the visualization.
I want to ignore all of the traffic that has this data:
"dest.ip" of < XYZ >
and
"source.ip" of < IP of server running packetbeat >
I have the "packetbeat.ignore_outgoing: true" set up in my packetbeat.yml file. I am running this on CentOS and outputting the packetbeat data straight to Logstash.
Is there any way to do this?
What I ended up doing was writing a Logstash filter.
filter {
if[type] == "flow" and [dest][ip] == "192.168.X.Y" and [packet_source][ip] == "192.168.Z.D" {
drop { }
}
}
I am upgrading from logstash-1.1.3 to logstash-1.3.3.
The problem is, that tags and fields configuration that were there in 1.1.3 are deprecated in version 1.3.3. These allowed to send only those events to the output which had given tags or contained given fields.
I just want to know what replaces these in logstash-1.3.3. How do I get the same functionality of sending selected events to an output. I don't want to send all the events to an output.
You can use if statement to do this.
output {
if [type] == "tech" {
stdout{}
}
}
This page has the introduction about how to configure.
I have a best practice question around Puppet when working is server/agent mode.
I have created a working solution using a manifest/sites.pp configuration that identifies the configuration using the hostname of the agent.
For example:
node 'puppetagent.somedomain.com' {
include my_module
notify { 'agent configuration applied':
}
}
This works great for configuring a single node but what if I had a scenario in which I had multiple applications servers all with differing hostnames but all of which needed the same configuration.
Adding multiple node entries, comma separated hostname list or regular expressions doesn't feel like the 'right' way to do this.
Are there alternative ways? Can you define node 'types'? What do the community consider best practice for this?
Many thanks
If all the servers have the same configuration, inheritance, or the hieara hierarchy are the easiest ways to achieve this.
Once you need to maintain a larger set of systems where certain nodes have types such as 'web server' or 'database server' the configurations will diverge and the single inheritance model is not entirely sufficient.
You can use composition in those places. Take a peak at this article for more details.
Regular expressions might not be so bad, but I suppose the current trend is to use hiera_include.
You can do something dirty like this :
$roles = { 'webserver' => [ 'server1', 'server2', 'server3' ]
, 'smtp' => [ 'gw1', 'gw2' ]
}
node default {
$roles . filter |$k,$v| { $hostname in $v }
. each |$k,$v| { hiera_include($k) }
}
I would suggest taking a look at the concept of "roles and profiles" here: http://www.craigdunn.org/2012/05/239/
You can have multiple nodes all of which include the same configuration with a "profile" that includes one or more "roles".
As for defining multiple nodes with the same configuration or a "profile" containing "role(s)", I would suggest using hiera_include like #bartavelle mentioned. Except to use a common environment variable for identifying the nodes rather than using regular expressions.
I am trying few features of CFENGINE 3.5 and stuck with a very basic issue.
I want to copy certain files which are kept in cfengine Policy hub to various cfengine clients. These files are spread into various locations and further cfengine should copy these files to targeted machines on same location as master server has.
How to do this ?
If you want to copy certain files from the hub onto the same location on the clients, you can do something like this:
vars:
"files" slist => { "/some/file", "/other/file", "/one/more/file" };
files:
"$(files)"
copy_from => secure_cp("$(files)", "$(sys.policy_hub)");
This will loop over the files, copying each one in turn. Make sure you include the appropriate standard library file to secure_cp(), something like this:
body common control
{
inputs => { "lib/3.5/files.cf" };
bundlesequence => { ... };
}
https://cfengine.com/docs/3.5/examples-policy-copy-single-files.html
This might help.
Thanks & Regards,
Alok Thaker