So I am doing a data visualization of netflow traffic, and I am running packetbeat in "af mode" to gather all of the netflow data.
The problem is that the IP that I am connecting to the box with packetbeat on it, is something I want to ignore. Since I know what it is and it is just cluttering things up in the visualization.
I want to ignore all of the traffic that has this data:
"dest.ip" of < XYZ >
and
"source.ip" of < IP of server running packetbeat >
I have the "packetbeat.ignore_outgoing: true" set up in my packetbeat.yml file. I am running this on CentOS and outputting the packetbeat data straight to Logstash.
Is there any way to do this?
What I ended up doing was writing a Logstash filter.
filter {
if[type] == "flow" and [dest][ip] == "192.168.X.Y" and [packet_source][ip] == "192.168.Z.D" {
drop { }
}
}
Related
Trying to figure out how to enrich the data being ingested (Network Data) Zeek/Suricata. I would like to either show Hostname vice IP, or more preferably add another field for hostname based off the IP address.
Have a file with IP -> Hostnames (CSV) currently could be anything other format if required. Unable to get IP to Hostname with DNS or Active Directory or any other connected means.
I know in Splunk you could do lookup tables, but unsure how to accomplish the same in the ELK stack to view the results in Kibana.
You could do this in logstash using a translate filter, which requires a two-column CSV file (or YML, or JSON). You could try
translate {
source => "[fieldWithIP]"
dictionary_path => "/path/to/mapping.csv"
target => "[fieldForHostname]"
}
this would add a new field called [fieldForHostname] if the value of [fieldWithIP] is found in column 1 of the mapping.csv
I am using logstash 1.5 in my ELK stack environment.
with the following filter configuration:
filter {
mutate {
add_filed => { "src_ip" => "%{src}" }
add_filed => { "dst_ip" => "%{dst}" }
}
dns {
reverse => [ "src", "dst" ]
action => "replace"
}
}
I have 2 problems:
The filter is missing or skip the dns reverse proccess on many logs - I mean each log that going in the filter process or that both dst and src fields reverse or not at all and remain with the ip
( when i test with nslookup all the ip fields has names in the dns).
I dont know how and why but some of my logs has multiple values and i get the following error:
DNS: skipping reverse, can't deal with multiple values, :field=>"src"
, :value=>["10.0.0.1","20.0.0.2"], : level=> warn
It looks like my (ELK) logstash cant handle with a lot of logs and resolve them fast enough. also its looks that he create array keys of multiple value from different logs.
any idea?
maybe you guys encounter this problem?
I noticed a typo in your configuration - add_filed should be add_field
We are using syslog-ng to send access-log file to remote servers via tcp. And I already know that multiple destination can be configured to do this job, just like:
source s_xxx { file("/xxx/access.log"); };
destination d_one {tcp("1.2.3.4", port(1234));};
destination d_two {tcp("1.2.3.5", port(1234));};
log {source(s_xxx); destination(d_one); destination(d_two);};
What I am going to figure out is that how to poll my content to these two destinations(such as round-robin). In other words, my content is either sent to d_one or d_two, not both of them.
thanks very much.
My scenario is very similar: I have a syslog-ng collector that forwards messages to an analytic application. It became overloaded and I needed to split the load. I have no requirement for traffic on which to filter and I did not want to maintain a list of types. I simply wanted message by message to round-robin as you are seeking. I decided to use mod(%) to achieve this.
Syslog-ng OSE v3.7.2:
destination d_net_qr1 { network("ip1"); };
destination d_net_qr2 { network("ip2"); };
filter f_qr1 { "$(% ${RCPTID} 2)" eq "0" };
filter f_qr2 { "$(% ${RCPTID} 2)" eq "1" };
log { source(s_net); filter(f_qr1); destination(d_net_qr1); };
log { source(s_net); filter(f_qr2); destination(d_net_qr2); };
syslog-ng Open Source Edition does not currently have a straightforward way to send the messages in round-robin fashion. If you want to do this for load-balancing, you can probably come up with a filter that switches between the destinations every few second using the $SEC macro and comparing macro values, see http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.6-guides/en/syslog-ng-ose-v3.6-guide-admin/html/filters-comparing.html
HTH,
Regards,
Robert
I have read that it is possible to assign dynamic names to the indexes like this:
elasticsearch {
cluster => "logstash"
index => "logstash-%{clientid}-%{+YYYY.MM.dd}"
}
What I am wondering is if it is possible to assign the template dynamically as well:
elasticsearch {
cluster => "logstash"
template => "/etc/logstash/conf.d/%{clientid}-template.json"
}
Also where does the variable %{clientid} come from?
Thanks!
After some testing and feedback from other users, thanks Ben Lim, it seems this is not possible to do so far.
The closest thing would be to do something like this:
if [type] == "redis-input" {
elasticsearch {
cluster => "logstash"
index => "%{type}-logstash-%{+YYYY.MM.dd}"
template => "/etc/logstash/conf.d/elasticsearch-template.json"
template_name => "redis"
}
} else if [type] == "syslog" {
elasticsearch {
cluster => "logstash"
index => "%{type}-logstash-%{+YYYY.MM.dd}"
template => "/etc/logstash/conf.d/syslog-template.json"
template_name => "syslog"
}
}
Full disclosure: I am a Logstash developer at Elastic
You cannot dynamically assign a template because templates are uploaded only once, at Logstash initialization. Without the flow of traffic, deterministic variable completion does not happen. Since there is no traffic flow during initialization, there is nothing there which can "fill in the blank" for %{clientid}.
It is also important to remember that Elasticsearch index templates are only used when a new index is created, and so it is that templates are not uploaded every time a document reached the Elasticsearch output block in Logstash--can you imagine how much slower it would be if Logstash had to do that? If you intend to have multiple templates, they need to be uploaded to Elasticsearch before any data gets sent there. You can do this with a script of your own making using curl and Elasticsearch API calls. This also permits you to update templates without having to restart Logstash. You could run the script any time before index rollover, and when the new indices get created, they'll have the new template settings.
Logstash can send data to a dynamically configured index name, just as you have above. If there is no template present, Elasticsearch will create a best-guess mapping, rather than what you wanted. Templates can and ought to be completely independent of Logstash. This functionality was added for an improved out-of-the-box experience for brand new users. The default template is less than ideal for advanced use cases, and Logstash is not a good tool for template management if you have more than one index template.
I want to have an output for Influx DB from Logstash, is there any such plugin available?
The output is set to graphite.. This is the influx config:
[input_plugins]
# Configure the graphite api
[input_plugins.graphite]
enabled = true
port = 2003
database = "AirAnalytics" # store graphite data in this database
# udp_enabled = true # enable udp interface on the same port as the tcp interface
This is the logstash config:
output {
stdout {}
graphite {
host => "localhost"
port => 2003
}
}
I see the output in the console (stdout) but no other message and nothing gets posted into influx. I checked the influx logs as well, nothing.
I tried posting the same message directly via http to influx and it works, so there's no issue with the message or influx install.
Solved it. I needed to pass on the already prepared influx compatible string to influx via logstash.
Following is the logstash configuration snippet which did the trick:
output {
http {
url => "http://localhost:8086/db/<influx db name>/series?u=<user name>&p=<pwd>"
format => "message"
content_type => "application/json"
http_method => "post"
message => "%{message}"
verify_ssl => false
}
stdout {}
}
Note: If you use the format "json" then logstash wraps the body around a "message" field which was causing a problem.
It's available via logstash-contrib as an output: https://github.com/elasticsearch/logstash-contrib/blob/master/lib/logstash/outputs/influxdb.rb
There is an influxdb output in logstash-contrib, however, this was added after 1.4.2 was released.
With logstash 1.5, there is a new plugin management system. If you're using 1.5, you can install the influxdb output with:
# assuming you're in the logstash directory
$ ./bin/plugin install logstash-output-influxdb
Maybe this help:
http://influxdb.com/docs/v0.8/api/reading_and_writing_data.html
Look at the section: Writing data through Graphite Protocol
maybe you can use the graphite output of logstash.
I think I am going to try that this weekend.
The accepted answer, while it works, is not very flexible because:
It requires the actual JSON payload to be in %{message} or whatever logstash variable you end up using
it doesn't submit the data points in batch where possible (of course, unless you have it in the JSON payload...which...in such case...why are you even using logstash in the first place?)
As noted by Paul and Wilfred, there is support for influxdb written by Jordan Sissel himself, but it was released after 1.4.2...good thing is that it works with 1.4.2 (i've tried it myself)...all you need to do is copy the influxdb.rb file to the /lib/logstash/outputs and configure your logstash accordingly. As for the documentation, you can find it here ...it did take me a bit more effort to find it because googling "influxdb logstash" doesn't take have this link on the first page results.