I have ganglia set up on a cluster of servers, all of which have gmond, one of which has gmetad, and one which has log stash and elasticsearch. I’d like to use Logstash’s ganglia input plugin to collect data directly from the monitoring daemons, but I’ve been unsuccessful so far. My logstash logs always show:
{:timestamp=>"2015-07-14T14:33:25.192000+0000", :message=>"ganglia udp listener died", :address=>"10.1.10.178:8664", :exception=>#, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in bind'", "/opt/logstash/lib/logstash/inputs/ganglia.rb:61:inudp_listener'", "/opt/logstash/lib/logstash/inputs/ganglia.rb:39:in run'", "/opt/logstash/lib/logstash/pipeline.rb:163:ininputworker'", "/opt/logstash/lib/logstash/pipeline.rb:157:in `start_input'"], :level=>:warn}
Here's the input config I've been testing with:
input {
ganglia {
host => "10.1.10.178" #ip of logstash node
port => 8666
type => "ganglia_test"
}
}
and I have this in gmond.conf on one of the gmond nodes
udp_send_channel {
host = 10.1.10.178 #logstash node
port = 8666
bind_hostname = yes
}
I've found this problem too. It looks like there's a bug in the Ganglia listener since about version 1.2 (I know it used to work in 1.1..)
I managed to work around the problem by adding an explicit 'UDP' listener. This seems to satisfy logstash and allows the Ganglia listener to keep running.
e.g.
input {
udp {
port => "1112"
type => "dummy"
}
ganglia {
port => "8666"
type => "ganglia"
}
}
Related
I am trying to get up to speed with js-libp2p and I am poking around with this code snippet:
const Libp2p = require("libp2p")
const WebSockets = require('libp2p-websockets')
const Tcp = require('libp2p-tcp')
const { NOISE } = require('libp2p-noise')
const MPLEX = require('libp2p-mplex')
const Mdns = require('libp2p-mdns')
async function main() {
const node = await Libp2p.create({
modules: {
transport: [WebSockets, Tcp],
connEncryption: [NOISE],
streamMuxer: [MPLEX],
peerDiscovery: [Mdns]
},
addresses: {
listen: [
'/ip4/127.0.0.1/tcp/0/ws',
'/ip4/127.0.0.1/tcp/0',
]
}
})
await node.start()
node.on('peer:discovery', (peer) => {
console.log(`Discovered ${peer}`)
})
node.connectionManager.on('peer:connect', (connection) => {
console.log(`Connection established ${connection.remotePeer.toB58String()}`)
})
}
main()
If I run it from two consoles on the same computer, on starting the second console I get the following expected output:
Discovered bafzbeia7bnohivu4lnyvzd5f4vix2xsru4sidmgitcp5u2ysepxwct5rwu
Connection established QmQRqPvVKDYUKrNYc8SsFwyK2kkxq3zUSojvmDUqNzoy32
Connection established QmQRqPvVKDYUKrNYc8SsFwyK2kkxq3zUSojvmDUqNzoy32
Connection established QmQRqPvVKDYUKrNYc8SsFwyK2kkxq3zUSojvmDUqNzoy32
and from the other console also:
Discovered bafzbeieeqfz3whgaj7hbxbczkxs3vue2yxid4cwpdqkrtzcrpkmepuqzj4
Connection established QmXFu1kP9wWu42vtREEdXhdWc4uLHZ4t5r3fYtfbmb66y4
Connection established QmXFu1kP9wWu42vtREEdXhdWc4uLHZ4t5r3fYtfbmb66y4
Connection established QmXFu1kP9wWu42vtREEdXhdWc4uLHZ4t5r3fYtfbmb66y4
But if I move the same script and run it on another computer on the same network, I only see the Discovered log. The connection log never gets printed out. For example I only see this:
➜ my_node node node.js
Discovered bafzbeienf65d3slqvblq3sqoedltuqorakqfkdnyodocul52gptdrqhqx4
Does anybody have an explanation why this is the case? And also what do I need to do to have full connection? Because as I poke around with libp2p I would prefer to have the nodes run on two computers.
Also maybe explain the difference between peer:discovery event and peer:connect event? I mean how is peer discovery different from peer connected?
PS: Also I noticed that the connection log keeps getting printed at interval. Why is that also the case?
Ok so it turned out the problem was me listening to the loopback address. That is 127.0.0.1. Replacing it with 0.0.0.0 solved the problem. Hence changing from:
'/ip4/127.0.0.1/tcp/0/ws',
'/ip4/127.0.0.1/tcp/0',
to
'/ip4/0.0.0.0/tcp/0/ws',
'/ip4/0.0.0.0/tcp/0',
I have a node express app with prom-client to monitor a serial connection and report the values to a http endpoint, the serial speed is 9600baud and is transferring some statistics over.
A Prometheus instance is configured with a job 10milliseconds interval to target that end point and grab the metrics.
I want to be able to see this metrics in at least 10 milliseconds resolution but it seems the Prometheus graph resolution does not accepts less than 1 seconds.
What should I do to get Prometheus collect data with at least 10 milliseconds res.
Is there a config I miss?
I have searched for hours
this is my node js app, a serial port listener is waiting for json messages, parses them and updates gauge metric types from 'prom-client' to be represented by express!
const serialPath = '/dev/tty.usbmodem14201';
const port = new SerialPort(serialPath, {
baudRate: 9600
});
const parser = new Readline();
port.pipe(parser);
parser.on('data', (line) => {
try {
const obj = JSON.parse(line);
if (obj.command !== undefined) {
console.log(obj);
}
if (obj.a) {
obj.a.forEach((analog) => {
analogGuage.set({
pin: analog.i
}, analog.v);
})
}
} catch (ex) {
console.log('Exception in parsing serial json:', ex);
console.log('Exception in parsing serial json:', line);
}
});
metrics endpoint for prometheus to call each 10ms
expressApp.get('/metrics', (req, res) => {
const metrics = client.register.metrics();
res.set('Content-Type', client.register.contentType);
res.end(metrics);
});
It is critical to mention all this is for an experimental personal embedded system :) so, no bottleneck or performance considerations are in place except to be able to transfer and parse serial reading in less than 10ms
since right now the Prometheus and the node exporter app are running on my PC, so 10ms intervals seems easy for Prom.
Please help.
Answer Edit: so I decided to drop Prometheus instead of InfluxDB, as both licenses allow source access and they promote millisec, nanosec monitoring,
but for future reference 9600baud was not enough either, but still after 115200baud rate and 150millisec reporting loops Prom. still did not manage to show less than 1sec,
So InfluxDB did it beatifullty , here is some pictures:
bellow is a 30sec window of Prom. on 115200baud
and about 10 second on same 115200baud in InfluxDB
While you can set scrape intervals less than a second, this isn't what Prometheus is designed for as that's hard real time monitoring at that point and e.g. the kernel scheduler may cause Prometheus to stop running briefly and miss some scrapes, which wouldn't be an issue with more typical scrape intervals.
I'd suggest looking at a custom solution if you need such a high resolution.
I am Working on Logstash input Jdbc with different databases, Primarily I am trying for cassandra and I am using DBSchema driver, the driver is properly working with my java jdbc code, but when coming to integrate with Logstash it was failing to connect, I am trying this from many days, i did n't find any exact solution in forum, i tried all the things, but not worked, with maria db was also i tried it is also failing.
Here I am providing Logs while starting Logstash
Error: com.dbschema.CassandraJdbcDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
Exception: LogStash::ConfigurationError
Stack: /opt/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'
Logstash conf file: (Cassandra)
#Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
jdbc {
clean_run => true
jdbc_connection_string => "jdbc:cassandra://localhost:9042/cloud"
jdbc_user => "cassandra"
jdbc_password => "cassandra"
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/cassandrajdbc1.2.jar"
jdbc_validate_connection => true
jdbc_driver_class => "com.dbschema.CassandraJdbcDriver"
statement => "SELECT * FROM cloud.event_history_all"
}
}
output {
elasticsearch { hosts => ["localhost:9200"]
index => "log_cassandra" }
stdout { codec => rubydebug }
}
Cassandra version - 3.11.5
elk versions - 7.1.1
java version - 11.0.5
Thanks in advance
I've got a problem with running logstash. My conf looks like this:
input {
udp {
port => 1514
type => docker
}
}
filter {
grok {
match => {
"message" => "<%{NUMBER}>%{DATA}(?:\s+)%{DATA:hostname}(?:\s+)%{DATA:imageName}(?:\s+)%{DATA:containerName}(?:\s*\[%{NUMBER}\]:) (\s+(?<logDate>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s+%{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL:logLevel}(?:\s*);* %{DATA:logAdditionalInfo};*)?%{GREEDYDATA:logMsg}"
}
keep_empty_captures => false
remove_field => ["message"]
}
}
output {
if [type] == "gelf" {
elasticsearch {
index => "phpteam-%{+YYYY.MM.dd}"
}
} else {
elasticsearch { }
}
}
The configuration is correct, but after running it /var/log/logstash/logstash.log shows the following output:
{:timestamp=>"2016-06-22T11:43:03.105000+0200", :message=>"SIGTERM
received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-06-22T11:43:03.532000+0200", :message=>"UDP
listener died", :exception=>#,
:backtrace=>["org/jruby/RubyIO.java:3682:in select'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:77:in
udp_listener'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:50:in
run'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:206:in
inputworker'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:199:in
`start_input'"], :level=>:warn}
The only thing I found to workaround this error is to edit those .rb files, but sadly I have no idea how to do it. Could you help me somehow?
Thanks in advance.
I found solution that is not perfect, but works, so maybe it will help somebody.
After installing the whole instance on new server everything works fine.
Everything crashed after upgrade'ing logstash/elasticsearch/kibana, so maybe there's something wrong with configuration files, but I couldn't figure out which ones.
I know there are quite a few node.js modules that implement a Kafka consumer that gets msgs and writes to elastic. But I only need some of the fields from each msg and not all of them. Is there an existing solution I don't know about?
The question is asking for an example from node.js. The kafka-node module provides a very nice mechanism for getting a Consumer, which you can combine with the elasticsearch-js module:
// configure Elasticsearch client
var elasticsearch = require('elasticsearch');
var esClient = new elasticsearch.Client({
// ... connection details ...
});
// configure Kafka Consumer
var kafka = require('kafka-node');
var Consumer = kafka.Consumer;
var client = new kafka.Client();
var consumer = new Consumer(
client,
[
// ... topics / partitions ...
],
{ autoCommit: false }
);
consumer.on('message', function(message) {
if (message.some_special_field === "drop") {
return; // skip it
}
// drop fields (you can use delete message['field1'] syntax if you need
// to parse a more dynamic structure)
delete message.field1;
delete message.field2;
delete message.field3;
esClient.index({
index: 'index-name',
type: 'type-name',
id: message.id_field, // ID will be auto generated if none/unset
body: message
}, function(err, res) {
if (err) {
throw err;
}
});
});
consumer.on('error', function(err) {
console.log(err);
});
NOTE: Using the Index API is not a good practice when you have tons of messages being sent through because it requires that Elasticsearch create a thread per operation, which is obviously wasteful and it will eventually lead to rejected requests if the thread pool is exhausted as a result. In any bulk ingestion situation, a better solution is to look into using something like Elasticsearch Streams (or Elasticsearch Bulk Index Stream that builds on top of it), which builds on top of the official elasticsearch-js client. However, I've never used those client extensions so I don't really know how well they do or do not work, but usage would simply replace the part where I am showing the indexing happening.
I'm not convinced that the node.js approach is actually better than the Logstash one below in terms of maintenance and complexity, so I've left both here for reference.
The better approach is probably to consume Kafka from Logstash, then ship it off to Elasticsearch.
You should be able to use Logstash to do this in a straight forward way using the Kafka input and Elasticsearch output.
Each document in the Logstash pipeline is called an "event". The Kafka input assumes that it will receive JSON coming in (configurable by its codec), which will populate a single event with all of the fields from that message.
You can then drop those fields that you have no interest in handling, or conditionally the entire event.
input {
# Receive from Kafka
kafka {
# ...
}
}
filter {
if [some_special_field] == "drop" {
drop { } # skip the entire event
}
# drop specific fields
mutate {
remove_field => [
"field1", "field2", ...
]
}
}
output {
# send to Elasticsearch
elasticsearch {
# ...
}
}
Naturally, you'll need to configure the Kafka input (from the first link) and the Elasticsearch output (and the second link).
The previous answer is not scaleable for production.
You will have to use ElasticSearch bulk API. You can use this NPM package https://www.npmjs.com/package/elasticsearch-kafka-connect It allows you to send data from Kafka to ES (duplex connection ES to kafka is still in development as per May 2019)