im used filebeat to many server, shipped nginx log to logstash
in this time and months my elk server is very good worked
but, my 1 line added grok pattern to syslog-filter.conf, and restart logstash ,,,
my elk and Concerning Logstash not worked
this is wehn, my elasticsearch and logatash and kibana ... services this up and enable and active!
but ...
my nginx servers ...
telnet to 5044
and
telnet to 5443
connection refused
this log, 1 server Logs (filebeat logs)
> 2018-01-23T10:21:21+03:30 ERR Failed to connect: dial tcp 172.17.11.202:5443: getsockopt: connection refused
> 2018-01-23T10:21:28+03:30 INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=11769216 beat.memstats.memory_alloc=5935656 beat.memstats.memory_total=73881024 filebeat.harvester.open_files=5 filebeat.harvester.running=6 libbeat.config.module.running=0 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.retry=2048 registrar.states.current=35
> 2018-01-23T10:47:16+03:30 INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=11297792 beat.memstats.memory_alloc=5872352 beat.memstats.memory_total=26557112 filebeat.harvester.open_files=5 filebeat.harvester.running=6 libbeat.config.module.running=0 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 registrar.states.current=37
> 2018-01-23T10:47:22+03:30 ERR Failed to connect: dial tcp 172.17.11.202:5443: getsockopt: connection refused
> 2018-01-23T10:47:46+03:30 INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=11297792 beat.memstats.memory_alloc=6012704 beat.memstats.memory_total=26697464 filebeat.harvester.open_files=5 filebeat.harvester.running=6 libbeat.config.module.running=0 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 libbeat.pipeline.events.retry=2048 registrar.states.current=37
> 2018-01-23T14:22:45+03:30 INFO Non-zero metrics in the last 30s: beat.info.uptime.ms=30000 beat.memstats.gc_next=11490800 beat.memstats.memory_alloc=5802160 beat.memstats.memory_total=153496216 filebeat.harvester.open_files=3 filebeat.harvester.running=2 libbeat.config.module.running=0 libbeat.pipeline.clients=1 libbeat.pipeline.events.active=4117 registrar.states.current=3
Related
I don't have an access to the etcd part of the project's source code, however I do have access to the /var/log/syslog.
The goal is to setup up 3-node cluster.
(1)The very first etcd error that comes up is:
rafthttp: failed to dial 76e7ffhh20007a98 on stream MsgApp v2 (dial tcp 10.0.0.134:2380: i/o timeout)
Before continuing, I would say that I can ping all three nodes from each of the nodes. As well as I have tried to open the 2380 TCP ports and still no success - same error.
(2)So, before that error I had following messages from the etcd, which in my opinion confirm that cluster is setup correctly:
etcdserver/membership: added member 76e7ffhh20007a98 [https://server2:2380]
etcdserver/membership: added member 222e88db3803e816 [https://server1:2380]
etcdserver/membership: added member 999115e00e17123d [https://server3:2380]
In /etc/hosts file these DNS names are resolved as:
server2 10.0.0.135
server1 10.0.0.134
server3 10.0.0.136
(3)The initial setup, however (on each nodes looks like this):
embed: listening for peers on https://127.0.0.1:2380
embed: listening for client requests on 127.0.0.1:2379
So, to sum up, each node have got this initial setup log (3) and then adds members (2) then once these steps are done it fails with (1). As I know the etcd cluster creation is following this pattern: https://etcd.io/docs/v3.5/tutorials/how-to-setup-cluster/
Without knowing the source code is really hard to debug, however maybe some ideas on the error and what could cause it?
UPD: etcdctl cluster-health output (ETCDCTL_ENDPOINT is exported):
cluster may be unhealthy: failed to list members Error: client: etcd
cluster is unavailable or misconfigured; error #0: client: endpoint
http://127.0.0.1:2379 exceeded header timeout ; error #1: dial tcp
127.0.0.1:4001: connect: connection refused
error #0: client: endpoint http://127.0.0.1:2379 exceeded header
timeout error #1: dial tcp 127.0.0.1:4001: connect: connection refused
kubernetes version: v1.16.3
linux version: 7.3.1611
Starting Vitess cluster on kubernetes using default operator.yaml and 101_initial_cluster.yaml, one of example-vttablet-zone1-xxx pod is restarting forever.
using kubectl logs -f example-vttablet-zone1-2548885007-46a852d0 -c vttablet to see the logs, i got
W0706 07:42:02.200507 1 tm_init.go:531] Cannot get current mysql port, will keep retrying every 1s: net.Dial(/vt/socket/mysql.sock) to local server failed: dial unix /vt/socket/mysql.sock: connect: no such file or directory (errno 2002) (sqlstate HY000)
E0706 07:42:02.285406 1 engine.go:213] Error starting vreplication engine: error in connecting to mysql db with connection <nil>, err net.Dial(/vt/socket/mysql.sock) to local server failed: dial unix /vt/socket/mysql.sock: connect: no such file or directory (errno 2002) (sqlstate HY000), will keep retrying.
E0706 07:42:02.285504 1 state_manager.go:276] Error transitioning to the desired state: MASTER, Serving, will keep retrying: net.Dial(/vt/socket/mysql.sock) to local server failed: dial unix /vt/socket/mysql.sock: connect: no such file or directory (errno 2002) (sqlstate HY000)
I0706 07:42:02.285527 1 state_manager.go:661] State: exiting lameduck
E0706 07:42:02.285539 1 tm_state.go:258] Cannot start query service: net.Dial(/vt/socket/mysql.sock) to local server failed: dial unix /vt/socket/mysql.sock: connect: no such file or directory (errno 2002) (sqlstate HY000)
I0706 07:42:02.285553 1 tm_state.go:305] Publishing state: alias:<cell:"zone1" uid:2548885007 > hostname:"10.233.107.217" port_map:<key:"grpc" value:15999 > port_map:<key:"vt" value:15000 > keyspace:"commerce" shard:"-" key_range:<> type:MASTER db_name_override:"vt_commerce" mysql_hostname:"10.233.107.217" master_term_start_time:<seconds:1625527268 nanoseconds:196807555 >
I didn't change any yaml in operator directory, anyone know why is this?
I have a log forwarding pipeline consists of filebeat and logstash. Somehow, they stopped working together recently. How can I check if Filebeat is correctly connected to Logstash?
Check your Filebeat log file, default location: /var/log/filebeat/filebeat
Example of errors:
ERR Failed to publish events
Connecting error publishing events (retrying): dial tcp x.x.x.x:5044: getsockopt: connection refused
Extra:
Troubleshooting Filebeat and Logstash
My logstash version is:
# /opt/logstash/bin/logstash --version
logstash 2.2.4
it is configured to receive input from port 5044 according to the filebeat file:
/etc/logstash/conf.d/02-beats-input.conf
input {
beats {
port => 5044
ssl => false
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
I have set ssl to false as I am not using it
but when I start the logstash service normally with systemctl it start and checking the status confirms it is running
systemctl status logstash
● logstash.service - LSB: Starts Logstash as a daemon.
Loaded: loaded (/etc/rc.d/init.d/logstash)
Active: active (exited) since Mon 2016-07-18 19:14:51 BST; 15h ago
Docs: man:systemd-sysv-generator(8)
Process: 19965 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
Process: 19970 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
...
logstash started
The problem is that logstash does not seem to be receiving input on port 5044. hosts sending filebeats encounter:
single.go:126: INFO Connecting error publishing events (retrying): dial tcp 192.72.0.92:5044: getsockopt: connection refused
when I check the port
# netstat -an | grep 5044
I get nothing. So even though logstash is running, I can't tell what port it is bound to and listening on.
Also the firewall is stopped temporarily to investigate this.
The strange thing is that is I run logstash is debug mode like so:
# ./logstash --debug -f /etc/logstash/conf.d/02-beats-input.conf
I can see
# netstat -an | grep 5044
tcp6 0 0 :::5044 :::* LISTEN
tcp6 0 0 192.72.0.92:5044 192.168.36.70:53720 ESTABLISHED
tcp6 0 0 192.72.0.92:5044 192.72.0.90:45980 ESTABLISHED
tcp6 0 0 192.72.0.92:5044 192.72.0.90:45975 ESTABLISHED
tcp6 0 0 192.72.0.92:5044 192.72.0.90:45976 ESTABLISHED
or
# lsof -i :5044
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 15136 root 7u IPv6 7191510 0t0 TCP *:lxi-evntsvc (LISTEN)
java 15136 root 33u IPv6 7192379 0t0 TCP hostname:lxi-evntsvc->192.72.0.90:45975 (ESTABLISHED)
and the host sending filebeats can connect
output.go:87: DBG output worker: publish 7 events
2016/07/19 10:02:08.017890 client.go:146: DBG Try to publish 7 events to logstash with window size 10
2016/07/19 10:02:08.038579 client.go:124: DBG 7 events out of 7 events sent to logstash. Continue sending ...
2016/07/19 10:02:08.038615 single.go:135: DBG send completed
Please help point out what I may be doing wrong with this configuration. Thanks
Based on the hing provided by #LiGhTx117
I think
The startup script used by logstash in:
/etc/init.d/logstash
has the following variables among others:
LS_USER=logstash
LS_GROUP=logstash
LS_HOME=/var/lib/logstash
LS_LOG_DIR=/var/log/logstash
LS_LOG_FILE="${LS_LOG_DIR}/$name.log"
LS_CONF_DIR=/etc/logstash/conf.d
The ownership and permission on these seem to be the issue.
I ensured that the directories where recursively accessible to the
user logstash as well as the group logstash
and
Then I also ensured that the log_file: logstash.log was writeable by
the user/group logstash
restarted logstash
Can some help me why i'm facing the below issue and how to fix when I'm trying to start my cqlsh (cassandra).
Connection error: ('Unable to connect to any servers',
{'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)].
Last error: Connection refused")})
When I type below command:
sudo service cassandra status
cassandra (pid 1xxxx) is running...
Which indicates my cassandra is running properly.
But unable to run cqlsh. But was able to run yesterday without any issues.
Coming to my cassandra.yaml file
my seed, listen_address, and rpc_address all are set to my public ip address 10.x.xx.xxx.
native_transport_port: 9042
I'm using single node cluster.
How are you starting cqlsh? If you want it to connect to an address other than 127.0.0.1, you need to specify it. Specifically, you should try the 10.x.xx.xxx address that you set in your yaml.
$ cqlsh 10.x.xx.xxx
Are you specifying anything for listen_interface or rpc_interface? Remember that you can set either the address or the interface, but not both.
To figure for sure out which address Cassandra is listening on, check your system.log file:
$ grep listening /var/log/cassandra/system.log
INFO [main] 2015-12-03 21:06:27,581 Server.java:182 - Starting listening for CQL clients on /192.168.0.100:9042...
Assuming that everything is configured properly, and you do not have any errors during startup, the address returned is the one you should be providing when you start cqlsh.
Also, are you trying to connect from the same machine? Or are you trying to remotely connect to your single node? Or is your Cassandra node running on a VM on your machine? Double-check your firewall rules, and ensure that traffic on 9042 can get from your client to your node.
I got below output when i ran $ grep listening /var/log/cassandra/system.log
INFO [main] 2015-12-02 12:49:20,334 Server.java:182 - Starting listening for CQL clients on localhost/127.0.0.1:9042...
INFO [StorageServiceShutdownHook] 2015-12-02 15:59:11,730 ThriftServer.java:142 - Stop listening to thrift clients
INFO [StorageServiceShutdownHook] 2015-12-02 15:59:11,771 Server.java:213 - Stop listening for CQL clients
INFO [main] 2015-12-02 17:21:28,775 Server.java:182 - Starting listening for CQL clients on /10.x.x.xxx:9042...
INFO [StorageServiceShutdownHook] 2015-12-03 17:12:12,840 ThriftServer.java:142 - Stop listening to thrift clients
INFO [StorageServiceShutdownHook] 2015-12-03 17:12:12,882 Server.java:213 - Stop listening for CQL clients
INFO [main] 2015-12-03 17:12:41,337 Server.java:182 - Starting listening for CQL clients on /10.x.x.xxx:9042...
INFO [StorageServiceShutdownHook] 2015-12-03 17:33:35,996 ThriftServer.java:142 - Stop listening to thrift clients
INFO [StorageServiceShutdownHook] 2015-12-03 17:33:36,100 Server.java:213 - Stop listening for CQL clients
INFO [main] 2015-12-03 17:34:00,741 Server.java:182 - Starting listening for CQL clients on /10.x.x.xxx:9042...
Also i'm trying to connect remotely through VPN. I'm using openstack.How to check for firewall issues?
Edit:
Finally I'm able to fix this issue. Ran netstat -tuplen command and found the address to be ::ffff:10.x.x.xxx:9042.
So ran cqlsh ::ffff:10.x.x.xxx:9042 and it started working.