cassandra nodetool : Failed to connect to '127.0.0.1:7199' - cassandra

i installed cassandra 3.11.3-1 on centos7 & vmware
i didn't have error while installing cassandra.
i started cassandra and faced this logs.
[root#localhost ~]# service cassandra start
Starting cassandra (via systemctl): [ OK ]
[root#localhost ~]# systemctl status cassandra
cassandra.service - LSB: distributed storage system for structured data
Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
Active: deactivating (stop) (Result: exit-code) since 2018-08-02 15:15:45
KST; 6s ago
Docs: man:systemd-sysv-generator(8)
Process: 10366 ExecStart=/etc/rc.d/init.d/cassandra start (code=exited,
status=0/SUCCESS)
Main PID: 10450 (code=exited, status=3); : 10478 (cassandra)
Tasks: 2
CGroup: /system.slice/cassandra.service
└─control
├─10478 /bin/bash /etc/rc.d/init.d/cassandra stop
└─10549 sleep 0.5
02 15:15:39 localhost.localdomain systemd[1]: Starting LSB: distributed
stora....
02 15:15:39 localhost.localdomain su[10376]: (to cassandra) root on none
02 15:15:41 localhost.localdomain cassandra[10366]: Starting Cassandra: OK
02 15:15:41 localhost.localdomain systemd[1]: Started LSB: distributed
storag....
02 15:15:45 localhost.localdomain systemd[1]: cassandra.service: main
process...D
02 15:15:45 localhost.localdomain su[10489]: (to cassandra) root on none
02 15:15:45 localhost.localdomain cassandra[10478]: Shutdown Cassandra:
bash: …
Hint: Some lines were ellipsized, use -l to show in full.
it means 'cassandra starting is ok' right?
but when i check the node status like this
"nodetool status"
then i met this logs.
"] nodetool status"
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException:
Connection refused)
so i searched a lot on google.
i found some information.
so i tried this.
edit cassandra-env.sh
"JVM_OPTS -Djava.rmi.server.hostname=127.0.0.1"
expend memory size : 1GB -> 2GB
but i still faced same error.
someone help me please.
------------- system.log ----------------------------------
INFO [main] 2018-08-02 15:15:44,866 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/cassandra/default.conf/cassandra.yaml
ERROR [main] 2018-08-02 15:15:45,043 CassandraDaemon.java:708 - Exception encountered during startup: Invalid yaml: file:/etc/cassandra/default.conf/cassandra.yaml
Error: while scanning a simple key; could not found expected ':'; in 'reader', line 601, column 1:
Set listen_address OR listen_i ...

First, you need to make sure that the activated version of OpenJDK should be openjdk-8-jdk. If you have multiple versions of OpenJDK on your machine, then you could follow this tutorial to set a default version (in this case openjdk-8-jdk).
Then, you need to check the status of cassandra service again. The result for activating cassandra service should be as following
After that, you can follow this instruction to modify JVM_OPTS in /etc/cassandra/cassandra-env.sh. In my case, I do not need to follow 2nd step. Finally, when checking status of a node, you should see the result as following

The cassandra service takes some time to start. After the install, check whether the service has started by using
service --status-all
and you will see something like this
[ + ] cassandra
[ - ] dbus
[ ? ] hwclock.sh
[ ? ] kmod
[ - ] ntp
[ - ] procps
[ - ] rsync
[ - ] udev
[ - ] x11-common
if you see a - sign instead of a + sign next to cassandra, it would mean that service has not yet started. you can start it by issuing this command
service cassandra restart
keep checking the status until you get the a +. Now you should be able to execute the nodetool command
nodetool status
and now you should get the desired result, someething like this
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 127.0.0.1 84.76 KiB 16 100.0% 7615cf7e-14cc-4475-bf46-ceeb122b6a12 rack1
this worked for me

When I checked this "systemctl status cassandra" I could see Active: failed as seen below
● cassandra.service - LSB: distributed storage system for structured
data Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor
preset: disabled) Active: failed (Result: signal) since Fri
2022-01-07 02:28:07 UTC; 10min ago
Docs: man:systemd-sysv-generator(8) Main PID: 8239 (code=killed, signal=KILL)
So I changed below parameter in cassandra-env.sh
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=localhost"
After I changed the parameter I could see this
[root#ip-172-31-28-163 default.conf]# systemctl status cassandra ●
cassandra.service - LSB: distributed storage system for structured
data Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor
preset: disabled) Active: active (running) since Fri
2022-01-07 02:47:27 UTC; 12min ago
Docs: man:systemd-sysv-generator(8)
Be sure to do this as root user. Be careful as well

Check in the file cassandra-env.sh (/etc/cassandra/cassandra-env.sh) if the parameters system_memory_in_mb and system_cpu_cores values is configured acoording your machine capabilites

Another solution. In my case I was installing Cassandra 41x with older Java 8 Version which caused an issue and in case to solve this I installed new Java 11. Just provision right version of Java.

Related

Cassandra issue while adding jmx_prometheus

I want to add Cassandra monitoring using Prometheus. ref https://blog.pythian.com/step-step-monitoring-cassandra-prometheus-grafana/
When I add /etc/cassandra/cassandra-env.sh
JVM_OPTS="$JVM_OPTS -javaagent:/opt/jmx_prometheus/jmx_prometheus_javaagent-0.3.0.jar=7070:/opt/jmx_prometheus/cassandra.yml"
I get an error :
ubuntu#ip-172-21-0-111:~$ sudo service cassandra status
● cassandra.service - LSB: distributed storage system for structured data
Loaded: loaded (/etc/init.d/cassandra; bad; vendor preset: enabled)
Active: active (exited) since Mon 2020-04-13 05:43:38 UTC; 3s ago
Docs: man:systemd-sysv-generator(8)
Process: 3557 ExecStop=/etc/init.d/cassandra stop (code=exited, status=0/SUCCESS)
Process: 3570 ExecStart=/etc/init.d/cassandra start (code=exited, status=0/SUCCESS)
Apr 13 05:43:38 ip-172-21-0-111 systemd[1]: Starting LSB: distributed storage system for structured data...
Apr 13 05:43:38 ip-172-21-0-111 systemd[1]: Started LSB: distributed storage system for structured data.
ubuntu#ip-172-21-0-111:~$ nodetool status
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
when I remove jmx_prometheus entry I get it working :
ubuntu#ip-172-21-0-111:~$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.21.0.111 1.83 GiB 128 100.0% b52324d0-c57f-46e3-bc10-a6dc07bae17a rack1
ubuntu#ip-172-21-0-111:~$ tail -f /var/log/cassandra/system.log
INFO [main] 2020-04-13 05:37:36,609 StorageService.java:2169 - Node /172.21.0.111 state jump to NORMAL
INFO [main] 2020-04-13 05:37:36,617 CassandraDaemon.java:673 - Waiting for gossip to settle before accepting client requests...
INFO [main] 2020-04-13 05:37:44,621 CassandraDaemon.java:704 - No gossip backlog; proceeding
INFO [main] 2020-04-13 05:37:44,713 NativeTransportService.java:70 - Netty using native Epoll event loop
INFO [main] 2020-04-13 05:37:44,773 Server.java:161 - Using Netty Version: [netty-buffer=netty-buffer-4.0.36.Final.e8fa848, netty-codec=netty-codec-4.0.36.Final.e8fa848, netty-codec-haproxy=netty-codec-haproxy-4.0.36.Final.e8fa848, netty-codec-http=netty-codec-http-4.0.36.Final.e8fa848, netty-codec-socks=netty-codec-socks-4.0.36.Final.e8fa848, netty-common=netty-common-4.0.36.Final.e8fa848, netty-handler=netty-handler-4.0.36.Final.e8fa848, netty-tcnative=netty-tcnative-1.1.33.Fork15.906a8ca, netty-transport=netty-transport-4.0.36.Final.e8fa848, netty-transport-native-epoll=netty-transport-native-epoll-4.0.36.Final.e8fa848, netty-transport-rxtx=netty-transport-rxtx-4.0.36.Final.e8fa848, netty-transport-sctp=netty-transport-sctp-4.0.36.Final.e8fa848, netty-transport-udt=netty-transport-udt-4.0.36.Final.e8fa848]
INFO [main] 2020-04-13 05:37:44,773 Server.java:162 - Starting listening for CQL clients on /172.21.0.111:9042 (unencrypted)...
INFO [main] 2020-04-13 05:37:44,811 CassandraDaemon.java:505 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
INFO [SharedPool-Worker-1] 2020-04-13 05:37:46,625 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
INFO [OptionalTasks:1] 2020-04-13 05:37:46,752 CassandraRoleManager.java:339 - Created default superuser role 'cassandra'
It worked! Changed port to 7071 from 7070 in JVM_OPTS="$JVM_OPTS -javaagent:/opt/jmx_prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/jmx_prometheus/cassandra.yml"

Restore fails on ovirt whether engine is on or off - how to get around

Wondering whether this is an issue;
I had an engine where i created a backup and stored elsewhere. This engine was reinstalled
and the backup file (called file_name) used to restore below. However i get 2 different
errors based on whether engine is on....
[root#dub-mgrfarm113 ~]# service ovirt-engine status
Redirecting to /bin/systemctl status ovirt-engine.service
● ovirt-engine.service - oVirt Engine
Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; disabled; vendor preset:
disabled)
Active: active (running) since Thu 2020-01-16 16:13:41 GMT; 2h 5min ago
Main PID: 5621 (ovirt-engine.py)
CGroup: /system.slice/ovirt-engine.service
├─5621 /usr/bin/python
/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output
--systemd=notify st...
└─5661 ovirt-engine -server -XX:+TieredCompilation -Xms1991M -Xmx1991M -Xss1M
-Djava.awt.headless=true -Dsun.rmi.dgc.clien...
Jan 16 16:13:40 dub-mgrfarm113.ie.oracle.com systemd[1]: Starting oVirt Engine...
Jan 16 16:13:40 dub-mgrfarm113.ie.oracle.com ovirt-engine.py[5621]: 2020-01-16
16:13:40,816+0000 ovirt-engine: INFO _detectJBossV...l=36
Jan 16 16:13:41 dub-mgrfarm113.ie.oracle.com ovirt-engine.py[5621]: 2020-01-16
16:13:41,795+0000 ovirt-engine: INFO _detectJBossV...'[]'
Jan 16 16:13:41 dub-mgrfarm113.ie.oracle.com systemd[1]: Started oVirt Engine.
Hint: Some lines were ellipsized, use -l to show in full.
[root#dub-mgrfarm113 ~]# engine-backup --mode=restore --file=file_name
--restore-permissions
Start of engine-backup with mode 'restore'
scope: all
archive file: file_name
log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20200116181932.log
Preparing to restore:
FATAL: Engine service is active - can not restore backup
or off.......
[root#dub-mgrfarm113 ~]# service ovirt-engine stop
Redirecting to /bin/systemctl stop ovirt-engine.service
[root#dub-mgrfarm113 ~]# engine-backup --mode=restore --file=file_name
--restore-permissions
Start of engine-backup with mode 'restore'
scope: all
archive file: file_name
log file: /var/log/ovirt-engine-backup/ovirt-engine-restore-20200116181952.log
Preparing to restore:
- Unpacking file 'file_name'
Restoring:
- Files
FATAL: Can't connect to database 'engine'. Please see
'/usr/bin/engine-backup --help'.
In addition, possibly related, the web page for the server will no longer launch even when
ovirt-engine and httpd services are running while.
Any troubleshooting tips or has anyone seen same before?

puppet server suddenly stop working

I am new to puppet, it was working just fine till some days ago. I got back to start the server today, running
sudo /etc/init.d/puppetserver start
I then got
Starting puppetserver (via systemctl): puppetserver.serviceJob for puppetserver.service failed. See 'systemctl status puppetserver.service' and 'journalctl -xn' for details.
and journalctl -xn throws
No journal files were found.
failed!
When I check the logs in /var/log/puppetlabs/puppetserver, I get no specific informations. I also noticed that the puppet command that was located in /opt/puppetlabs/puppet/bin but I can't find the bin folder anymore.
Does anyone has an idea ?
EDIT :
Here's the output of systemctl status puppetserver.service
puppetserver.service - LSB: puppetserver
Loaded: loaded (/etc/init.d/puppetserver)
Active: failed (Result: exit-code) since Fri 2017-11-10 10:20:13 UTC; 3h 54min ago
Process: 5490 ExecStart=/etc/init.d/puppetserver start (code=exited, status=2)
and

cant connect to Datastax OpsCenter after brand new install

Can anyone help, I did a fresh install of Datastax OpsCenter/Lifecylce Manager, installed via apg-get on clean Ubuntu VMs. But, I can not connect via browser, "site not found", on port 8888.
ps returns a running process and Status seems started. Any ideas?
root#unigmadevdse1:/var/log/opscenter# ps -ef | grep opscenter root
19381 1890 0 20:49 pts/0 00:00:00 grep --color=auto opscenter
root#unigmadevdse1:/var/log/opscenter# service opscenterd status ●
opscenterd.service - LSB: Cassandra cluster manager Loaded: loaded
(/etc/init.d/opscenterd; bad; vendor preset: enabled) Active:
active (exited) since Tue 2017-03-07 20:39:54 UTC; 9min ago
Docs: man:systemd-sysv-generator(8) Process: 18848 ExecStart=/etc/init.d/opscenterd start (code=exited, status=0/SUCCESS)
Mar 07 20:39:53 unigmadevdse1.unigmagroup.com systemd[1]: Starting
LSB: Cassandra cluster manager...
Mar 07 20:39:54 unigmadevdse1.unigmagroup.com systemd[1]: Started LSB: >Cassandra cluster manager.
Have a look in the opscenter.log file for any error messages [if you are not sure, run a find / -iname opscenter.log
or try a service opscenter start -f [will output to screen]

Elasticsearch connection error in Ubuntu 16.4

In my ubuntu machine when I run the command curl -X GET 'http://localhost:9200' to test connection it show following message.
curl: (7) Failed to connect to localhost port 9200: Connection refused
When i check server status with sudo systemctl start elasticsearch it show following message.
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2016-11-20 16:32:30 BDT; 44s ago
Docs: http://www.elastic.co
Process: 8653 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefa
Process: 8649 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 8653 (code=exited, status=1/FAILURE)
Nov 20 16:32:29 bahar elasticsearch[8653]: 2016-11-20 16:32:25,579 main ERROR Null object returned for RollingFile in Appenders.
Nov 20 16:32:29 bahar elasticsearch[8653]: 2016-11-20 16:32:25,579 main ERROR Null object returned for RollingFile in Appenders.
Nov 20 16:32:29 bahar elasticsearch[8653]: 2016-11-20 16:32:25,580 main ERROR Unable to locate appender "rolling" for logger config "root"
Nov 20 16:32:29 bahar elasticsearch[8653]: 2016-11-20 16:32:25,580 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logge
Nov 20 16:32:29 bahar elasticsearch[8653]: 2016-11-20 16:32:25,581 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger
Nov 20 16:32:29 bahar elasticsearch[8653]: 2016-11-20 16:32:25,581 main ERROR Unable to locate appender "deprecation_rolling" for logger config "o
Nov 20 16:32:29 bahar elasticsearch[8653]: [2016-11-20T16:32:25,592][WARN ][o.e.c.l.LogConfigurator ] ignoring unsupported logging configuration
Nov 20 16:32:30 bahar systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Nov 20 16:32:30 bahar systemd[1]: elasticsearch.service: Unit entered failed state.
Nov 20 16:32:30 bahar systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
This is the error for the PATH and LOgs in the elasticsearch.yml (etc/elasticsearch/elasticsearch.yml)
Uncheck these path and your error will be removed.
That means elasticsearch is not running. And from what I see, there is a problem with starting it. Check your elasticsearch configuration.
check if Elasticsearch is running,run the follwing command:
$ ps aux|grep elasticsearch
if Elasticsearch is not started,check your JAVA Environment,download a new Elasticsearch and install it again:
1.check if JAVA is correctly installed:
$ java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)
if your JAVA version is lower 1.7,change a new one.
2.download Elasticsearch install package,unzip it:
$ tar -zxvf elasticsearch-2.3.3.gz
3. run Elasticsearch
$ cd elasticsearch-2.3.3
$ ./bin/elasticsearch
Usually it's the write permission issue for the log directory (default as /var/log/elasticsearch), use ls -l to check the permission and change mode to 777 for the log directory and files if necessary.
Long story short: a system reboot might get it OK.
It has been a while since the question is asked. Anyway, I ran into a similar problem recently.
The elasticsearch service on one of my nodes died, with error saying similar to those posted in the question when restart the service. It says the log folder to write is read-only file system. But these files and directories are indeed owned by user elasticsearch (version 5.5, deployed on Cent OS 6.5), there should not be a read-only problem.
I checked and didn't find a clue. So, I just reboot the system. After rebooting, everything goes all right without any further tuning: elasticsearch service starts on boot as configured, it finds the cluster and all the other nodes, and the cluster health status turns green after a little while.
I guess, the root reason might be some hardware failure in my case. All data and logs managed by elasticsearch cluster are stored in a 2TB SSD driver mounted on each node. And our hardware team just managed to recover from an external storage failure recently. All the nodes restarted during that recovery. Chances are there are some lagged issues caused the problem.

Resources