cassandra.service: main process exited, code=killed, status=11/SEGV
env:
apache-cassandra-4.0.0
jdk-11.0.12
ZGC
jvm:
-Xms31G
-Xmx31G
host:
16core 128G
/var/log/message:
Jul 4 13:57:10 iZ2zec1q29sosy4bdv893qZ systemd-logind: Removed session 277.
Jul 4 13:57:12 iZ2zec1q29sosy4bdv893qZ cassandra: INFO [CompactionExecutor:4] 2022-07-04 13:57:12,074 CompactionTask.java:245 - Compacted (24af5250-fb5e-11ec-aa2a-6b96728ba428)
4 sstables to [/data/cassandra/data/data/spaceport/xm_coupon_code_realtime1-d77e7f10ebcc11ecae252faeea3c28c4/nb-6494-big,] to level=0. 27.414MiB to 27.412MiB (~99% of original) i
n 1,812ms. Read Throughput = 15.127MiB/s, Write Throughput = 15.126MiB/s, Row Throughput = ~123,625/s. 32,718 total partitions merged to 32,689. Partition merge counts were {1:
32663, 2:23, 3:3, }
Jul 4 13:57:12 iZ2zec1q29sosy4bdv893qZ cassandra: INFO [NonPeriodicTasks:1] 2022-07-04 13:57:12,083 SSTable.java:111 - Deleting sstable: /data/cassandra/data/data/spaceport/xm_c
oupon_code_realtime1-d77e7f10ebcc11ecae252faeea3c28c4/nb-6490-big
Jul 4 13:57:12 iZ2zec1q29sosy4bdv893qZ cassandra: INFO [NonPeriodicTasks:1] 2022-07-04 13:57:12,084 SSTable.java:111 - Deleting sstable: /data/cassandra/data/data/spaceport/xm_c
oupon_code_realtime1-d77e7f10ebcc11ecae252faeea3c28c4/nb-6493-big
Jul 4 13:57:12 iZ2zec1q29sosy4bdv893qZ cassandra: INFO [NonPeriodicTasks:1] 2022-07-04 13:57:12,085 SSTable.java:111 - Deleting sstable: /data/cassandra/data/data/spaceport/xm_c
oupon_code_realtime1-d77e7f10ebcc11ecae252faeea3c28c4/nb-6491-big
Jul 4 13:57:12 iZ2zec1q29sosy4bdv893qZ cassandra: INFO [NonPeriodicTasks:1] 2022-07-04 13:57:12,085 SSTable.java:111 - Deleting sstable: /data/cassandra/data/data/spaceport/xm_c
oupon_code_realtime1-d77e7f10ebcc11ecae252faeea3c28c4/nb-6492-big
Jul 4 14:00:01 iZ2zec1q29sosy4bdv893qZ systemd: Started Session 293 of user root.
Jul 4 14:01:01 iZ2zec1q29sosy4bdv893qZ systemd: Started Session 294 of user root.
Jul 4 14:01:59 iZ2zec1q29sosy4bdv893qZ systemd: cassandra.service: main process exited, code=killed, status=11/SEGV
Jul 4 14:02:00 iZ2zec1q29sosy4bdv893qZ systemd: Unit cassandra.service entered failed state.
Jul 4 14:02:00 iZ2zec1q29sosy4bdv893qZ systemd: cassandra.service failed.
Jul 4 14:02:05 iZ2zec1q29sosy4bdv893qZ systemd: cassandra.service holdoff time over, scheduling restart.
Jul 4 14:02:05 iZ2zec1q29sosy4bdv893qZ systemd: Stopped Cassandra Server Service.
Jul 4 14:02:05 iZ2zec1q29sosy4bdv893qZ systemd: Started Cassandra Server Service.
Jul 4 14:02:55 iZ2zec1q29sosy4bdv893qZ cassandra: CompileCommand: dontinline org/apache/cassandra/db/Columns$Serializer.deserializeLargeSubset(Lorg/apache/cassandra/io/util/DataI
nputPlus;Lorg/apache/cassandra/db/Columns;I)Lorg/apache/cassandra/db/Columns;
The log entries you posted on their own don't explain what the problem is. You will need to review the Cassandra system.log for clues.
A friendly reminder that Stack Overflow is for getting help with coding, algorithm, or programming language problems. For future reference, you should post DB admin/ops questions on dba.stackexchange.com. If you post it there, I'd be happy to help. Cheers!
Related
I created a three Cassandra node initially it's working fine, but now 2 nodes are stop working.
I tried
sudo service dse stop
and
sudo service dse start
got below error
Job for dse.service failed because the control process exited with error code.
See "systemctl status dse.service" and "journalctl -xe" for details.
systemctl status dse.service
● dse.service - LSB: DataStax Enterprise
Loaded: loaded (/etc/init.d/dse; generated)
Active: failed (Result: exit-code) since Tue 2020-03-17 04:34:24 UTC; 4min 43s ago
Docs: man:systemd-sysv-generator(8)
Process: 4263 ExecStop=/etc/init.d/dse stop (code=exited, status=0/SUCCESS)
Process: 11273 ExecStart=/etc/init.d/dse start (code=exited, status=1/FAILURE)
Tasks: 0 (limit: 4915)
CGroup: /system.slice/dse.service
Mar 17 04:34:14 cstar-node1 su[11442]: pam_unix(su:session): session closed for user cassandra
Mar 17 04:34:14 cstar-node1 su[11456]: Successful su for cassandra by root
Mar 17 04:34:14 cstar-node1 su[11456]: + ??? root:cassandra
Mar 17 04:34:14 cstar-node1 su[11456]: pam_unix(su:session): session opened for user cassandra by (uid=0)
Mar 17 04:34:14 cstar-node1 su[11456]: pam_unix(su:session): session closed for user cassandra
Mar 17 04:34:24 cstar-node1 dse[11273]: ERROR: DSE failed to start. Please check your logs.
Mar 17 04:34:24 cstar-node1 dse[11273]: ...fail!
Mar 17 04:34:24 cstar-node1 systemd[1]: dse.service: Control process exited, code=exited status=1
Mar 17 04:34:24 cstar-node1 systemd[1]: dse.service: Failed with result 'exit-code'.
Mar 17 04:34:24 cstar-node1 systemd[1]: Failed to start LSB: DataStax Enterprise.
only one node is UP
nodetool status
Datacenter: Cassandra
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
DN X.X.X.X ? 1 ? 46fdfb5e-238c-476b-a243-184a530fg30e rack1
UN X.X.X.Y 207.4 KiB 1 ? 7fasd242-891d-4ecf-ggef-0f8hffarr434 rack1
DN X.X.X.Z ? 1 ? 34ffda2f-46d2-443d-4546-33c55cface2c rack1
how to resolve this error? can anyone help me.
Thanks in advance.
It's some time ago - so even if it is not of help for you anymore, it might for others. I had the same issue but there hasn't been any entry in the cassandra logs nor in the system logs. Also the start process failed with that non-descriptive message above. To resolve the issue I've been stopping (as root):
the agent: systemctl stop datastax-agent and
the dse service: systemctl stop dse
Then deleted the directories where the PIDs are located:
/var/run/datastax-agent
/var/run/dse
And finally restarted both services. That did the trick for me. I cannot say if the deletion of the PIDs or restarting the datastax-agent actually resolved the problem but I my blind guess would fall on the PIDs.
Below is my filebeat.yml file , I am unable to start service getting error
filebeat.prospectors:
- input_type: log
** paths:**
** - /opt/apache-tomcat-7.0.82/logs/*.log**
document_type: apache-access
fields_under_root: true
output.logstash:
** hosts: '${host}'**
host is environment variable i did export host="10.2.3.1:5044"
Apr 10 06:59:35 node1 filebeat[401]: Exiting: error initializing publisher: missing field accessing 'output.logstash.hosts' (source:'/etc/filebeat/filebeat.yml')
Apr 10 06:59:35 node1 systemd[1]: filebeat.service: main process exited, code=exited, status=1/FAILURE
Apr 10 06:59:35 node1 systemd[1]: Unit filebeat.service entered failed state.
Apr 10 06:59:35 node1 systemd[1]: filebeat.service failed.
Apr 10 06:59:35 node1 systemd[1]: filebeat.service holdoff time over, scheduling restart.
Apr 10 06:59:35 node1 systemd[1]: start request repeated too quickly for filebeat.service
Apr 10 06:59:35 node1 systemd[1]: Failed to start filebeat.
Apr 10 06:59:35 node1 systemd[1]: Unit filebeat.service entered failed state.
Apr 10 06:59:35 node1 systemd[1]: filebeat.service failed.``
I found solution, That's the issue is Systemd does not pass environment variables. I started filebeat as ./filebeat -e -c filebeat.yml it worked for me
By looking at this I see multiple syntax differences to your configuration. There are some examples at the bottom of the page even for hosts.
https://www.elastic.co/guide/en/beats/filebeat/1.2/using-environ-vars.html
Hope it helps.
We have an application that is running on RHEL6/32 bit and RHEL6/64 bit. This application uses postgresql 8.4 from the beginning. Now, we want to provide support for this application on RHEL7/64 bit. RHEL7 comes with default postgresql 9.2 in its yum list and this is getting installed and its related services are running properly as well. But after installing postgresql 8.4 on RHEL7, it seems like the services are never running. Please find below the logs:
[root#linpubn218 postgres]# service postgresql status
postgresql.service - SYSV: PostgreSQL database server.
Loaded: loaded (/etc/rc.d/init.d/postgresql)
Active: failed (Result: resources) since Mon 2016-07-25 12:40:28 IST; 2h 0min ago
Docs: man:systemd-sysv-generator(8)
Jul 25 12:40:26 linpubn218.gl.avaya.com systemd[1]: Starting SYSV: PostgreSQL database server....
Jul 25 12:40:28 linpubn218.gl.avaya.com postgresql[26957]: Starting postgresql service: [ OK ]
Jul 25 12:40:28 linpubn218.gl.avaya.com systemd[1]: PID file /var/run/postmaster-8.4.pid not readable (yet?) after start.
Jul 25 12:40:28 linpubn218.gl.avaya.com systemd[1]: Failed to start SYSV: PostgreSQL database server..
Jul 25 12:40:28 linpubn218.gl.avaya.com systemd[1]: Unit postgresql.service entered failed state.
Jul 25 12:40:28 linpubn218.gl.avaya.com systemd[1]: postgresql.service failed.
Jul 25 14:33:45 linpubn218.gl.avaya.com systemd[1]: Unit postgresql.service cannot be reloaded because it is inactive.
Jul 25 14:33:45 linpubn218.gl.avaya.com systemd[1]: Unit postgresql.service cannot be reloaded because it is inactive.
After looking at the logs in journalctl -xe
[root#linpubn218 postgres]# journalctl -xe
Jul 25 14:39:21 linpubn218.gl.avaya.com yum[29260]: Installed: postgresql84-libs-8.4.17-1PGDG.rhel6.x86_64
Jul 25 14:39:45 linpubn218.gl.avaya.com yum[29275]: Installed: postgresql84-8.4.17-1PGDG.rhel6.x86_64
Jul 25 14:40:01 linpubn218.gl.avaya.com useradd[29316]: failed adding user 'postgres', exit code: 9
Jul 25 14:40:02 linpubn218.gl.avaya.com CROND[29320]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jul 25 14:40:02 linpubn218.gl.avaya.com systemd[1]: Reloading.
Jul 25 14:40:03 linpubn218.gl.avaya.com systemd[1]: Configuration file /usr/lib/systemd/system/auditd.service is marked world-inaccessible. This has no effect as config
Jul 25 14:40:03 linpubn218.gl.avaya.com yum[29309]: Installed: postgresql84-server-8.4.17-1PGDG.rhel6.x86_64
Jul 25 14:42:05 linpubn218.gl.avaya.com polkitd[819]: Registered Authentication Agent for unix-process:29459:43987285 (system bus name :1.292 [/usr/bin/pkttyagent --not
Jul 25 14:42:05 linpubn218.gl.avaya.com systemd[1]: Starting SYSV: PostgreSQL database server....
Jul 25 14:42:06 linpubn218.gl.avaya.com runuser[29473]: pam_unix(runuser-l:session): session closed for user postgres
Jul 25 14:42:08 linpubn218.gl.avaya.com postgresql[29464]: Starting postgresql service: [ OK ]
Jul 25 14:42:08 linpubn218.gl.avaya.com systemd[1]: PID file /var/run/postmaster-8.4.pid not readable (yet?) after start.
Jul 25 14:42:08 linpubn218.gl.avaya.com systemd[1]: Failed to start SYSV: PostgreSQL database server..
Can postgresql 8.4 be installed on RHEL7, which is a systemd based OS? If yes, then what should I do to remove the above error?
I noticed that in /etc/init.d/postgresql-8.4 there is a declared variable:
pidfile="/var/run/postmaster-${PGMAJORVERSION}.${PGPORT}.pid"
But in systemctl, PIDfile is not the same:
# systemctl show postgresql-8.4.service -p PIDFile
PIDFile=/var/run/postmaster-8.4.pid
So, to fix the problem edit /etc/init.d/postgresql-8.4 and replace
pidfile="/var/run/postmaster-${PGMAJORVERSION}.${PGPORT}.pid"
with
pidfile="/var/run/postmaster-${PGMAJORVERSION}.pid"
then reload systemctl:
# systemctl daemon-reload
#/etc/init.d/postgresql-8.4 start
Starting postgresql-8.4 (via systemctl): [ OK ]
Generally permissions caused this type of error
su - postgres
After that:
chmod 700 -R <data_directory>
And you should check SELinux as well.
I'm having a dilemma here. I was required to (attempt to) upgrade MongoDB on my CentOS 7 server from 2.6.X to 3.0+. I tried following the basic guide from Mongo (replacing the binaries directly) and this worked perfectly well... in local. On the server my MongoDB service is totally flipping out and I have no idea. And on top of that Mongo Shell is still at 2.6 somehow XD
systemctl status mongo* reveals this catastrophe:
root#staging:~# systemctl status mongo*
● mongod.service - SYSV: Mongo is a scalable, document-oriented database.
Loaded: loaded (/etc/rc.d/init.d/mongod)
Active: failed (Result: exit-code) since 一 2016-01-25 16:57:13 CST; 18h ago
Docs: man:systemd-sysv-generator(8)
1月 25 16:57:13 staging systemd[1]: Starting SYSV: Mongo is a scalable, document-oriented database....
1月 25 16:57:13 staging runuser[5310]: pam_unix(runuser:session): session opened for user mongod by (uid=0)
1月 25 16:57:13 staging runuser[5310]: pam_unix(runuser:session): session closed for user mongod
1月 25 16:57:13 staging mongod[5301]: Starting mongod: [FAILED]
1月 25 16:57:13 staging systemd[1]: mongod.service: control process exited, code=exited status=1
1月 25 16:57:13 staging systemd[1]: Failed to start SYSV: Mongo is a scalable, document-oriented database..
1月 25 16:57:13 staging systemd[1]: Unit mongod.service entered failed state.
1月 25 16:57:13 staging systemd[1]: mongod.service failed.
1月 26 11:03:04 staging systemd[1]: Stopped SYSV: Mongo is a scalable, document-oriented database..
1月 26 11:04:52 staging systemd[1]: Stopped SYSV: Mongo is a scalable, document-oriented database..
● mongos.service
Loaded: not-found (Reason: No such file or directory)
Active: failed (Result: exit-code) since 一 2016-01-25 15:46:20 CST; 20h ago
1月 25 15:46:20 staging systemd[1]: Starting High-performance, schema-free document-oriented database...
1月 25 15:46:20 staging mongos[2712]: /usr/bin/mongos: error while loading shared libraries: libssl.so.1.0.0: cannot open shared object file: No such f... directory
1月 25 15:46:20 staging systemd[1]: mongos.service: control process exited, code=exited status=127
1月 25 15:46:20 staging systemd[1]: Failed to start High-performance, schema-free document-oriented database.
1月 25 15:46:20 staging systemd[1]: Unit mongos.service entered failed state.
1月 25 15:46:20 staging systemd[1]: mongos.service failed.
1月 25 16:04:23 staging systemd[1]: Stopped High-performance, schema-free document-oriented database.
1月 26 11:18:04 staging systemd[1]: Stopped mongos.service.
Hint: Some lines were ellipsized, use -l to show in full.
Any assistance at all would be greatly appreciated!
Thanks again, as always.
This was ultimately solved by yum remove mongo* followed by manually removing ANYTHING referring to Mongo in any way (found using locate mongo*). Then adding an up to date Mongo repo and installing v3.2.1 via yum (contrary to the more commonplace suggestion from MongoDB to simply replace the binaries directly).
When I try to start the apache this happens:
Job for apache2.service failed. See ‘systemctl status apache2.service’ and ‘journalctl -xn’ for details.
system.ctl status apache2.service -l returns me this:
Mar 24 23:41:57 glauber-pc.site start_apache2[3249]: httpd2-prefork: Syntax error on line 179 of /etc/apache2/httpd.conf: Syntax error on line 102 of /etc/apache2/default-server.conf: Syntax error on line 1 of /etc/apache2/conf.d/mod_evasive.conf: Cannot load /usr/lib64/apache2/mod_evasive20.so into server: /usr/lib64/apache2/mod_evasive20.so: cannot open shared object file: No such file or directory
Mar 24 23:41:57 glauber-pc.site systemd[1]: apache2.service: main process exited, code=exited, status=1/FAILURE
Mar 24 23:41:57 glauber-pc.site systemd[1]: Failed to start The Apache Webserver.
Mar 24 23:41:57 glauber-pc.site systemd[1]: Unit apache2.service entered failed state.
Everything seems right in those lines, any clue what it can be?
Change the first line in
/etc/apache2/conf.d/mod_evasive.conf
to
LoadModule evasive20_module /usr/lib64/apache2/mod_evasive24.so
(...24.so instead of ...20.so)