Recently a memsql leaf had a major failure and was not able to be brought up again. The corresponding instance was terminated (aws instance) and the ip of the node / leaf id does not appear in the leaves node anymore. However the ip/id appears in the followers list memsql-ops agent-list -q -r follower shows all my followers + this one that i want to remove.
Trying to run unmonitor on the specific id fails
memsql-ops memsql-unmonitor Af...
Failed to find MemSQL node matching id Af.... -> i assume this is normal given the fact the the specific instance was terminated
Is there any way to get rid of this id from the followers list and webUI?
memsql-ops agent-uninstall --force
https://docs.memsql.com/docs/agent-uninstall
Unless you are using HA the data on the leaf is likely gone and the MA has dropped it (if you have enterprise this would not happen). So you were just seeing ops -a separate process- trying to reach the ops-agent and failing. Since agent-uninstall expects to hear back from the agent, it will not succeed without --force
memsql-ops agent-uninstall --agent-id --force
The --force does the trick here
Related
I just updated a mariadb/galera-cluster to db version 10.3.15. It won't work correctly without at least 2 nodes up, but trying to start any node past the 1st runs into strange error messages, like: .
0 [Warning] WSREP: SST position can't be set in past. Requested: 0, Current: 14422308.
0 [Warning] WSREP: Can't continue.
This bug may be related:
https://jira.mariadb.org/browse/MDEV-17458?attachmentViewMode=list
However, I notice one peculiarity: the requested state is 0, quite possibly because it's lost somewhere along the way, or because I'm experiencing an entirely different issue.
I also know what it should be: the value that it thinks is 'current'.
In other words, reality is the exact opposite of what this node thinks is true: the 'current' should be 0, the 'requested' should be 14422308.
In a related issue:
https://jira.mariadb.org/browse/MDEV-19193
someone off-hand comments about deleting some files in order to start from a pristine case, but isn't exactly clear what exactly to do where.
I do not mind starting from the data on one node, ignoring everything on the other nodes and copying everything over.
I tried deleting the following file(s) from the offending nodes. (I believe the data directory they're mentioning is /var/lib/mysql/ on most linux systems):
galera.cache
ib_logfile0
ib_logfile1
This has no effect.
Someone over at this question: Unable to complete SST transfer due to "WSREP: SST position can't be set in past." error suggests changing the SST number on the node that's still OK. But that won't work: I can only start that node if I use the 'galera_new_cluster' script, which resets its SST number to '-1', no matter what it was. If I start it normally, I get an error like this:
[ERROR] WSREP: wsrep::connect(gcomm://<IP1>,<IP2>,<IP3>,...) failed: 7
In other words, there's not enough other nodes online to join the cluster. So in order to change the SST on the primary node, another node needs to be online, but in order to start up the other node, I need to change the SST on the primary? Catch-22, won't work.
It's nice that they fixed the bug, but how do I fix my now broken cluster?
One more question I've asked myself is this: Does this 'SST number' of 14422308 originate from the node that's trying to re-join the cluster, or is it retrieved from the cluster? Apparently, the second thing is true, for even completely reinstalling the secondary node from scratch and trying to re-join the cluster with it will not solve the problem. The exact same error message stays.
Somehow, the cluster appears to have gotten confused as to its own state. The JOINER nodes in each synchronization step think they have a more advanced state than the DONOR nodes.
The solution to this problem is to trick the cluster; to force it to recognize some node as 'more advanced'.
Suppose we can identify one node that has complete cluster data. Denote this to be the '1st node'. Pick one node to be the 2nd, one to be the 3rd, etc. (These choices can be at-random).
Then, stop mysql on all nodes. Edit the configuration file for the cluster and change the value for 'wsrep_cluster_address' on each node. It should be the following:
+------+---------------------------+
| Node | wsrep_cluster_address |
+------+---------------------------+
| 1 | gcomm:// |
| 2 | gcomm://<IP1>,<IP2> |
| 3 | gcomm://<IP1>,<IP2>,<IP3> |
+------+---------------------------+
(The pattern continues like this for the fourth and any further nodes in the cluster).
Now remove all cached data from the nodes other than the first. These are the files:
ib_logfile*
grastate.dat
gvwstate.dat
galera.cache
situated in the data dir of the mysql installation. (Example; /var/lib/mysql/ on debian systems).
Then edit the "grastate.dat" file on node #1. In our example, the most advanced state the cluster has yet seen is 14422308. Thus set it to 14422309 (or: old state + 1). Also set safe_to_bootstrap to 0 on all nodes (so we don't accidentally try to bootstrap and lose our seqno, running into the same bug again).
Now start mysql on node #1 (example, via systemd: systemctl start mysql).
Once it's running, do the same on node #2. Wait for all the data to transfer (this may take a while depending on the inter-node connection speed and the size of the database in question), then repeat for node 3, and any further nodes.
Afterwards, restore the value for wsrep_cluster_address in every configuration to what it should be (which is equal to the value for the last node).
I am using community edition of memsql. I got this error while i was running a query today. So i just restarted my cluster and got this error solved.
memsql-ops cluster-restart
But what happened and what should i do in future to avoid this error ?
NOTE
I donot want to buy the Enterprise edition.
Question
Is this problem of Availability ?
I got this error when experimenting with performance.
VM had 24 CPUs and 25 nodes: 1 Master Agg, 24 Leaf nodes
Reduced VM to 4 CPUs and restarted cluster.
All the leaves did not recover.
All except 4 recovered in < 5 minutes.
20 minutes later, 4 leaf nodes still were not connected.
From MySQL/MemSQL prompt:
use db;
show partitions;
I notice some partitions with ordinal from 0-71 for me have null instead Host, Port, Role defined for a given partition.
In memsql ops UI http://server:9000 > Settings > Config > Manual Cluster Control I checked "ENABLE MANUAL CONTROL" while I tried to run various commands with no real benefit.
Then 15 minutes later, I unchecked the box, Memsql-ops tried attaching all the leaf nodes again and was finally successful.
Perhaps a cluster restart would have done the same thing.
This happened because a leaf in your cluster has failed a health check heartbeat for some reason (loss of network connectivity, hardware failure, OS issue, machine overloaded, out of memory, etc.) and its partitions are no longer accessible to query. MemSQL Community Edition only supports redundancy 1 so there are no other copies of the data on the failed leaf node in your cluster (thus the error about missing a partition of data - MemSQL can't complete a query that needs to read data on any partitions on the problem leaf).
Given that a restart repaired things, the most likely answer is that linux "out of memory" killed you: MemSQL Linux OOM killer docs
You can also check the tracelog on the leaf that ran into issues to see if there is any clue there about what happened (It's usually at /var/lib/memsql/leaf_3306/tracelogs/memsql.log)
-Adam
I too have faced this error, that was because some of the slave ordinals had no corresponding masters. My error message looked like:
ERROR 1772 (HY000) at line 1: Leaf Error (10.0.0.112:3306): Partition database `<db_name>_0` can't be promoted to master because it is provisioning replication
My memsql> SHOW PARTITIONS; command returned the following.
So what approach I followed was to remove each of such cases (where the role was either Slave or NULL).
DROP PARTITION <db_name>:4 ON "10.0.0.193":3306;
..
DROP PARTITION <db_name>:46 ON "10.0.0.193":3306;
And then created a new partition with each of the dropped partition.
CREATE PARTITION <db_name>:4 ON "10.0.0.193":3306;
..
CREATE PARTITION <db_name>:46 ON "10.0.0.193":3306;
And this was the result of memsql> SHOW PARTITIONS; after that.
You can refer to the MemSQL Documentation regarding partitions, here if the above steps doesn't seem to solve your problem.
I was hitting the same problem. Using the following command in the master node, solved the problem:
REBALANCE PARTITIONS ON db_name
Optionally you can force it using FORCE:
REBALANCE PARTITIONS ON db_name FORCE
And to see the list of operations when rebalancing is going to execute, use above command with EXPLAIN:
EXPLAIN REBALANCE PARTITIONS ON db_name [FORCE]
I've upgraded cassandra from 1.2.13 to 2.0.4 on a cluster of 5 nodes.
when i run nodetool -h localhost ring I see this errormessage in the end:
ERROR 10:33:28,324 Unable to initialize MemoryMeter (jamm not specified as javaagent). This means Cassandra will be unable to measure object sizes accurately and may consequently OOM.
according to this:
https://issues.apache.org/jira/browse/CASSANDRA-6404
it should be fixed.
i'm running java-1.7.0-oracle-1.7.0.45-1jpp.2.el6_4.x86_64.
this is the beginning of the process options:
/usr/lib/jvm/java-1.7.0-oracle-1.7.0.45.x86_64/jre/bin/java -ea -javaagent:/usr/share/cassandra//lib/jamm-0.2.5.jar
is there anyone who could point me in a direction where too look for a solution?
these error's, are they serious or mere cosmetic?
//john
If this error only emits from tools, then ignore. If you also see it in your output.log or system.log, then it can be problematic. It seems your version of JVM is good. If the command prompt you pasted here is from Cassandra process, then you are good. Tools have a different script to initialize their environment. Check cassandra/bin folder and inspect tools' scripts to see if they include this change:
https://issues.apache.org/jira/secure/attachment/12615604/0001-Set-javaagent-when-running-tools-in-bin.patch
Chances are somehow your upgrade process didn't change them.
I tried adding a job to sqe by qsub. But it seems to be stuck. The state is shown as 'dt'. What could be wrong? I cannot add run any more jobs due to this. How can I remove the job from queue?
Looks like a grid engine status. If that is the case the 'd' means the job is in the process of being deleted while the 't' means the job is being transfered to the node where it is supposed to run. This combination usually occurs only if the node crashed while the job was being transfered to it.
You should be able to delete it with qdel -f JOBID if you are the administrator or the cluster is appropriately configured. If not ask your sysadmin/support to do it for you.
I run:
gluster volume heal myvol info heal-failed
and I get back a whole bunch of entries. Is this normal? Is anyone else out there seeing this in their implementation of glusterfs? If so, how do you go about resolving this?
List of entries from "gluster volume heal myvol info heal-failed" can be real failure or it could just list the entries which self-heal-daemon failed to self-heal in that crawl.
Gradually the file/directory which is listed under "heal-failed" entry would be self-healed by self-heal-daemon.
It is normal to see heal-failed entries.