I just updated a mariadb/galera-cluster to db version 10.3.15. It won't work correctly without at least 2 nodes up, but trying to start any node past the 1st runs into strange error messages, like: .
0 [Warning] WSREP: SST position can't be set in past. Requested: 0, Current: 14422308.
0 [Warning] WSREP: Can't continue.
This bug may be related:
https://jira.mariadb.org/browse/MDEV-17458?attachmentViewMode=list
However, I notice one peculiarity: the requested state is 0, quite possibly because it's lost somewhere along the way, or because I'm experiencing an entirely different issue.
I also know what it should be: the value that it thinks is 'current'.
In other words, reality is the exact opposite of what this node thinks is true: the 'current' should be 0, the 'requested' should be 14422308.
In a related issue:
https://jira.mariadb.org/browse/MDEV-19193
someone off-hand comments about deleting some files in order to start from a pristine case, but isn't exactly clear what exactly to do where.
I do not mind starting from the data on one node, ignoring everything on the other nodes and copying everything over.
I tried deleting the following file(s) from the offending nodes. (I believe the data directory they're mentioning is /var/lib/mysql/ on most linux systems):
galera.cache
ib_logfile0
ib_logfile1
This has no effect.
Someone over at this question: Unable to complete SST transfer due to "WSREP: SST position can't be set in past." error suggests changing the SST number on the node that's still OK. But that won't work: I can only start that node if I use the 'galera_new_cluster' script, which resets its SST number to '-1', no matter what it was. If I start it normally, I get an error like this:
[ERROR] WSREP: wsrep::connect(gcomm://<IP1>,<IP2>,<IP3>,...) failed: 7
In other words, there's not enough other nodes online to join the cluster. So in order to change the SST on the primary node, another node needs to be online, but in order to start up the other node, I need to change the SST on the primary? Catch-22, won't work.
It's nice that they fixed the bug, but how do I fix my now broken cluster?
One more question I've asked myself is this: Does this 'SST number' of 14422308 originate from the node that's trying to re-join the cluster, or is it retrieved from the cluster? Apparently, the second thing is true, for even completely reinstalling the secondary node from scratch and trying to re-join the cluster with it will not solve the problem. The exact same error message stays.
Somehow, the cluster appears to have gotten confused as to its own state. The JOINER nodes in each synchronization step think they have a more advanced state than the DONOR nodes.
The solution to this problem is to trick the cluster; to force it to recognize some node as 'more advanced'.
Suppose we can identify one node that has complete cluster data. Denote this to be the '1st node'. Pick one node to be the 2nd, one to be the 3rd, etc. (These choices can be at-random).
Then, stop mysql on all nodes. Edit the configuration file for the cluster and change the value for 'wsrep_cluster_address' on each node. It should be the following:
+------+---------------------------+
| Node | wsrep_cluster_address |
+------+---------------------------+
| 1 | gcomm:// |
| 2 | gcomm://<IP1>,<IP2> |
| 3 | gcomm://<IP1>,<IP2>,<IP3> |
+------+---------------------------+
(The pattern continues like this for the fourth and any further nodes in the cluster).
Now remove all cached data from the nodes other than the first. These are the files:
ib_logfile*
grastate.dat
gvwstate.dat
galera.cache
situated in the data dir of the mysql installation. (Example; /var/lib/mysql/ on debian systems).
Then edit the "grastate.dat" file on node #1. In our example, the most advanced state the cluster has yet seen is 14422308. Thus set it to 14422309 (or: old state + 1). Also set safe_to_bootstrap to 0 on all nodes (so we don't accidentally try to bootstrap and lose our seqno, running into the same bug again).
Now start mysql on node #1 (example, via systemd: systemctl start mysql).
Once it's running, do the same on node #2. Wait for all the data to transfer (this may take a while depending on the inter-node connection speed and the size of the database in question), then repeat for node 3, and any further nodes.
Afterwards, restore the value for wsrep_cluster_address in every configuration to what it should be (which is equal to the value for the last node).
Related
I'm using the latest Spark 3.3.0 but still got the exception: java.io.FileNotFoundException: File does not exist: /LOG_DIR/application_1657344020931_1400038_1.inprogress
which contradicts https://github.com/apache/spark/pull/29350 , how could it be?
In LOG_DIR there are 70k eventlogs.
One SHS is running on nodeA with Spark2.4.5.
In order to resolve the exception,I run a new SHS one node B with Spark 3.3.0 ,but still got the exception.
The basic problem is that some finished spark applications randomly cannot be seen from the SHS.I think the FileNotFoundException is the main cause and the PR is aimed to resolve it.
I re-checked the whole FsHistoryProvider.checkForLogs(...) and found that I misunderstood the PR.
My question: SHS always shows less (COMPLETED) apps than the number of event logs(exculde inprogress) in my LOG_DIR
This situation will always happens because when SHS found an untracked log eg. aaa._inprogress, at the moment checkForLogs is writing the aaa._inprogress to LevelDB, Spark renames it to aaa, thus FileNotFoundException happens and the loop is aborted.
So the rest of the logs which may be 10% of all apps in the loop are missed. That's why my SHS less so many than the LOG_DIR.
THE PR above skip such problematic aaa event logs in this loop, but next time, since aaa is not in LevelDB, it will be tracked.
SHS still show less than the num of eventlogs, but just 40~70 apps in my cluster and these apps will show soon
I am trying to sum continuous stream of numbers from a file using hazelcast jet
pipe
.drawFrom(Sources.fileWatcher)<dir>))
.map(s->Integer.parseInt(s))
.addTimestamps()
.window(WindowDefinition.sliding(10000,1000))
.aggregate(AggregateOperations.summingDouble(x->x))
.drainTo(Sinks.logger());
Few questions
It doesn't give the expected output, my expectation is as soon as new number appears in the file, it should just add it to the existing sum
To do this why i need to give window and addTimestamp method, i just need to do sum of infinite stream
How can we achieve fault tolerance, i. e. if server restarts will it save the aggregated result and when it comes up it will aggregate from the last computed sum?
if the server is down and few numbers come in file now when the server comes up, will it read from last point from when the server went down or will it miss the numbers when it was down and will only read the number it got after the server was up.
Answer to Q1 & Q2:
You're looking for rollingAggregate, you don't need timestamps or windows.
pipe
.drawFrom(Sources.fileWatcher(<dir>))
.rollingAggregate(AggregateOperations.summingDouble(Double::parseDouble))
.drainTo(Sinks.logger());
Answer to Q3 & Q4: the fileWatcher source isn't fault tolerant. The reason is that it reads local files and when a member dies, the local files won't be available anyway. When the job restarts, it will start reading from current position and will miss numbers added while the job was down.
Also, since you use global aggregation, data from all files will be routed to single cluster member and other members will be idle.
We have a spark 1.6.1 application, which takes input from two kafka topics and writes the result to another kafka topic. The application receives some large (approximately 1MB) files in the first input topic and some simple conditions from the second input topic. If the condition is satisfied, the file is written to output topic else held in state (we use mapWithState).
The logic works fine for less (few hundred) number of input files, but fails with org.apache.spark.rpc.RpcTimeoutException and recommendation is to increase spark.rpc.askTimeout. After increasing from default (120s) to 300s the ran fine longer but crashed with the same error after 1 hour. After changing the value to 500s, the job ran fine for more than 2 hours.
Note: We are running the spark job in local mode and kafka is also running locally in the machine. Also, some time I see warning "[2016-09-06 17:36:05,491] [WARN] - [org.apache.spark.storage.MemoryStore] - Not enough space to cache rdd_2123_0 in memory! (computed 2.6 GB so far)"
Now, 300s seemed large enough a timeout considering all local configuration. But any idea, how to come up to an ideal timeout value instead of just using 500s or higher based on testing, as I see crashed cases using 800s and cases suggesting to use 60000s?
I was facing the same problem, I found this page saying that under heavy workloads it is wise to set spark.network.timeout(which controls all the timeouts, also the RPC one) to 800. At the moment it solved my problem.
I am using community edition of memsql. I got this error while i was running a query today. So i just restarted my cluster and got this error solved.
memsql-ops cluster-restart
But what happened and what should i do in future to avoid this error ?
NOTE
I donot want to buy the Enterprise edition.
Question
Is this problem of Availability ?
I got this error when experimenting with performance.
VM had 24 CPUs and 25 nodes: 1 Master Agg, 24 Leaf nodes
Reduced VM to 4 CPUs and restarted cluster.
All the leaves did not recover.
All except 4 recovered in < 5 minutes.
20 minutes later, 4 leaf nodes still were not connected.
From MySQL/MemSQL prompt:
use db;
show partitions;
I notice some partitions with ordinal from 0-71 for me have null instead Host, Port, Role defined for a given partition.
In memsql ops UI http://server:9000 > Settings > Config > Manual Cluster Control I checked "ENABLE MANUAL CONTROL" while I tried to run various commands with no real benefit.
Then 15 minutes later, I unchecked the box, Memsql-ops tried attaching all the leaf nodes again and was finally successful.
Perhaps a cluster restart would have done the same thing.
This happened because a leaf in your cluster has failed a health check heartbeat for some reason (loss of network connectivity, hardware failure, OS issue, machine overloaded, out of memory, etc.) and its partitions are no longer accessible to query. MemSQL Community Edition only supports redundancy 1 so there are no other copies of the data on the failed leaf node in your cluster (thus the error about missing a partition of data - MemSQL can't complete a query that needs to read data on any partitions on the problem leaf).
Given that a restart repaired things, the most likely answer is that linux "out of memory" killed you: MemSQL Linux OOM killer docs
You can also check the tracelog on the leaf that ran into issues to see if there is any clue there about what happened (It's usually at /var/lib/memsql/leaf_3306/tracelogs/memsql.log)
-Adam
I too have faced this error, that was because some of the slave ordinals had no corresponding masters. My error message looked like:
ERROR 1772 (HY000) at line 1: Leaf Error (10.0.0.112:3306): Partition database `<db_name>_0` can't be promoted to master because it is provisioning replication
My memsql> SHOW PARTITIONS; command returned the following.
So what approach I followed was to remove each of such cases (where the role was either Slave or NULL).
DROP PARTITION <db_name>:4 ON "10.0.0.193":3306;
..
DROP PARTITION <db_name>:46 ON "10.0.0.193":3306;
And then created a new partition with each of the dropped partition.
CREATE PARTITION <db_name>:4 ON "10.0.0.193":3306;
..
CREATE PARTITION <db_name>:46 ON "10.0.0.193":3306;
And this was the result of memsql> SHOW PARTITIONS; after that.
You can refer to the MemSQL Documentation regarding partitions, here if the above steps doesn't seem to solve your problem.
I was hitting the same problem. Using the following command in the master node, solved the problem:
REBALANCE PARTITIONS ON db_name
Optionally you can force it using FORCE:
REBALANCE PARTITIONS ON db_name FORCE
And to see the list of operations when rebalancing is going to execute, use above command with EXPLAIN:
EXPLAIN REBALANCE PARTITIONS ON db_name [FORCE]
Been using a 6GB dataset with each source record being ~1KB in length when I accidentally added an index on a column that I am pretty sure has a 100% cardinality.
Tried dropping the index from cqlsh but by that point the two node cluster had gone into a run away death spiral with loadavg surpassing 20 on each node and cqlsh hung on the drop command for 30 minutes. Since this was just a test setup, I shut-down and destroyed the cluster and restarted.
This is a fairly disconcerting problem as it makes me fear a scenario where a junior developer is on a production cluster and they set an index on a similar high cardinality column. I scanned through the documentation and looked at the options in nodetool but there didn't seem to be anything along the lines of "abort job or abort building index".
Test environment:
2x m1.xlarge EC2 instances with 2 Raid 0 ephemeral disks
Dataset was 6GB, 1KB per record.
My question in summary: Is it possible to abort the process of building a secondary index AND or possible to stop/postpone running builds (indexing, compaction) for a later date.
nodetool -h node_address stop index_build
See: http://www.datastax.com/docs/1.2/references/nodetool#nodetool-stop