I'm using memsql 5.0.8 community version. I just randomly get the ER_STMT_CACHE_FULL error, All I can do is reboot the server. How can I increase it.
MemSQL doesn't have a statement cache and never generates this error. You are likely running your application against MySQL or MariaDB
Related
I have started an offline upgrade process to upgrade my MemSql Cluster from 5.8 To 6.5, Data size is around 300G it's been 5 hours already but i have lost all access to cluster and also there is no way to check the status.
memsql-ops memsql-list returns all leaves and aggregator shows online.
But, memsql> SHOW LEAVES; return empty set, my master aggregator automatically converted to child aggregator, so now i don't have any master aggregator.
I can't execute any command (Like AGGREGATOR SET AS MASTER) to child aggregator, it says 'memsql is not running as an aggregator', Or 'memsql node is not running', and sql query returns "The database 'xxx' is not available for queries, as it is waiting for the Master Aggregator to bring it online. Run SHOW DATABASES EXTENDED ..."
Also performing any management command like memsql-ops restart returns "Job cannot run because there is a MemSql upgrade intention with ID xxx is in progress"
Any information about this will be helpful as i am not able to find any related information online.
Thanks in advance...
We debugged the issue in MemSQL public chat and it was found that the Master Agg was running an unsupported beta version of MemSQL (6.0.0) which prevented the upgrade and then corrupted the database post upgrade.
For future readers please audit that you are not running beta versions of MemSQL on production clusters. If you are, not only will upgrade likely break, but it may not be possible to recover your data on a non-beta cluster.
I just have one reason to restart cluster below :
All the nodes have the same hardware configuration
1. When i update file cassandra.yaml
Are there other reasons ?
The thing you are asking for is Rolling Restart a cassandra cluster. There are so many reason to restart a cassandra cluster. I'm just mentioning some below-
when you update any value in cassandra.yaml. (As you mentioned above)
When your nodetool got stucked somehow. such as- you gave command nodetool repair and cancelled the command but it got stucked behind, then you won't be able to give another nodetool repair command.
When you are adding a new node to cluster and you got stream_failed due to nproc limit. That time your running cluster nodes could be down to this issue and going to hold the status.
When you don't want to use sstableloader and you need to restore your data from snapshots. That time you need to provide your snapshots to the data_directory on each node and rolling restart.
When you are about to upgrade your cassandra_version.
For example when you upgrading Cassandra version.
After restart my cassandra node does not start anymore. Ends with following error message.
ERROR 18:39:37 Unknown exception caught while attempting to update MaterializedView! findkita.kitas
java.lang.AssertionError: We shouldn't have got there is the base row had no associated entry
cassandra has heavy cpu usage and use 2,1 gb of memory there is be 1gb more available. I run nodetool cleanup and repair, but did not help.
I have 5 materialzied views on this table, but the amount of rows in table is under 2000, that is not much.
The cassandra runs in a docker container. The container is access able, but can not call cqlsh and my website cound not connect too
How can I fix the error? Is it possible to fix it?
I do not really fix it, but I get it run. My first container is now complitly crashed and is not start able anymore. But I had same problem with other container, that are enter able. I run apt-get update and apt-get upgrade and get cassandra work again.
It is not importand if there are any upgrades, only to run the command make cassandra call able again. Have to do it at each restart, but better as a completle crashed database.
We're hosting Cassandra 2.0.2 cluster on AWS. We've recently started upgrading from normal to SSD drives, by bootstrapping new and decommissioning old nodes. It went fairly well, aside from two nodes hanging forever on decommission. Now, after the new 6 nodes are operational, we noticed that some of our old tools, using phpcassa stopped working. Nothing has changed with security groups, all ports TCP/UDP are open, telnet can connect via 9160, cqlsh can 'use' a cluster, select data, however, 'describe cluster' fails, in cli, 'show keyspaces' also fails - and by fail, I mean never exits back to prompt, nor returns any results. The queries work perfectly from the new nodes, but even the old nodes waiting to be decommissioned cannot perform them. The production system, also using phpcassa, does normal data requests - it works fine.
All cassandras have the same config, the same versions, the same package they were installed from. All nodes were recently restarted, due to seed node change.
Versions:
Connected to ### at ####.compute-1.amazonaws.com:9160.
[cqlsh 4.1.0 | Cassandra 2.0.2 | CQL spec 3.1.1 | Thrift protocol 19.38.0]
I've run out out of ideas. Any hints would be greatly appreciated.
Update:
After a bit of random investigating, here's a bit more detailed description.
If I cassandra-cli to any machine, and do "show keyspaces", it works.
If I cassandra-cli to a remote machine, and do "show keyspaces", it hangs indefinitely.
If I cqlsh to a remote cassandra, and do a describe keyspaces, it hangs. ctrl+c, repeat the same query, it instantly responds.
If I cqlsh to a local cassandra, and do a describe keyspaces, it works.
If I cqlsh to a local cassandra, and do a select * from Keyspace limit x, it will return data up to a certain limit. I was able to return data with limit 760, the 761 would fail.
If I do a consistency all, and select the same, it hangs.
If I do a trace, different machines return the data, though sometimes source_elapsed is "null"
Not to forget, applications querying the cluster sometimes do get results, after several attempts.
Update 2
Further playing introduced failed bootstrapping of two nodes, one hanging on bootstrap for 4 days, and eventually failing, possibly due to a rolling restart, and the other plain failing after 2 days. Repairs wouldn't function, and introduced "Stream failed" errors, as well as "Exception in thread Thread[StorageServiceShutdownHook,5,main] java.lang.NullPointerException". Also, after executing repair, started getting "Read an invalid frame size of 0. Are you using tframedtransport on the client side?", so..
Solution
Switch rpc_server_type from hsha to sync. All problems gone. We worked with hsha for months without issues.
If someone also stumbles here:
http://planetcassandra.org/blog/post/hsha-thrift-server-corruption-cassandra-2-0-2-5/
In cassandra.yaml:
Switch rpc_server_type from hsha to sync.
I am currently planning for an upgrade to 2.0.7 cassandra version . My base version is 2.0.3. I have not done an upgrade so far and hence want to be absolutely sure about what am doing . Can someone explain what needs to be done apart front this.
Do a nodetool drain to stop all writes to the particular node.
Stop the cassandra node(I have a 8 node , 2 data center network topology. I am bringing down one node in DC1)
Change the cassandra.yaml accordingly in the new binary tarball.
Make the required changes for the new node(using gossiping property file snitch. So , making changes for that)
Start off the new cassandra binary(2.0.7)
Question striking me the most
Do I have to copy the data from 2.0.3 to 2.0.7?
2.Even if it's a rolling upgrade , I think the following steps will do( Except moving from one version to another ) . My assumption is right?
Am going to do this operation on a running application. Am planning to have the application running while doing this as I have enough replicas in local quorum to satisfy reads and writes. Does this idea have any disadvantages ? I loved cassandra for this kind of operation but would like to know of there are any potential problems ?
I will be having the existing 2.0.3 in my running machine while doing this. If there is a problem in 2.0.7 , I shall start off 2.0.3 version again right? Just wanted to know whether there will be any data conflicts with other nodes in the cluster? Or having a snapshot to recover the data is the best option?
5.Apart from this, any other thing I have bear in mind?
Do I have to copy the data from 2.0.3 to 2.0.7? 2.Even if it's a rolling upgrade , I think the following steps will do( Except moving from one version to another ) . My assumption is right?
If you just upgrade the binaries, you can leave all of the data in place and it will use it automatically.
Am going to do this operation on a running application. Am planning to have the application running while doing this as I have enough replicas in local quorum to satisfy reads and writes. Does this idea have any disadvantages ? I loved cassandra for this kind of operation but would like to know of there are any potential problems ?
Normal read and write operations are fine. While you are temporarily running a mixed-version cluster, it's best to avoid doing anything that involves streaming (repairs) or topology changes (bootstrapping or decommissioning nodes). They might work, but they're not officially supported and you're more likely to have problems.
I will be having the existing 2.0.3 in my running machine while doing this. If there is a problem in 2.0.7 , I shall start off 2.0.3 version again right? Just wanted to know whether there will be any data conflicts with other nodes in the cluster? Or having a snapshot to recover the data is the best option?
You want to have a snapshot to recover from. Newer versions of Cassandra may use new SSTable or commitlog formats which the older version will not be able to read.