We are planning to Use datastax 2.0. driver in our application . We have following scenario in our application .There are two different transactions, one for increasing the consumption and the other for decreasing the consumption that can be done at the same time.
For Example:
Let us assume Repair_Qty = 10 From Machine 1 - I am doing a new repair and so the new Repair_Qty should be 10 + 1 .. i.e. 11 From Machine 2 at the same time, someone else is canceling a Repair. The Repair_Qty should be 11-1 = 10. However, as the transaction happened at same time and as there is no Transaction lock, the new Repair Qty will be 10-1 = 9 which is wrong.
I want to know if there is some mechanism for WRITE-READ_WRITE lock support in the datastax java driver.
Please help.
Regards,
Arun
I would suggest you to do that at your application level somehow. As cassandra is eventually consistent, those kind of operations tend to fail.
Related
My postgres instance CPU utilisation has shot up recently. I'm trying to identify the root cause. I will add the details below.
My postgres database instance running on GCP has the following configuration:
v PostgreSQL 9.6
vCPUs-1
Memory-3.75 GB
SSD storage-15 GB
I'm running 5 databases in the above DB server which are connected with a nodejs app.
I use sequelize as my ORM and recently upgraded my sequeliz from 4.6.x to 5.8.6".
Before this upgrade the CPU utilization would usually remain less than 20 percent. But after the upgrade, I see a lot of fluctuation in CPU utilization graph. And it hits 100 percent too often as well. Also, when it hits 100%, my services start wont work as expected ( because they cant interact with the DB).
I tried running this query .
SELECT "usesysid", "backend_start", "xact_start","query_start", "state_change", "state", "query" FROM pg_stat_activity ORDER BY "query_start" DESC
And, it returns the following:
But I'm not sure if this info is enough for me to find out which query could be causing this issue.
I also ran this query:
SELECT max(now() - xact_start) FROM pg_stat_activity WHERE state IN ('idle in transaction', 'active');
and it returns max = 1 day 01:42:10.987635. I think this is something alarming, but i dont know how to put this info to use.
Another thing which i think is worth mentioning is, I have started using sequelize's bulk update.
Its syntax is something like this:
Model.bulkCreate(scalesToUpdate, {
updateOnDuplicate: [
'field1',
'field2'
],
})
And, this gets translated into SQL like below:
INSERT INTO "mymodel" ("id","field1","field2","field3","field4","field5","field6","field7") VALUES (') ON CONFLICT ("id") DO UPDATE SET "field3"=EXCLUDED."field3","field4"=EXCLUDED."field4","field6"=EXCLUDED."field6","field7"=EXCLUDED."field7"
And, this query gets fired 5 times per second. Could this be the culprit?
Any insight into this is highly appreciable.
You could try the next things:
Increase ht machine type to have one core more having vCPUs= 2
It might be that sequelize 5.8.6 requires more resources than the old version, you could try to install one of the tools and run it, run the queries that you typed, to review which query has more resource usage.
If you have that query running 5 times per second, that could be using more resources. Test using one of the tools in order to be able to have a better approach.
We use hbase 1.2.4 in pseudo-destributed mode and java api to read information. Our client makes 20k rps to hbase, but it loads only 30% of cpu and computation takes about 5 hours. I tried to split data and run 4 clients on same machine in separate jvm and had 80k rps with computation time of approximately one hour. This is not the solution to satisfy me. Profiling has shown many blocking of connection threads.
I have also tried to use ipc pool options of hbase client, but it had not improved performance too much.
If anyone had some similar propblem, please give me some advice.
Setting connection pool size much larger then count of reading threads did the trick.
Configuration conf;
...
String poolSize = "128";
conf.set(HConstants.HBASE_CLIENT_IPC_POOL_SIZE, poolSize);
conf.set(HConstants.HBASE_CLIENT_IPC_POOL_TYPE, "RoundRobin");
...
Connection conn = ConnectionFactory.createConnection(conf);
...
I’d appreciate any guidance on optimal setup for multi-threaded, high throughput low latency Java Client using DataStax Java Driver for Apache Cassandra. I appreciate ‘roll-your-own’ bench-marking is not recommended, but this task is also aimed at a proof-of-concept for a real-world application to achieve high TPS.
Setup:
Client Side : Java 8 Client, configurable number of multi-threaded executor threads (facilitated by lmax disruptor), cassandra-driver-core-3.0.0.jar, running on Redhat 6.3, 24 core machine, dl360s
Server side : 3 node Cassandra Cluster (apache-cassandra-2.2.4, on Redhat 6 with Java 8) , Replication Factor = 3 , running on Redhat 6.3, 24 core machine dl360s
Testing
With cl=LOCAL_QUORUM tests have been in the region of 3.5K INSERTS and 6.5K READS per second from a relatively simple schema, with latency circa 6 and 2 milliseconds respectively, with CPU usage circa 20% across the box.
Problem
However the problem I can not solve is that - when I create multiple separate instances of my load client-application I can achieve significantly higher TPS summed across instances, and greater CPU usage than I can ever achieve within a single JVM. This suggests that my Java Client Application is neither IO or CPU bound, nor is the server-side Cassandra cluster the bottleneck. Likewise when I stub out the Cassandra call, I achieve much higher TPS thus giving me confidence that the application is not suffering from any contention.
So my question is: Is this a common problem – that one single Java Client using DataStax Java Driver for Apache Cassandra is somehow limited on it’s throughput? and assuming not can anyone point me in the right direction to investigate.
I have tested multiple sequences (READs and WRITEs), and also both execute and executeAsync, with variable number of concurrent threads. As you’d expect I see higher numbers with executeAsync but still the same limitation within my app.
I have tested with multiple Connection Pooling settings, and have tried creating/building 1 Cluster Instance per client-application, and multiple cluster instances per application, and varying CoreConnections, maxRequestsPerConnection and newConnectionThreshold values but thus far with no success.
My current best results were with 50 executor threads, 5 instances ;MaxRequestsPerConnection(L) = 1024; ;NewConnectionThreshold(L) = 800; CoreConnectionsPerHost(L) = 20
This yielded ~4K TPS BUT only using 18% of the CPU, and when I start a separate Application Instance I achieve 7.5K TPS across both using 30% CPU, but I can not achieve this 7.5K within the save JVM
Code: Create Cluster
LoadBalancingPolicy tokenAwarePolicy =
new TokenAwarePolicy(new RoundRobinPolicy());
Cluster cluster = Cluster.builder()
.addContactPoints(node)
.withLoadBalancingPolicy(tokenAwarePolicy)
.withPoolingOptions(new PoolingOptions()) // Have tried various options here
.build();
Code: Prepare Statement (once)
String insertSqlString = "INSERT INTO " + keySpaceName + ".test_three ("
+ "user_id, field_a, field_b, field_c, field_d) values "
+ "( ?, ?, ?, ?);";
statementInsertDataTablePS = session.prepare(insertSqlString);
statementInsertDataTablePS.setConsistencyLevel(configuredConsistencyLevel); //2
Code: Execute
BoundStatement boundStatement = new BoundStatement(statementInsertDataTablePS);
session.executeAsync(boundStatement.bind(
sequence, // userID
sequence + "value_for_field_a",
sequence + "value_for_field_b",
sequence + "value_for_field_c",
sequence + "value_for_field_d") );
I am using community edition of memsql. I got this error while i was running a query today. So i just restarted my cluster and got this error solved.
memsql-ops cluster-restart
But what happened and what should i do in future to avoid this error ?
NOTE
I donot want to buy the Enterprise edition.
Question
Is this problem of Availability ?
I got this error when experimenting with performance.
VM had 24 CPUs and 25 nodes: 1 Master Agg, 24 Leaf nodes
Reduced VM to 4 CPUs and restarted cluster.
All the leaves did not recover.
All except 4 recovered in < 5 minutes.
20 minutes later, 4 leaf nodes still were not connected.
From MySQL/MemSQL prompt:
use db;
show partitions;
I notice some partitions with ordinal from 0-71 for me have null instead Host, Port, Role defined for a given partition.
In memsql ops UI http://server:9000 > Settings > Config > Manual Cluster Control I checked "ENABLE MANUAL CONTROL" while I tried to run various commands with no real benefit.
Then 15 minutes later, I unchecked the box, Memsql-ops tried attaching all the leaf nodes again and was finally successful.
Perhaps a cluster restart would have done the same thing.
This happened because a leaf in your cluster has failed a health check heartbeat for some reason (loss of network connectivity, hardware failure, OS issue, machine overloaded, out of memory, etc.) and its partitions are no longer accessible to query. MemSQL Community Edition only supports redundancy 1 so there are no other copies of the data on the failed leaf node in your cluster (thus the error about missing a partition of data - MemSQL can't complete a query that needs to read data on any partitions on the problem leaf).
Given that a restart repaired things, the most likely answer is that linux "out of memory" killed you: MemSQL Linux OOM killer docs
You can also check the tracelog on the leaf that ran into issues to see if there is any clue there about what happened (It's usually at /var/lib/memsql/leaf_3306/tracelogs/memsql.log)
-Adam
I too have faced this error, that was because some of the slave ordinals had no corresponding masters. My error message looked like:
ERROR 1772 (HY000) at line 1: Leaf Error (10.0.0.112:3306): Partition database `<db_name>_0` can't be promoted to master because it is provisioning replication
My memsql> SHOW PARTITIONS; command returned the following.
So what approach I followed was to remove each of such cases (where the role was either Slave or NULL).
DROP PARTITION <db_name>:4 ON "10.0.0.193":3306;
..
DROP PARTITION <db_name>:46 ON "10.0.0.193":3306;
And then created a new partition with each of the dropped partition.
CREATE PARTITION <db_name>:4 ON "10.0.0.193":3306;
..
CREATE PARTITION <db_name>:46 ON "10.0.0.193":3306;
And this was the result of memsql> SHOW PARTITIONS; after that.
You can refer to the MemSQL Documentation regarding partitions, here if the above steps doesn't seem to solve your problem.
I was hitting the same problem. Using the following command in the master node, solved the problem:
REBALANCE PARTITIONS ON db_name
Optionally you can force it using FORCE:
REBALANCE PARTITIONS ON db_name FORCE
And to see the list of operations when rebalancing is going to execute, use above command with EXPLAIN:
EXPLAIN REBALANCE PARTITIONS ON db_name [FORCE]
I would like to know if there is a way to check sql_ids that were downgraded to either serial or lesser degree in an Oracle 4-node RAC Data warehouse, version 11.2.0.3. I want to write a script and check the queries that are downgraded.
SELECT NAME, inst_id, VALUE FROM GV$SYSSTAT
WHERE UPPER (NAME) LIKE '%PARALLEL OPERATIONS%'
OR UPPER (NAME) LIKE '%PARALLELIZED%' OR UPPER (NAME) LIKE '%PX%'
NAME VALUE
queries parallelized 56083
DML statements parallelized 6
DDL statements parallelized 160
DFO trees parallelized 56249
Parallel operations not downgraded 56128
Parallel operations downgraded to serial 951
Parallel operations downgraded 75 to 99 pct 0
Parallel operations downgraded 50 to 75 pct 0
Parallel operations downgraded 25 to 50 pct 119
Parallel operations downgraded 1 to 25 pct 2
Does it ever refresh? What conclusion can be drawn from above output? Is it for a day? month? hour? since startup?
This information is stored as part of Real-Time SQL Monitoring. But it requires licensing the Diagnostics and Tuning packs, and it only stores data for a short period of time.
Oracle 12c can supposedly store SQL Monitoring data for longer periods of time. If you don't have Oracle 12c, or if you don't have those options licensed, you'll need to create your own monitoring tool.
Real-Time SQL Monitoring of Parallel Downgrades
select /*+ parallel(1000) */ * from dba_objects;
select sql_id, sql_text, px_servers_requested, px_servers_allocated
from v$sql_monitor
where px_servers_requested <> px_servers_allocated;
SQL_ID SQL_TEXT PX_SERVERS_REQUESTED PX_SERVERS_ALLOCATED
6gtf8np006p9g select /*+ parallel ... 3000 64
Creating a (Simple) Historical Monitoring Tool
Simplicity is the key here. Real-Time SQL Monitoring is deceptively simple and you could easily spend weeks trying to recreate even a tiny portion of it. Keep in mind that you only need to sample a very small amount of all activity to get enough information to troubleshoot. For example, just store the results of GV$SESSION or GV$SQL_MONITOR (if you have the license) every minute. If the query doesn't show up from sampling every minute then it's not a performance issue and can be ignored.
For example: create a table create table downgrade_check(sql_id varchar2(100), total number), and create a job with DBMS_SCHEDULER to run insert into downgrade_check select sql_id, count(*) total from gv$session where sql_id is not null group by sql_id;. Although the count from GV$SESSION will rarely be exactly the same as the DOP.
Other Questions
V$SYSSTAT is updated pretty frequently (every few seconds?), and represents the total number of events since the instance started.
It's difficult to draw many conclusions from those numbers. From my experience, having only 2% of your statements downgraded is a good sign. You likely either have good (usually default) settings and not too many parallel jobs running at once.
However, some parallel queries run for seconds and some run for weeks. If the wrong job is downgraded even a single downgrade can be disastrous. Storing some historical session information (or using DBA_HIST_ACTIVE_SESSION_HISTORY) may help you find out if your critical jobs were affected.