I want to explore read-only replicas in Spanner. I have options to create nam-3 or nam-6 multi-region spanner( read-only can be created only on multi-regional spanner). When I select either of the options on the right panel I get replica details as follows
For num-6
Replicas
2 read-write replicas in us-central1 (Iowa) - default leader region
2 read-write replicas in us-east1 (South Carolina)
1 read-write replica in us-west1 (Oregon)
1 read-write replica in us-west2 (Los Angeles)
For num-3
Replicas
2 read-write replicas in us-east4 (Northern Virginia) - default leader region
2 read-write replicas in us-east1 (South Carolina)
I do not see read-only replicas in both of the options. Do I need to anything different here?
Authoritative information about available configurations is available here:
https://cloud.google.com/spanner/docs/instances#available-configurations-multi-region
Only nam6 and nam-eur-asia1 multi-regional configurations offer read-only replicas.
I've verified that descriptions in Cloud Console are in line with your observations and this is incorrect. Thank you for noticing, I'll pass the report to the dedicated team to address it.
UPDATE 2019-10-04: this is fixed now.
Related
We have a 2 datacenters ( One in EU and one in US ) cluster with 4 nodes each deployed in AWS.
The nodes are separated in 3 racks ( Availability zones ) each.
In the cluster we have a keyspace test with replication: NetworkTopologyStrategy, eu-west:3, us-east:3
In the keyspace we have a table called mytable that has only one row 'id' text
Now, we were doing some tests on the performance of the database.
In CQLSH with a consistency level of LOCAL_QUORUM we were doing some inserts with TRACING ON and we noticed that the requests were not working as we expected them.
From the tracing data we found out that the coordinator node was hitting as expected 2 other local nodes and was also sending a request to one of the remote datacenter nodes. Now the problem here was that the coordinator was waiting not only for the local nodes ( who finished in no time ) but for the remote nodes too.
Now since our 2 datacenters are geographically far away from each other, our requests were taking a very long time to complete.
Notes:
- This does not happen with DSE but our understanding was we don't need to pay crazy money for LOCAL_QUORUM to work as is expected
There is a high probability that you're hitting CASSANDRA-9753 when the non-zero dclocal_read_repair_chance will trigger a query against remote DC. You need to check the trace for hint about triggering of read repair for your query. If you really get it, then you can set dclocal_read_repair_chance to 0 - this parameter is deprecated anyway...
For functional and performance tests it would be better to use the driver instead of CQLSH, as most of the time that will be the way that you are interacting with the database.
For this case, you may use a DC-aware policy like
Cluster cluster = Cluster.builder()
.addContactPoint("127.0.0.1")
.withLoadBalancingPolicy(
DCAwareRoundRobinPolicy.builder()
.withLocalDc("myLocalDC")
.build()
).build();
This is modified from the example here, where all the clauses that allow to interact with remote datacenters are removed, as your purpose is to isolate the calls to local.
Lets say, I have several Hazelcast members (servers) spread across world (e.g. Germany, Russia and etc).
It was required to store/split data in database by region and all data should be accessible from any server via IMap backed by MapStore.
I recently read this article which fulfills my requirement, but I am not sure about how will MapStore behave.
Crucial moment is that, if member1 (e.g. Russia) requests data from IMap with key owned by member2 (e.g. Germany), on which side MapStore.load() will be called?
You should not split members of the same cluster across different data centers. Members of a cluster depend on a regular heartbeat message to detect the health of the cluster; wide area networks cannot reliably deliver these in a consistent fashion and you will almost certainly have network partition issues (split brain syndrome).
Each data center (Germany, Russia, etc.) should have a separate cluster with region-specific maps. These maps can then be replicated (WAN replication) to the remote data centers both for disaster recovery and to provide a geographically close server to support users in that region needing access to the other region's data.
Since the data in the database is already split by region, matching this split on the Hazelcast side means that the MapLoader will always be loading from a database in the same location.
I would like to know is there any way to connect to one of readonly replicas?
I know if Azure SQL data base is Premium type and Read Scale-out is Enabled, I can have two readnonly replicas. For connecting to ANY of those replicas, I just need to put ApplicationIntent=ReadOnly; in my connection string.
Attached link about Read scale-out replicas
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-read-scale-out
Also I know that there are some load balancer that can switch from request from one readonly replica to another.
So, is there some approach to define what replica should I use, Replica1 or Replica2 for (for ex.) Analytics and PowerBi reports accordingly at the same time?
Server=tcp:.database.windows.net;Database=;ApplicationIntent=ReadOnly;User ID=;Password=;Trusted_Connection=False; Encrypt=True;
At any given time only one of the replicas is accessible by the ReadOnly sessions and you cannot specify which one.
No, not at this time - Per the documentation (in a 'Note' about half way down: https://learn.microsoft.com/en-us/azure/sql-database/sql-database-read-scale-out)
At any given time only one of the AlwaysON replicas is accessible by the ReadOnly sessions.
It seems that the system will choose which replica to use for the connection/session when the connection is initiated.
After setting node type i,e Reliability tier to sliver to bronze on azure service fabric, error on cluster health is waring here below is the error evaluation from service fabric.(Even in vmss of service fabric
Services
Warning
Unhealthy services: 100% (1/1), ServiceType='ClusterManagerServiceType', MaxPercentUnhealthyServices=0%.
Service
Warning
Unhealthy service: ServiceName='fabric:/System/ClusterManagerService', AggregatedHealthState='Warning'.
Event
Warning
Unhealthy event: SourceId='System.PLB', Property='ServiceReplicaUnplacedHealth_Secondary_00000000-0000-0000-0000-000000002000', HealthState='Warning', ConsiderWarningAsError=false.
The Load Balancer was unable to find a placement for one or more of the Service's Replicas:
ClusterManagerServiceName Secondary Partition 00000000-0000-0000-0000-000000002000 could not be placed, possibly, due to the following constraints and properties:
TargetReplicaSetSize: 5
Placement Constraint: NodeTypeName==NOde
Depended Service: N/A
Constraint Elimination Sequence:
ReplicaExclusionStatic eliminated 2 possible node(s) for placement -- 1/3 node(s) remain.
ReplicaExclusionDynamic eliminated 1 possible node(s) for placement -- 0/3 node(s) remain.
Nodes Eliminated By Constraints:
ReplicaExclusionStatic -- No Colocations with Partition's Existing Secondaries/Instances:
FaultDomain:fd:/0 NodeName:_NOde_0 NodeType:NOde NodeTypeName:NOde UpgradeDomain:0 UpgradeDomain: ud:/0 Deactivation Intent/Status: None/None
FaultDomain:fd:/2 NodeName:_NOde_2 NodeType:NOde NodeTypeName:NOde UpgradeDomain:2 UpgradeDomain: ud:/2 Deactivation Intent/Status: None/None
ReplicaExclusionDynamic -- No Colocations with Partition's Existing Primary or Potential Secondaries:
FaultDomain:fd:/1 NodeName:_NOde_1 NodeType:NOde NodeTypeName:NOde UpgradeDomain:1 UpgradeDomain: ud:/1 Deactivation Intent/Status: None/None
Help me to slove this problem
When you create your cluster with Reliability tier Silver it will provision 5 replicas of the system services, i.e. the services that essentially are Service Fabric.
Downgrading from Silver to Bronze means that you change the target replica count of these services from 5 to 3.
In order for SF to place replicas on nodes it evaluates a set of constraints, on of these being that it does not want two replicas of the same service partition to end up on the same node.
As it looks from your error you have one Node Type with 3 nodes in it but you still have Silver reliabilty tier, that means that SF is unable to find a node for the last two of your replicas for the system services (in your log it is System/ClusterManagerService, but same applies for all system services).
Make sure that your cluster has at least as many nodes as your reliability tier needs, i.e. 3 nodes for a Bronze tier, 5 for a Silver and so on.
Also, what you are seeing is a warning that the cluster is not able to uphold it's characteristics, but it should still be running, right?
I have two Data Centers, each one with replication factor 3.
Will write with CL.ALL block until data is stored in both DCs (6 nodes, or 3 + 1)?
I would assume, that it blocks until all 3 replicas in local DC has acknowledged successful write.
I would like to have something like CL.ALL_LOCAL, which stores data on all replicas in single DC, so I can read with CL.ONE. The idea is, that write blocks until all replicas in single DC has persisted data, and following read will have high probability to read fresh data
There isn't currently a consistency level that provides what you are describing. The closest is LOCAL_QUORUM which will return after a quorum of nodes in the local datacenter respond.
You can file a ticket on jira to add this functionality if you would like.
https://issues.apache.org/jira/browse/CASSANDRA
I've checked Cassandra 1.1 code and noticed interesting behavior when writing with CL.ALL in multi DC deployment. Probably I've interpreted code wrong.... anyway:
on the beginning they are collecting IP addresses of nodes to send row mutation - this is independent from consistency level provided by the client. In 1.0 it were all nodes from all DCs, from 1.1 they get all nodes from local DC plus one node from each remote DC (the remaining nodes are as "forward to" in the message). Each mutation will be send by separate thread, so the requests can run in parallel. Each such mutation is being handled as a message by messaging service. When node in remote DC receives message, it forwards it to remaining nodes, which are provided in "forward to".
The consistency level provided by the client, defines number of nodes which must acknowledge received message. In case of CL.ALL this number is determined by replication factor - now is getting interesting: since we've send message to all nodes from local DC and to nodes from remote DCs, we will get also acknowledgement from those remove nodes too - yes this is still the number which is defined by replication factor, but depending on notwork latency, we can not be sure which nodes has conformed received message - could be mix from nodes from local and remote DC, but could be also only nodes from local DC. In the worst case, it could happen, that none of the local nodes got the message, and confirmation come from remote DCs (if you have many). This means - writing with CL.ALL does not grantee, that you can immediately read message from your local DC.