I'm using AWS-RDS(Aurora MySQL5.6) and It's a cluster, it has one writer instance and one reader instance. I find each instance would be down for nearly 10 minutes when I change its type, it's unacceptable, what should I do to avoid downtime?
Here are more details information:
Change type from db.t2.small to db.t2.medium
Engine version: 5.6.10a
Freeable Memory: 300M
DB Connections (Count): ~400
CPU Utilization (Percent): ~20%
Instead of changing the type of the current writer instance, add a new writer with the desired instance type:
Add an Aurora reader with the desired new instance type (for example db.t2.medium) to the cluster. Be sure to assign the reader the best priority so that it will be promoted to writer during failover (see Aurora Documentation for more in-depth information).
Wait until the new instance is up and running.
Then failover the cluster so that the new reader gets promoted to a writer.
Finally delete the previous Aurora instance of the old instance type.
This way you will have nearly no downtime.
If you want to change the type of a reader instance:
Add an Aurora reader with the desired new instance type.
Wait until it's up and running.
Delete the previous reader instance.
I am assuming you are using cloud formation or terraform to manage db cluster and instances. I think, adding a "dependsOn" in cloud formation might help you in doing a sequential update.
e.g. instance 1(reader) depends on instance 2 (writer)
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-dependson.html
Related
Let's say I have a Cosmos DB set up in three regions A, B, and C, where A is the write region. Also I have Automatic Failover turned off.
When using the .NET SDK in a client, if I set ConnectionPolicy.EnableEndpointDiscovery = true but don't specify anything for the ConnectionPolicy.PreferredLocations property, what will happen to reads (and writes), from the client, if region A has an outage?
Also, suppose I do specify ConnectionPolicy.PreferredLocations = {A, B} and both those regions have outages, what happens then?
I can't find any documentation that describes and explains the outcome of these scenarios.
If have EnableEndpointDiscovery enabled, writes will get automatically routed to the new primary region (if not, they will fail during a primary region failure). If you have PreferredLocations, reads will get routed to the regions in order of preferrence. If this is not specified, then reads will fail when the primary is unavailable. In other words, you must enable both to have high availability in case of a regional failure.
If you have PreferredLocations = {A, B} and both regions fail, then reads will be unavailable till the services are brought back online. Generally, this scenario is very unlikely as it involves multiple regions failing simultaneously.
Some additional details here: https://learn.microsoft.com/en-us/azure/cosmos-db/regional-failover
I am working in a specific project to change my repository to hazelcast.
I need find some documents by data range, store type and store ids.
During my tests i got 90k throughput using one instance c3.large, but when i execute the same test with more instances the result decrease significantly (10 instances 500k and 20 instances 700k).
These numbers were the best i could tuning some properties:
hazelcast.query.predicate.parallel.evaluation
hazelcast.operation.generic.thread.count
hz:query
I have tried to change instance to c3.2xlarge to get more processing but but the numbers don't justify the price.
How can i optimize hazelcast to be more fast in this scenario?
My user case don't use map.get(key), only map.values(predicate).
Settings:
Hazelcast 3.7.1
Map as Data Structure;
Complex object using IdentifiedDataSerializable;
Map index configured;
Only 2000 documents on map;
Hazelcast embedded configured by Spring Boot Application (singleton);
All instances in same region.
Test
Gatling
New Relic as service monitor.
Any help is welcome. Thanks.
If your use-case only contains map.values with a predicate, I would strongly suggest to use object type as in in-memory storage model. This way, there will not be any serialization involved during Query execution.
On the other end, it is normal to get very high numbers when you only have 1 member. Because, there is no data moving across network. Potentially to improve, I would check EC2 instances with high network capacity. For example c3.8xlarge has 10 Gbit network, compared to High that comes with c3.2xlarge.
I can't promise, how much increase you can get, but I would definitely try these changes first.
(I could not find a good source explaining this, so if it is available elsewhere, you could just point me to it)
Hazelcast replicates data across all nodes in clusters. So, if data is changed in one of the nodes, does the node update its own copy and then propagate it to other nodes?
I read somewhere that each data is owned by a node, how does Hazelcast determine the owner? Is the owner determined per datastructure or per key in the datastructure?
Does Hazelcast follow "eventually consistent" principle? (When the data is being propagated across the nodes, there could be a small window during which the data might be inconsistent between the nodes)
How are conflicts handled? (Two nodes update the same key-value simultaneously)
Hazelcast does not replicate (with exception of the ReplicatedMap, obviously ;-)) but partitions data. That means you have one node that owns a given key. All updates to that key will go to the owner and he notifies possible updates.
The owner is determined by consistent hashing using the following formula:
partitionId = hash(serialize(key)) % partitionCount
Since there is only one owner per key it is not eventually consistent but consistent whenever the mutating operations is returned. All following read operations will see the new value. Under normal operational circumstances. When any kind of failure happens (network, host, ...) we choose availability over consistency and it might happen that a not yet updated backup is reactivated (especially if you use async backups).
Conflicts can happen after split-brain when the split cluster re-merge. For this case you have to configure (or use the default one) MergePolicy to define the behavior on how conflicting elements are merged together or which one of both wins.
Triggers are defined on a table at the keyspace level and so I'm guessing every replica for the table needs to have access to the trigger class.
Is it possible to have a given DC with the trigger class resident whilst the other DCs don't?
It is possible to only have a subset of the replicas within a DC have the trigger class?
I suppose a quick and dirty workaround would be to have "empty" classes on the nodes upon which we don't want any triggering but this seems a real hack -- hence my question.
Thanks in advance.
So here is the problem. I need to update about 40 million entities in an azure table. Doing this with a single instance (select -> delete original -> insert with new partitionkey) will take until about Christmas.
My thought is use an azure worker role with many instances running. The problem here is the query grabs the top 1000 records. That's fine with one instance but with 20 running their selects will obviously overlap.. a lot. this would result in a lot of wasted compute trying to delete records that were already deleted by another instance and updating a record that has already been updated.
I've run through a few ideas, but the best option I have is to have the roles fill up a queue with partition and row keys then have the workers dequeue and do the actual processing?
Any better ideas?
Very interesting question!!! Extending #Brian Reischl's answer (and a lot of it is thinking out loud, so please bear with me :))
Assumptions:
Your entities are serializable in some shape or form. I would assume that you'll get raw data in XML format.
You have one separate worker role which is doing all the reading of entities.
You know how many worker roles would be needed to write modified entities. For the sake of argument, let's assume it is 20 as you mentioned.
Possible Solution:
First you will create 20 blob containers. Let's name them container-00, container-01, ... container-19.
Then you start reading entities - 1000 at a time. Since you're getting raw data in XML format out of table storage, you create an XML file and store those 1000 entities in container-00. You fetch next set of entities and save them in XML format in container-01 and so on and so forth till the time you hit container-19. Then the next set of entities go into container-00. This way you're evenly distributing your entities across all the 20 containers.
Once all the entities are written, your worker role for processing these entities would come into picture. Since we know that instances in Windows Azure are sequentially ordered, you get instance names like WorkerRole_IN_0, WorkerRole_IN_1, ... and so on.
What you would do is take the instance name, get the number "0", "1" etc. Based on this you would determine which worker role instance will read from which blob container...WorkerRole_IN_0 will read files from container-00, WorkerRole_IN_1 will read files from container-01 and so on.
Now your individual worker role instance will read the XML file, create the entities from that XML file, update those entities and save it back into table storage. Once this process is done, you would then delete the XML file and you move on to next file in that container. Once all files are read and processed, you can just delete the container.
As I said earlier, this is a lot "thinking out loud" kind of solution and some things must be considered like what happens when "reader" worker role goes down and other things.
If your PartitionKeys and/or RowKeys fall into a known range, you could attempt to divide them into disjoint sets of roughly equal size for each worker to handle. eg, Worker1 handles keys starting with 'A' through 'C', Worker2 handles keys starting with 'D' through 'F', etc.
If that's not feasible, then your queuing solution would probably work. But again, I would suggest that each queue message represent a range of keys if possible. eg, a single queue message specifies deleting everything in the range 'A' through 'C', or something like that.
In any case, if you have multiple entities in the same PartitionKey then use batch transactions to your advantage for both inserting and deleting. That could cut down the number of transactions by almost a factor of ten in the best case. You should also use parallelism within each worker role. Ideally use the async methods (either Begin/End or *Async) to do the writing, and run several transactions (12 is probably a good number) in parallel. You can also run multiple threads, but that's somewhat less efficient. In either case, a single worker can push a lot of transactions with table storage.
As a side note, your process should go "Select -> Insert New -> Delete Old". Going "Select -> Delete Old -> Insert New" could result in permanent data loss if a failure occurs between steps 2 & 3.
I think you should mark your question as the answer ;) I cant think of a better solution since I don't know what your partition and row keys look like. But to enhance your solution, you may choose to pump multiple partition/row keys into each queue message to save on transaction cost. Also when consuming from the queue, get them in batches of 32. Process asynchronously. I was able to transfer 170 million records from SQL server (Azure) to Table storage in less than a day.