When I try to use clone/restore point in time from amazon console. It clones cluster as well as all the instances which belongs to that. But when I consume the same functionality using amazon API, it clones only cluster alone.
Is there any other API to clone cluster alone with their instances, security/parameter group and other settings?
Console adds a convenience layer where in it internally makes multiple API calls to make the experience better. Restoring from a snapshot or from point in time is done in 2 steps:
RestoreDBClusterFromSnapshot or RestoreDBClusterToPointInTime API - To create a new cluster, backed by a new distributed aurora volume. No DB instances are added when then API is issued.
CreateDBInstance API - To add instances to the cluster.
So in short, if you want to do it via CLI, you need to issue both these API calls. The same is true while creating a cluster with instances as well. Console would create a cluster and add instances in the same UX workflow, but behind the scenes, it is actually issuing a CreateDBCluster API followed by one or more CreateDBInstance API call(s).
Hope this helps.
Related
I have managed to get the C# and db setup using ListMappings. However, when I try to deploy the split/merge tool to Azure cloud classic the service it states 'The requested VM tier is currently not available in East US for this subscription. Please try another tier or deploy to a different location.' We tried a few other regions with the same result. Do you know if there is a workaround or updated version? Is the split / merge service even still relevant? Has anyone got this service to run on Azure lately?
https://learn.microsoft.com/en-us/azure/azure-sql/database/elastic-scale-overview-split-and-merge
The answer to the question on whether it is still relevant, in my opinion is ...no. Split\merge is no longer relevant with the maturation of elastic pools. Elastic pools with one data base per tenant seem the sustainable way to implement multi tenancy with legacy code. The initial plan was to add keys to each of our tables to have multiple tenants per database. Elastic pools give us the same flexibility without having to make breaking changes our existing code.
Late post here, but we are implementing ElasticScale for a client to split ~50 clients into a database-per-tenant model. I don't think the SplitMerge tool will be used over the long term, just for the initial data migration from one db to many shards, but it has been handy for that purpose. We are using the ElasticScale SDK to allow a single API to route queries to the appropriate shard(s) based on sharding key. Happy to compare notes with you if you are still working on this.
I’ve finished working on my node application which provide a series of APIs endpoint for a mobile application. I’ve made it with node and mongodb as db. Now I’ve reached the point where I should pick the right deployment environment.
Initially I’ll make a private beta but I need to choose a service I can scale easily (I’m not a devop) with the right price balance.
My initial choice is amazon aws (elastic beanstalk?). What’s about the db? I’ve not used dynamodb in order to be more service agnostic but now I don’t know how to create a reliable db infrastructure. Any suggestion to deploy both app and dB in order to make easy scaling in case it will become necessary?
title describes pretty much what we are trying to accomplish in our organization.
We have a very database intensive application, and our single SQL Server machine is struggling.
We are reading articles about Azure, Docker and Kubernetes but we are afraid of trying these technologies.
Our problem is data replication.
How can we have scalability here? If we have three different SQL Server instances in three different containers, How does data get replicated across them? (meaning, user inserts a new product into a shared library, other user accessing a different node/container should be able to see that product).
Maybe we don't need containers at all and Azure provides another way to scale databases?
We really appreciate any help from you guys.
Regards, Cris.
I would advise against trying to run your databases on K8s. Kubernetes Containers should generally be stateless application, and were not designed for persistent data storage. Azure provides a Database as a Service, which will be able to scale appropriately with your needs(Azure Pricing for Cloud SQL
We once experimented with running our Postgres DB inside of a Kubernetes pod, but I was terrified to change anything. Not worth it, and not what the system was designed for.
If you are really really committed to this path, you can check out MySQL NDB ClusterMySQL for distributed environments. It should be adaptable to the Kubernetes paradigm.
We are using AWS for our infra requirement, and for billing and costing purpose we need to know the exact amount of data transferred to our EC2 instances for a particular client. Is there any such utility available in AWS or how should I approach this problem.
Our Architecture is simple we have a api server which is a Node.js® server on one of the ec2 instance, this talks to the db server which is a MongoDB® on another ec2, apart from this we also have a web application server which runs angular web application in Node.js® again.
Currently we don't use ELB and we Identified the client by there login informations i.e the organisation id in the JWT Token.
Given your current architecture, you will need to create some form of Node middleware that extracts the client ID and content-length from the request (and/or response) and writes them to persistent storage. Within the AWS ecosystem, you could write to DynamoDB, or Kinesis, or even SQS. Outside the AWS ecosystem you could write to a relational DB, or perhaps the console log with some form of log agent to move the information to persistent store.
However, capturing the data here has a few issues:
Other than logging to the console, it adds time to each request.
If logging to the console, there will be a time delay between the actual request and the time that the log is shipped to persistent storage. If the machine crashes in that interval you've lost data.
When using AWS services you must be prepared for rate limiting (this is one area where SQS is better than Kinesis or DynamoDB).
Regardless of the approach you use, you will have to write additional code to process the logs.
A better approach, IMO, would be to add the client ID to the URL and an ELB for front-end load distribution. Then turn on request logging and do after-the-fact analysis of the logs using AWS Athena or some other tool.
If you run these EC2 instances in VPC, you can use VPC Flow Logs to get insight into how much data each of the instances transfers.
Recently I'm considering to use Amazon RDS Multi-AZ deployment for a service in production environment, and I've read the related documents.
However, I have a question about the failover. In the FAQ of Amazon RDS, failover is described as follows:
Q: What happens during Multi-AZ failover and how long does it take?
Failover is automatically handled by Amazon RDS so that you can resume
database operations as quickly as possible without administrative
intervention. When failing over, Amazon RDS simply flips the canonical
name record (CNAME) for your DB Instance to point at the standby,
which is in turn promoted to become the new primary. We encourage you
to follow best practices and implement database connection retry at
the application layer. Failover times are a function of the time it
takes crash recovery to complete. Start-to-finish, failover typically
completes within three minutes.
From the above description, I guess there must be a monitoring service which could detect failure of primary instance and do the flipping.
My question is, which AZ does this monitoring service host in? There are 3 possibilities:
1. Same AZ as the primary
2. Same AZ as the standby
3. Another AZ
Apparently 1&2 won't be the case, since it could not handle the situation that entire AZ being unavailable. So, if 3 is the case, what if the AZ of the monitoring service goes down? Is there another service to monitor this monitoring service? It seems to be an endless domino.
So, how is Amazon ensuring the availability of RDS in Multi-AZ deployment?
So, how is Amazon ensuring the availability of RDS in Multi-AZ deployment?
I think that the "how" in this case is abstracted by design away from the user, given that RDS is a PaaS service. A multi-AZ deployment has a great deal that is hidden, however, the following are true:
You don't have any access to the secondary instance, unless a failover occurs
You are guaranteed that a secondary instance is located in a separate AZ from the primary
In his blog post, John Gemignani mentions the notion of an observer managing which RDS instance is active in the multi-AZ architecture. But to your point, what is the observer? And where is it observing from?
Here's my guess, based upon my experience with AWS:
The observer in an RDS multi-AZ deployment is a highly available service that is deployed throughout every AZ in every region that RDS multi-AZ is available, and makes use of existing AWS platform services to monitor the health and state of all of the infrastructure that may affect an RDS instance. Some of the services that make up the observer may be part of the AWS platform itself, and otherwise hidden from the user.
I would be willing to bet that the same underlying services that comprise CloudWatch Events is used in some capacity for the RDS multi-AZ observer. From Jeff Barr's blog post announcing CloudWatch Events, he describes the service this way:
You can think of CloudWatch Events as the central nervous system for your AWS environment. It is wired in to every nook and cranny of the supported services, and becomes aware of operational changes as they happen. Then, driven by your rules, it activates functions and sends messages (activating muscles, if you will) to respond to the environment, making changes, capturing state information, or taking corrective action.
Think of the observer the same way - it's a component of the AWS platform that provides a function that we, as the users of the platform do not need to think about. It's part of AWS's responsibility in the Shared Responsibility Model.
Educated guess - the monitoring service runs on all the AZs and refers to a shared list of running instances (which is sync-replicated across the AZs). As soon as a monitoring service on one AZ notices that another AZ is down, it flips the CNAMES of all the running instances to an AZ which is currently up.
We did not get to determine where the fail-over instance resides, but our primary is in US-West-2c and secondary is in US-West-2b.
Using PostgreSQL, our data became corrupted because of a physical problem with the Amazon volume (as near as we could tell). We did not have a multi-AZ set up at the time, so to recover, we had to perform a point-in-time restore as close in time as we could to the event. Amazon support assured us that had we gone ahead with the Multi-AZ, they would have automatically rolled over to the other AZ. This begs the questions how they could have determined that, and would the data corruption propagated to the other AZ?
Because of that shisaster, we also added a read-only replica, which seems to make a lot more sense to me. We also use the RO replica for read and other functions. My understanding from my Amazon rep is that one can think of the Multi-AZ setting as more like a RAID situation.
From the docs, fail over occurs if the following conditions are met:
Loss of availability in primary Availability Zone
Loss of network connectivity to primary
Compute unit failure on primary
Storage failure on primary
This infers that the monitoring is not located in the same AZ. Most likely, the read replica is using mysql functions (https://dev.mysql.com/doc/refman/5.7/en/replication-administration-status.html) to monitor the status of the master, and taking action if the master becomes unreachable.
Of course, this bears the question what happens if the replica AZ fails? Amazon most likely has checks in the replica's failure detection to figure out whether it's failing or the primary is.