I am wrapping elastic search on windows in a service fabric stateless service, such it runs the elastic search node on each node that the service is running.
Elastic Search is distributed in the code package and can be updated with the application.
So in a 3 node service fabric, the elastic node will get the name of each node in service fabric.
What would be the best approach to locate the data for elastic node?
My own idea would be to locate it on the VM temp disks, as long as more nodes are up, then elastic should replicate data internal such one node can die.
Then i would also do daily backups copying all data to external storage to be able to restore it.
Is there any other options that I should consider?
Related
Azure functions can be containerized but what are the actual use cases for it. Is it portability and ease of running it in any Kubernetes environment on prem or otherwise? Or anything further?
As far as I Know,
We can run the Azure Functions in a serverless fashion i.e., backend VMs and servers managed by the Vendor (Azure). Also, I believe there are 2 Azure Container Services like Container Instances and Kubernetes Service.
Azure Kubernetes Service handles large volume of containers.
Much like running multiple virtual machines on a single physical host, you can run multiple containers in a single physical or virtual host.
In VMs, you look at OS, disk, internet, updating the VM and patching, updating the applications present in VM and all you have to manage, whereas in containers, you don’t have to look at OS, you can easily provision the services like databases, python runtime in the container and utilize them.
Example:
You have control over the VM, but containers are not like that.
Let’s say If I’m the web developer / data scientist / data analyst who wants to work only on SQL Database.
It can be installed on the Virtual Machine, and it is also available through containers.
The primary difference would be,
When you deploy on containers, it would be a simple package which would let you only focus on SQL Database, all the other configuration like dependencies like OS, Configuration comes as part of that package can be taken care by that Container Service.
But in the VM, the moment you install SQL Database, there are other dependencies you need to look at.
The objective is to create a highly available redis cluster using kubernetes for a nodeJS client. I have already created the architecture as below:
Created a Kubernetes cluster of Kmaster with 3 nodes (slaves).
Then I created statefulsets and persistent volumes (6 - one for each POD).
Then created Redis pods 2 on each node (3 Master, 3 replicas of respective master).
I need to understand the role of Redis Sentinel hereafter, how does it manage the monitoring, scaling, HA for the redis-cluster PODs across the nodes. I understand Sentinel should be on each node and doing its job but what should be the right architecture here?
P.S. I have created a local setup for now, but ultimately this goes on Azure so any suggestions w.r.to az is also welcome.
Thanks!
From an Azure perspective, you have two options and if you are very specific to option two but are looking for the Sentinel architecture piece, there is business continuity and high availability options in both IaaS (Linux VM scale sets) and PaaS services that go beyond the Sentinel component.
Azure Cache for Redis (PaaS) where you choose & deploy your desired service tier (Premium Tier required for HA) and connect your client applications. Please see: Azure Cache for Redis FAQ and Caching Best Practice.
The second option is to deploy a solution (as you have detailed) as an IaaS solution built from Azure VMs. There are a number of Redis Linux VM images to choose from the Azure Marketplace or there is the option to create a Linux VM OS image from your on-premise solution and migrate that to Azure. The Sentinel component is enabled on each server (master, slavea, and slaveb, ...). There are networking and other considerations too. For building a system from scratch, please see: How to Setup Redis Replication (with Cluster-Mode Disabled) in CentOS 8 – Part 1 and How to Setup Redis For High Availability with Sentinel in CentOS 8 – Part 2
problem statement.
as per my understanding, we can run an elastic search, kibana and logstash etc as a pod in kubernates cluster for log management. but it is also memory heavy intensive application. AWS provides various managed services like Cloudwatch, cloud trail and ELK stack for log management.
do we have a similar substitute in Azure as well i.e. some managed service?
you can use AKS with Azure Monitor (reading). I'm not sure you can apply this to not AKS cluster (at least not in a straight forward fashion).
Onboarding (for AKS clusters) is really simple and can be done using various methods (portal included).
You can read more on the docs I've linked (for example, about capabilities).
Azure Monitor for Containers is available now and once integrated some cluster metrics as well as the logs will be automatically collected and made available through log analytics.
I have checked the source code, the elasticsearch worker role install the elasticsearch and create the cloud drive from blob "ElasticStorage.vhd", what will happen if i have multiple elasticsearch worker role, then multiple elasticsearch instances will be installed, but i want to know can index data be shared? or each elasticsearch instance hold it's index data?
The elasticsearch instances form a cluster (you can see in the source, as soon as the instance is started, the list of ip is passed as part of the configuration). So the index data is shared. The cloud drive is used to give some persistence in case instances are crashed. It would be better to setup snapshots instead.
You can also setup elastic search using official plugin instead: https://github.com/elasticsearch/elasticsearch-cloud-azure
I recently got a trial version of Windows Azure and wanted to know if there is any way I can deploy an application using Cassandra.
I can't speak specifically to Cassandra working or not in Azure unfortuantly. That's likely a question for that product's development team.
But the challenge you'll face with this, mySQL, or any other role hosted database is persistence. Azure Roles are in and of themselves not persistent so whatever back end store Cassandra is using would need to be placed onto soemthing like an Azure Drive (which is persisted to Azure Blob Storage). However, this would limit the scalability of the solution.
Basically, you run Cassandra as a worker role in Azure. Then, you can mount an Azure drive when a worker starts up and unmount when it shuts down.
This provides some insight re: how to use Cassandra on Azure: http://things.smarx.com/#Run Cassandra
Some help w/ Azure drives: http://azurescope.cloudapp.net/CodeSamples/cs/792ce345-256b-4230-a62f-903f79c63a67/
This should not limit your scalability at all. Just spin up another Cassandra instance whenever processing throughput or contiguous storage become an issue.
You might want to check out AppHarbor. AppHarbor is a .Net PaaS built on top of Amazon. It gives users the portability and infrastructure of Amazon and they provide a number of the rich services that Azure offers such as background tasks & load balancing plus some that it doesn't like 3rd party add-ons, dead-simple deployment and more. They already have add-ons for CouchDB, MongoDB and Redis if Cassandra got high enough on the requested features I'm sure they could set it up.