We are trying to do GR setup for our application which has casandra pods(deployed within application namespace) deployed in two different clusters on two
sites of webscale environment. All external routing from the cluster is done via F5/spk.
How can we setup casandra data replication via F5/SPK between primary and secondary .
Multus is not allowed on platform ...
Related
I have installed cassandra(three nodes) on kubernetes in aws account.
I want to open the same cassandra via aws endpoint(through NLB) to different aws account so that i can access this cassandra for read/write purpose.
I am using spark(in different AWS account) to load data in cassandra but i am getting this WARN while loading the data.
WARN ChannelPool: [s0|/10.0.246.140:32034] Error while opening new channel (ConnectionInitException: [s0|connecting...] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.7.2, CLIENT_ID=b52c9022-561a-48d3-bd98-893c6c17f0c3, APPLICATION_NAME=Spark-Cassandra-Connector-application_1606197155514_0510}): failed to send request (java.nio.channels.NotYetConnectedException))
Has anybody open Cassandra via NLB , do i need to make separate routes for each node in cassandra in NLB? if yes, how to do that?
You need to define a K8s service and expose it through an Ingress controller such as Traefik so clients (such as your Spark app) can connect to your Cassandra cluster from outside the Kubernetes cluster.
If you're using the DataStax Cassandra Operator (cass-operator), it makes it a lot easier since it comes pre-configured with a service that you can use. See the Ingress examples we have included in Connecting to Cassandra from outside the Kubernetes cluster.
If you weren't already aware, have a look at open-source K8ssandra. It is a ready-made platform for running Apache Cassandra in Kubernetes using the DataStax Cassandra Operator under the hood but with all the tools bundled together:
Reaper for automated repairs
Medusa for backups and restores
Metrics Collector for monitoring with Prometheus + Grafana
Traefik templates for k8s cluster ingress
Since all these components are open-source, they are all free to use and don't require a licence or paid subscription but still comes with a robust community support. Cheers!
we installed the follwing presto cluster on Linux redhat 7.2 version
presto latest version - 0.216
1 presto coordinator
231 presto workers
on each worker machine we can use the follwing command in order to verify the status
/app/presto/presto-server-0.216/bin/launcher status
Running as 61824
and also stop/start as the follwing
/app/presto/presto-server-0.216/bin/launcher stop
/app/presto/presto-server-0.216/bin/launcher start
I also searches in google about UI that can manage the presto status/stop/start
but not seen any thing about this
its very strange that presto not comes with some user interface that can show the cluster status and do stop/start action if we need to do so
as all know the only user interface of presto is show status and not have the actions as stop/start
in the above example screen we can see that the active presto worker are only 5 from 231 , but this UI not support stop/start actions and not show on which worker presto isn't active
so what we can do about it?
its very bad idea to access each worker machine and see if presto is up or down
why presto not have centralized UI that can do stop/start action ?
example what we are expecting from the UI , - partial list
.
.
.
Presto currently uses discovery service where workers announce themselves to join the cluster, so if a worker node is not registered there is no way for coordinator or discovery server to know about its presence and/or restart it.
At Qubole, we use an external service alongside presto master that tracks nodes which do not register with discovery service within a certain interval. This service is responsible for removing such nodes from the cluster.
One more thing we do is use monit service on each of presto worker nodes, which ensures that presto server is restarted whenever it goes down.
You may have to do something similar for cluster management , as presto does not provide it right now.
In my opinion and experience managing prestosql cluster, it matters of service discovery in architecture patterns.
So far, it uses following patterns in the open source release of prestodb/prestosql:
server-side service discovery - it means a client app like presto cli or any app uses presto sdk just need to reach a coordinator w/o awareness of worker nodes.
service registry - a place to keep tracking available instances.
self-registration - A service instance is responsible for registering itself with the service registry. This is the key part that it forces several behaviors:
Service instances must be registered with the service registry on startup and unregistered on shutdown
Service instances that crash must be unregistered from the service registry
Service instances that are running but incapable of handling requests must be unregistered from the service registry
So it keeps the life-cycle management of each presto worker to each instance itself.
so what we can do about it?
It provides some observability from presto cluster itself like HTTP API /v1/node and /v1/service/presto to see instance status. Personally I recommend using another cluster manager like k8s or nomad to manage presto cluster members.
its very bad idea to access each worker machine and see if presto is up or down
why presto not have centralized UI that can do stop/start action ?
No opinion on good/bad. Take k8s for example, you can manage all presto workers as one k8s deployment and manage each presto worker in one pod. It can use Liveness, Readiness and Startup Probes to automate the instance lifecycle with a few YAML code. E.g., the design of livenessProbe of helm chart stable/presto. And cluster manageer like k8s does provide web UI so that you can touch resources to act like an admin. . Or you can choose to write more Java code to extend Presto.
I am wrapping elastic search on windows in a service fabric stateless service, such it runs the elastic search node on each node that the service is running.
Elastic Search is distributed in the code package and can be updated with the application.
So in a 3 node service fabric, the elastic node will get the name of each node in service fabric.
What would be the best approach to locate the data for elastic node?
My own idea would be to locate it on the VM temp disks, as long as more nodes are up, then elastic should replicate data internal such one node can die.
Then i would also do daily backups copying all data to external storage to be able to restore it.
Is there any other options that I should consider?
Want to deploy datastax Cassandra cluster(Multiple DC's - in multiple regions) in google cloud compute engine with auto scale capability.
I have deployed datastax cassandra cluster in google cloud but not sure how to configure auto scaling capability .
In google cloud compute engines auto scaling is only possible with managed instances group. I.e. all the instances should be in a particular region. Since cassandra cluster deployed in multiple regions, do we need maintain separate auto scaling(managed instance template) feature for each region.
Do we need to use google cloud auto scaling concept or datastax has its own in build one?
How to add new VM's to cluster if we provision a VM's based on google cloud managed instance group concept.
Thanks,
You need to maintain separate auto scaling.
And Let Google Cloud maintain the auto scaling because the machines that need to be added are obviously handled by Google Cloud. DataStax Enterprise has no way of doing it automatically.
I don't know each and every step (for Google Cloud) but when you spin up a machine,you can keep a script to run which starts Cassandra with some set of 'seeds'.
Can we create a multi datacenter cluster using OpsCenter alone? I am able to create one ring, but it is not clear how I can specify the data center settings for the nodes in the second ring.
Currently it's not possible to create a multi datacenter cluster with OpsCenter. It can manage such a cluster if you create it yourself, but cannot create one.
Here are the relevant docs for doing a multi-DC install:
Cassandra: Initializing a multiple node cluster (multiple data centers)
DSE: Multiple data center deployment per workload type
FYI regarding #phact comment above: OpsCenter does automatically create separate logical datacenters when using DataStax Enterprise, separating the workload into Cassandra, Solr, and Analytics DCs. However it does not support creating multiple Cassandra-only datacenters, for example.