I have two Presto clusters and I want to bring them under Presto gateway.
What are the challenges and what unexpected can happen if I have only one cluster in each group?
I want all of my users to use Gateway connection instead of coordinator.
Related
I have a cluster of microservices to be hosted on Azure Kubernetes Service.
These microservices are .NET Core based and will
talk to on-premises services via gRPC
stream data using SignalR Core to client apps(Websockets)
The problem I can't find a good solution for is "How to persist gRPC" connections as pods are created and destroyed.
This seems like a very trivial problem for hosting microservices on a hybrid network. I would love to hear how others have addressed this issue.
Persistance of grpc would be difficult in such environment as pods are not persistant at all. I would suggest two approach to handle this scenario
Use/build a proxy between EKS and On Premises Service which can keep persistant connection open with on-premises service but connection to proxy can be added/removed as the pods are created/destroyed. This proxy can act as connection pool and provide higher throughput to on-premises service invocation.
Don't worry about the persistance of connection with on-premises services (treat this like a rdbms connection which can be created or destroyed on demand but has some cost in order to create new). This approach would work in case the pods are created or destroyed not too frequently.
I would suggest second approach in case the pods are not created/destroyed too frequently (few every hour) as it has less moving parts. But if pods are scaled up too frequently, approach one should be used.
I am developing an application from scratch using Springboot (Jhipster) and intend to deploy in a cloudfoundry with multiple nodes. Planning to setup prometheus server which can pull metrics from the cluster.
I am trying to understand how can I setup Prometheus to query individual nodes of the cluster. As the application is deployed in cloud foundry, it may not be possible to obtain the ip address of individual nodes.
As I am newbie with Prometheus, want to make sure I am solutioning appropriately.
I would recommend using Micrometer.io
You can use either service discovery (e.g. Netflix Eureka) or use the Pushgateway in some form.
https://prometheus.io/docs/instrumenting/pushing/
https://prometheus.github.io/client_java/io/prometheus/client/exporter/PushGateway.html
We have hosted a Hazelcast cluster on a server and a different app on a different server in the same region uses Hazelcast Jet client instance to aggregate the data from Kafka source using pipeline.
In this setup, does Jet client instance send data which it receives from Kafka source Hazelcast cluster which will involve lot of IO or when we create pipeline, the Hazelcast cluster itself creates the connection to Kafka and this connection is from the Jet cluster rather than from the client app?
#Abhishek, you need to have a Jet cluster somewhere. The job itself run on the Jet cluster. If you have Kafka as a source, your app will submit the job to Jet cluster and Jet cluster will connect to data. If your sink is a Hazelcast map on the same Jet cluster, it'll be a local write, if it's a separate Hazelcast cluster, then you need to use Sinks.remoteMap configuring a client to connect to remote Hazelcast cluster.
I have a StatefulSet Cassandra deployment that works great for services deployed to Kubernetes with namespace access, but I also have an ETL job that runs in EMR and needs to load data into that Cassandra cluster.
What would be the main approach/Kubernetes way of doing this?
I can think of two options.
Simple one is you can create the server with Type: NodePort, with this you can connect server with Node IP Address:PortNumber.
Second option is you can create the Ingress Load Balancer and connect to Cassandra cluster.
trying to figure out how to create Cassandra cluster in Azure across more than one datacentre.
I am not much interested in Cassandra topology settings yet, but more in how to set Azure endpoints or inter-datacentre communication to allow nodes to connect remotely. Do I need to set endpoint for node communication inside one datacentre?
What about security of azure endpoints?
Thank you.