I would like to know more about how to connect to a Kafka MSK Cluster.
You need a Kafka client.
Since you have node.js, start with kafkajs or node-rdkafka. I'd recommend kafkajs since it supports SASL consumption with AWS IAM
Beyond that, MSK is an implementation detail, and any steps will work for all Kafka clusters (depending on other security implementations). However, cloud providers will require that you allow your client IP address space within the VPC, and you can follow the official getting started guide for verifying a basic client will work.
Related
I am developing an application from scratch using Springboot (Jhipster) and intend to deploy in a cloudfoundry with multiple nodes. Planning to setup prometheus server which can pull metrics from the cluster.
I am trying to understand how can I setup Prometheus to query individual nodes of the cluster. As the application is deployed in cloud foundry, it may not be possible to obtain the ip address of individual nodes.
As I am newbie with Prometheus, want to make sure I am solutioning appropriately.
I would recommend using Micrometer.io
You can use either service discovery (e.g. Netflix Eureka) or use the Pushgateway in some form.
https://prometheus.io/docs/instrumenting/pushing/
https://prometheus.github.io/client_java/io/prometheus/client/exporter/PushGateway.html
We have a Kafka cluster and various producers and consumers running across at least 5 differe servers. I have been asked to secure Kafka environment using SSL. I have configured SSL on brokers, now it's time to secure clients. So in the document it's mentioned that below config needs to be placed in producer/consumer config
security.protocol=SSL
ssl.truststore.location=/var/private/ssl/kafka.client.truststore.jks
ssl.truststore.password=test1234
Now the problem is I have distributed producer and consumer application jars to different teams and they have deployed them on their application servers. How can I create truststore for all of them? Can anyone explain me this concept. In case clients run on hundreds of server how can one create truststore on each server? Please help.
We are currently running two Kafka clusters. One is inside our corporate network and another is in AWS. We need to stream data from the internal Kafka to the cluster in AWS. Mirror Maker is currently our first choice based on simplicity. I've been investigating on how to do this in a secure fashion. I would prefer not to place Kafka on a public subnet as it seems that is not a secure option but, some limitations of Kafka have made this somewhat difficult. Namely that Kafka producers/consumers need to be able to see all instances of the target cluster on the network. This means that all Kafka instances in AWS need a public IP address.
The alternative is to place Mirror Maker in AWS, but then how to we expose our internal servers to the web in a secure manner? This seems like it might be a common use case, but I cannot find anything on the web that relates. Can someone provide recommendations or correct me if any of my assumptions are incorrect?
Looking the documentation of AWS Elastic Cache I can see they support Redis Cluster and talk about key/value data and Redis operations in general.
However is not clear for me if this will support replication of Redis' pub/sub along the different servers.
We are building a chat server on node-xmpp. We will have many application servers for handling chat connections and we are relying on Redis pub/sub for handling the communication between chat threads. We require that regardless the actual Redis instance each chat server is communicating to, they can share the same pub/sub channel.
At AWS Elastic cache white paper (page 7) they indicate to use Redis if you want pub/sub. I understand from this that AWS Elastic Cache will actual support pub/sub scalability but I'm not convinced yet.
I have tried it and yes, AWS manages pub/sub channels among Redis instances within a cluster. We have deployed now three chat platforms using node-xmpp and redis-cluster on AWS Elastic Cache for managing the real-time capabilities of the chat, and it works neatly.
trying to figure out how to create Cassandra cluster in Azure across more than one datacentre.
I am not much interested in Cassandra topology settings yet, but more in how to set Azure endpoints or inter-datacentre communication to allow nodes to connect remotely. Do I need to set endpoint for node communication inside one datacentre?
What about security of azure endpoints?
Thank you.