I have a Kubernetes cluster with 50+ pods on it, and I want to grab the logs from all of these pods, then store logs in the ElasticSearch and visualize that using Kibana, but ElasticSearch and Kibana should be outside Kubernetes, on another virtual machine in the same network.
How can I configure the Fluentd to grab and send logs to Non-Kubernetes ElasticSearch?
It is totally possible. In the kubernetes cluster, you would need to expose the Fluentd service on an external IP reachable from the virtual machine outside the cluster, and run ElasticSearch and Kibana on this virtual machine.
ElasticSearch (outside the kubernetes cluster) will access Fluentd (inside the kubernetes cluster) using the Fluentd service in k8s and pull the logs.
There are four ways to expose the Fluentd service in k8s for external access by ElasticSearch:
LoadBalancer service type which sets the ExternalIP automatically. This is used when there is an external non-k8s, cloud-provider's load-balancer like CGE, AWS or Azure, and this external load-balancer would provide the ExternalIP for the nginx ingress service.
ExternalIPs per https://kubernetes.io/docs/concepts/services-networking/service/#external-ips.
NodePort: In this approach, the service can be accessed from outside the cluster using NodeIP:NodePort/url/of/the/service.
Ingress: https://kubernetes.io/docs/concepts/services-networking/ingress/
Related
I develop microservices (using nodejs) for each microservice I serve with a random port, for that, I need a discovery service.
The question is does kubernetes have a service discovery solution or should I implement this myself?
The short answer is yes, Kubernetes does have a service discovery solution. Kubernetes has a built-in DNS server that can be used to discover services. For example, if you have a service named my-service in the my-namespace namespace, you can access it from any pod using http://my-service.my-namespace.svc.cluster.local.
The long answer is that Kubernetes provides two different types of service discovery:
DNS-based service discovery
Environment variable-based service discovery
DNS-based service discovery
DNS-based service discovery is the most common service discovery mechanism in Kubernetes. It is based on the built-in DNS server that is part of the Kubernetes cluster. Kubernetes automatically creates DNS records for each service. The DNS records are named after the service name and the service namespace. The DNS records are stored in the cluster DNS server, which is a special type of DNS server that is part of the Kubernetes cluster. The cluster DNS server is automatically configured as the default DNS server for all pods in the cluster. If you have a service named my-service in the my-namespace namespace, you can access it from any pod using http://my-service.my-namespace.svc.cluster.local.
Environment variable-based service discovery
Environment variable-based service discovery is a more advanced service discovery mechanism. It is based on the fact that Kubernetes automatically creates environment variables for each service. The environment variables are named after the service name and the service namespace. The environment variables are stored in the container's environment. If you have a service named my-service in the my-namespace namespace, you can access it from any pod using the MY_SERVICE_MY_NAMESPACE_SERVICE_HOST and MY_SERVICE_MY_NAMESPACE_SERVICE_PORT environment variables.
DNS-based service discovery vs. environment variable-based service discovery
DNS-based service discovery is the most common service discovery mechanism in Kubernetes. It is very easy to use and it is supported by all Kubernetes client libraries. Environment variable-based service discovery is a more advanced service discovery mechanism. It is more flexible and it is supported by all Kubernetes client libraries. The main drawback of environment variable-based service discovery is that it requires the client application to be aware of the Kubernetes API.
I have a separate VM in the same network as my kubernetes in Azure.
I have a kafka pod and I am able to reach this pod using the IP. The problem is that the pod IP is changing all the time.
Is there any way to get the correct IP each time the pod IP is changing?
I would suggest using a kubernetes service to expose pod. This avoids the problem with change in POD IP because service IP does not change.
Kubernetes ServiceTypes allow you to specify what kind of Service you want. The default is ClusterIP.
Type values and their behaviors are:
ClusterIP: Exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType.
NodePort: Exposes the Service on each Node's IP at a static port (the NodePort). A ClusterIP Service, to which the NodePort Service routes, is automatically created. You'll be able to contact the NodePort Service, from outside the cluster, by requesting :.
LoadBalancer: Exposes the Service externally using a cloud provider's load balancer. NodePort and ClusterIP Services, to which the external load balancer routes, are automatically created.
ExternalName: Maps the Service to the contents of the externalName field (e.g. foo.bar.example.com), by returning a CNAME record
Since you are accessing the POD from outside the kubernetes cluster itself use NodePort or LoadBalancer type service.
As mentioned by #arghya-sadhu already going for Kubernetes service, is the best option. The kubernetes service has an IP depending on the type of kubernetes service.
For services of type ClusterIP, you get a cluster IP address
For services of type Load Balancer, you get a Loadbalancer IP address (i.e) public IP address
For services of type NodePort, you can access using the node's address.
But, whatever the type of service is, you can access the service using the kube-DNS within the cluster. So, let's say your service name is other-service and it exposes port 8080, running on namespace abc, then you can access the service as follows:
http://other-service.abc:8080
Since, your VM runs outside the cluster, it is better to use Loadbalancer and access the pod using Loadbalancer url or IP address. You can set an ingress in case there are multiple pods in the cluster that you want to connect to.
We have a system in an azure kubernetes cluster, consisting of 7 nodes. 3 of those nodes are consul servers, forming a quorum. We are encountering a problem, where when the pods restart, their IP address changes. Thus we are forced to re-configure the consul servers manually.
Consul is installed using the Hashicorp helm chart for our consul cluster. all of its files are stored in a persistent volume (/data) and it does store the nodeid in StatefulSet.
IF there is a way where consul can reconfigure itself or kubernetes can provide a static IP for the consul servers to connect with each other, I would appreciate it if it could be shared!
Did you install Consul on your cluster using the Hashicorp helm chart? Their architecture uses a StatefulSet for the Consul server pods and persistent volume claims to store the node-id so the pods can move around. (ref: https://www.consul.io/docs/k8s/installation/overview#server-agents)
If you have used another installation method, do you have persistent volumes so the node-id does not change between restarts? Please expand on your Consul installation method, current configuration and re-configuration steps that are required as well.
I was going through Docker and Kubernetes . I want to create two Python web servers and need to access them using public URL and these requests should be balanced between two servers.
I created one Python server and initially deployed that with Docker containers and all this I'm doing using AWS ec2 instance so when I tried to send a request I used ec2publicip:port. This is working which means I created one web server and similarly I will do the same for the second server.
My question is If I deploy this with Kubernetes - Is there any way to do load balancing the Python web servers within the pod. If so, can someone tell me how to do this?
If you create two replicas of the pod via a kubernetes deployment and create a service of type LoadBalancer an ELB on AWS is automatically provisioned.Then whenever a request comes to the ELB on AWS it will distribute the traffic to the replicas of the pod. With a loadbalancer type service you get advanced load balancing capabilities at layer 7. Without a loadbalancer type service or an ingress you get round robin load balancing at layer 4 offered by kube proxy.
Problem with loadbalancer type service is that it will create new ELB for each service which is costly. So I recommend using ingress controller such as Nginx and expose the Nginx Ingress controller via a single loadbalancer on AWS. Then create ingress resource and use path or host based routing to send traffic to pods behind a clusterIP type service.
We have an application that runs on an Ubuntu VM. This application connects to Azure Redis, Azure Postgres and Azure CosmosDB(mongoDB) services.
I am currently working on moving this application to Azure AKS and intend to access all the above services from the cluster. The services will continue to be external and will not reside inside the cluster.
I am trying to understand how the network/firewall of both the services and aks should be configured so that pods inside the cluster can access the above services or any Azure service in general.
I tried the following:
Created a configMap containing the connection params(public ip/address, username/pwd, port, etc) of all the services and used this configMap in the deployment resource.
Hardcoded the connection params of all the services as env vars inside the container image
In the firewall/inbound rules of the services, I added the AKS API ip, individual node ips
None of the above worked. Did I miss anything? What else should be configured?
I tested the setup locally on minikube with all the services running on my local machine and it worked fine.
I am currently working on moving this application to Azure AKS and
intend to access all the above services from the cluster.
I assume that you would like to make all services to access each other and all the services are in AKS cluster? If so, I advise you configure the internal load balancer in AKS cluster.
Internal load balancing makes a Kubernetes service accessible to
applications running in the same virtual network as the Kubernetes
cluster.
You can take a try and follow the following document: Use an internal load balancer with Azure Kubernetes Service (AKS). In the end, good luck to you!
Outbound traffic in azure is SNAT-translated as stated in this article. If you already have a service in your AKS cluster, the outbound connection from all pods in your cluster will come thru the first LoadBalancer type service IP; I strongly suggest you create one for the sole purpose to have a consistent outbound IP. You can also pre-create a Public IP and use it as stated in this article using the LoadBalancerIP spec.
On a side note, rather than a ConfigMap, due to the sensitiveness of the connection string, I'd suggest you create a Secret and pass that down to your Deployment to be mounted or exported as environment variable.