I would like to add additional name servers to kube-dns in the kube-system namespace (solution provided here: https://stackoverflow.com/a/34085756/2461761), however doing this in an automated manner.
So I know I can create my own dns addon via https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns and launch it in the system's namespace, however, I am trying to provide a simple script to developers to spin up their own clusters with DNS resolution built in and don't want them to concern with the system namespace.
Is there a way to modify/set the SKYDNS_NAMESERVERS environment variable without having to making a copy of the replication controller?
Or even set it via a command and re-launch the pods of the kube-dns deployment?
Thank you in advance.
I still think that "adding SKYDNS_NAMESERVERS to the manifest file" solution is a good choice.
Suppose the developers still need to spin up the cluster, it would be better to set up the upstream DNS servers ahead through the manifest file instead of changing them on the fly. Or is there any requirement that need this to be done after the cluster is up?
If this has to be done while everything is running, one way to do so is to modify the manifest file locates on the master node. For current version kubernetes(1.4), you will also need to modify the ReplicationController name to a new one and the Addon Manager will then update the resources for you. But notice that there would be kube-dns outage(probably seconds) in between because current Addon Manager executes the update in the delete->create manner.
Related
I have successfully deployed a multi master Kubernetes cluster using the repo https://github.com/kubernetes-sigs/kubespray and everything works fine. But when I stop/terminate a node in the cluster, new node is not joining to the cluster.I had deployed kubernetes using KOPS, but the nodes were created automatically, when one deletes. Is this the expected behaviour in kubespray? Please help..
It is expected behavior because kubespray doesn't create any ASGs, which are AWS-specific resources. One will observe that kubespray only deals with existing machines; they do offer some terraform toys in their repo for provisioning machines, but kubespray itself does not get into that business.
You have a few options available to you:
Post-provision using scale.yml
Provision the new Node using your favorite mechanism
Create an inventory file containing it, and the etcd machines (presumably so kubespray can issue etcd certificates for the new Node
Invoke the scale.yml playbook
You may enjoy AWX in support of that.
Using plain kubeadm join
This is the mechanism I use for my clusters, FWIW
Create a kubeadm join token using kubeadm token create --ttl 0 (or whatever TTL you feel comfortable using)
You'll only need to do this once, or perhaps once per ASG, depending on your security tolerances
Use the cloud-init mechanism to ensure that docker, kubeadm, and kubelet binaries are present on the machine
You are welcome to use an AMI for doing that, too, if you enjoy building AMIs
Then invoke kubeadm join as described here: https://kubernetes.io/docs/setup/independent/high-availability/#install-workers
Use a Machine Controller
There are plenty of "machine controller" components that aim to use custom controllers inside Kubernetes to manage your node pools declaratively. I don't have experience with them, but I believe they do work. That link was just the first one that came to mind, but there are others, too
Our friends over at Kubedex have an entire page devoted to this question
Well, for the last 2 days I battled this documentation:
https://learn.microsoft.com/en-au/azure/aks/static-ip
and
https://learn.microsoft.com/en-au/azure/aks/ingress-own-tls
First of all I ensured that I had my aks k8s cluster upgraded to 1.11.5, so there is no question about having the static IP in a different resource group.
Overall, I could not get the static IP really working. With dynamic everything sounds fine, but I cannot add a A record for a dynamic IP.
I managed to deploy everything successfully, but any curl ip.. does not work. I did run exec -ti locally, and locally everything is fine.
Could someone please point me to a GitHub config or article that has this configuration running? As a disclaimer I know azure very well, so well the service principal assignments are well done, etc. However, I am new, only a few months on k8s.
Thanks in advance for any suggestion.
I can share logs if needed but believe I did check everything from dns to ingress routes. I am worried that this doc is not good and I am just loosing my time.
Answering myself this question, after quite a journey, for when I get older and I forget what I've done, and maybe my nephew will save some hours someday.
First, it's important:
In the values provided to nginx-ingress chart template, there are 2 annotations that are important:
service.beta.kubernetes.io/azure-load-balancer-resource-group: "your IP's resource group"
externalTrafficPolicy: "Local"
Here are all the values documented: https://github.com/helm/charts/blob/master/stable/nginx-ingress/values.yaml
The chart can be deployed near your service's namespace, it should not be in kube-system (with my current knowledge I don't find a reason to have it in system).
Second, could be misleading
There is a delay of ~30+ seconds (in my case) from the moment when IP appeared in the kubectl get services --watch and till the moment curl -i IP was able to answer the call. So, if you have automation or health probes then ensure that you have 1 - 2 mins added to wait. Or maybe take better nodes, bare metal machines.
Look at GCE and DO for the same setup as might help:
https://cloud.google.com/community/tutorials/nginx-ingress-gke
https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes
The guys at DO, are good writes as well.
Good luck!
Based on your comments, it seems that you are trying to override the externalIPs but use the default value of the helm chart for controller.service.type which is LoadBalancer. What you might want to do is to keep controller.service.type to LoadBalancer and set controller.service.loadBalancerIP with your static IP instead of overriding externalIPs.
Here some documentation from microsoft.
When launching EC2 using Terraform (or cloud formation), we can configure EC2 by putting some scripts in user_data/remote-exec. Alternatively, we can configure EC2 using Ansible/Chef, etc. What are the difference of configuring EC2 in user_data/remote-exec and do that with Ansible/Chef? when to use the former, when to use the latter (I know Ansible/Chef is idempotent)?
In my case, the EC2 is originally manually launched, then manually configured using a lot of linux commands. and the commands are not configured by me. Now I am the person to automate the whole structure using terraform, and configure EC2s. Using user_data/remote-exec to configure EC2 is straightforward. I just need to put all the existing linux commands they have in some scripts with a little change. And if the configuration result using my script is not successful, at least I can quickly figure out whether I miss some commands by comparing my script and the original linux commands. But if I use ansible/chef, I have to rewrite all the steps using different language. And if the configuration is not what expected, it is hard for me to figure out which steps are not correct, because the syntax of ansible/chef and linux commands are totally different.
My question is, in my case, should I use ansible/chef or user_data/remote-exec for configuration?
User Data is good for initial configuration of the system. If you need longer term maintenance a configuration management software like Ansible/Chef/Salt/Puppet is a great option.
Packer can be used for immutable infrastructure, i.e. doesn't change after creation. You can run all the scripts and installs on the system for it to be ready to just boot, this is also faster because you don't have to wait for user data to run.
A few questions you have to ask as well, how often are you going to patch these? Are you going to just update existing or replace with new. Ansible is great for configuration since it's just yaml files an
Blue/Green deployments generally replace servers with all new ones and gradually move traffic over to the new servers.
Some more things to consider with your Infrastructure as code
I have been using terraform to create a CoreOs Cluster on Digital Ocean just fine. My question was addressed here but nearly a year has passed
which seems like 10 on a fast pace projects like etcd2 and terraform. IMHO, if the master fails terraform will create another instance with the exact same configurantion, but according to the free discovery coreos service the cluster will be full and all the slaves will have the wrong ip to connect to the etcd2 master. In the case of a minion failure, the master ip wont be an issue, but I still wont be able to join a full cluster.
How does terraform deal with this kind of problem? Is there a solution or am I still binded to a hacky solution like the link above?
If I run terraform taint node1. It there a way to notify the dicovery service this change?
Terraform doesn't replace configuration management tools like Ansible, Chef and Puppet.
This can be solved using a setup where, say, a Ansible run is triggered to reconfigure the slaves when the master is reprovisioned. The ansible inventory in this case, would have been update by terraform with the right ip, and the slave ansible role can pick this up and configure appropriately.
There are obviously other ways to do this, but it is highly recommended that you couple a proper CM tool with Terraform and propagate such changes.
What is the best way to set yaml settings? I am using docker containers and want to automate the process of setting cassandra.yaml settings like seeds, listen_address & rpc_address.
I have seen something like this in other yaml tools: <%= ENV['envsomething'] %>
Thanks in advance
I don't know about the "best" way but when I set up a scripted cluster of cassandra servers on a few vagrant vms I used puppet to set the seed and so on in cassandra.yaml.
I did write some scripting than used puppetdb to keep track of the addresses of the hosts but this wasn't terrifically successful. The trouble was the node that came up first only had itself in the list of seeds and so tended to make a cluster on it's own. Then the rest would come up as a seperate cluster. So I had to take down the solo node, clear it out and restart it with correct config
If I did it now I would set the addresses as static ip, then use them to fill in the templates for the cassandra.yaml files on all the nodes. Then hopefully the nodes would come up with the right idea about the other cluster members.
I don't have any experience with Docker but they do say the way to use puppet+Docker is to use puppet on the Docker container before starting it up
Please note that you need a lot of memory to make this work. I had a machine with 16GB and that was a bit dubious.
Thank you for information.
I was considering using https://github.com/go-yaml/yaml
But this guy did the trick: https://github.com/abh1nav/docker-cassandra
Thanks
If you're running Cassandra in Docker use this as an example: https://github.com/tobert/cassandra-docker You can override cluster name/seeds when launching so whatever config management tool you use for deploying your containers could do something similar.