What is the best way to set yaml settings? I am using docker containers and want to automate the process of setting cassandra.yaml settings like seeds, listen_address & rpc_address.
I have seen something like this in other yaml tools: <%= ENV['envsomething'] %>
Thanks in advance
I don't know about the "best" way but when I set up a scripted cluster of cassandra servers on a few vagrant vms I used puppet to set the seed and so on in cassandra.yaml.
I did write some scripting than used puppetdb to keep track of the addresses of the hosts but this wasn't terrifically successful. The trouble was the node that came up first only had itself in the list of seeds and so tended to make a cluster on it's own. Then the rest would come up as a seperate cluster. So I had to take down the solo node, clear it out and restart it with correct config
If I did it now I would set the addresses as static ip, then use them to fill in the templates for the cassandra.yaml files on all the nodes. Then hopefully the nodes would come up with the right idea about the other cluster members.
I don't have any experience with Docker but they do say the way to use puppet+Docker is to use puppet on the Docker container before starting it up
Please note that you need a lot of memory to make this work. I had a machine with 16GB and that was a bit dubious.
Thank you for information.
I was considering using https://github.com/go-yaml/yaml
But this guy did the trick: https://github.com/abh1nav/docker-cassandra
Thanks
If you're running Cassandra in Docker use this as an example: https://github.com/tobert/cassandra-docker You can override cluster name/seeds when launching so whatever config management tool you use for deploying your containers could do something similar.
Related
I've an ELK stack 7.6.2 with logstash, an elasticsearch cluster with 3 nodes and kibana. I would like to add security but the only doc I can fin always start 'from scratch' I would like to have an example on an already running cluster in order not te mess up with it. Thanks for your help.
Guillaume
You can not enable security features on an already running cluster. Security-settings are classified as static, meaning that they can not be dynamically updated on the fly:
static:
These settings must be set at the node level, either in the elasticsearch.yml file, or as an environment variable or on the command line when starting a node. They must be set on every relevant node in the cluster.
dynamic:
These settings can be dynamically updated on a live cluster with the cluster-update-settings API.
See https://www.elastic.co/guide/en/elasticsearch/reference/7.6/modules.html for reference and for all settings that can be dynamically updated (you won't find security settings there).
Also, from this guide (https://www.elastic.co/guide/en/elasticsearch/reference/current/get-started-enable-security.html) one can tell that you need to stop your running elasticsearch and kibana instances in order to enable security.
I hope I could help you.
I have successfully deployed a multi master Kubernetes cluster using the repo https://github.com/kubernetes-sigs/kubespray and everything works fine. But when I stop/terminate a node in the cluster, new node is not joining to the cluster.I had deployed kubernetes using KOPS, but the nodes were created automatically, when one deletes. Is this the expected behaviour in kubespray? Please help..
It is expected behavior because kubespray doesn't create any ASGs, which are AWS-specific resources. One will observe that kubespray only deals with existing machines; they do offer some terraform toys in their repo for provisioning machines, but kubespray itself does not get into that business.
You have a few options available to you:
Post-provision using scale.yml
Provision the new Node using your favorite mechanism
Create an inventory file containing it, and the etcd machines (presumably so kubespray can issue etcd certificates for the new Node
Invoke the scale.yml playbook
You may enjoy AWX in support of that.
Using plain kubeadm join
This is the mechanism I use for my clusters, FWIW
Create a kubeadm join token using kubeadm token create --ttl 0 (or whatever TTL you feel comfortable using)
You'll only need to do this once, or perhaps once per ASG, depending on your security tolerances
Use the cloud-init mechanism to ensure that docker, kubeadm, and kubelet binaries are present on the machine
You are welcome to use an AMI for doing that, too, if you enjoy building AMIs
Then invoke kubeadm join as described here: https://kubernetes.io/docs/setup/independent/high-availability/#install-workers
Use a Machine Controller
There are plenty of "machine controller" components that aim to use custom controllers inside Kubernetes to manage your node pools declaratively. I don't have experience with them, but I believe they do work. That link was just the first one that came to mind, but there are others, too
Our friends over at Kubedex have an entire page devoted to this question
I would like to add additional name servers to kube-dns in the kube-system namespace (solution provided here: https://stackoverflow.com/a/34085756/2461761), however doing this in an automated manner.
So I know I can create my own dns addon via https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns and launch it in the system's namespace, however, I am trying to provide a simple script to developers to spin up their own clusters with DNS resolution built in and don't want them to concern with the system namespace.
Is there a way to modify/set the SKYDNS_NAMESERVERS environment variable without having to making a copy of the replication controller?
Or even set it via a command and re-launch the pods of the kube-dns deployment?
Thank you in advance.
I still think that "adding SKYDNS_NAMESERVERS to the manifest file" solution is a good choice.
Suppose the developers still need to spin up the cluster, it would be better to set up the upstream DNS servers ahead through the manifest file instead of changing them on the fly. Or is there any requirement that need this to be done after the cluster is up?
If this has to be done while everything is running, one way to do so is to modify the manifest file locates on the master node. For current version kubernetes(1.4), you will also need to modify the ReplicationController name to a new one and the Addon Manager will then update the resources for you. But notice that there would be kube-dns outage(probably seconds) in between because current Addon Manager executes the update in the delete->create manner.
I have been using terraform to create a CoreOs Cluster on Digital Ocean just fine. My question was addressed here but nearly a year has passed
which seems like 10 on a fast pace projects like etcd2 and terraform. IMHO, if the master fails terraform will create another instance with the exact same configurantion, but according to the free discovery coreos service the cluster will be full and all the slaves will have the wrong ip to connect to the etcd2 master. In the case of a minion failure, the master ip wont be an issue, but I still wont be able to join a full cluster.
How does terraform deal with this kind of problem? Is there a solution or am I still binded to a hacky solution like the link above?
If I run terraform taint node1. It there a way to notify the dicovery service this change?
Terraform doesn't replace configuration management tools like Ansible, Chef and Puppet.
This can be solved using a setup where, say, a Ansible run is triggered to reconfigure the slaves when the master is reprovisioned. The ansible inventory in this case, would have been update by terraform with the right ip, and the slave ansible role can pick this up and configure appropriately.
There are obviously other ways to do this, but it is highly recommended that you couple a proper CM tool with Terraform and propagate such changes.
this question might be a silly one but since i am new in hadoop and there are very few material available online which can be used as a reference point so i thought this might be the best place to ask this question .
i have successfully configured few computers in multi node configuration. during the setup process i have to change many hadoop file .now i am wondering can i use every single computer as an single node configuration with out changing any settings or hadoop file ?
You can make your each node as separate instance. But you have to modify the configuration files surely and restart all the instances.
You can do that
Follow below steps
Remove IP or Hostname from masters file
Remove IP's or hostname's from slaves file
change fs.defaultFS property IP address in core-site.xml
As well as Resource Manager IP