I would like to get help with relabeling in prometheus.
I am trying to replace instance with nodename, so here in prometheus before relabeling I would like to have this Nodename
We are using kube-prometheus and we have 5 clusters where is cassandra running in 6 pods.
I want to use this because this dashboard has label with instances dashboard
My attempts to workaround this with * on(pod) group_left(node) kube_pod_info
failed, it will duplicate give false number and so I guess the best approach would be to replace the instance with the name or maybe just add another label with the nodename instead of replacing, I have tried a lot of things but this dashboard is just not good fit for multiple clusters and multiple instances of cassandra with same names. Is it possible to use relabeling even when I would not specify the target because the targets are being auto discovered? Or what do you think is the best approach in this case?
Thanks
Edit:
In the end I just edited ip address to pod names instead of node names, which eventually was bad idea.
Can be closed
Related
I have cluster about 100 nodes and it grows. I need to add 10-50 on request. As I know by default cassandra has cassandra.consistent.rangemovement=true this means multiple nodes can't to bootstrap in a moment.
Anyway when I add many nodes using Terraform and some kind of default configuration (using Puppet) at least 2-3 becomes UJ state and eventually only one bootstrap successfully. Earlier I used random time delay before start cassandra.service, but it doesn't work adding 10+ nodes.
I'm trying to figure out how to implement kind of "lock" for bootstrap.
I have Consul and can get kind of lock for bootstrap in KV. For instance get lock using ExecPreStart systemd feature but I can't get how to release it after bootstrap.
I'm looking for any solutions for that.
I've done something similar using Rundeck before. Basically, we had Rundeck kick off a bash script, taking parameters about the deployment of our nodes as well as how many.
What we did, was parse the output of nodetool status. We'd count the number of nodes as well as the number of UN indicators. If those two numbers didn't match, we'd do a sleep 30s and try again.
Once those numbers matched, we knew that it was safe to add another node. The total operation could take a while to add all nodes, but it worked.
I need to run identical jobs in schedule, and they differ only in few strings.
As you may know, there is no a convenient way to create identical jobs with different parameters. For now i prefer so "codeless" way to do so, or with "as less code as possilbe".
So lets imagine they are stored in a rows of JobsConfigurations table of the website-related database.
How I can get the Job name of job being running to pick the right configuration from the table?
Thanks for help!
See https://github.com/projectkudu/kudu/wiki/Web-Jobs#environment-settings
The WEBJOBS_NAME environment variable will give you the name of the current WebJob.
We are deploying our application to production this month and our stack will include a 3 node, single datacenter Cassandra version 1.2 cluster. In anticipation of this, we have been getting our initial Cassandra.yaml settings worked out. While doing this I ran into a interesting situation for which I haven't been able to find an answer.
This has to do with setting the -seeds parameter in each of the nodes Cassandra.yaml files. All of the reading I've done say it is best practice to:
Have at least 2 seeds per datacenter. This makes sense so that one of the nodes can come down and other nodes can be seeded by the second seed.
These two seeds should be the same for all (in our case 3) nodes.
In the deployment I tested this on, I started out with all three nodes having a single seed, node 1's IP address. My intention was to change the seeds of all three nodes to the IP address of node1 and node2. First I did node 3 by:
decommissioning the node.
Shutting down Cassandra.
changing the -seeds value to ip_node1,ip_node2
starting up Cassandra.
running nodetool status to ensure the node was added back to the cluster.
Next I did node 2, following the exact same steps I did for node 3. But something unexpected happened. When I restarted Cassandra on node 2, it did not join the existing ring. Instead it started its own single node ring. It seems pretty obvious that of the two seed parameters I passed it, it used its own IP address and thus believed it was the first node in a new ring.
I was surprised Cassandra didn't select the seed argument of the other seed value I passed it (node 2's). The only way I could get it to join the existing datacenter was to set its seeds to one or both of the other nodes in the cluster.
An obvious work around to this is to configure each of my three nodes seeds value to the IP addresses of the other two nodes in the cluster. But since several sources have suggested this isn't a "Best Practice" I thought I'd ask how this should be handled. So my question is:
Is it normal for Cassandra to always use its own IP address as a seed if it is in the seed list?
Is configuring the cluster the way I've suggested, which goes against best practice a huge issue?
This might not be the solution to your question but did you compare all your cassandra.yaml files?
They should all be the same, apart from things like listen_address.
Is it possible you might have had a whitespace or typo in the cluster name also?
I just thought I'd mention it as something good to check.
I am planning to create an application that will use just 1 cassandra table. Replication factor will be probably 2 or 3. I might start initially with 2 cassandra server and then keep adding servers as needed. But I am not sure if I need to pre-plan anything so that the table is distributed uniformly when I add more servers. Are there any best practices or things I need to be aware? I read about tokens , http://www.datastax.com/docs/1.1/initialize/token_generation , but I am not sure what I need to do.
I suppose the keys have to be distrubuted uniformly in the cluster, so:
how will that happen i.e. when I add the 2nd server and say the 1st one already has 1 million keys
do I need to pre-plan the keyspace or tables?
I can suggest two things.
First, when designing your schema, pick a good partition key (1st column in the primary key). You need to ensure a couple of things:
There are enough values such that you can distribute it to an arbitrary amount of nodes. For example, sex would be a bad partition key, because you only have two values and therefore can only distribute it to two nodes.
The distribution across different partition key values is more or less uniform. For example, country might not be best, because you will most likely have most of your rows in just a few unique countries.
Secondly, to ease deployment of new nodes later consider setting up your cluster to use virtual nodes (vnodes). If you do that you will be able to skip a few steps when expanding your cluster.
To configure virtual nodes, set num_tokens in cassandra.yaml to more than 1. This will decide how many virtual nodes your node will have. A recommended value is 256.
Later, when you add new nodes, you need to make sure add_bootstrap is true in cassandra.yaml for your new nodes. Then you configure network parameters as usual to match your cluster, and finally start your node. It should automatically bootstrap and start streaming appropriate data. After everything is settled down, you can run cleanup (nodetool clean) on your other nodes to make sure they purge redundant data that they're no longer responsible for.
For more detailed documentation, please see http://www.datastax.com/documentation/cassandra/2.0/cassandra/operations/ops_add_node_to_cluster_t.html
To best explain my goal, I will simplify the problem to the basics of my requirements. Please let me know if more details are needed for clarity.
Suppose I have 10 unique numbers (0-9) that are available for assignment. Which numbers are reserved or free is contained by the database. The goal of a running Front End web service is to successfully request a number for assignment. Once the number has been assigned to the specific node, it is reserved and no other node can have it assigned.
Keep in mind that this is to be a distributed system with no single point of failure.
The caveat that is giving me trouble is Cassandra's notion of eventual consistency. Note that I could tune Cassandra to be fully consistent at the cost of higher latency. If that is my best (and possibly only) option, I can do it, but I'd like to preserve the concept of consistency tuning.
My thoughts on the strategy is to do the following on the node:
1) Query Cassandra to get a list of free numbers.
2) Randomly select one of the free numbers.
3) Perform a Put to Cassandra saying that this node has reserved that number.
4) Continuously query Cassandra to see which node successfully reserved the number. (Continuously request, as the read may not immediately reflect the assignment.)
5) If the returned node name is the name in which this node's reservation was filed, then the reservation was a success.
6) If the returned node name is a different node name, it means that another node requested the number at around the same time as this one, and was given the assignment. This node must go back to step 1 and try again.
I have an odd feeling that specific circumstances will cause errors (double assignment, etc.) if I use the above strategy.
Can anyone else comment on my proposed strategy, and possibly offer their own? Thanks.
You may want to check out Apache ZooKeeper. It seems like it's exactly what you need. You can use ZooKeeper to get new assignments, and store the existing assignments in Cassandra.
Depending on your application, do you really need to choose one from a fixed set of numbers? If all you need is to choose a unique number that no other node has chosen, you could do it by generating a UUID (Universally Unique IDentifier) for each node, e.g. with http://docs.python.org/library/uuid.html.