I was creating Databricks job with API. Just want to know if instance pool Id and driver instance pool Id can be same ?
Yes, it's ok to use the same Instance Pool for both worker & driver nodes. And really, if you don't specify a separate pool for driver, then the worker's instance pool will be used if it's configured. This is described in the documentation for clusters, in the description of instance_pool_id setting:
The optional ID of the instance pool to use for cluster nodes. If driver_instance_pool_id is present, instance_pool_id is used for worker nodes only. Otherwise, it is used for both the driver and worker nodes.
Related
I have a case where I want to perform an inplace upgrade of the AKS cluster node pools vmSize, deletion of the full cluster is not possible.
One alternative that I have looked into is to perform `az aks nodepool delete' and then recreate it with a new vmSize.
Question here is: What is really happening under the hood, drain all and delete?
Should we first drain all the nodes in sequence, and then run the command to achieve 0 downtime? We are running multiple node pools
Anyother suggestion?
why you dont add a new node pool and migrate your workload to the new nodepool, then delete the old node pool.
You could also import this new node pool with Terraform if you use it.
Or is it the system node pool you are talking about?
Foreword
When you create a Kubernetes cluster on AKS you specify the type of VMs you want to use for your nodes (--node-vm-size). I read that you can't change this after you create the Kubernetes cluster, which would mean that you'd be scaling vertically instead of horizontally whenever you add resources.
However, you can create different node pools in an AKS cluster that use different types of VMs for your nodes. So, I thought, if you want to "change" the type of VM that you chose initially, maybe add a new node pool and remove the old one ("nodepool1")?
I tried that through the following steps:
Create a node pool named "stda1v2" with a VM type of "Standard_A1_v2"
Delete "nodepool1" (az aks nodepool delete --cluster-name ... -g ... -n nodepool1
Unfortunately I was met with Primary agentpool cannot be deleted.
Question
What is the purpose of the "primary agentpool" which cannot be deleted, and does it matter (a lot) what type of VM I choose when I create the AKS cluster (in a real world scenario)?
Can I create other node pools and let the primary one live its life? Will it cause trouble in the future if I have node pools that use larger VMs for its nodes but the primary one still using "Standard_A1_v2" for example?
Primary node pool is the first nodepool in the cluster and you cannot delete it, because its currently not supported. You can create and delete additional node pools and just let primary be as it is. It will not create any trouble.
For the primary node pool I suggest picking a VM size that makes more sense in a long run (since you cannot change it). B-series would be a good fit, since they are cheap and CPU\mem ratio is good for average workloads.
ps. You can always scale primary node pool to 0 nodes, cordon the node and shut it down. You will have to repeat this post upgrade, but otherwise it will work
It looks like this functionality was introduced around the time of your question, allowing you to add new system nodepools and delete old ones, including the initial nodepool. After encountering the same error message myself while trying to tidy up a cluster, I discovered I had to set another nodepool to a system type in order to delete the first.
There's more info about it here, but in short, Azure nodepools are split into two types ('modes' as they call it): System and User. When creating a single pool to begin with, it will be of a system type (favouring system pod scheduling -- so it might be good to have a dedicated pool of a node or two for system use, then a second user nodepool for the actual app pods).
So if you wish to delete your only system pool, you need to first create another nodepool with the --mode switch set to 'system' (with your preferred VM size etc.), then you'll be able to delete the first (and nodepool modes can't be changed after the fact, only on creation).
I have created 6 disks of 256GB each on 2 windows server 2016 VMs. I need to implement Active-Active SQL failover cluster on these 2 VMs using S2D.
I am getting error while creating storage pool for 3 disks , below is the error
Cluster resource 'Cluster Pool 1' of type 'Storage Pool' in clustered role xxxxxx failed. The error code was '0x16' ('The device does not recognize the command.').
Based on the failure policies for the resource and role, the cluster service may try to bring the resource online on this node or move the group to another node of the cluster and then restart it. Check the resource and group state using Failover Cluster Manager or the Get-ClusterResource Windows PowerShell cmdlet
[Problem start date and time]
S2D is new in Windows Server 2016. You can check what to have before you process with building your failover cluster. It's strongly to validate cluster first and then enable S2D following Configure the Windows Failover Cluster with S2D.
this error is appearing because i tried to create storage pool again..basically, enable-s2dcluster has created the pool already for me..i didnt notice it and was trying to create the pool using Failove cluster manager
In order to achieve an active-active solution, you should configure a host/VM per location. For Azure, S2D does not work between two locations. It requires RDMA support for the performance that cannot be configured in Azure. So, to get HA for SQL FCI to check StarWind vSAN free that can be configured between sites replicating/mirroring storage. https://www.starwindsoftware.com/resource-library/installing-and-configuring-a-sql-server-failover-clustered-instance-on-microsoft-azure-virtual-machines
I see the following configuration: Storage Spaces provides disk redundancy configuring mirror or parity for each VM and StarWind distributes HA storage on top of underlying Storage Spaces.
I am creating an EMR cluster using the aws_emr_cluster resource in terraform.
I need to get access to the instance ID of the underlying EC2 hardware, specifically the MASTER node.
It does not appear in the attributes and neither when I perform an terraform show
The data definitely exists and is available in AWS.
Does anyone know how I can get at this value and how to do it it using terraform?
You won't be able to access the nodes (EC2 Instances) in an EMR Cluster through terraform. It is the same case for AutoScaling Groups too.
If terraform includes EMR or ASG nodes, state file will be changed everytime a change happens in EMR/ASG. So, storing the instance information won't be ideal for terraform.
Instead, you can use AWS SDK/CLI/boto3 to see them.
Thanks.
In the Hazelcast docs it is stated about Cluster Groups:
You can create cluster groups. To do this, use the group configuration element.
By specifying a group name and group password, you can separate your clusters in a simple way. Example groupings can be by development, production, test, app, etc. <...> Each Hazelcast instance can only participate in one group. Each Hazelcast instance only joins to its own group and does not interact with other groups.
<...>
The cluster members (nodes) and clients having the same group configuration (i.e., the same group name and password) forms a private cluster.
Each cluster will have its own group and it will not interfere with other clusters.
But there are no details about data partitioning.
If I have 5 nodes and 2 cluster groups:
node1, node2 and node3 are members of GroupA
node4 and node5 are members of GroupB
does it means that no data from GroupA will be stored at nodes4 and node5?
Yeah that's what it means. Those groups are independent clusters and have nothing in common (except maybe the network ;-)).
If you look for data partitioning, Hazelcast distributes information based on keys but you can have some kind of influence by utilizing data affinity (http://docs.hazelcast.org/docs/3.8/manual/html-single/index.html#data-affinity).
If you're looking for backup distribution you might be interested in partition groups (http://docs.hazelcast.org/docs/3.8/manual/html-single/index.html#partition-group-configuration).