sorry to ask my questions on stackoverflow instead of ask.openstack. I really need answers and ask.openstack is useless. my questions are:
what is openstack heat?
how to setup dynamic resource allocation for an instance using dashboard?
is there any way to automatically live migrate instances on openstack when dynamic resource allocation is active and openstack node runs out of resources?
Answer-1:
OpenStack Heat:
Heat is a service to orchestrate composite cloud applications using a
declarative template format through an OpenStack-native REST API.
Heat is a project under OpenStack tent which is used for orchestration of resources using templates in yaml format
Heat template is defined with sections like definition, parameters, resources, outputs
Based on the topology and requirement of resources like VM, port, network, subnet ... the resources are defined in template. All the supported resource types are available at https://docs.openstack.org/heat/latest/template_guide/openstack.html
Official documentation of heat: https://docs.openstack.org/heat/latest/
Answer-2:
Dynamic resource allocation for instance using dashboard
Dynamic resource allocation is allocating/changing the resources after instance is up and running, for example attach the port to instance, attach volume to instance, resize the instance... . Yes these can be done from dashboard/horizon.
Answer-3:
Live-migration of instance:
Yes, live migration can be done and step by step procedure available at https://docs.openstack.org/nova/pike/admin/live-migration-usage.html
Related
Am using a combination of these tools
Terraform - To deploy the Application specific AWS resources I need
(For instance a secret)
Skaffold - To help with the inner
development loop, surrounding the deployment of K8s resources to
local and remote cluster
Kustomize - To help with templating of
different configurations for different environment
My github action steps are as follows
Terraform to create the AWS resources. At this point it creates a AWS
secrets arn.
Skaffold to deploy the k8s manifests. Skaffold in-turn delegates K8s manifest generation to Kustomize. Within the Kustomize overlay files i need to be able to access the Secrets arn that was created earlier, this arn needs to be injected into the container that is being deployed. How do I achieve this?
Rephrasing the question: How do I pass resources that were created by terraform to be consumed by something like Kustomize (Which is used by skaffold)
(p.s, I really like the choice of my tools thus far as each one excels at one thing. I realize that terraform can possibly do all of it, but that is a choice that I dont want to make unless there are no easier options)
Here is what I have learnt:
I don't think there are any industry standards in terms of how to share this data between the tools across different steps within github actions. That being said here are some of the options
Have the Terraform store the secrets arn in a parameter store. Retrieve the arn from the parameter store in later steps. This means that the steps have to share a static key
Have Terraform update the kustomize files directly (or use kustomize_overlays as datasource)
There could be other similar approaches, but none of these tools have a native way of passing/sharing data
As described in https://learn.microsoft.com/en-us/azure/templates/microsoft.machinelearningservices/workspaces/computes?tabs=bicep there are the properties location, sku, tags and identity.
For me it is not clear whether these properties relate to the parent workspace or to the compute resource (e.g. as the there is also computeLocation or sku as far as I can see should have the same value as the workspace)...
It would be great when someone can clarify to which resource these properties and related values belong to (workspace vs. compute resource).
EDIT:
Also: which properties are actually required for attach versus create? E.g. do I need identity or computeLocation for attach, and if yes what is the purpose of it as the compute resource is being created in another context?
I also figured out that location as well as disableLocalAuth are required for the attach operation - why when the resource is being deployed in another context and only attached?
And why do I get unsupported compute type when checking for the compute resources via Azure CLI for the attached AKS?
{
"description": "Default AKS Cluster",
"id": "/subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.MachineLearningServices/workspaces/xxx/computes/DefaultAKS",
"location": "westeurope",
"name": "DefaultAKS",
"provisioning_state": "Succeeded",
"resourceGroup": "xxx",
"resource_id": "/subscriptions/xxx/resourcegroups/xxx/providers/Microsoft.ContainerService/managedClusters/xxx",
"type": "*** unsupported compute type ***"
}
EDIT-2:
So based on the response from #SairamTadepalli-MT all the properties actually relate to the compute resource - what makes sense. Still, I don't understand the purpose of a few of these properties. For instance why is there a "location" and a "computeLocation" or what is the meaning of "sku" (e.g. I tried "AmlCompute" and provided the value "Basic" - but "Basic" is the "sku" of the workspace and for "AmlCompute" the size is actually defined by the "vmSize" or?...).
What brings me to the next point: the current documentation currently lacks a detailed description in which scenarios which properties can have which values respectively need to be provided (beside "properties").
This is also true for attach (i.e. providing a "resourceId") vs. create (i.e. providing "properties"): which properties are actually required for attach? For what I figured out it requires "location" and "disableLocalAuth" - why do I need these properties as I would assume "name" and "resourceId" (and maybe "computeType") should be sufficient to attach a compute resource? What is the purpose of properties like "sku", "tags" or "identity" when I attach an existing compute resource?
Finally regarding "unsupported compute type": not sure if your response really helps me. The AKS is successfully attached, so I don't understand why I get "unsupported compute type". This should be fixed.
Before identifying the properties and their uses, we need to understand a small difference between workspaces and computer resources
Workspaces: The workspace is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. The workspace keeps a history of all training runs, including logs, metrics, output, and a snapshot of your scripts. You use this information to determine which training run produces the best model.
https://learn.microsoft.com/en-us/azure/machine-learning/concept-workspace
on the other hand let's check about computer resources
Compute Resources: An Azure Machine Learning compute instance is a managed cloud-based workstation for data scientists. Compute instances make it easy to get started with Azure Machine Learning development as well as provide management and enterprise readiness capabilities for IT administrators.
https://learn.microsoft.com/en-us/azure/machine-learning/concept-compute-instance
Workspaces are used to work on machine learning resources and identify all the operations which are completed and in progress or even failed as a log.
Compute resources are the kind of administration operations which can handle the authority on the resource which was shared and created.
Moving on to the next part of the question. Regarding the properties.
Location: This is used to identify where the compute resources are created for ML operations. Comes under compute resources.
SKU: This is regarding the tier and price of each tier which we have chosen in our subscription. This is related to the computer resources.
Tags: Each resource created need to be recognized. Tags are compute resource identifiers.
Identity: Subscription details of the compute resource. This can be further need to be used with workspace which deploying ML application.
Location:: The location where the parent code is located. Where the output is stored will be defined
disableLocalAuth: Azure Automation provides Microsoft Azure Active Directory (Azure AD) authentication support for all Automation service public endpoints. This critical security enhancement removes certificate dependencies and gives organizations control to disable local authentication methods. This feature provides you with seamless integration when centralized control and management of identities and resource credentials through Azure AD is required.
unsupported compute type: Azure Kubernetes Service (AKS) clusters, you might occasionally come across problems. Refer the link mentioned below for better trouble shooting
https://learn.microsoft.com/en-us/azure/aks/troubleshooting
This would appear to be a fairly simple and basic scenario but I'm frankly at a loss on how to get around this using Terraform and would appreciate any suggestions.
The issue is this. In Azure, I have a number of resource groups, each containing a number of resources, including virtual networks, subnets, storage accounts, etc. What I would now like to do is add new resources to one or two of the resource groups. Typical example, I would like to provision a new virtual machine in each of the resource groups.
Now, so far all of the documentation and blogs I seem to come across only provide guidance on how to create resources whereby you also create a new resource group, vnet, subnet, from scratch. This is definitely not what I wish to do.
All I'm looking to do is get Terraform to add a single virtual machine to an existing resource group, going on to configure it to connect to existing networking resources such as a VNet, Subnet, etc. Any ideas?
I tested for ECS by destroying the launch configuration.
terraform destroy -target module.ecs.module.ec2_alb.aws_launch_configuration.launchcfg
I recreated the launch configuration and it worked:
terraform plan -target=module.ecs.module.ec2_alb.aws_launch_configuration
terraform apply -target=module.ecs.module.ec2_alb.aws_launch_configuration
Also, you can go read more on Terraform target here: https://learn.hashicorp.com/tutorials/terraform/resource-targeting
If you just want to be able to reference your existing resources in your TF script, you normally would use data sources in TF to fetch their information.
So for resource group, you would use data source azurerm_resource_group, for vnet there is azurerm_virtual_network and so forth.
These data sources would allow you to only reference and get details of existing resources, not to manage them in your TF script. Thus if you would like to actually manage these resources using TF (modify, delete, etc), you would have to import them first to TF.
I am trying to replicate an existing Azure resource group using arm templates from the automation script section but running into a warning message about 10 resources types cannot be exported yet. One being Azure data factory. After searching online I found that templates for some resource types are not yet available. While looking at the Deployment section in my resource group, I am able to see a deployment details download link for each resource.
Can those be used to recreate each resource into a new resource group? Or is my best bet downloading the template from the automation script section and manually adding the resources for which the template are not yet available?
Any help would be appreciated.
Some of the warnings I got:
ResourceTypeSchemaNotFound:
***The schema of resource type 'Microsoft.DataFactory/dataFactories' is not available. Resources of this type will not be exported to the
template. (Code: ResourceTypeSchemaNotFound, Target:
Microsoft.DataFactory/dataFactories
ExportTemplateProviderError: all related to sql server
***Could not get resources of the type 'Microsoft.Sql/servers/connectionPolicies'. Resources of this type
will not be exported. (Code: ExportTemplateProviderError, Target:
Microsoft.Sql/servers/connectionPolicies)
***Could not get resources of the type 'Microsoft.Sql/servers/backupLongTermRetentionVaults'
***Could not get resources of the type 'Microsoft.Sql/servers/extendedAuditingSettings'.
The Azure resource group ARM templates from the automation script section are not as comprehensive as they should probably be. For example, it is quite possible to deploy a Data Factory instance from ARM. It provides a reasonable starting point but you will find resources that don't deploy as well as a lot of superfluous cruft.
The approach I take is to start from a clean template in Visual Studio and then construct a project from a blank template or quick-start, then refer to the template documentation and build up from scratch, using your exported ARM template from your resource group to help work out how it should look. This results in a nice clean reusable template project.
Unfortunately I have not found an better way yet.
I want to monitor Perfomance metrics of a existing Service Fabric Cluster.
Here is the link of Performance metrics -
https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-diagnostics-event-generation-perf
I went through this Microsoft documentation -
https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-diagnostics-perf-wad
My problem is, The ARM template I downloaded during Service Fabric creation time is quite big and contains lot of params and I don't have the template-params file. I think it is possible to build the params file but it will be time consuming.
Is it possible to download template and template-params file of
existing service fabric cluster ?
If no, Is it possible to just update the "WadCfg" section to add new
performance counters ?
Your can export your entire resource group with all definitions and parameters, there you can find all parameters(as default parameters) for the resources deployed in the resource group. I've never done for SF cluster, but a quick look to an existing resource group I have I could see the cluster definition included.
This link explain how: https://learn.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-export-template
In Summary:
Find the resource group where your cluster is deployed
Open the resource group and navigate to 'Automation Scripts'
Click 'Download' on top bar
Open the ARM template with all definitions
Make the modifications and save
Publish the updates
1:
2:
You could also add it to a library and deploy from there, as guided in the link above.
From the docs: Not all resource types support the export template function. To resolve this issue, manually add the missing resources back into your template.
To be honest, I've never deployed this way other than test environments, so I am not sure if it is safe for production.