I have a running AML endpoint deployed on AKS cluster, looking like http://[IP address]:80/api/v1/service/modelscoring36/score.
This is the POST endpoint to respond any incoming model scoring request. Is there any way to check if the relevant AML service or attached AKS cluster is healthy?
In Asp.Net we is used to some dummy GET method constructed in one controller for this health check purpose.
Does AML service provide such health check approach? or any other suggestion?
Thank you.
To get the health check of the end points
install Azure ML SDK using Command Prompt
pip install azureml-core
Get complete workspace details like subscription ID and other details. The requirements are mentioned below:
subscription_id = '<your_subscription_id>'
resource_group = '<your_resource_group>'
workspace_name = '<your_workspace_name>'
ws = Workspace(subscription_id, resource_group, workspace_name)
Using intermediate _get() method, we can get the health check details like below
for service in Webservice.list(workspace= ws):
service_properties = Webservice._get(workspace= ws, name = service.name)
print('Service Name: {} - {}'.format(service_properties['name'], service_properties['state']))
The results look like below:
Service Name: Model Name - Failed
Service Name: Model Name - Healthy
Service Name: Model name - Healthy
Related
I have enabled health check with path "/" for App Service slot.
This path in browser and using curl return 200 status code.
But App Service health check status displays 0.00% (Healthy 0 / Degraded 1)
How is this possible?
Just to highlight that if your application depends on a database and a messaging system, the Health check endpoint should connect to those components. If the application can't connect to a critical component, then the path should return a 500-level response code to indicate the app is unhealthy. See: Monitor App Service instances using Health check
Also: Please confirm to make sure that your health check path was added successfully:
From Azure Portal:
To enable Health check, browse to the Azure portal and select your App
Service app.
Under Monitoring, select Health check.
Select Enable and provide a valid URL path on your application, such as
/health or /api/health.
Select Save.
From Resource url:
You may use the following URL: https://resources.azure.com/ to check that you added the path correctly in your web application.
Follow the steps here:
Go to https://resources.azure.com/
Expand your subscription > Go to Resource Groups > Select the Resource
Group where the app service is hosted.
The select providers > Then select Microsoft Web > Open sites And open your
app service > Then open config > And lastly web.
You will need to scroll to the bottom to find the variable called
‘healthCheckPath’ and you will see the value there for the path you set in
the portal.
Also refer to this detailed document on Health check here might be helpful: https://learn.microsoft.com/en-us/azure/app-service/monitor-instances-health-check?tabs=dotnet#frequently-asked-questions
Addition:
Using curl to the heatlhCheckPath in the case of multiple running instances could be misleading as you might not receive status of each running instance of the app and might get result for random instance the load balancer directs to you.
Suggest you to fetch the values of Health Check Status metric by using the Azure CLI.
az monitor metrics list --resource myresource --resource-group myresourcegroup --resource-type "Microsoft.Web/sites" --metric "HealthCheckStatus" --interval 5m
Also please Note that the --interval property is important as health checks do not support the default 1m interval used by az monitor metrics list
I understand there is a difference between a service account and a service agent for different services such as composer.
How do you enable a service agent via terraform?
What I'm trying to do is this :
# TODO : Maybe enable this service agent somehow via gcloud? It got enabled when trying to manually create the composer env from the console
# Step 4 (Src2) - Host project GKE service account: service-<some-numeric-id>#container-engine-robot.iam.gserviceaccount.com
# Need 'container.serviceAgent' in the host project
resource "google_project_iam_member" "dev-omni-orch-gke-project-lvl-roles-t2" {
provider = google.as_super_admin
for_each = toset([
"roles/container.serviceAgent",
])
role = each.value
member = "serviceAccount:service-<some-numeric-id>#container-engine-robot.iam.gserviceaccount.com"
# member = "serviceAccount:service-${google_project.main-shared-vpc-host.number}#container-engine-robot.iam.gserviceaccount.com"
# project = google_project.dev-main-code-base.project_id
project = google_project.main-shared-vpc-host.project_id
}
I get
Request `Create IAM Members roles/container.serviceAgent serviceAccount:service-<some-numeric-id>#container-engine-robot.iam.gserviceaccount.com for project "<shared-vpc-host-project-id>"` returned error: Batch request and retried single request "Create IAM Members roles/container.serviceAgent serviceAccount:service-<some-numeric-id>#container-engine-robot.iam.gserviceaccount.com for project \"<shared-vpc-host-project-id>\"" both failed. Final error: Error applying IAM policy for project "<shared-vpc-host-project-id>": Error setting IAM policy for project "<shared-vpc-host-project-id>": googleapi: Error 400: Service account service-<some-numeric-id>#container-engine-robot.iam.gserviceaccount.com does not exist., badRequest
But when I try to do it via the console manually, there is a prompt that asks me if I want to enable this service agent, which I do, but I want to be able to do this on terraform.
The said prompt :
The service-[PROJECT_ID]#cloudcomposer-accounts.iam.gserviceaccount.com service agent will only exist after the Cloud Composer API has been enabled.
This can be done in Terraform using the google_project_service resource, for example:
resource "google_project_service" "project" {
project = "your-project-id"
service = "composer.googleapis.com"
}
Once the API has been enabled, the service agent should exist and you should be able to grant it the required permissions.
When we deploy a model using Azure Machine Learning Service, we need a scoring script with at least two functions init which is run once when a model is first deployed and a run function which is run against any data we want to be scored.
The endpoint for this is: http://<IP-Address>/api/v1/service/<Webservice-Name>/score
Is there a way to include a health-check function in the deployed web service? Maybe adding another function healthcheck() and a corresponding endpoint http://<IP-Address>/api/v1/service/<Webservice-Name>/health which when pinged sends us a custom JSON containing information about the health of our service. This may even include information about the Azure Health Check too:
Apparently, the Azure Machine Learning SDK has to show the properties of the class Webservice. I haven't found any mention to this issue online.
To reproduce fully the experiment, you have to install AML SDK:
pip install azureml-core
And get the workspace:
# get the workspace
subscription_id = '<your_subscription_id>'
resource_group = '<your_resource_group>'
workspace_name = '<your_workspace_name>'
ws = Workspace(subscription_id, resource_group, workspace_name)
Using the Webservice Class, it should work. I have 3 services, but I can't see the status:
# this should work, but it isn't returning the state
for service in Webservice.list(workspace= ws):
print('Service Name: {} - {}'.format(service.name, service.state))
>> Service Name: akstest - None
>> Service Name: regression-model-1 - None
>> Service Name: xgboost-classification-1 - None
Using an intermediate step using the _get() method, it works:
# this works
for service in Webservice.list(workspace= ws):
service_properties = Webservice._get(workspace= ws, name = service.name)
print('Service Name: {} - {}'.format(service_properties['name'], service_properties['state']))
>> Service Name: akstest - Failed
>> Service Name: regression-model-1 - Healthy
>> Service Name: xgboost-classification-1 - Healthy
I am trying to provision some IBM Watson Assistant services using the terraform provider for IBM.
I am currently using the ibm_service_instance along with ibm_service_key resources from the IBM provider.
The relevant piece of terraform code can be found below.
# create service
resource "ibm_service_instance" "wa_test_service_instance_name" {
count = var.wa_template_service_counter
name = "Test-${var.wa_test_service_instance_name}-${var.app_name}-${count.index + 1}"
space_guid = var.space_guid
service = var.service_offering
plan = var.plan
}
# create credentials
resource "ibm_service_key" "wa_test_service_key" {
count = var.wa_template_service_counter
name = var.service_key_name
service_instance_guid = ibm_service_instance.wa_test_service_instance_name[count.index].id
depends_on = [ibm_service_instance.wa_test_service_instance_name]
}
The service instance is created successfully along with the credentials and the CF alias. The problem is that for the CF alias, the created credentials are not having iam_role_crn manager which is the required setup in my configuration.
If I manually add the credentials from IBM cloud dashboard for the CF alias, they are created with the iam_role_crn Manager. Also, the resource instance for which is this alias has in its credentials iam_role_crn = Manager.
I could not find a way of specifying such a parameter when the ibm_service_key or ibm_service_instance gets created.
https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/service_key
Is there a way to create the credentials for the alias of the service with this parameter iam_role_crn setup as Manager or is there a work-around to achieve this?
I'm using Azure Monitoring Service API and need to pass DEPLOYTMENT NAME as a parameter to BuildVirtualMachineResourceId API method.
At the moment its not clear to me where/how to locate this piece of information so it can be passed to the method. Both cloud service name and vm name are easily available.
String vmResourceId = ResourceIdBuilder.BuildVirtualMachineResourceId(
CLOUD_SERVICE_NAME, DEPLOYMENT_NAME, VM_NAME);
Try Deployment ID... should be available on your Azure portal under the "Dashboard" screen
Use Get-AzureDeployment Cmdlet that gives you the deployment name. More details: http://msdn.microsoft.com/en-us/library/azure/dn495146.aspx