I am trying to register a model inside one of my azure ml experiments. I am able to register it via Model.register but not via run_context.register_model
This are the two code sentences I use. The commented one is the one that fails
learn.path = Path('./outputs').absolute()
Model.register(run_context.experiment.workspace, "outputs/login_classification.pkl","login_classification", tags=metrics)
run_context.register_model("login_classification", "outputs/login_classification.pkl", tags=metrics)
I received the next error:
Message: Could not locate the provided model_path outputs/login_classification.pkl
But model is stored in this path:
Before implementing run_context.register_model() implement run_context = Run.get_context()
I was able to fix the problem by explicitly uploading the model into the run history record before trying for registering the model.
run.upload_file("output/model.pickle", "output/model.pickle")
Check the documentation for Message: Could not locate the provided model_path outputs/login_classification.pkl
To check about Run Class
Related
I am trying to train a model in databricks with mlflow and roberta transformers. I am able to register the model but when I call it for testing, I got the following error:
OSError: We couldn't connect to 'https://huggingface.co/' to load this model and it looks like dbfs:/databricks/mlflow-tracking/3638851642935524/cd4eae6034684211933b97b178e5f062/artifacts/checkpoint-36132/artifacts/checkpoint-36132 is not the path to a directory containing a config.json file.
However, when I check for the model saved, I can see that the config.json and other files are saved in the mentioned artifact but with the following error:
Couldn't load model information due to an error.
For more details, I have followed the following link for creating an mlflow with transformers on databricks:
https://gitlab.com/juliensimon/huggingface-demos/-/tree/main/mlflow
I'm relatively new to Azure ML and trying to run a model via PythonScriptStep
I can publish pipelines and run the model. However, once it has run once I can't re-submit the step as it states "This run reused the output from a previous run".
My code declares allow_reuse to be False, but this doesn't seem to make a difference and I can simply not resubmit the step even though the underlying data is changing.
train_step = PythonScriptStep(
name='model_train',
script_name="model_train.py",
compute_target=aml_compute,
runconfig=pipeline_run_config,
source_directory=train_source_dir,
allow_reuse=False)
Many thanks for your help
I am trying to deploy a machine learning model through an ACI (Azure Container Instances) service. I am working in Python and I followed the following code (from the official documentation : https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-and-where?tabs=azcli) :
The entry script file is the following (score.py):
import os
import dill
import joblib
def init():
global model
# Get the path where the deployed model can be found
model_path = os.getenv('AZUREML_MODEL_DIR')
# Load existing model
model = joblib.load('model.pkl')
# Handle request to the service
def run(data):
try:
# Pick out the text property of the JSON request
# Expected JSON details {"text": "some text to evaluate"}
data = json.loads(data)
prediction = model.predict(data['text'])
return prediction
except Exception as e:
error = str(e)
return error
And the model deployment workflow is as:
from azureml.core import Workspace
# Connect to workspace
ws = Workspace(subscription_id="my-subscription-id",
resource_group="my-ressource-group-name",
workspace_name="my-workspace-name")
from azureml.core.model import Model
model = Model.register(workspace = ws,
model_path= 'model.pkl',
model_name = 'my-model',
description = 'my-description')
from azureml.core.environment import Environment
# Name environment and call requirements file
# requirements: numpy, tensorflow
myenv = Environment.from_pip_requirements(name = 'myenv', file_path = 'requirements.txt')
from azureml.core.model import InferenceConfig
# Create inference configuration
inference_config = InferenceConfig(environment=myenv, entry_script='score.py')
from azureml.core.webservice import AciWebservice #AksWebservice
# Set the virtual machine capabilities
deployment_config = AciWebservice.deploy_configuration(cpu_cores = 0.5, memory_gb = 3)
from azureml.core.model import Model
# Deploy ML model (Azure Container Instances)
service = Model.deploy(workspace=ws,
name='my-service-name',
models=[model],
inference_config=inference_config,
deployment_config=deployment_config)
service.wait_for_deployment(show_output = True)
I succeded once with the previous code. I noticed that during the deployment the Model.deploy created a container registry with a specific name (6e07ce2cc4ac4838b42d35cda8d38616).
The problem:
The API was working well and I wanted to deploy an other model from scratch. I deleted the API service and model from Azure ML Studio and the container registry from Azure ressources.
Unfortunately I am not able to deploy again anything.
Everything goes fine until the last step (the Model.deploy step), I have the following error message :
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 46243f9b-3833-4650-8d47-3ac54a39dc5e
More information can be found here: https://machinelearnin2812599115.blob.core.windows.net/azureml/ImageLogs/46245f8b-3833-4659-8d47-3ac54a39dc5e/build.log?sv=2019-07-07&sr=b&sig=45kgNS4sbSZrQH%2Fp29Rhxzb7qC5Nf1hJ%2BLbRDpXJolk%3D&st=2021-10-25T17%3A20%3A49Z&se=2021-10-27T01%3A24%3A49Z&sp=r
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 404,
"message": "No definition exists for Environment with Name: myenv Version: Autosave_2021-10-25T17:24:43Z_b1d066bf Reason: Container > registry 6e07ce2cc4ac4838b42d35cda8d38616.azurecr.io not found. If private link is enabled in workspace, please verify ACR is part of private > link and retry..",
"details": []
}
I do not understand why the first time a new container registry was well created, but now it seems that it is sought (the message is saying that container registry identified by name 6e07ce2cc4ac4838b42d35cda8d38616 is missing). I never found where I can force the creation of a new container registry ressource in Python, neither specify a name for it in AciWebservice.deploy_configuration or Model.deploy.
Does anyone could help me moving on with this? The best solution would be I think to delete totally this 6e07ce2cc4ac4838b42d35cda8d38616 container registry but I can't find where the reference is set so Model.deploy always fall to find it.
An other solution would be to force Model.deploy to generate a new container registry, but I could find how to make that.
It's been 2 days that I am on this and I really need your help !
PS : I am not at all a DEVOPS/MLOPS guy, I make data science and good models, but infrastructure and deployment is not really my thing so please be gentle on this part ! :-)
What I tried
Creating the container registry with same name
I tried to create the container registry by hand, but this time, this is the container that cannot be created. The Python output of the Model.deploy is the following :
Tips: You can try get_logs(): https://aka.ms/debugimage#dockerlog or local deployment: https://aka.ms/debugimage#debug-locally to debug if deployment takes longer than 10 minutes.
Running
2021-10-25 19:25:10+02:00 Creating Container Registry if not exists.
2021-10-25 19:25:10+02:00 Registering the environment.
2021-10-25 19:25:13+02:00 Building image..
2021-10-25 19:30:45+02:00 Generating deployment configuration.
2021-10-25 19:30:46+02:00 Submitting deployment to compute.
Failed
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 93780de6-7662-40d8-ab9e-4e1556ef880f
Current sub-operation type not known, more logs unavailable.
Error:
{
"code": "InaccessibleImage",
"statusCode": 400,
"message": "ACI Service request failed. Reason: The image '6e07ce2cc4ac4838b42d35cda8d38616.azurecr.io/azureml/azureml_684133370d8916c87f6230d213976ca5' in container group 'my-service-name-LM4HbqzEBEi0LTXNqNOGFQ' is not accessible. Please check the image and registry credential.. Refer to https://learn.microsoft.com/azure/container-registry/container-registry-authentication#admin-account and make sure Admin user is enabled for your container registry."
}
Setting admin user enabled
I tried to follow the recommandation of the last message saying to set Admin user enabled for the container registry. All what I saw in Azure interface is that a username and password appeared when enabling on user admin.
Unfortunately the same error message appears again if I try to relaunche my code and I am stucked here...
Changing name of the environment and model
This does not produces any change. Same errors.
As you tried with first attempt it was worked. After deleting the API service and model from Azure ML Studio and the container registry from Azure resources you are not able to redeploy again.
My assumption is your first attempt you are already register the Model Environment variable. So when you try to reregister by using the same model name while deploying it will gives you the error.
Thanks # anders swanson Your solution worked for me.
If you have already registered your env, myenv, and none of the details of the your environment have changed, there is no need re-register it with myenv.register(). You can simply get the already register env using Environment.get() like so:
myenv = Environment.get(ws, name='myenv', version=11)
My Suggestion is to name your environment as new value.
"model_scoring_env". Register it once, then pass it to the InferenceConfig.
Refer here
With an sagemaker.estimator.Estimator, I want to re-deploy a model after retraining (calling fit with new data).
When I call this
estimator.deploy(initial_instance_count=1, instance_type='ml.m5.xlarge')
I get an error
botocore.exceptions.ClientError: An error occurred (ValidationException)
when calling the CreateEndpoint operation:
Cannot create already existing endpoint "arn:aws:sagemaker:eu-east-
1:1776401913911:endpoint/zyx".
Apparently I want to use functionality like UpdateEndpoint. How do I access that functionality from this API?
Yes, under the hood the model.deploy creates a model, an endpoint configuration and an endpoint. When you call again the method from an already-deployed, trained estimator it will create an error because a similarly-configured endpoint is already deployed. What I encourage you to try:
use the update_endpoint=True parameter. From the SageMaker SDK doc:
"Additionally, it is possible to deploy a different endpoint configuration, which links to your model, to an already existing
SageMaker endpoint. This can be done by specifying the existing
endpoint name for the endpoint_name parameter along with the
update_endpoint parameter as True within your deploy() call."
Alternatively, if you want to create a separate model you can specify a new model_name in your deploy
update_endpoint has been deprecated since AFAIK . To re-create the UpdateEndpoint functionality from this API itself and deploy a newly fit training job to an existing endpoint , we could do something like this (this example uses the sagemaker sklearn API):
from sagemaker.sklearn.estimator import SKLearn
sklearn_estimator = SKLearn(
entry_point=model.py,
instance_type=<instance_type>,
framework_version=<framework_version>,
role=<role>,
dependencies=[
<comma seperated names of files>
],
hyperparameters={
'key_1':value,
'key_2':value,
...
}
)
sklearn_estimator.fit()
sm_client = boto3.client('sagemaker')
# Create the model
sklearn_model = sklearn_estimator.create_model()
# Define an endpoint config and an endpoint
endpoint_config_name = 'endpoint-' + datetime.utcnow().strftime("%Y%m%d%H%m%s")
current_endpoint = endpoint_config_name
# From the Model : create the endpoint config and the endpoint
sklearn_model.deploy(
initial_instance_count=<count>,
instance_type=<instance_type>,
endpoint_name=current_endpoint
)
# Update the existing endpoint if it exists or create a new one
try:
sm_client.update_endpoint(
EndpointName=DESIRED_ENDPOINT_NAME, # The Prod/Existing Endpoint Name
EndpointConfigName=endpoint_config_name
)
except Exception as e:
try:
sm_client.create_endpoint(
EndpointName=DESIRED_ENDPOINT_NAME, # The Prod Endpoint name
EndpointConfigName=endpoint_config_name
)
except Exception as e:
logger.info(e)
sm_client.delete_endpoint(EndpointName=current_endpoint)
I was trying to register an ONNX model to Azure Machine Learning service workspace in two different ways, but I am getting errors I couldn't solve.
First method: Via Jupyter Notebook and python Script
model = Model.register(model_path = MODEL_FILENAME,
model_name = "MyONNXmodel",
tags = {"onnx":"V0"},
description = "test",
workspace = ws)
The error is : HttpOperationError: Operation returned an invalid status code 'Service invocation failed!Request: GET https://cert-westeurope.experiments.azureml.net/rp/workspaces'
Second method: Via Azure Portal
Anyone can help please?
error 413 means the payload is too large. Using Azure portal, you can only upload a model upto 25MB in size. Please use python SDK to upload models larger than 25MB.