ModelAssetPathNotFoundInStorage error when mlflow.sklearn.autolog() used to train within an Azure ML YAML Pipeline - scikit-learn

Question

YAML appears correct, there are no validation issues, and the pipeline can be seen in the Azure ML Studio GUI
I'm assuming the error is thrown by mlflow.sklearn.autolog() when the fit() method is called
Full stacktrace not available, the exception shown below is the full exception raised in Azue ML Studio GUI
I've commented out the function save_outputs() and the same error is raised, leading to my assumption regarding MLFlow SDK attempting to autolog the model
I haven't included the code below for the predict job in the pipeline, this step doesn't get to execute as the seep job in the pipeline fails first
Exception raised in Azure ML GUI
UserErrorException:
Message: Model asset creation API failed with {'additional_properties': {'message': 'The request is invalid.', 'details': [{'code': 'ModelAssetPathNotFoundInStorage', 'message': 'No blobs found in storage at model asset path: azureml/HD_9b8798ab-c0cb-4c5d-8822-1411c01af249_0/model/'}], 'code': 'BadRequest', 'statusCode': 400}, 'error': <data_capability._restclient.model.models._models_py3.RootError object at 0x7f210407e310>, 'correlation': {'operation': '214b5eca2adce7f52fdba06fb5003437', 'request': '9515d04215048b81', 'RequestId': '9515d04215048b81'}, 'environment': '<REDACTED>', 'location': '<REDACTED>', 'time': datetime.datetime(2023, 1, 12, 23, 1, 27, 38782, tzinfo=<FixedOffset '+00:00'>), 'component_name': 'modelregistry'}
InnerException None
ErrorResponse
{
"error": {
"code": "UserError",
"message": "Model asset creation API failed with {'additional_properties': {'message': 'The request is invalid.', 'details': [{'code': 'ModelAssetPathNotFoundInStorage', 'message': 'No blobs found in storage at model asset path: azureml/HD_9b8798ab-c0cb-4c5d-8822-1411c01af249_0/model/'}], 'code': 'BadRequest', 'statusCode': 400}, 'error': <data_capability._restclient.model.models._models_py3.RootError object at 0x7f210407e310>, 'correlation': {'operation': '214b5eca2adce7f52fdba06fb5003437', 'request': '9515d04215048b81', 'RequestId': '9515d04215048b81'}, 'environment': '<REDACTED>', 'location': '<REDACTED>', 'time': datetime.datetime(2023, 1, 12, 23, 1, 27, 38782, tzinfo=<FixedOffset '+00:00'>), 'component_name': 'modelregistry'}"
}
} Marking the experiment as failed because initial child jobs have failed due to user error
CLI Command
$ az ml job create --subscription <REDACTED> --resource-group <REDACTED> --workspace-name <REDACTED> --file /home/azureuser/cloudfiles/code/Users/<REDACTED>/repos/<REDACTED>/src/assets/pipeline_tune.yml --stream
RunId: quirky_bone_tf250gdlfg
Web View: https://ml.azure.com/runs/<REDACTED>?wsid=/subscriptions/<REDACTED>/resourcegroups/<REDACTED>/workspaces/<REDACTED>
Streaming logs/azureml/executionlogs.txt
========================================
[2023-01-12 22:58:42Z] Submitting 1 runs, first five are: <REDACTED>
[2023-01-12 23:03:46Z] Execution of experiment failed, update experiment status and cancel running nodes.
Execution Summary
=================
RunId: <REDACTED>
Web View: https://ml.azure.com/runs/<REDACTED>?wsid=/subscriptions/<REDACTED>/resourcegroups/<REDACTED>/workspaces/<REDACTED>
Exception :
{
"error": {
"code": "UserError",
"message": "Pipeline has some failed steps. See child run or execution logs for more details.",
"message_format": "Pipeline has some failed steps. {0}",
"message_parameters": {},
"reference_code": "PipelineHasStepJobFailed",
"details": []
},
"environment": "<REDACTED>",
"location": "<REDACTED>",
"time": "2023-01-12T23:03:45.982134Z",
"component_name": ""
}
Folder structure
src/
assets/
component_train.yml
pipeline_tune.yml
train.py
src/assets/pipeline_tune.yml
# References
# ----------
# - How to create component pipelines
# - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-component-pipelines-cli
# - Pattern reference
# - https://github.com/Azure/azureml-examples/tree/main/cli/jobs/pipelines-with-components/pipeline_with_hyperparameter_sweep
# - Pipeline schema
# - https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-job-pipeline
# - Sweep Job schema (hyperparameter tuning)
# - https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-job-sweep
# - Core Azure ML YAML syntax
# - https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-core-syntax#binding-inputs-and-outputs-between-steps-in-a-pipeline-job
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
# -------------------------------------------------------------------
# Pipeline settings
# - Having inputs defined at the pipelione level, instead of the first
# job, allows for parameterisation of the pipeline via both CLI/SDK
experiment_name: <REDACTED>
description: Tune hyperparemeters for training a scikit-learn SVM on the Iris dataset.
settings:
default_compute: azureml:aml-compute-cpu
default_datastore: azureml:workspaceblobstore
inputs:
data:
type: uri_file
mode: ro_mount
path: wasbs://datasets#azuremlexamples.blob.core.windows.net/iris.csv
outputs:
predict:
type: uri_folder
mode: rw_mount
path: azureml://datastores/workspaceblobstore/paths/<REDACTED>
# -------------------------------------------------------------------
# Jobs
jobs:
# Tune job
tune:
type: sweep
inputs:
data: ${{parent.inputs.data}}
outputs:
model:
type: mlflow_model
test_data:
type: uri_folder
trial: ./component_train.yml
search_space:
c_value:
type: uniform
min_value: 0.5
max_value: 0.9
kernel:
type: choice
values:
- rbf
- linear
- poly
coef0:
type: uniform
min_value: 0.1
max_value: 1
sampling_algorithm: random
objective:
goal: minimize
primary_metric: training_f1_score
limits:
max_total_trials: 20
max_concurrent_trials: 10
timeout: 7200
# Score test data
predict:
type: command
inputs:
model: ${{parent.jobs.tune.outputs.model}}
test_data: ${{parent.jobs.tune.outputs.test_data}}
outputs:
predictions: ${{parent.outputs.predict}}
component: ./component_predict.yml
src/assets/component_train.yml
# References
# ----------
# - How to create component pipelines
# - https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-component-pipelines-cli
# - Command schema
# - https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-job-command
# - Core Azure ML YAML syntax
# - https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-core-syntax#binding-inputs-and-outputs-between-steps-in-a-pipeline-job
$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json
type: command
# -------------------------------------------------------------------
# Command settings
version: 1
description: Training a scikit-learn SVM on the Iris dataset
environment: azureml:<REDACTED>:1
inputs:
data:
type: uri_file
c_value:
type: number
default: 1.0
kernel:
type: string
default: rbf
coef0:
type: number
default: 0
outputs:
model:
type: mlflow_model
test_data:
type: uri_folder
# -------------------------------------------------------------------
# Job
code: ..
command: >-
python train.py
--data ${{inputs.data}}
--C ${{inputs.c_value}}
--kernel ${{inputs.kernel}}
--coef0 ${{inputs.coef0}}
--outputs_model ${{outputs.model}}
--outputs_test_data ${{outputs.test_data}}
src/train.py
"""
Notes
-----
- Imports in this file must match the imports in `score.py` to allow
pickle objects to be loaded correctly by `score.py`
References
----------
- Azure ML Environments and ScriptRunConfig for training
- i.e How to execute this script against Azure ML compute cluster
- https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-environments#use-environments-for-training
- Azure ML - Hyperparameter tuning
- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-tune-hyperparameters
- Azure ML - Hyperparameter tuning in Azure Machine Learning pipeline
- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-sweep-in-pipeline#how-to-do-hyperparameter-tuning-in-azure-machine-learning-pipeline
- Azure ML - Random, Grid, Bayesian sampling
- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-tune-hyperparameters#sampling-the-hyperparameter-space
- Azure ML - How to Train scikit-learn
- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-train-scikit-learn#prepare-the-training-script
- Azure ML - How to train Tesnorflow
- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-train-tensorflow
- Azure ML - How to train Keras
- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-train-keras
- Azure ML - How to train PyTorch
- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-train-pytorch
- Azure ML - Logging with MLFlow
- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-log-view-metrics?tabs=jobs#getting-started
- MLFlow - Autologging of frameworks
- https://mlflow.org/docs/latest/tracking.html#automatic-logging
"""
import argparse
from distutils.dir_util import copy_tree
from pathlib import Path
import mlflow.sklearn
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
def get_data(data):
"""Get data and return train/test splits"""
df = pd.read_csv(data)
X = df.iloc[:, :-1]
y = df.iloc[:, -1]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=0
)
# Return split data
return X_train, X_test, y_train, y_test
def get_hyperparameters(**kwargs):
"""Set hyperparameters here
References
----------
- Understand how to pass hyperparameters to a sklearn pipeline
- https://stackoverflow.com/questions/66388056/why-does-sklearn-pipeline-set-params-not-work
"""
hyperparameters = {
"estimator__C": kwargs.get("c_value"),
"estimator__kernel": kwargs.get("kernel"),
"estimator__coef0": kwargs.get("coef0"),
}
return hyperparameters
def save_outputs(model, model_dir, X_test, y_test, test_data_dir):
"""Save outputs of the training process"""
# Save model
local_dir = "model"
mlflow.sklearn.save_model(model, local_dir)
copy_tree(local_dir, model_dir)
# Save test data
X_test.to_csv(Path(test_data_dir) / "X_test.csv", index=False)
y_test.to_csv(Path(test_data_dir) / "y_test.csv", index=False)
def train_model(hyperparameters, X_train, y_train):
"""Train the model with your chosen framework here"""
# Model architecture
model = Pipeline(
steps=[
("scaler", StandardScaler()),
("estimator", SVC()),
]
)
# Set hyperparameters for training run
model.set_params(**hyperparameters)
# Train model
model.fit(X_train, y_train)
return model
def parse_args():
"""Parse args and hyperparameters"""
parser = argparse.ArgumentParser()
# Parse mandatory args
parser.add_argument("data", help="Path to data for training", type=str)
parser.add_argument("outputs_model", help="Name of the model", type=str)
parser.add_argument("outputs_test_data", help="Path to data for testing", type=str)
# Parse hyperparameter args
parser.add_argument("c_value", help="Coeffiecient for the estimator", type=str)
parser.add_argument("kernel", help="Kernel for the estimator", type=str)
parser.add_argument("coef0", help="Coeffiecient for the estimator", type=str)
# Get args
args = parser.parse_args()
return args
def main(**kwargs):
"""Train the model(s)
Parameters
----------
kwargs : dict
Dictionary of all parsed arguments
"""
# Logging
# - Autologging works with (0.22.1 <= scikit-learn <= 1.1.3)
# - See: https://www.mlflow.org/docs/latest/python_api/mlflow.sklearn.html#mlflow.sklearn.autolog
mlflow.sklearn.autolog()
# Setup
X_train, X_test, y_train, y_test = get_data(kwargs.get("data"))
hyperparameters = get_hyperparameters(**kwargs)
# Train
model = train_model(hyperparameters, X_train, y_train)
# Output
save_outputs(model, kwargs.get("outputs_model"), X_test, y_test, kwargs.get("outputs_test_data"))
if __name__ == "__main__":
"""Entrypoint for training the model(s)"""
main(**vars(parse_args()))