MLFlow RestException: RESOURCE_ALREADY_EXISTS error when starting runs - mlflow

Question

Previously was using ML FLow with Databricks on Azure Machine Learning to register and track model Hyperparameter tuning with both SKLearn and Stats model models from start of September with no issues. But since about 23rd October, I started getting these kinds of errors:
RestException: RESOURCE_ALREADY_EXISTS: Failed to create AML experiment for experiment id=863468136127724, name=/my-experiment3, artifactLocation=dbfs:/databricks/mlflow-tracking/863468136127724. There is an existing AML experiment with id=c74bdea3-e382-4cdf-868a-ee1421de078e and name='/adb/5909321886823418/863468136127724/my-experiment3' and artifactLocation='' that is not compatible.
Even when running a newly created experiment, it will throw this error
We recently have updated to ml flow v1.21.0 but it doesn't seem to be a bug as there is nothing on the ML Flow github that is similar, just wondering if anyone has encountered anything similar as I run out of ideas of things to look for the issue.

Answer 1

I contact the Microsoft Support team. The problem seems to be that azure databricks mistakenly link to the AML workspace and they provide the following ARM template where you should fill the parameters that it include:
{"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceName": {
"type": "string",
"defaultValue": "",
"metadata": {
"description": "The name of the Azure Databricks workspace to create or update."
}
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for all resources."
}
},
"amlWorkspaceId": {
"type": "string",
"metadata": {
"description": "The resource id of the Azure Machine Learning service workspace."
},
"defaultValue": ""
}
},
"variables": {
"managedResourceGroupName": "[concat('databricks-rg-', parameters('workspaceName'), '-', uniqueString(parameters('workspaceName'), resourceGroup().id))]"
},
"resources": [
{
"type": "Microsoft.Databricks/workspaces",
"name": "[parameters('workspaceName')]",
"location": "[parameters('location')]",
"apiVersion": "2018-04-01",
"properties": {
"ManagedResourceGroupId": "[concat(subscription().id, '/resourceGroups/', variables('managedResourceGroupName'))]",
"parameters": {
"amlWorkspaceId": {
"value": "[if(equals(parameters('amlWorkspaceId'),''), json('null'), parameters('amlWorkspaceId'))]"
}
}
}
}
],
"outputs": {
"workspace": {
"type": "object",
"value": "[reference(resourceId('Microsoft.Databricks/workspaces', parameters('workspaceName')))]"
}
}}
But I can not deploy this ARM Template in my subscription, so I decided to delete de Azure Machine Learning Service that is linked automatically with my Databricks following the solution in this thread https://github.com/MicrosoftDocs/azure-docs/issues/80298 and the problem was solved.