Not all data Stores (workspaceblobstore) specified in the run configuration exist - azure-machine-learning-service

Question

I am submitting a run to an AML workspace programmatically, and it is failing with the error message:
"error": {
"code": "UserError",
"message": "Not all data Stores (workspaceblobstore) specified in the run configuration exist." }
Inspecting the run configuration object, the sourceDirectoryDataStore is set to null, and the other two data-related properties are empty.
"dataReferences": {},
"data": {},
"sourceDirectoryDataStore": null
The script that I am submitting does not make use of any datastores registered through aml, it is just the simple diabetes regression, using the sklearn diabetes dataset.
The training script is copied to the snapshots container of the storage account linked with the AML workspace.
What would be the next steps on troubleshooting this?
SDK Version: 1.0.85.
When retrieving the datastores, through ws.datastores (suggested in the comments), I get another exception about the storage service missing name or key, but there is a storage account that got deployed with the AML workspace. Inspecting the ARM template of the AML workspace the storage account id is in the properties of the AML template, and the usual containers (revisions, snapshots, snapshotzips, azureml-bloblstore-GUID) are created and the *.py files that I am attempting to run are being uploaded.
print(ws.datastores)
File "C:\ProgramData\Anaconda3\lib\site-packages\azureml\core\workspace.py", line 789, in datastores
return {datastore.name: datastore for datastore in _DatastoreClient.list(self)}
File "C:\ProgramData\Anaconda3\lib\site-packages\azureml\data\datastore_client.py", line 486, in list
dss, ct = _DatastoreClient._list(workspace, ct, 100)
File "C:\ProgramData\Anaconda3\lib\site-packages\azureml\data\datastore_client.py", line 688, in _list
return list(datastores), datastore_dtos.continuation_token
File "C:\ProgramData\Anaconda3\lib\site-packages\azureml\data\datastore_client.py", line 687, in <lambda>
map(lambda dto: _DatastoreClient._dto_to_datastore(ws, dto), datastore_dtos.value))
File "C:\ProgramData\Anaconda3\lib\site-packages\azureml\data\datastore_client.py", line 760, in _dto_to_datastore
as_section.sas_token, as_section.account_key, as_section.protocol, as_section.endpoint)
File "C:\ProgramData\Anaconda3\lib\site-packages\azureml\data\azure_storage_datastore.py", line 390, in __init__
endpoint_suffix=endpoint
File "C:\ProgramData\Anaconda3\lib\site-packages\azureml\_vendor\azure_storage\file\fileservice.py", line 184, in __init__
raise ValueError(_ERROR_STORAGE_MISSING_INFO)
ValueError: You need to provide an account name and either an account_key or sas_token when creating a storage service.

Answer 1

I had created the workspace that gives this problem through the powershell ARM deployment scripts, specifying its ARM template. The deployment succeeded fine.
I tried deploying the workspace through the AzureML SDK, and that creates the datastores and suceeds the run.

Answer 2

It might be linked to the ARM template (or Terraform in our case).
We updated the infrastructure (ML workspace), and for some reason the user principal's access to the associated keyvault got removed...
Trying to launch a new run resulted in a very similar error:
Not all data Stores (workspaceblobstore) specified in the run configuration exist. If the Datastore has secret registered, please make sure that RBAC action "metadata/secrets/read" is permitted
In our case, the machine learning workspace resource was configured like this (only option at the time):
identity {
type = "SystemAssigned"
}
So, we fixed this issue by explicitly adding a key vault access policy for the machine learning workspace:
resource "azurerm_key_vault_access_policy" "principal_id_access" {
key_vault_id = azurerm_key_vault.example.id
tenant_id = var.azure_tenant_id
object_id = azurerm_machine_learning_workspace.example.identity[0].principal_id
key_permissions = [
"backup", "create", "decrypt", "delete", "encrypt",
"get", "import", "list", "purge", "recover", "restore",
"sign", "unwrapKey", "update", "verify", "wrapKey"
]
secret_permissions = [
"backup", "delete", "get", "list", "purge", "recover", "restore", "set"
]
certificate_permissions = [
"backup", "create", "delete", "deleteissuers", "get", "getissuers",
"import", "list", "listissuers", "managecontacts", "manageissuers", "purge",
"recover", "restore", "setissuers", "update"
]
}