Azure ML Studio Pipeline is run under Service Principle - azure-machine-learning-service

Question

I have a draft pipeline created inside the azure machine learning service workspace (Designer Mode). I try to run pipeline from python using Python Azure ML SDK. It starts but quickly fails on the second step.
Trace from step:
Traceback (most recent call last):
File "invoker.py", line 81, in <module>
execute(args)
File "invoker.py", line 71, in execute
ret = run(generate_run_command(args))
File "invoker.py", line 52, in run
return subprocess.Popen(command, stdout=sys.stdout, stderr=sys.stderr).wait(timeout=timeout)
File "/azureml-envs/azureml_b05af1507517824d92fd90bb8ce7897a/lib/python3.8/subprocess.py", line 858, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/azureml-envs/azureml_b05af1507517824d92fd90bb8ce7897a/lib/python3.8/subprocess.py", line 1704, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: ''
When I submit a job for the drafted pipeline in UI there is no problem. When I submit a job for the same draft pipeline from Python SDK, it fails with "Permission denied" on the second step, which actually "Apply SQL Transformation", the first step is Import Dataset. When I resubmit the failed job from UI there is also no problem. It is clear that the problem is in Service Principle. I granted all possible permissions to SP for the workspace. It didn't help. Does anybody have luck running Azure ML Drafted Pipeline from Python?

Answer 1

Service principal method works for me when using Azure ML Drafted Pipeline from Python. I am Using-python 3.7 and azureml-sdk 1.47.0
My code
from azureml.core import Workspace
from azureml.core.authentication import ServicePrincipalAuthentication
from dotenv import load_dotenv
load_dotenv()
def getMLWorkspace():
# Connect to the Azure ML Service Workspace using a service principal
svcpr = ServicePrincipalAuthentication(
tenant_id = os.environ['TENANT_ID'],
service_principal_id = os.environ['SERVICE_PRINCIPAL_ID'],
service_principal_password = os.environ['SERVICE_PRINCIPAL_PWD'])
subscription_id = os.environ['SUBSCRIPTION_ID']
resource_group = os.environ['RESOURCE_GROUP']
workspace_name = os.environ['WORKSPACE_NAME']
ws = Workspace(
subscription_id = subscription_id,
resource_group = resource_group,
workspace_name = workspace_name,
auth = svcpr)
print('Workspace configuration succeeded')
return ws