I have a KubeFlow Pipeline up and running on a VM in GCP with KF.
I create the pipeline using a Jupyter Notebook server with image jupyter-kale and python.
The first part of the pipeline is doing the dataprep, it downloads images and saves them to a PVC. This all works just fine but I run out of storage space so I decided to save the images downloaded directly to a google Bucket instead using the PVC.
I modified my pipeline as shown in the code below:
import kfp
import kfp.dsl as dsl
import kfp.onprem as onprem
import kfp.compiler as compiler
import os
#dsl.pipeline(
name='try_mount',
description='...'
)
def one_d_pipe(pvc_name = "gs://xxx-images/my_folder/"):
trymount = dsl.ContainerOp(
name="trymount",
#image = "sprintname3:0.2.0",
image = "eu.gcr.io/xxx-admin/kubeflow/trymount_1:0.1"
)
steps = [trymount]
for step in steps:
step.apply(onprem.mount_pvc(pvc_name, "gs://xxx-images/my_folder/", '/home/jovyan/data'))
But this code resulting in an error message right after start saying that the volume has invalid value and could not be found:
This step is in Error state with this message: Pod "try-mount-75vrt-3151677017" is invalid: [spec.volumes[2].name: Invalid value: "gs://xxx-images/my_folder/": a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for validation is 'a-z0-9?'), spec.containers[0].volumeMounts[3].name: Not found: "gs://xxx-images/my_folder/", spec.containers[1].volumeMounts[0].name: Not found: "gs://xxx-images/my_folder/"]
So, my question:
How to mount a google bucket in Kubeflow Pipelines?
You can't mount a bucket as a volume. It's not a file system. However, I'm sure that you can cheat by use gcsfuse on your VM.
On your VM, mount the GCS bucket with fuse
gcsfuse xxx-images /path/to/mount-gcs
Then in your code, use this directory. No mount required, the GCS is already mount with GCSFuse.
Related
I am trying to set up a ML pipeline on Azure ML using the Python SDK.
I have scripted the creation of a custom environment from a DockerFile as follows
from azureml.core import Environment
from azureml.core.environment import ImageBuildDetails
from other_modules import workspace, env_name, dockerfile
custom_env : Environment = Environment.from_dockerfile(name=env_name, dockerfile=dockerfile)
custom_env.register(workspace=workspace)
build : ImageBuildDetails = custom_env.build(workspace=workspace)
build.wait_for_completion()
However, the ImageBuildDetails object that the build method returns invariably times out while executing the last wait_for_completion() line, ... likely due to network constraints that I cannot change.
So, how can I possibly check the build status via the SDK in a way that doesn't exclusively depend on the returned ImageBuildDetails object?
My first suggestion would be to use:
build.wait_for_completion(show_output=True)
This will help you debug better rather than assuming you have network issues, as the images can take quite a long time to build, and from my experience creating environments it's very likely you may have an issue with related to your Dockerfile.
A good alternative option is to build your docker image locally and optionally push it to the container registry associated with the workspace:
from azureml.core import Environment
myenv = Environment(name="myenv")
registered_env = myenv.register(workspace)
registered_env.build_local(workspace, useDocker=True, pushImageToWorkspaceAcr=True)
However another preferred method is to create an environment object from an environment specification YAML file:
from_conda_specification(name, file_path)
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-environments#use-conda-dependencies-or-pip-requirements-files
This should return the Environment and to verify it has been created:
for name,env in ws.environments.items():
print("Name {} \t version {}".format(name,env.version))
restored_environment = Environment.get(workspace=ws,name="myenv",version="1")
print("Attributes of restored environment")
restored_environment
I am trying to deploy a machine learning model through an ACI (Azure Container Instances) service. I am working in Python and I followed the following code (from the official documentation : https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-and-where?tabs=azcli) :
The entry script file is the following (score.py):
import os
import dill
import joblib
def init():
global model
# Get the path where the deployed model can be found
model_path = os.getenv('AZUREML_MODEL_DIR')
# Load existing model
model = joblib.load('model.pkl')
# Handle request to the service
def run(data):
try:
# Pick out the text property of the JSON request
# Expected JSON details {"text": "some text to evaluate"}
data = json.loads(data)
prediction = model.predict(data['text'])
return prediction
except Exception as e:
error = str(e)
return error
And the model deployment workflow is as:
from azureml.core import Workspace
# Connect to workspace
ws = Workspace(subscription_id="my-subscription-id",
resource_group="my-ressource-group-name",
workspace_name="my-workspace-name")
from azureml.core.model import Model
model = Model.register(workspace = ws,
model_path= 'model.pkl',
model_name = 'my-model',
description = 'my-description')
from azureml.core.environment import Environment
# Name environment and call requirements file
# requirements: numpy, tensorflow
myenv = Environment.from_pip_requirements(name = 'myenv', file_path = 'requirements.txt')
from azureml.core.model import InferenceConfig
# Create inference configuration
inference_config = InferenceConfig(environment=myenv, entry_script='score.py')
from azureml.core.webservice import AciWebservice #AksWebservice
# Set the virtual machine capabilities
deployment_config = AciWebservice.deploy_configuration(cpu_cores = 0.5, memory_gb = 3)
from azureml.core.model import Model
# Deploy ML model (Azure Container Instances)
service = Model.deploy(workspace=ws,
name='my-service-name',
models=[model],
inference_config=inference_config,
deployment_config=deployment_config)
service.wait_for_deployment(show_output = True)
I succeded once with the previous code. I noticed that during the deployment the Model.deploy created a container registry with a specific name (6e07ce2cc4ac4838b42d35cda8d38616).
The problem:
The API was working well and I wanted to deploy an other model from scratch. I deleted the API service and model from Azure ML Studio and the container registry from Azure ressources.
Unfortunately I am not able to deploy again anything.
Everything goes fine until the last step (the Model.deploy step), I have the following error message :
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 46243f9b-3833-4650-8d47-3ac54a39dc5e
More information can be found here: https://machinelearnin2812599115.blob.core.windows.net/azureml/ImageLogs/46245f8b-3833-4659-8d47-3ac54a39dc5e/build.log?sv=2019-07-07&sr=b&sig=45kgNS4sbSZrQH%2Fp29Rhxzb7qC5Nf1hJ%2BLbRDpXJolk%3D&st=2021-10-25T17%3A20%3A49Z&se=2021-10-27T01%3A24%3A49Z&sp=r
Error:
{
"code": "AciDeploymentFailed",
"statusCode": 404,
"message": "No definition exists for Environment with Name: myenv Version: Autosave_2021-10-25T17:24:43Z_b1d066bf Reason: Container > registry 6e07ce2cc4ac4838b42d35cda8d38616.azurecr.io not found. If private link is enabled in workspace, please verify ACR is part of private > link and retry..",
"details": []
}
I do not understand why the first time a new container registry was well created, but now it seems that it is sought (the message is saying that container registry identified by name 6e07ce2cc4ac4838b42d35cda8d38616 is missing). I never found where I can force the creation of a new container registry ressource in Python, neither specify a name for it in AciWebservice.deploy_configuration or Model.deploy.
Does anyone could help me moving on with this? The best solution would be I think to delete totally this 6e07ce2cc4ac4838b42d35cda8d38616 container registry but I can't find where the reference is set so Model.deploy always fall to find it.
An other solution would be to force Model.deploy to generate a new container registry, but I could find how to make that.
It's been 2 days that I am on this and I really need your help !
PS : I am not at all a DEVOPS/MLOPS guy, I make data science and good models, but infrastructure and deployment is not really my thing so please be gentle on this part ! :-)
What I tried
Creating the container registry with same name
I tried to create the container registry by hand, but this time, this is the container that cannot be created. The Python output of the Model.deploy is the following :
Tips: You can try get_logs(): https://aka.ms/debugimage#dockerlog or local deployment: https://aka.ms/debugimage#debug-locally to debug if deployment takes longer than 10 minutes.
Running
2021-10-25 19:25:10+02:00 Creating Container Registry if not exists.
2021-10-25 19:25:10+02:00 Registering the environment.
2021-10-25 19:25:13+02:00 Building image..
2021-10-25 19:30:45+02:00 Generating deployment configuration.
2021-10-25 19:30:46+02:00 Submitting deployment to compute.
Failed
Service deployment polling reached non-successful terminal state, current service state: Unhealthy
Operation ID: 93780de6-7662-40d8-ab9e-4e1556ef880f
Current sub-operation type not known, more logs unavailable.
Error:
{
"code": "InaccessibleImage",
"statusCode": 400,
"message": "ACI Service request failed. Reason: The image '6e07ce2cc4ac4838b42d35cda8d38616.azurecr.io/azureml/azureml_684133370d8916c87f6230d213976ca5' in container group 'my-service-name-LM4HbqzEBEi0LTXNqNOGFQ' is not accessible. Please check the image and registry credential.. Refer to https://learn.microsoft.com/azure/container-registry/container-registry-authentication#admin-account and make sure Admin user is enabled for your container registry."
}
Setting admin user enabled
I tried to follow the recommandation of the last message saying to set Admin user enabled for the container registry. All what I saw in Azure interface is that a username and password appeared when enabling on user admin.
Unfortunately the same error message appears again if I try to relaunche my code and I am stucked here...
Changing name of the environment and model
This does not produces any change. Same errors.
As you tried with first attempt it was worked. After deleting the API service and model from Azure ML Studio and the container registry from Azure resources you are not able to redeploy again.
My assumption is your first attempt you are already register the Model Environment variable. So when you try to reregister by using the same model name while deploying it will gives you the error.
Thanks # anders swanson Your solution worked for me.
If you have already registered your env, myenv, and none of the details of the your environment have changed, there is no need re-register it with myenv.register(). You can simply get the already register env using Environment.get() like so:
myenv = Environment.get(ws, name='myenv', version=11)
My Suggestion is to name your environment as new value.
"model_scoring_env". Register it once, then pass it to the InferenceConfig.
Refer here
I have an EC2 instance running pyspark and I'm able to connect to it (ssh) and run interactive code within a Jupyter Notebook.
I have a S3 bucket with a csv file that I want to read, when I attempt to read it with:
spark = SparkSession.builder.appName('Basics').getOrCreate()
df = spark.read.csv('https://s3.us-east-2.amazonaws.com/bucketname/filename.csv')
Which throws a long Python error message and then something related to:
Py4JJavaError: An error occurred while calling o131.csv.
Specify S3 path along with access key and secret key as following:
's3n://<AWS_ACCESS_KEY_ID>:<AWS_SECRET_ACCESS_KEY>#my.bucket/folder/input_data.csv'
Access key-related information can be introduced in the typical username + password manner for URLs. As a rule, the access protocol should be s3a, the successor to s3n (see Technically what is the difference between s3n, s3a and s3?). Putting this together, you get
spark.read.csv("s3a://<AWS_ACCESS_KEY_ID>:<AWS_SECRET_ACCESS_KEY>#bucketname/filename.csv")
As an aside, some Spark execution environments, e.g., Databricks, allow S3 buckets to be mounted as part of the file system. You can do the same when you build a cluster using something like s3fs.
I have a model that was trained on a Machine Learning Compute on Azure Machine Learning Service. The registered model already lives my workspace and I would like to deploy it to a pre-existing AKS instance that I previously provisioned in my workspace. I am able to successfully configure and register the container image:
# retrieve cloud representations of the models
rf = Model(workspace=ws, name='pumps_rf')
le = Model(workspace=ws, name='pumps_le')
ohc = Model(workspace=ws, name='pumps_ohc')
print(rf); print(le); print(ohc)
<azureml.core.model.Model object at 0x7f66ab3b1f98>
<azureml.core.model.Model object at 0x7f66ab7e49b0>
<azureml.core.model.Model object at 0x7f66ab85e710>
package_list = [
'category-encoders==1.3.0',
'numpy==1.15.0',
'pandas==0.24.1',
'scikit-learn==0.20.2']
# Conda environment configuration
myenv = CondaDependencies.create(pip_packages=package_list)
conda_yml = 'file:'+os.getcwd()+'/myenv.yml'
with open(conda_yml,"w") as f:
f.write(myenv.serialize_to_string())
Configuring and registering the image works:
# Image configuration
image_config = ContainerImage.image_configuration(execution_script='score.py',
runtime='python',
conda_file='myenv.yml',
description='Pumps Random Forest model')
# Register the image from the image configuration
# to Azure Container Registry
image = ContainerImage.create(name = Config.IMAGE_NAME,
models = [rf, le, ohc],
image_config = image_config,
workspace = ws)
Creating image
Running....................
SucceededImage creation operation finished for image pumpsrfimage:2, operation "Succeeded"
Attaching to an existing cluster also works:
# Attach the cluster to your workgroup
attach_config = AksCompute.attach_configuration(resource_group = Config.RESOURCE_GROUP,
cluster_name = Config.DEPLOY_COMPUTE)
aks_target = ComputeTarget.attach(workspace=ws,
name=Config.DEPLOY_COMPUTE,
attach_configuration=attach_config)
# Wait for the operation to complete
aks_target.wait_for_completion(True)
SucceededProvisioning operation finished, operation "Succeeded"
However, when I try to deploy the image to the existing cluster, it fails with a WebserviceException.
# Set configuration and service name
aks_config = AksWebservice.deploy_configuration()
# Deploy from image
service = Webservice.deploy_from_image(workspace = ws,
name = 'pumps-aks-service-1' ,
image = image,
deployment_config = aks_config,
deployment_target = aks_target)
# Wait for the deployment to complete
service.wait_for_deployment(show_output = True)
print(service.state)
WebserviceException: Unable to create service with image pumpsrfimage:1 in non "Succeeded" creation state.
---------------------------------------------------------------------------
WebserviceException Traceback (most recent call last)
<command-201219424688503> in <module>()
7 image = image,
8 deployment_config = aks_config,
----> 9 deployment_target = aks_target)
10 # Wait for the deployment to complete
11 service.wait_for_deployment(show_output = True)
/databricks/python/lib/python3.5/site-packages/azureml/core/webservice/webservice.py in deploy_from_image(workspace, name, image, deployment_config, deployment_target)
284 return child._deploy(workspace, name, image, deployment_config, deployment_target)
285
--> 286 return deployment_config._webservice_type._deploy(workspace, name, image, deployment_config, deployment_target)
287
288 #staticmethod
/databricks/python/lib/python3.5/site-packages/azureml/core/webservice/aks.py in _deploy(workspace, name, image, deployment_config, deployment_target)
Any ideas on how to solve this issue? I am writing the code in a Databricks notebook. Also, I am able to create and deploy the cluster using Azure Portal no problem so this appears to be an issue with my code/Python SDK or the way Databricks works with AMLS.
UPDATE:
I was able to deploy my image to AKS using Azure Portal and the webservice works as expected. This means the issue lies somewhere between Databricks, the Azureml Python SDK and Machine Learning Service.
UPDATE 2:
I'm working with Microsoft to fix this issue. Will report back once we have a solution.
In my initial code, when creating the image, I was not using:
image.wait_for_creation(show_output=True)
As a consequence, I was calling CreateImage and DeployImage before the image was created which errored out. Can't believe it was that simple..
UPDATED IMAGE CREATION SNIPPET:
# Register the image from the image configuration
# to Azure Container Registry
image = ContainerImage.create(name = Config.IMAGE_NAME,
models = [rf, le, ohc],
image_config = image_config,
workspace = ws)
image.wait_for_creation(show_output=True)
From personal experience I would say that the error message you see might suggest that there is some error with the script inside the image. Such errors doesn't necessary prevent the image from being created successfully, but it might prevent the image from being used in a service. However, if you have successfully been able to deploy the image in other services, then you should be able to rule out this option.
You can follow this guide for more information on how to debug the Docker image locally, as well as finding logs and other useful information.
Agreed on Arvid's answer. Were you able to succesfully run it? You can also try and deploy it to ACI, but if the problem is in the score.py, you'd have the same issue but it's quick to try. Also, a bit more painful but if you want to debug the deployment, but you can expose port tcp 5678 on your local docker deployment and use VSCode and PTVSD to connect to it and debug step by step.
I am trying to connect to AWS S3 without using credentials. I attached the Role S3 fullaccess for my instance to check if the file exists or not; if it is not, upload it into S3 bucket. If is isn't I want to check md5sum and if it is different from the local local file, upload a new version.
I try to get key of file in S3 via boto by using bucket.get_key('mykey') and get this error:
File "/usr/local/lib/python3.5/dist-packages/boto/s3/bucket.py", line 193, in get_key key, resp = self._get_key_internal(key_name, headers, query_args_l)
File "/usr/local/lib/python3.5/dist-packages/boto/s3/bucket.py", line 232, in _get_key_internal response.status, response.reason, '') boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden"
I searched and added "validate=False" when getting the bucket, but this didn't resolve my issue. I'm using Python 3.5, boto and boto3.
Here is my code:
import boto3
import boto
from boto import ec2
import os
import boto.s3.connection
from boto.s3.key import Key
bucket_name = "abc"
conn = boto.s3.connect_to_region('us-west-1', is_secure = True, calling_format = boto.s3.connection.OrdinaryCallingFormat())
bucket = conn.get_bucket(bucket_name, validate=False)
key = bucket.get_key('xxxx')
print (key)
I don't know why I get that error. Please help me to clearly this problem. Thanks!
Updated
I've just find root cause this problem. Cause by "The difference between the request time and the current time is too large".
Then it didn't get key of file from S3 bucket. I updated ntp service to synchronize local time and UTC time. It run success.
Synchronization time by:
sudo service ntp stop
sudo ntpdate -s 0.ubuntu.pool.ntp.org
sudo service ntp start
Thanks!
IAM role is the last in the order of search. I bet you have the credentials stored before the search order which doesn't have full S3 access. Check Configuration Settings and Precedence and make sure no credentials is present so that IAM role is used to fetch the credentials. Though it is for CLI, it applies to scripts too.
The AWS CLI looks for credentials and configuration settings in the following order:
Command line options – region, output format and profile can be specified as command options to override default settings.
Environment variables – AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN.
The AWS credentials file – located at ~/.aws/credentials on Linux, macOS, or Unix, or at C:\Users\USERNAME .aws\credentials on Windows. This file can contain multiple named profiles in addition to a default profile.
The CLI configuration file – typically located at ~/.aws/config on Linux, macOS, or Unix, or at C:\Users\USERNAME .aws\config on Windows. This file can contain a default profile, named profiles, and CLI specific configuration parameters for each.
Container credentials – provided by Amazon Elastic Container Service on container instances when you assign a role to your task.
Instance profile credentials – these credentials can be used on EC2 instances with an assigned instance role, and are delivered through the Amazon EC2 metadata service.