Azure function failing : "statusCode": 413, "message": "request entity too large" - python-3.x

I have an Python script that creates a CSV file and loads it into an Azure Container. I want to run it as an Azure Function, but it is failing once I deploy it to Azure.
The code works fine in Google Colab (code here minus the connection string). It also works fine when I run it locally as an Azure function via the CLI (func start command).
Here is the code from the init.py file that is deployed
import logging
import azure.functions as func
import numpy as np
import pandas as pd
from datetime import datetime
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__
import tempfile
def main(mytimer: func.TimerRequest) -> None:
logging.info('Python trigger function.')
temp_path = tempfile.gettempdir()
dateTimeObj = datetime.now()
timestampStr = dateTimeObj.strftime("%d%b%Y%H%M%S")
filename =f"{timestampStr}.csv"
df = pd.DataFrame(np.random.randn(5, 3), columns=['Column1','Column2','Colum3'])
df.to_csv(f"{temp_path}{filename}", index=False)
blob = BlobClient.from_connection_string(
conn_str="My connection string",
container_name="container2",
blob_name=filename)
with open(f"{temp_path}{filename}", "rb") as data:
blob.upload_blob(data)
The deployment to Azure is successful, but the functions fails in Azure. When I look in the Azure Function portal, the output from the Code+ Test menu says
{
"statusCode": 413,
"message": "request entity too large"
}
The CSV file produced by the script is 327B - so tiny. Besides that error message I can't see any good information on what is causing the failure. Can anyone suggest a solution / way forward?
Here is the requirements file contents
# Do not include azure-functions-worker as it may conflict with the Azure Functions platform
azure-functions
azure-storage-blob==12.8.1
logging
numpy==1.19.3
pandas==1.3.0
DateTime==4.3
backports.tempfile==1.0
azure-storage-blob==12.8.1
Here is what I have tried.
This article suggests this issue is linked to CORS (Cross Origin Resource sharing) issue. However adding : https://functions.azure.com to my CORS allowed domains in the the Resource sharing menu in the storage settings of my storage account didn't solve the problem (I also republished the Azure function).
Any help, or suggestions would be appreciated.

I tried reproducing the issue with all the information which you have provided and got the same 404 error as below:
Later after adding a value * in CORS, we can get rid of this issue, below is the fixed screenshot and CORS values:
CORS:
Test/Run:
Also we can add the following domain names to avoid 404 error
https://functions.azure.com
https://functions-staging.azure.com
https://functions.azure.com
This will also fix the issue.

Related

How to use AWS Secrets Manager Caching for Python Lambda?

I am referring the aws-secretsmanager-caching-python documentation and trying to cache the retrieved secret from secrets manager however, for some reason, i am always getting timeout without any helpful errors to troubleshoot this further. I am able to retrieve the secrets properly if i retrieve the secrets from secrets manager (without caching).
My main function in lambda function looks like this:
import botocore
import botocore.session
from aws_secretsmanager_caching import SecretCache, SecretCacheConfig
from cacheSecret import getCachedSecrets
def lambda_handler(event, context):
result = getCachedSecrets()
print(result)
and i have created cacheSecret as following.
from aws_secretsmanager_caching import SecretCache
from aws_secretsmanager_caching import InjectKeywordedSecretString, InjectSecretString
cache = SecretCache()
#InjectKeywordedSecretString(secret_id='my_secret_name', cache=cache, secretKey1='keyname1', secretKey2='keyname2')
def getCachedSecrets(secretKey1, secretKey2):
print(secretKey1)
print(secretKey2)
return secretKey1
In the above code, my_secret_name is the name of the secret created in secret manager and secretKey1 and secretKey1 are the secret key names which have string values.
Error:
{
"errorMessage": "2021-03-31T15:29:08.598Z 01f5ded3-7658-4zb5-ae66-6f300098a6e47 Task timed out after 3.00 seconds"
}
Can someone please suggest what needs to be fixed in the above to make this work. Also, i am not sure where to define the secret_name, secret key names in case if we dont use decorators.
The lambda needs to ack the response within 3 seconds but the code can run longer. The timeout can be configured in the function config: https://docs.aws.amazon.com/lambda/latest/dg/configuration-function-common.html

Azure ML: how to access logs of a failed Model deployment

I'm deploying a Keras model that is failing with the error below. The exception says that I can retrieve the logs by running "print(service.get_logs())", but that's giving me empty results. I am deploying the model from my AzureNotebook and I'm using the same "service" var to retrieve the logs.
Also, how can i retrieve the logs from the container instance? I'm deploying to an AKS compute cluster I created. Sadly, the docs link in the exception also doesnt detail how to retrieve these logs.
More information can be found using '.get_logs()' Error:
{ "code":
"KubernetesDeploymentFailed", "statusCode": 400, "message":
"Kubernetes Deployment failed", "details": [
{
"code": "CrashLoopBackOff",
"message": "Your container application crashed. This may be caused by errors in your scoring file's init() function.\nPlease check
the logs for your container instance: my-model-service. From
the AML SDK, you can run print(service.get_logs()) if you have service
object to fetch the logs. \nYou can also try to run image
mlwks.azurecr.io/azureml/azureml_3c0c34b65cf18c8644e8d745943ab7d2:latest
locally. Please refer to http://aka.ms/debugimage#service-launch-fails
for more information."
} ] }
UPDATE
Here's my code to deploy the model:
environment = Environment('my-environment')
environment.python.conda_dependencies = CondaDependencies.create(pip_packages=["azureml-defaults","azureml-dataprep[pandas,fuse]","tensorflow", "keras", "matplotlib"])
service_name = 'my-model-service'
# Remove any existing service under the same name.
try:
Webservice(ws, service_name).delete()
except WebserviceException:
pass
inference_config = InferenceConfig(entry_script='score.py', environment=environment)
comp = ComputeTarget(workspace=ws, name="ml-inference-dev")
service = Model.deploy(workspace=ws,
name=service_name,
models=[model],
inference_config=inference_config,
deployment_target=comp
)
service.wait_for_deployment(show_output=True)
And my score.py
import joblib
import numpy as np
import os
import keras
from keras.models import load_model
from inference_schema.schema_decorators import input_schema, output_schema
from inference_schema.parameter_types.numpy_parameter_type import NumpyParameterType
def init():
global model
model_path = Model.get_model_path('model.h5')
model = load_model(model_path)
model = keras.models.load_model(model_path)
# The run() method is called each time a request is made to the scoring API.
#
# Shown here are the optional input_schema and output_schema decorators
# from the inference-schema pip package. Using these decorators on your
# run() method parses and validates the incoming payload against
# the example input you provide here. This will also generate a Swagger
# API document for your web service.
#input_schema('data', NumpyParameterType(np.array([[0.1, 1.2, 2.3, 3.4, 4.5, 5.6, 6.7, 7.8, 8.9, 9.0]])))
#output_schema(NumpyParameterType(np.array([4429.929236457418])))
def run(data):
return [123] #test
Update 2:
Here is a screencap of the endpoint page. Is it normal for the CPU to be .1? Also, when i hit the swagger url in the browser, i get the error: "No ready replicas for service doc-classify-env-service"
Update 3
After finally getting to the container logs, it turns out that it was choking with this error on my score.py
ModuleNotFoundError: No module named 'inference_schema'
I then ran a test that commented out the refs for "input_schema" and "output_schema" and also simplified my pip_packages and the REST endpoint come up! I was also able to get a prediction out of the model.
pip_packages=["azureml-defaults","tensorflow", "keras"])
So my question is, how should I have my pip_packages for the scoring file to utilize the inference_schema decorators? I'm assuming I need to include azureml-sdk[auotml] pip package, but when i do so, the image creation fails and I see several dependency conflicts.
Try retrieving your service from the workspace directly
ws.webservices[service_name].get_logs()
Also, I found deploying an image as an endpoint to be easier than inference+deploy model (depending on your use case)
my_image = Image(ws, name='test', version='26')
service = AksWebservice.deploy_from_image(ws, "test1", my_image, deployment_config, aks_target)

Crashing of a Cloud Functions HTTP Trigger

I have written a HTTP trigger that takes the ECG database name and record no as the arguments and reads the record from the Cloud Storage, calculates the parameters and writes them to Firestore. I observed a very strange thing, the code crashes without indicating the reason in the console. This is all I get in the console:
Function execution took 58017 ms, finished with status: 'crash'`.
It generally stops when it reads the record from the Cloud Storage. I am using MIT-BIH Cloud Storage to read the records.
from google.cloud import storage
from flask import escape
import firebase_admin
from firebase_admin import credentials
from firebase_admin import firestore
import numpy as np
import os
from pathlib import Path
from os import listdir
from os.path import isfile, join
from random import randint
def GCFname(request):
recordno = request.args['recordno']
database = request.args['database']
client = storage.Client()
bucket = client.get_bucket('bucket_name')
# it crashes here
record = wfdb.rdrecord(recordno, channels=[0],pb_dir='mitdb')
sig = record.p_signal[:,0]
test_qrs = processing.gqrs_detect(record.p_signal[:,0], fs=record.fs)
ann_test= wfdb.rdann(recordno, 'atr',pb_dir='mitdb')
##Calculate Parameters
cred = credentials.ApplicationDefault()
firebase_admin.initialize_app(cred, {
'projectId': 'project_name',
})
db = firestore.client()
doc_ref = db.collection('xyz').document(database).collection('abc').document(recordno)
doc_ref.set({
u'fieldname': fieldvalue
})
I have deployed using gcloud,
gcloud functions deploy GCFname --runtime python37 --trigger-http --allow-unauthenticated --timeout 540s
But on using the same URL after some time it works. What could be reason for this? Its definitely not timeout issue.
It would be difficult to see why the Cloud Function is crashing without the logs as it can be due to many reasons. Currently, there is a bug open for Cloud Functions crashing before flushing log entries or writing any error trace to Stackdriver. You can follow this issue tracker here for any updates: https://issuetracker.google.com/155215191

Error: 503 DNS resolution failed in Google Translate API but ONLY when executing the Python file via terminal

I am running into a strange issue when using Google Translate API with a JSON authorization key. I can run the file without any issue directly in my editor of choice (VSCode). No errors.
But when I run the same file via terminal, the file executes up until after loading the Google Translate credentials from disk. The call then throws below error message.
Since this only ever happens when running the file from terminal, I find this bug hard to tackle.
The goal is to then have this file collect data and translate some of the fields using Google services, then store the data in a data base.
Below is the error message:
Error: 503 DNS resolution failed
Traceback (most recent call last):
File "/home/MY-USER-NAME/anaconda3/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
File "/home/MY-USER-NAME/anaconda3/lib/python3.6/site-packages/grpc/_channel.py", line 690, in __call__
File "/home/MY-USER-NAME/anaconda3/lib/python3.6/site-packages/grpc/_channel.py", line 592, in _end_unary_response_blocking
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "DNS resolution failed"
debug_error_string = "{"created":"#1584629013.091398712","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3934,"referenced_errors":[{"created":"#1584629013.091395769","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/resolving_lb_policy.cc","file_line":262,"referenced_errors":[{"created":"#1584629013.091394954","description":"DNS resolution failed","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":370,"grpc_status":14,"referenced_errors":[{"created":"#1584629013.091389655","description":"C-ares status is not ARES_SUCCESS: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244,"referenced_errors":[{"created":"#1584629013.091380513","description":"C-ares status is not ARES_SUCCESS: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244}]}]}]}]}"
>
This is the relevant part of my code:
Dedicated CRON file (I use this two-step setup in many other projects without any issue)
#! anaconda3/bin/python
import os
os.chdir(r'/home/MY-USER-NAME/path-to-project')
import code-file
Code file (simple .py script)
[...]
from google.oauth2 import service_account
from google.cloud import translate
key_path = 'path-to-file/credentials.json'
credentials = service_account.Credentials.from_service_account_file(
key_path, scopes=['https://www.googleapis.com/auth/cloud-platform'])
def translate_to_en(contents, credentials=None, verbose=False, use_pinyin=False):
[...]
client = translate.TranslationServiceClient(credentials=credentials)
parent = client.location_path(credentials.project_id, 'global')
[...]
response = client.translate_text(
parent=parent,
contents=[contents],
mime_type='text/plain', # mime types: text/plain, text/html
target_language_code='en-US')
[...]
[...]
for c in trans_cols:
df[f'{c}__trans'] = df[c].apply(lambda x: translate_to_en(contents=x, credentials=credentials, verbose=False))
[...]
If there is anyone with a good idea to solve this, your help is greatly appreciated.
It seems the issue is relevant to the grpc bug reported in github. For gRPC name resolution, you can follow this doc on github.

Google Cloud Functions Python Logging issue

I'm not sure how to say this but, I'm feeling like there is something under the hood that was changed by Google without me knowing about it. I used to get my logs from my python Cloud Functions in the Google Cloud Console within the logging dashboard. And now, it just stopped working.
So I went investigating for a long time, I just made a log hello world python Cloud Function:
import logging
def cf_endpoint(req):
logging.debug('log debug')
logging.info('log info')
logging.warning('log warning')
logging.error('log error')
logging.critical('log critical')
return 'ok'
So this is my main.py that I deploy as a Cloud Function with an http trigger.
Since I was having a log ingestion exclusion filter with all the "debug" level logs I wasn't seeing anything in the logging dashboard. But when I removed it I discovered this :
So it seems like something that was parsing the python built-in log records into stackdriver stopped parsing the log severity parameter! I'm sorry if I look stupid but that's the only thing I can think about :/
Do you guys have any explanations or solutions for this ? am I doing it the wrong way ?
Thank you in advance for your help.
UPDATE 2022/01:
The output now looks for example like:
[INFO]: Connecting to DB ...
And the drop-down menu for the severity looks like:
With "Default" as the filter that is needed to show the Python logging logs, which means to show just any log available, and all of the Python logs are under "Default", the severity is still dropped.
Stackdriver Logging severity filters are no longer supported when using the Python native logging module.
However, you can still create logs with certain severity by using the Stackdriver Logging Client Libraries. Check this documentation in reference to the Python libraries, and this one for some usage-case examples.
Notice that in order to let the logs be under the correct resource, you will have to manually configure them, see this list for the supported resource types.
As well, each resource type has some required labels that need to be present in the log structure.
As an example, the following code will write a log to the Cloud Function resource, in Stackdriver Logging, with an ERROR severity:
from google.cloud import logging
from google.cloud.logging.resource import Resource
log_client = logging.Client()
# This is the resource type of the log
log_name = 'cloudfunctions.googleapis.com%2Fcloud-functions'
# Inside the resource, nest the required labels specific to the resource type
res = Resource(type="cloud_function",
labels={
"function_name": "YOUR-CLOUD-FUNCTION-NAME",
"region": "YOUR-FUNCTION-LOCATION"
},
)
logger = log_client.logger(log_name.format("YOUR-PROJECT-ID"))
logger.log_struct(
{"message": "message string to log"}, resource=res, severity='ERROR')
return 'Wrote logs to {}.'.format(logger.name) # Return cloud function response
Notice that the strings in YOUR-CLOUD-FUNCTION-NAME, YOUR-FUNCTION-LOCATION and YOUR-PROJECT-ID, need to be specific to your project/resource.
I encountered the same issue.
In the link that #joan Grau shared, I also see there is a way to integrate cloud logger with Python logging module, so that you could use Python root logger as usually, and all logs will be sent to StackDriver Logging.
https://googleapis.github.io/google-cloud-python/latest/logging/usage.html#integration-with-python-logging-module
...
I tried it and it works. In short, you could do it two ways
One simple way that bind cloud logger to root logging
from google.cloud import logging as cloudlogging
import logging
lg_client = cloudlogging.Client()
lg_client.setup_logging(log_level=logging.INFO) # to attach the handler to the root Python logger, so that for example a plain logging.warn call would be sent to Stackdriver Logging, as well as any other loggers created.
Alternatively, you could set logger with more fine-grain control
from google.cloud import logging as cloudlogging
import logging
lg_client = cloudlogging.Client()
lg_handler = lg_client.get_default_handler()
cloud_logger = logging.getLogger("cloudLogger")
cloud_logger.setLevel(logging.INFO)
cloud_logger.addHandler(lg_handler)
cloud_logger.info("test out logger carrying normal news")
cloud_logger.error("test out logger carrying bad news")
Not wanting to deal with cloud logging libraries, I created a custom Formatter that emits a structured log with the right fields, as cloud logging expects it.
class CloudLoggingFormatter(logging.Formatter):
"""Produces messages compatible with google cloud logging"""
def format(self, record: logging.LogRecord) -> str:
s = super().format(record)
return json.dumps(
{
"message": s,
"severity": record.levelname,
"timestamp": {"seconds": int(record.created), "nanos": 0},
}
)
Attaching this handler to a logger results in logs being parsed and shown properly in the logging console. In cloud functions I would configure the root logger to send to stdout and attach the formatter to it.
# setup logging
root = logging.getLogger()
handler = logging.StreamHandler(sys.stdout)
formatter = CloudLoggingFormatter(fmt="[%(name)s] %(message)s")
handler.setFormatter(formatter)
root.addHandler(handler)
root.setLevel(logging.DEBUG)
From Python 3.8 onwards you can simply print a JSON structure with severity and message properties. For example:
print(
json.dumps(
dict(
severity="ERROR",
message="This is an error message",
custom_property="I will appear inside the log's jsonPayload field",
)
)
)
Official documentation: https://cloud.google.com/functions/docs/monitoring/logging#writing_structured_logs
To use the standard python logging module on GCP (tested on python 3.9), you can do the following:
import google.cloud.logging
logging_client = google.cloud.logging.Client()
logging_client.setup_logging()
import logging
logging.warning("A warning")
See also: https://cloud.google.com/logging/docs/setup/python
I use a very simple custom logging function to log to Cloud Logging:
import json
def cloud_logging(severity, message):
print(json.dumps({"severity": severity, "message": message}))
cloud_logging(severity="INFO", message="Your logging message")

Resources