The following python code is a batch_write into Dynamodb using boto3:
try:
chunk_size = 500
for i in range(0, len(items), chunk_size):
chunk = items[i:i+chunk_size]
with table.batch_writer() as batch:
for item in chunk:
dynamodb_item = {
'itemId': item['itemId'],
'time': item['time'],
'value': item['value']
}
batch.put_item( Item = dynamodb_item )
except botocore.exceptions.ClientError as error:
print(error.message)
It throws the following exception which is about "exceeding the provisioned throughput", but try-catch does not catch it.
Why it does not catch it?
DEBUG:botocore.parsers:Response body:
b'{"__type":"com.amazonaws.dynamodb.v20120810#ProvisionedThroughputExceededException",
"message":"The level of configured provisioned throughput for the table was exceeded.
Consider increasing your provisioning level with the UpdateTable API."}'
You are attempting to catch a botocore.exceptions.ClientError while the error being thrown is ProvisionedThroughputExceededException. Try changing the type of exception to just Exception.
Related
I have a main function which prepare a customised bigquery and pass it to the another function where actual execution of the query happens:
Main function:
def main(request):
try:
client = bigquery.Client(project=flat_rate_project)
# Executing query and loading data into temporary table.
job_config = {"destination": temp_bq_project_details,
"create_disposition": "CREATE_IF_NEEDED",
"write_disposition": "WRITE_APPEND"}
response = execute_select_bq_query(static_query, bq_location, job_config)
print(response['message'])
except RuntimeError:
print("Exception occurred {}".format(RuntimeError))
except ValueError as error:
print("This is the ValueError", str(error))
finally:
Print("Functionality completed !")
Another function [execute_select_bq_query]:
from google.cloud import bigquery
import math
def execute_select_bq_query(query, location, jobconfig):
try:
# Construct a BigQuery client object.'
client = bigquery.Client(project=queryexcproject)
job_config = bigquery.QueryJobConfig()
dest_table = bq_table_reference(client, jobconfig['destination'])
job_config.destination = dest_table
job_config.create_disposition = jobconfig['create_disposition']
job_config.write_disposition = jobconfig['write_disposition']
query_job = client.query(query, location=location, job_config=job_config)
math.sqrt(-10) # To test if it raises ValueError exception
results = query_job.result()
response = {"message": "Data loaded into BQ table {}".format(jobconfig['destination']), "responseCode": 200}
return response
except RuntimeError:
print("Exception occurred {}".format(RuntimeError))
response = "Exception occurred {}".format(RuntimeError)
return response
except ValueError as error:
raise "Exception occurred while loading data into BQ table {} with error {}".format(jobconfig['destination'], error)
Whenever there is ValueError exception occurred in 'execute_select_bq_query' function I am raising it with customise message and I want my main function should catch that. But it is giving me 'During handling of the above exception, another exception occurred' error.
I have try-catch that behaved differently, Once an exception is raised, the next time the try-catch block is called, it's always raising an exception even if the params passed are valid
class Some
def some_method(variable_id: str):
response = requests.request('https://...........)
if response.status_code != 200:
if response.status_code == 404:
raise ClientError(
'_NOT_FOUND', 'Consignment does not exist. Please provide a valid variable_id', 404)
elif response.status_code == 400:
raise ClientError(
'_ALREADY_CANCELLED', f"Cannot print {variable_id}", 400)
raise ClientError(
'ERROR', "ERROR", 400)
return response
try catch
class Other:
def __init__(self):
self.error = False
def somefunc(id: str):
//id = '123' //working one
try:
response = Some().some_method(id)
return self.error, response
except Exception as e:
self.error = True
return error,[]
The weird thing is if I first called the try-catch using the id that the API check as valid, it will return the response in the try-catch block as expected. But when I replace the id with a value that the API is returning 400 status_code, it will always raise the 400 status code exception even if I called the try-catch block with the previous id that should return 200.
So I tried to print the response.status_code, it is correctly 200 for a valid id, and 400 for an invalid id, but I don't know why it always raises the exception after I call the try-catch block with an invalid id, and change again with a valid id instead of returning the response in Some.some_method class method.
What have I done wrong here?
Ah my bad. I need to reset the error
def somefunc(id: str):
//id = '123'
try:
response = Some().some_method(id)
return self.error, response
except Exception as e:
self.error = True
return error,[]
finally:
self.error = False
I have a python service which sends and receives message from azure service bus queue. As per my setup the received will take long time to do some precess and finally completing the message.
To tackle this scenario added auto lock renewal with max_lock_renewal_duration=3600 and set message lock duration as 1 minute.
when i check the log for the first time the lock renewal worked fine but for the second time it errored out. But when i printed the error it just logging the received message not the error.
Code:
class AzureMessageQueue(MessageQueue):
async def on_renew_error(renewable, error, _):
print("On renew error -\n renewable: ", renewable,"\n error: ", error,"\n type: ", type(error), "\n message: ", error.message)
if type(error) == MessageAlreadySettled:
logger.warn("Message already settled")
else:
logger.warn("Error renewing lock: %s", error)
def __init__(self, conn_str: str, queue_name: str) -> None:
self.sb_client: ServiceBusClient = ServiceBusClient.from_connection_string(
conn_str=conn_str,
logging_enable=True,
retry_total=10,
retry_backoff_factor=1,
)
self.sender: ServiceBusSender = self.sb_client.get_queue_sender(queue_name)
self.receiver: ServiceBusReceiver = self.sb_client.get_queue_receiver(
queue_name
)
self.renewer: AutoLockRenewer = AutoLockRenewer(
max_lock_renewal_duration=3600, on_lock_renew_failure=self.on_renew_error
)
async def send(self, message: str) -> None:
sb_message = ServiceBusMessage(message)
await self.sender.send_messages(sb_message)
async def process(self, processor) -> AsyncIterator:
async with self.receiver:
async for msg in self.receiver:
self.renewer.register(self.receiver, msg)
message = str(msg)
try:
result = await processor(message)
await self.receiver.complete_message(msg)
yield message, None, result
except Exception as e:
yield message, e, None
Log:
On renew error -
renewable: <api.message_queue.AzureMessageQueue object at 0x7fe34bdaea90>
error: {"message": "test"}
type: <class 'azure.servicebus._common.message.ServiceBusReceivedMessage'>
message: {"message": "test"}
i'm trying to understand what caused the issue?
Im using python azure service bus sdk(azure-servicebus~=7.6.0).
Thanks in advance!
I'm new to Azure functions
Wished to know how to return exception from python worker log to the API caller .
In a HTTP Trigger with COSMOS DB binding , on firing an insert call to the binding , if data already exists , it fails with
"System.Private.CoreLib: Exception while executing function: Functions.insertEntityData. Microsoft.Azure.DocumentDB.Core: Entity with the specified id already exists in the system."
How can this message be sent back to the end user ? It is not getting captured anywhere.
def main(req: func.HttpRequest, cosmosdata: func.Out[func.Document]) -> func.HttpResponse:
try:
message = ""
logging.info('Python HTTP trigger function processed a request.')
entity_name = req.route_params['entity']
status_code = 500
payload = req.get_json()
if payload:
try:
logging.info(payload)
resultant = cosmosdata.set(func.Document.from_dict(payload))
logging.info(resultant)
status_code = 200
message = "Insert Successful to %s" % (entity_name)
except Exception as e:
return func.HttpResponse(str(e), status_code=500)
else:
status_code = 400
message = "Please pass data in the POST Request"
except Exception as e:
return func.HttpResponse(str(e), status_code=500)
return func.HttpResponse(message, status_code=500)
The try / catch block is not working because you're using an Output binding to Cosmos Db, which is the one that is failing. However, it also looks weird to me because by default it performs and Upsert operation.
I believe the problem relates to your partition Key defined in the function.json file.
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-cosmosdb-v2#input---python-examples
Im trying to handle errors with Python-Twitter, for example: when I do the below passing a Twitter account that returns a 404 I get the following response...
import twitter
# API connection
api = twitter.Api(consumer_key='...',
consumer_secret='...',
access_token_key='...',
access_token_secret='...')
try:
data = api.GetUser(screen_name='repcorrinebrown')
Except Exception as err:
print(err)
Response:
twitter.error.TwitterError: [{'code': 50, 'message': 'User not found.'}]
how can I iterate through this list on Except
The message property of the exception object should hold the data you are looking for. Try this:
import twitter
# API connection
api = twitter.Api(consumer_key='...',
consumer_secret='...',
access_token_key='...',
access_token_secret='...')
try:
data = api.GetUser(screen_name='repcorrinebrown')
except Exception as err:
print(err.message)
for i in err.message:
print(i)
However, you may want to except the specific exception rather than all exceptions:
except twitter.error.TwitterError as err:
...
For a more specific exception, I would use twitter.TweepError like this
try:
user = api.get_user('user_handle')
except twitter.TweepError as error: #Speficically twitter errors
print(error)