I'm getting a Boto3 InvalidParameterException while running the lambda function.
I'm trying to find out a way to handle this exception.
I came across the below solution:
from boto.exception import BotoServerError
class InvalidParameterException(BotoServerError):
pass
I'm using python3 and understood that boto is deprecated now and is replaced by boto3.
But i could not find an equivalent solution in boto3.
Can anyone help me out with this ?
As boto is deprecated all the modeled exceptions are available on the client. You can look it up same in the API docs as well , basically the code for the boto3 is straight away generated from the APIs. Earlier approach with boto was hard coded stuff and writing code for the same.
As you can see here
For example
import boto3
from botocore.exceptions import ClientError
def get_secret():
secret_name = "MySecretName"
region_name = "us-west-2"
session = boto3.session.Session()
client = session.client(
service_name='secretsmanager',
region_name=region_name,
)
try:
get_secret_value_response = client.get_secret_value(
SecretId=secret_name
)
except ClientError as e:
if e.response['Error']['Code'] == 'ResourceNotFoundException':
print("The requested secret " + secret_name + " was not found")
elif e.response['Error']['Code'] == 'InvalidRequestException':
print("The request was invalid due to:", e)
elif e.response['Error']['Code'] == 'InvalidParameterException':
print("The request had invalid params:", e)
elif e.response['Error']['Code'] == 'DecryptionFailure':
print("The requested secret can't be decrypted using the provided KMS key:", e)
elif e.response['Error']['Code'] == 'InternalServiceError':
print("An error occurred on service side:", e)
AWS Secrets Manager Example From the docss
How to handle errors with boto3
Related
Python3.7
I have the following GET endpoint to return the data in the database, but keeps getting
{
"code": 400,
"message": "Exception in _query: <sqlite3.Row object at 0x10fcbb4b0> is not JSON serializable"
}
Here is the code. I have been trying many different solutions but none of them works. Any thoughts?
Any way I can printout the sqlite3 data? When I do
print(entries)
I will return
as well. Any thoughts? Thank you!
# all the imports
import os
import sqlite3
from flask import Flask, request, session, g, redirect, url_for, abort, \
render_template, flash
from .response import Response
app = Flask(__name__) # create the application instance :)
app.config.from_object(__name__) # load config from this file , flaskr.py
import json
import requests
#app.route('/table_result', methods=['GET'])
def table_result():
try:
db = get_db()
cur = db.execute("SELECT name FROM mouse_tracking; ")
entries = cur.fetchall()
# return Response(200, json.dumps(entries)).payload()
return json.dumps(entries)
except sqlite3.Error as e:
return Response(400, "Database error: %s" % e).payload()
except Exception as e:
return Response(400, "Exception in _query: %s" % e).payload()
(i cant comment yet) did you use cursor?
def table():
try:
db = sqlite3.connect('db.db')
cursor = db.cursor()
cursor.execute("SELECT name FROM db")
entries = cursor.fetchall()
return jsonify(json.dumps(entries))
except Exception as e:
return Response(400, "Exception: %s" % e).payload()
you should use db.cursor() to execute commands on the database
I am trying to create bucket using aws python boto 3.
Here is my code:-
import boto3
response = S3_CLIENT.create_bucket(
Bucket='symbols3arg',
CreateBucketConfiguration={'LocationConstraint': 'eu-west-1'}
)
print(response)
I am getting below error:-
botocore.exceptions.ClientError: An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to.
This happens you configured a different region during aws configure in specifying a different region in s3 client object initiation.
Suppose my AWS config look like
$ aws configure
AWS Access Key ID [None]: AKIAIOSFODEXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json
and my python script for creating bucket
import logging
import boto3
from botocore.exceptions import ClientError
def create_bucket(bucket_name, region=None):
# Create bucket
try:
if region is None:
s3_client = boto3.client('s3')
s3_client.create_bucket(Bucket=bucket_name)
else:
s3_client = boto3.client('s3')
location = {'LocationConstraint': region}
s3_client.create_bucket(Bucket=bucket_name,
CreateBucketConfiguration=location)
except ClientError as e:
logging.error(e)
return False
return True
create_bucket("test-bucket-in-region","us-west-1")
This will throw the below error
ERROR:root:An error occurred (IllegalLocationConstraintException) when calling the CreateBucket operation: The us-west-1 location constraint is incompatible for the region specific endpoint this request was sent to.
To solve this issue all you need to specify the region in s3 client object initiation. A working example in different region regardless of aws configure
import logging
import boto3
from botocore.exceptions import ClientError
def create_bucket(bucket_name, region=None):
"""Create an S3 bucket in a specified region
If a region is not specified, the bucket is created in the S3 default
region (us-east-1).
:param bucket_name: Bucket to create
:param region: String region to create bucket in, e.g., 'us-west-2'
:return: True if bucket created, else False
"""
# Create bucket
try:
if region is None:
s3_client = boto3.client('s3')
s3_client.create_bucket(Bucket=bucket_name)
else:
s3_client = boto3.client('s3', region_name=region)
location = {'LocationConstraint': region}
s3_client.create_bucket(Bucket=bucket_name,
CreateBucketConfiguration=location)
except ClientError as e:
logging.error(e)
return False
return True
create_bucket("my-working-bucket","us-west-1")
create-an-amazon-s3-bucket
Send the command to S3 in the same region:
import boto3
s3_client = boto3.client('s3', region_name='eu-west-1')
response = s3_client.create_bucket(
Bucket='symbols3arg',
CreateBucketConfiguration={'LocationConstraint': 'eu-west-1'}
)
You can try the following code.
import boto3
client = boto3.client('s3',region_name="aws_region_code")
response = client.create_bucket(
Bucket='string'
)
Hope, it might helps.
I am trying to pass my secrets value to my SSM document from my lambda function. Though I am able to read from my lambda's output - I am not able to put it into my document to call it as a variable. Please Suggest.
import boto3 # Required to interact with AWS
import json # Required for return object parsing
from botocore.exceptions import ClientError
# Set required variables
secret_name = "***/***/***"
endpoint_url = "https://secretsmanager.eu-west-1.amazonaws.com"
region_name = "eu-west-1"
session = boto3.session.Session()
client = session.client(
service_name='secretsmanager',
region_name=region_name,
endpoint_url=endpoint_url
)
try:
get_secret_value_response = client.get_secret_value(
SecretId=secret_name
)
except ClientError as e:
if e.response['Error']['Code'] == 'ResourceNotFoundException':
print("The requested secret " + secret_name + " was not found")
elif e.response['Error']['Code'] == 'InvalidRequestException':
print("The request was invalid due to:", e)
elif e.response['Error']['Code'] == 'InvalidParameterException':
print("The request had invalid params:", e)
else:
# Decrypted secret using the associated KMS CMK
# Depending on whether the secret was a string or binary, one of these fields will be populated
if 'SecretString' in get_secret_value_response:
secret = json.loads(get_secret_value_response['SecretString'])
else:
binary_secret_data = get_secret_value_response['SecretBinary']
access_key = secret['AWS_ACCESS_KEY_ID']
secret_key = secret['AWS_SECRET_ACCESS_KEY']
region = secret['AWS_DEFAULT_REGION']
ssm = boto3.client('ssm')
ec2 = boto3.resource('ec2')
def lambda_handler(event, context):
running_with = []
running_without = []
for instance in ec2.instances.all():
if instance.state['Name'] != 'running':
continue
has_tag = False
for tag in instance.tags:
if tag['Key'] == 'AutoDiskGrowth' and tag['Value'] == 'True':
has_tag = True
break
if has_tag:
running_with.append(instance.id)
else:
running_without.append(instance.id)
print("access_key: %s" % access_key)
print("Instances found with AutoDiskGrowth Tag: %s" % running_with)
print("Instances without AutoDiskGrowth Tag: %s" % running_without)
ssmCommand = ssm.send_command(
Targets = [
{'Key': 'tag:AutoDiskGrowth',
'Values': [
'True']
}
],
DocumentName = 'Secrets_Management',
TimeoutSeconds = 6000,
Comment = 'Extending disk volume by 50%',
Parameters={
'AWS_ACCESS_KEY_ID': [
'secret_key',
]
}
)
Here, in the above print secret_key, I am able to see the value of the secret stored. But I need it to be sent to the Secrets_Management document as a variable.Here's what I get when I run this.
Current Output:
Response:
{
"errorMessage": "An error occurred (InvalidParameters) when calling the SendCommand operation: ",
"errorType": "InvalidParameters",
The InvalidParameters response indicates that your code is not sending the Parameters expected by the document.
If you go to the document in the Systems Manager console and look in the Parameters tab, you will see the list of Parameters that are permitted. They are case-sensitive.
just learning python mocking in general and struggling with using Magicmock and pytest with boto3.
Here is my code block
def upload_to_s3(self, local_file, bucket, dest_file):
self.local_file = local_file
self.bucket = bucket
self.dest_file = dest_file
s3_client = self.prime_s3_client() # this method returns the boto3 client
try:
s3_client.upload_file(local_file, bucket, dest_file)
LOG_IT.info('File uploaded to S3 from: %s to %s.', local_file, dest_file)
except Exception:
LOG_IT.critical('The %s failed to upload to S3.', local_file)
This is the test that's not working:
def test_upload_to_s3(self, monkeypatch, aws):
mock_s3_client = MagicMock()
monkeypatch.setattr(boto3, 'client', mock_s3_client)
mock_upload_file = MagicMock()
monkeypatch.setattr(mock_s3_client, 'upload_file', mock_upload_file)
push_to_s3 = aws.upload_to_s3('localfile', 'chumbucket', 'destfile')
mock_upload_file.assert_called()
The error returned:
E AssertionError: Expected 'upload_file' to have been called.
Thank you!
I am fetch all child account from the Master AWS Account by boto3 Organization.
Code is working fine. I am able to get child account list.
But if you run my AWS Lambda function again then it fail to get Child Accounts.
Getting following error:
Error while getting AWS Accounts : An error occurred (TooManyRequestsException) when calling the ListAccounts operation: AWS Organizations can't complete your request because another request is already in progress. Try again later
After 20 to 30 minutes, I can see my code work for once and again raise above exception.
I am Run this code by AWS Gateway + AWS Lambda.
Any idea?
Code:
import boto3
class Organizations(object):
"""AWS Organization"""
def __init__(self, access_key, secret_access_key, session_token=None):
self.client = boto3.client('organizations',
aws_access_key_id=access_key,
aws_secret_access_key=secret_access_key,
aws_session_token=session_token
)
def get_accounts(self, next_token=None, max_results=None):
"""Get Accounts List"""
if next_token and max_results:
result = self.client.list_accounts(NextToken=next_token,
MaxResults=max_results)
elif next_token:
result = self.client.list_accounts(NextToken=next_token)
elif max_results:
result = self.client.list_accounts(MaxResults=max_results)
else:
result = self.client.list_accounts()
return result
class AWSAccounts(object):
""" Return AWS Accounts information. """
def get_aws_accounts(self, access_key, secret_access_key, session_token):
""" Return List of AWS account Details."""
org_obj = Organizations(access_key=access_key,
secret_access_key=secret_access_key,
session_token=session_token)
aws_accounts = []
next_token = None
next_result = None
while True:
response = org_obj.get_accounts(next_token, next_result)
for account in response['Accounts']:
account_details = {"name": account["Name"],
"id": account["Id"],
"admin_role_name": self.account_role_name
}
aws_accounts.append(account_details)
if "NextToken" not in response:
break
next_token = response["NextToken"]
return aws_accounts
By Exception Handling, my code is running successfully.
Catch TooManyRequestsException exception by ClientError exception and retry to call AWS list_accounts API by boto3.
We can add time sleep of 0.1 seconds.
Code:
class AWSAccounts(object):
""" Return AWS Accounts information. """
def get_accounts(self, next_token=None, max_results=None):
"""Get Accounts List"""
# If Master AWS account contain more child accounts(150+) then
# Too-Many-Request Exception is raised by the AWS API(boto3).
# So to fix this issue, we are calling API again by Exception Handling.
result = None
while True:
try:
if next_token and max_results:
result = self.client.list_accounts(NextToken=next_token,
MaxResults=max_results)
elif next_token:
result = self.client.list_accounts(NextToken=next_token)
elif max_results:
result = self.client.list_accounts(MaxResults=max_results)
else:
result = self.client.list_accounts()
except botocore.exceptions.ClientError as err:
response = err.response
print("Failed to list accounts:", response)
if (response and response.get("Error", {}).get("Code") ==
"TooManyRequestsException"):
print("Continue for TooManyRequestsException exception.")
continue
break
return result
Configure your boto3 client to use the built-in standard retry mode:
import boto3
from botocore.config import Config
config = Config(
retries = {
'max_attempts': 10,
'mode': 'standard'
}
)
ec2 = boto3.client('ec2', config=config)
Per the documentation, the default mode is 'legacy' which doesn't handle TooManyRequestsException.
See boto3 documentation about retry configuration here: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html