Lambda function not taken into account in tests - python-3.x

I am trying to deploy my first function on AWS Lambda that should store results on my S3 instance following this tutorial. To be sure I have the permission to write files to the S3 bucket I've created I added the code below the commented one into the Function code section.
# import json
# def lambda_handler(event, context):
# # TODO implement
# return {
# 'statusCode': 200,
# 'body': json.dumps('Hello from fucking Lambda!')
# }
import boto3
def lambda_handler(event, context):
s3 = boto3.resource("s3") # pointer to AWS S3 service
bucket_name = "my-own-bucket" # bucket for saving our data
file_name = "HelloWorld.txt" # dummy file
body = event['key1'] + ' ' + event['key2'] + event['key3']
# "Hello World!" from event input
s3.Bucket(bucket_name).put_object(Key=file_name, Body=body)
# write object to bucket
return f"Succeed! Find your new {file_name} file in {bucket_name} bucket ;)"
I used the template test event:
My code automatically saved, so I opened the S3 bucket again and verified that a HelloWorld.txt file existed. But it didn't. And the execution results were:
Response:
{
"statusCode": 200,
"body": "\"Hello from Lambda!\""
}
Request ID:
"93c1c61c-73cf-4308-9a88-0e8771612b8e"
Function logs:
START RequestId: 93c1c61c-73cf-4308-9a88-0e8771612b8e Version: $LATEST
END RequestId: 93c1c61c-73cf-4308-9a88-0e8771612b8e
REPORT RequestId: 93c1c61c-73cf-4308-9a88-0e8771612b8e Duration: 0.32 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 43 MB
Like if the commented code had still been taken into account.

It could be due to following reasons:
You are invoking old lambda version, rather then your latest version.
You are not deploying your changes. In the new UI, there is no 'Save' button. Instead there is orange Deploy button.

Related

AWS Lambda function to enable Default Encryption on bucket creation

I want to write a lambda function with Python, to enable S3 bucket default encryption, if the newly created bucket is not encryption enabled
Need to have following steps
Trigger Lambda function when new S3 bucket is created
If Default encryption is not enabled, it will enable automatically
SNS topic will be triggered and send email to administrator & bucket creator/owner
Following lambda function, I have created will encrypt any existing bucket periodically. I want to extend it to trigger at new bucket creation
import json
import boto3
def lambda_handler(event, context):
s3 = boto3.client("s3")
response = s3.list_buckets()
buckets = [bucket['Name'] for bucket in response['Buckets']]
status = 401
unencrypted_buckets = []
for bucket in buckets:
try:
s3.get_bucket_encryption(Bucket=bucket)
print(f"Bucket {bucket} has already Encryption enabled")
except s3.exceptions.ClientError:
unencrypted_buckets.append(bucket)
encryption_enabled_buckets = []
for unencrypted_bucket in unencrypted_buckets:
try:
print(f"Bucket {unencrypted_bucket} has no Encryption enabled")
s3.put_bucket_encryption(
Bucket=unencrypted_bucket,
ServerSideEncryptionConfiguration={
'Rules': [
{
'ApplyServerSideEncryptionByDefault':
{
'SSEAlgorithm': 'AES256'
}
}
]
}
)
encryption_enabled_buckets.append(unencrypted_bucket)
status = 200
except s3.exceptions.ClientError:
status = 500
break
return {
'statusCode': status,
'details': 'Default encryption enabled',
'encryption enabling success': encryption_enabled_buckets,
'encryption enabling failed': list(set(unencrypted_buckets) - set(encryption_enabled_buckets)) + list(
set(encryption_enabled_buckets) - set(unencrypted_buckets))
}
You may not have to code this at all. Consider using AWS Config Rules for this, and other, compliance requirements.
See AWS Config managed rules:
s3-bucket-server-side-encryption-enabled.html
s3-default-encryption-kms.html
AWS Config can send notifications via SNS and here is an example of How can I be notified when an AWS resource is non-compliant using AWS Config?

Phone book search on AWS Lambda and S3

I want to make a serverless application in AWS Lambda for phone book searches.
What I've done:
Created a bucket and uploaded a CSV file to it.
Created a role with full access to the bucket.
Created a Lambda function
Created API Gateway with GET and POST methods
The Lambda function contains the following code:
import boto3
import json
s3 = boto3.client('s3')
resp = s3.select_object_content(
Bucket='namebbacket',
Key='sample_data.csv',
ExpressionType='SQL',
Expression="SELECT * FROM s3object s where s.\"Name\" = 'Jane'",
InputSerialization = {'CSV': {"FileHeaderInfo": "Use"}, 'CompressionType': 'NONE'},
OutputSerialization = {'CSV': {}},
)
for event in resp['Payload']:
if 'Records' in event:
records = event['Records']['Payload'].decode('utf-8')
print(records)
elif 'Stats' in event:
statsDetails = event['Stats']['Details']
print("Stats details bytesScanned: ")
print(statsDetails['BytesScanned'])
print("Stats details bytesProcessed: ")
print(statsDetails['BytesProcessed'])
print("Stats details bytesReturned: ")
print(statsDetails['BytesReturned'])
When I access the Invoke URL, I get the following error:
{errorMessage = Handler 'lambda_handler' missing on module 'lambda_function', errorType = Runtime.HandlerNotFound}
CSV structure: Name, PhoneNumber, City, Occupation
How to solve this problem?
Please refer to this documentation topic to learn how to write a Lambda function in Python. You are missing the Handler. See: AWS Lambda function handler in Python
Wecome to S.O. #smac2020 links you to the right place AWS Lambda function handler in Python. In short, AWS Lambda needs to know where to find your code, hence the "handler". Though a better way to think about it might be "entry-point."
Here is a close approximation of your function, refactored for use on AWS Lambda:
import json
import boto3
def function_to_be_called(event, context):
# TODO implement
s3 = boto3.client('s3')
resp = s3.select_object_content(
Bucket='stack-exchange',
Key='48836509/dogs.csv',
ExpressionType='SQL',
Expression="SELECT * FROM s3object s where s.\"breen_name\" = 'pug'",
InputSerialization = {'CSV': {"FileHeaderInfo": "Use"}, 'CompressionType': 'NONE'},
OutputSerialization = {'CSV': {}},
)
for event in resp['Payload']:
if 'Records' in event:
records = event['Records']['Payload'].decode('utf-8')
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!'),
'pugInfo': records
}
This function produces the following result:
Response
{
"statusCode": 200,
"body": "\"Hello from Lambda!\"",
"currentWorkdingDirectory": "/var/task",
"currentdirlist": [
"lambda_function.py"
],
"pugInfo": "1,pug,toy\r\n"
}
The "entry point" for this function is in a Python file called lambda_function.py and the function function_to_be_called. Together these are the "handler." We can see this in the Console:
or using the API through Boto3
import boto3
awslambda = boto3.client('lambda')
awslambda.get_function_configuration('s3SelectFunction')
Which returns:
{'CodeSha256': 'mFVVlakisUIIsLstQsJUpeBIeww4QhJjl7wJaXqsJ+Q=',
'CodeSize': 565,
'Description': '',
'FunctionArn': 'arn:aws:lambda:us-east-1:***********:function:s3SelectFunction',
'FunctionName': 's3SelectFunction',
'Handler': 'lambda_function.function_to_be_called',
'LastModified': '2021-03-10T00:57:48.651+0000',
'MemorySize': 128,
'ResponseMetadata': ...
'Version': '$LATEST'}

Python Boto3 put_object file from lambda in s3

I would like to send a json file in s3 from a lambda. I saw in the documentation that we can send with the function boto3 put_object a file or a bytes object (Body=b'bytes'|file).
But if I'm not wrong, if I send a file in s3 with Body=bytes and then I download my file the content will be not visible.
So in my lambda function, I receive messages from a SQS Queue, I created a file with the message content in the lambda temporary folder /tmp. And I want to get this json file to send it in my_bucket/folder/file.json
I saw many examples to create a file in s3 but Body parameter is in bytes and not a file.
This is my code (python3.7)
def alpaca_consent_customer_dev(event, context): # handler
# TODO implement
request_id = context.aws_request_id
print('START - RequestID: {}'.format(request_id))
# function to write json file
def write_json(target_path, target_file, data):
if not os.path.exists(target_path):
try:
os.makedirs(target_path)
except Exception as e:
print(e)
raise
with open(os.path.join(target_path, target_file), 'w') as f:
json.dump(data, f)
try:
s3 = boto3.client('s3', region_name="us-west-2")
request_id = context.aws_request_id
print('START - RequestID: {}'.format(request_id))
# Get message from SQS queue
for record in event['Records']:
data = record
# Get message from SQS
data_loaded = json.loads(data['body'])
sns_message_id = data_loaded['MessageId']
print('data loaded type:', type(data_loaded))
data_saved = json.dumps(data_loaded)
# Create json file in temporary folder
write_json('/tmp', sns_message_id+'.json', data_saved)
# Check if file exists
print(glob.glob("/tmp/*.json"))
# result: ['/tmp/3bb1c0bc-68d5-5c4d-b827-021301.json']
s3.put_object(Body='/tmp/'+sns_message_id + '.json', Bucket='mybucket', Key='my_sub_bucket/' + datetime.datetime.today().strftime('%Y%m%d')+ '/'+ sns_message_id + '.json')
except Exception as e:
raise Exception('ERROR lambda failed: {}'.format(str(e)))
Thanks for your help. Regards.
There's an official example in the boto3 docs:
import logging
import boto3
from botocore.exceptions import ClientError
def upload_file(file_name, bucket, object_name=None):
"""Upload a file to an S3 bucket
:param file_name: File to upload
:param bucket: Bucket to upload to
:param object_name: S3 object name. If not specified then file_name is used
:return: True if file was uploaded, else False
"""
# If S3 object_name was not specified, use file_name
if object_name is None:
object_name = file_name
# Upload the file
s3_client = boto3.client('s3')
try:
response = s3_client.upload_file(file_name, bucket, object_name)
except ClientError as e:
logging.error(e)
return False
return True
You can just use the upload_file method of the s3 client.

AWS Lambda Function Boto 3 filter_log_events Cannot Use endTime Parameter

I'm trying to extract information from Cloudwatch logs to send a more customised email from an alert based on a metric. I'm creating a lambda function to try and extract information at a specified timeframe. However, when I specify a startTime and endTime, nothing comes back in the repsonse. I've looked at the documentation for Boto3 but it doesn't say much. I'm relatively new to python and AWS so any help would be great. An example of the code can be seen below:
import boto3
import json
import time
from datetime import datetime
from calendar import timegm
# Create CloudWatch client
clw = boto3.client('logs')
def lambda_handler(event, context):
# User defined for testing purposes
name = 'Error Log Metric'
namespace = 'User Defined Metrics'
response = clw.describe_metric_filters(metricName=name,
metricNamespace=namespace)
LogGroupName = response['metricFilters'][0]['logGroupName']
FilterPattern = response['metricFilters'][0]['filterPattern']
StartTime = timegm(time.strptime('2020-06-15T00:00:00.000Z',
'%Y-%m-%dT%H:%M:%S.%fZ'))
EndTime = timegm(time.strptime('2020-06-16T23:59:59.000Z',
'%Y-%m-%dT%H:%M:%S.%fZ'))
filteredLogs = clw.filter_log_events(logGroupName=LogGroupName,
filterPattern=FilterPattern,
logStreamNamePrefix='TEST_PREFIX_NAME',
startTime=StartTime, endTime=EndTime)
print(filteredLogs)
Response
Response:
null
Request ID:
"610a2849-3fef-46b2-b75e-450c4f37ec25"
Function Logs:
START RequestId: 610a2849-3fef-46b2-b75e-450c4f37ec25 Version: $LATEST
{'events': [], 'searchedLogStreams': [], 'ResponseMetadata': {'RequestId': '04d59cfe-9069-4bb4-ad3b-7135a649d2e6', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '04d59cfe-9069-4bb4-ad3b-7135a649d2e6', 'content-type': 'application/x-amz-json-1.1', 'content-length': '121', 'date': 'Tue, 16 Jun 2020 00:42:23 GMT'}, 'RetryAttempts': 0}}
END RequestId: 610a2849-3fef-46b2-b75e-450c4f37ec25
REPORT RequestId: 610a2849-3fef-46b2-b75e-450c4f37ec25 Duration: 299.88 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 71 MB Init Duration: 315.90 ms
I think the issue is that timegm returns timestamps in seconds, not milliseconds as required by filter_log_events.
Assuming everything else is correct (all log streams names, dates, prefix), a quick fix could be:
filteredLogs = clw.filter_log_events(logGroupName=LogGroupName,
filterPattern=FilterPattern,
logStreamNamePrefix='TEST_PREFIX_NAME',
startTime=StartTime*1000, endTime=EndTime*1000)

get aws lambda to start transcribe job with filename without extension

I have a lambda function which will start a transcribe job when an object is put into the s3 bucket. I am having trouble getting setting the transcribe job to be the file name without the extension; also the file is not putting into the correct prefix folder in S3 bucket for some reason, here's what I have:
import json
import boto3
import time
import os
from urllib.request import urlopen
transcribe = boto3.client('transcribe')
def lambda_handler(event, context):
if event:
file_obj = event["Records"][0]
bucket_name = str(file_obj['s3']['bucket']['name'])
file_name = str(file_obj['s3']['object']['key'])
s3_uri = create_uri(bucket_name, file_name)
job_name = filename
print(os.path.splitext(file_name)[0])
transcribe.start_transcription_job(TranscriptionJobName = job_name,
Media = {'MediaFileUri': s3_uri},
MediaFormat = 'mp3',
LanguageCode = "en-US",
OutputBucketName = "sbox-digirepo-transcribe-us-east-1",
Settings={
# 'VocabularyName': 'string',
'ShowSpeakerLabels': True,
'MaxSpeakerLabels': 2,
'ChannelIdentification': False
})
while Ture:
status = transcribe.get_transcription_job(TranscriptionJobName=job_name)
if status["TranscriptionJob"]["TranscriptionJobStatus"] in ["COMPLETED", "FAILED"]:
break
print("Transcription in progress")
time.sleep(5)
s3.put_object(Bucket = bucket_name, Key="output/{}.json".format(job_name), Body=load_)
return {
'statusCode': 200,
'body': json.dumps('Transcription job created!')
}
def create_uri(bucket_name, file_name):
return "s3://"+bucket_name+"/"+file_name
the error i get is
[ERROR] BadRequestException: An error occurred (BadRequestException) when calling the StartTranscriptionJob operation: 1 validation error detected: Value 'input/7800533A.mp3' at 'transcriptionJobName' failed to satisfy constraint: Member must satisfy regular expression pattern: ^[0-9a-zA-Z._-]+
so my desired output should have the TranscriptionJobName value to be 7800533A for this case, and the result OutputBucketName to be in s3bucket/output. any help is appreciated, thanks in advance.
The TranscriptionJobName argument is a friendly name for your job and is pretty limited based on the regex. You're passing it the full object key, which contains the prefix input/, but / is a disallowed character in the job name. You could just split out the file name portion in your code:
job_name = file_name.split('/')[-1]
I put a full example of uploading media and starting an AWS Transcribe job on GitHub that puts all this in context.

Resources