AWS Lambda, saving S3 file to /tmp directory - python-3.x

I want to copy a set of files over from S3, and put them in the /tmp directory while my lambda function is running, to use and manipulate the contents. The following code excerpt works fine on my PC (which is running windows)
s3 = boto3.resource('s3')
BUCKET_NAME = 'car_sentiment'
keys = ['automated.csv', 'connected_automated.csv', 'connected.csv',
'summary.csv']
for KEY in keys:
try:
local_file_name = 'tmp/'+KEY
s3.Bucket(BUCKET_NAME).download_file(KEY, local_file_name)
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == "404":
continue
else:
raise
However, when I try to run on AWS lambda, I get:
{
"errorMessage": "[Errno 2] No such file or directory: 'tmp/automated.csv.4Bcd0bB9'",
"errorType": "FileNotFoundError",
"stackTrace": [
[
"/var/task/SentimentForAWS.py",
28,
"my_handler",
"s3.Bucket(BUCKET_NAME).download_file(KEY, local_file_name)"
],
[
"/var/runtime/boto3/s3/inject.py",
246,
"bucket_download_file",
"ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)"
],
[
"/var/runtime/boto3/s3/inject.py",
172,
"download_file",
"extra_args=ExtraArgs, callback=Callback)"
],
[
"/var/runtime/boto3/s3/transfer.py",
307,
"download_file",
"future.result()"
],
[
"/var/runtime/s3transfer/futures.py",
73,
"result",
"return self._coordinator.result()"
],
[
"/var/runtime/s3transfer/futures.py",
233,
"result",
"raise self._exception"
],
[
"/var/runtime/s3transfer/tasks.py",
126,
"__call__",
"return self._execute_main(kwargs)"
],
[
"/var/runtime/s3transfer/tasks.py",
150,
"_execute_main",
"return_value = self._main(**kwargs)"
],
[
"/var/runtime/s3transfer/download.py",
582,
"_main",
"fileobj.seek(offset)"
],
[
"/var/runtime/s3transfer/utils.py",
335,
"seek",
"self._open_if_needed()"
],
[
"/var/runtime/s3transfer/utils.py",
318,
"_open_if_needed",
"self._fileobj = self._open_function(self._filename, self._mode)"
],
[
"/var/runtime/s3transfer/utils.py",
244,
"open",
"return open(filename, mode)"
]
]
}
Why does it think the file name is tmp/automated.csv.4Bcd0bB9 rather than just tmp/automated.csv and how do I fix it? Been pulling my hair out on this one, trying multiple approaches, some of which generate a similar error when running locally on my PC. Thanks!

You should save in /tmp, rather than tmp/.
eg:
local_file_name = '/tmp/' + KEY

Well the reason why lambda gives the above error is because it doesn't allow to write in the heirarchial structure with in the /tmp/ directory. You can write the files directly to /tmp/example.txt dir but not to say /tmp/dir1/example.txt

Related

Can Azure Read Handle Equations

Can Microsoft Azure's Read API read equations and if so, what form would it return them in? I don't want to build an equation OCR myself for my app and would like to know if there is an Azure (or maybe AWS/GCP, but I would prefer Azure) service that can read images containing hand-draw equations.
Extra info
language: I am using python to build the app
pricing: free tier/open source, but up to $60 would still be expectable
usage: I don't think that the app that I am building will get more than 1K requests before I shut it down (it's basically a proof-of-concept app)
There is a way to solve this problem. If the application you are running is a web app or mobile app, we have predefined read API based computer vision portal designed for math operations recognition.
Reason: the read API is working fine for the normal form contents detection, as there are pre-defined structures for the forms. But there is no pre-defined structure for the equations.
In the same case if we need to solve the equation based (Math operations), we need not create OCR. As the application you are working with is python based, directly navigate the application for vision studio which will detect the handwritten equations and give the result. The result can be acquired in the form of JSON.
I have a handwritten image with equation on it.
Go to : https://portal.vision.cognitive.azure.com/gallery/ocr
Click on the only option available
We can take live pictures or upload images. I reproduced the issue by uploading the equation image
It detected successfully
[
{
"page": 1,
"angle": -0.5468,
"width": 1840,
"height": 1034,
"unit": "pixel",
"lines": [
{
"boundingBox": [
250,
399,
1612,
392,
1615,
552,
253,
571
],
"appearance": {
"style": {
"name": "handwriting",
"confidence": 1
}
},
"text": "(a+b) = a2+2ab+b2",
"words": [
{
"boundingBox": [
251,
404,
644,
394,
654,
559,
261,
572
],
"text": "(a+b)",
"confidence": 0.764
},
{
"boundingBox": [
674,
394,
785,
393,
794,
554,
684,
558
],
"text": "=",
"confidence": 0.619
},
{
"boundingBox": [
815,
392,
1594,
406,
1602,
530,
824,
553
],
"text": "a2+2ab+b2",
"confidence": 0.694
}
]
}
]
}
]
The above is the JSON response.

discord py roles creation with permissions

I have been working on this as a side project for like 2 months and for the life of me I can not get the bot to create roles with permissions. Here is what I have.
levels={
"Admin":{"name":"ADMIN","hoist":"1","colour":"0x6F0A0A","permissions":""},
"Moderator":{"name":"WATCHMEN","hoist":"1","colour":"0xFF611A","permissions":"discord.permissions(manage_roles=True,kick_members=True,ban_members=True,create_instant_invite=True,mention_everyone=True,change_nickname=True,manage_nicknames=True,read_message_history,send_messages=True,embed_links=True,send_tts_messages,attach_files=True,external_emojis=True,add-reactions=True)"},
"Member":{"name":"MEMBER","hoist":"0","colour":"0x52D41A","permissions":"discord.permissions(send_messages=True)"},
"Verify":{"name":"VERIFY","hoist":"1","colour":"0xFFFFFF","permissions":"discord.permissions(send_messages=True)"},
}
and
async def cook_roles(ctx):
for level in levels.keys():
guild=ctx.guild
name=levels[level]['name']
hoist=levels[level]['hoist']
colour=levels[level]['colour']
# perms=levels[level]['permissions']
if name == "Admin":
perms=discord.Permissions.all()
else:
perms=discord.Permissions(levels[level]['permissions'])
print=(perms)
await ctx.send(perms)
await guild.create_role(name=name,hoist=hoist,permissions=perms,colour=discord.Colour(int(colour,16)))
Any help is appriciated!
I tried taking away the discord.Permissions() and formatting in perms like this
perms=discord.Permissions(levles[level]['permissions'])
but that didn't work either. (I have tried a host of things and just haven't figured it out.)
Here is a traceback for the first provided answer:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/discord/ext/commands/core.py", line 83, in wrapped
ret = await coro(*args, **kwargs)
File "ny.py", line 483, in build
await cook_roles(ctx)
File "ny.py", line 551, in cook_roles
await guild.create_role(name=name,hoist=hoist,permissions=perms,colour=discord.Colour(int(colour,16)))
TypeError: int() can't convert non-string with explicit base
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/discord/ext/commands/bot.py", line 892, in invoke
await ctx.command.invoke(ctx)
File "/usr/local/lib/python3.6/dist-packages/discord/ext/commands/core.py", line 797, in invoke
await injected(*ctx.args, **ctx.kwargs)
File "/usr/local/lib/python3.6/dist-packages/discord/ext/commands/core.py", line 92, in wrapped
raise CommandInvokeError(exc) from exc
discord.ext.commands.errors.CommandInvokeError: Command raised an exception: TypeError: int() can't convert non-string with explicit base
You have a string values in dict, which is not considered as real objects.
You can store any type objects in dicts, so all you need to to do, basically, is to make values in dict actual objects with actual types.
You also was trying to use permissions module instead of Permissions class.
And always make sure that permissions names exists: discord.Permissions (your mistake was in some missing =True and add-reactions instead of add_reactions)
levels = {
"Admin": {"name": "ADMIN", "hoist": True, "colour": 0x6F0A0A, "permissions": discord.Permissions()},
"Moderator": {
"name": "WATCHMEN",
"hoist": True,
"colour": 0xFF611A,
"permissions": discord.Permissions(manage_roles=True,
kick_members=True,
ban_members=True,
create_instant_invite=True,
mention_everyone=True,
change_nickname=True,
manage_nicknames=True,
read_message_history=True,
send_messages=True,
embed_links=True,
send_tts_messages=True,
attach_files=True,
external_emojis=True,
add_reactions=True),
},
"Member": {
"name": "MEMBER",
"hoist": False,
"colour": 0x52D41A,
"permissions": discord.Permissions(send_messages=True),
},
"Verify": {
"name": "VERIFY",
"hoist": True,
"colour": 0xFFFFFF,
"permissions": discord.Permissions(send_messages=True),
},
}

ec2client.describe_instances returns UnauthorizedOperation after adding condition to IAM policy

My goals is to restrict access to ec2 using tag key. It works fine if I remove the condition from the IAM policy. However, if I add the aws:TagKeys condition then I get UnauthorizedOperation error. Need some assistance in fixing the IAM policy or either the code to work with tagkey.
Here's the IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstances",
"ec2:DescribeKeyPairs"
],
"Resource": "*",
"Condition": {
"ForAnyValue:StringEquals": {
"aws:TagKeys": "mytag"
}
}
}
]
}
Here's my python code:
import os
import boto3
import json
os.environ['AWS_DEFAULT_REGION'] = 'ap-south-1'
os.environ['AWS_ACCESS_KEY_ID'] = 'myacceskey'
os.environ['AWS_SECRET_ACCESS_KEY'] = 'secret'
def list_instances_by_tag_value(tagkey, tagvalue):
# When passed a tag key, tag value this will return a list of InstanceIds that were found.
ipdict={}
ec2client = boto3.client('ec2')
#response = ec2client.describe_key_pairs()
#print(response)
response = ec2client.describe_instances(
Filters=[
{
'Name':'tag:' + tagkey,
'Values':[tagvalue]
}
]
)
client_dict = {}
for reservation in (response["Reservations"]):
print(reservation)
#boto3.set_stream_logger(name='botocore')
output = list_instances_by_tag_value("mytag", "abcd")
Here's the exception:
Traceback (most recent call last):
File "test.py", line 29, in <module>
output = list_instances_by_tag_value("mytag", "abcd")
File "test.py", line 20, in list_instances_by_tag_value
'Values':[tagvalue]
File "C:\python35\lib\site-packages\botocore\client.py", line 272, in _api_call
return self._make_api_call(operation_name, kwargs)
File "C:\python35\lib\site-packages\botocore\client.py", line 576, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnauthorizedOperation) when calling the DescribeInstances operation: You are not authorized to perform this operation.
I have checked that tagkey is supported by describeinstances - https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html
Also checked couple of SO threads after which I changed my action to very specific DescribeInstances from Describe*
But its still not working for me.
Got it: Why does applying a condition to ec2:DescribeInstances in an IAM policy fail?
DescribeInstances does not support resource level permissions

Google Adwords script runs on AWS Lambda:[Errno 30] Read-only file system: '/home/sbx_user1051'

A python script fetch google adwords report, it working as expected in my local machine, but when deployed in AWS Lambda function, I got the following error
{
"errorMessage": "[Errno 30] Read-only file system: '/home/sbx_user1051'",
"errorType": "OSError",
"stackTrace": [
[
"/var/task/lambda_function.py",
24,
"lambda_handler",
"report_downloader = client.GetReportDownloader(version='v201809')"
],
[
"/var/task/googleads/adwords.py",
370,
"GetReportDownloader",
"return ReportDownloader(self, version, server)"
],
[
"/var/task/googleads/adwords.py",
1213,
"__init__",
"self.proxy_config, self._namespace, self._adwords_client.cache)"
],
[
"/var/task/googleads/common.py",
819,
"__init__",
"transport = _ZeepProxyTransport(timeout, proxy_config, cache)"
],
[
"/var/task/googleads/common.py",
667,
"__init__",
"cache = zeep.cache.SqliteCache()"
],
[
"/var/task/zeep/cache.py",
77,
"__init__",
"self._db_path = path if path else _get_default_cache_path()"
],
[
"/var/task/zeep/cache.py",
155,
"_get_default_cache_path",
"os.makedirs(path)"
],
[
"/var/lang/lib/python3.6/os.py",
210,
"makedirs",
"makedirs(head, mode, exist_ok)"
],
[
"/var/lang/lib/python3.6/os.py",
210,
"makedirs",
"makedirs(head, mode, exist_ok)"
],
[
"/var/lang/lib/python3.6/os.py",
220,
"makedirs",
"mkdir(name, mode)"
]
]
}
I know in Lambda it could only write files located in tem folder, but what confused me is that in my script, I don't write to any file at all, here is the main structure of my script:
client = adwords.AdWordsClient.LoadFromStorage('tmp/googleads.yaml')
report_downloader = client.GetReportDownloader(version='v201809')
report_query = (adwords.ReportQueryBuilder()
.Select( str)
.From('ACCOUNT_PERFORMANCE_REPORT')
.During('LAST_7_DAYS')
.Build())
results=report_downloader.DownloadReportAsStringWithAwql( report_query, 'TSV', skip_report_header=True, skip_column_header=True, skip_report_summary=True, include_zero_impressions=False)
campaigns=results.splitlines()
Please advise how to fix this issue. The env is python 3.6
It looks like Adwords is using a cache and, by default, that cache goes into the home directory of the user running your code. To fix this, set the environment variable XDG_CACHE_HOME to /tmp/.cache. You can set this in the Lambda environment variables.

boto3 - AWS lambda -copy files between buckets

I am trying to copy multiple files in a source bucket to a destination bucket using AWS lambda and am getting the error below. Bucket structures are as follows
Source Buckets
mysrcbucket/Input/daily/acctno_pin_xref/ABC_ACCTNO_PIN_XREF_FULL_20170926_0.csv.gz
mysrcbucket/Input/daily/acctno_pin_xref/ABC_ACCTNO_PIN_XREF_FULL_20170926_1.csv.gz
mysrcbucket/Input/daily/acctno_pin_xref/ABC_ACCTNO_PIN_XREF_count_20170926.inf
Destination Buckets
mydestbucket/Input/daily/acctno_pin_xref/ABC_ACCTNO_PIN_XREF_FULL_20170926_0.csv.gz
mydestbucket/Input/daily/acctno_pin_xref/ABC_ACCTNO_PIN_XREF_FULL_20170926_1.csv.gz
mydestbucket/Input/daily/acctno_pin_xref/ABC_ACCTNO_PIN_XREF_count_20170926.inf
I wrote the lambda function below but am getting the error below. Can someone help me explain what I am doing wrong
{ "errorMessage": "expected string or bytes-like object", "errorType": "TypeError", "stackTrace": [
[
"/var/task/lambda_function.py",
17,
"lambda_handler",
"s3.Object(dest_bucket,dest_key).copy_from(CopySource= { 'Bucket': obj.bucket_name , 'Key' : obj.key})"
],
[
"/var/runtime/boto3/resources/factory.py",
520,
"do_action",
"response = action(self, *args, **kwargs)"
],
[
"/var/runtime/boto3/resources/action.py",
83,
"__call__",
"response = getattr(parent.meta.client, operation_name)(**params)"
],
[
"/var/runtime/botocore/client.py",
312,
"_api_call",
"return self._make_api_call(operation_name, kwargs)"
],
[
"/var/runtime/botocore/client.py",
575,
"_make_api_call",
"api_params, operation_model, context=request_context)"
],
[
"/var/runtime/botocore/client.py",
627,
"_convert_to_request_dict",
"params=api_params, model=operation_model, context=context)"
],
[
"/var/runtime/botocore/hooks.py",
227,
"emit",
"return self._emit(event_name, kwargs)"
],
[
"/var/runtime/botocore/hooks.py",
210,
"_emit",
"response = handler(**kwargs)"
],
[
"/var/runtime/botocore/handlers.py",
208,
"validate_bucket_name",
"if VALID_BUCKET.search(bucket) is None:"
] ] }
Lambda Function Code
import boto3
import json
s3 = boto3.resource('s3')
def lambda_handler (event, context):
bucket = s3.Bucket('mysrcbucket')
dest_bucket=s3.Bucket('mydestbucket')
print(bucket)
print(dest_bucket)
for obj in bucket.objects.filter(Prefix='Input/daily/acctno_pin_xref/ABC_ACCTNO_PIN_XREF',Delimiter='/'):
dest_key=obj.key
print(dest_key)
s3.Object(dest_bucket,dest_key).copy_from(CopySource= { 'Bucket': obj.bucket_name , 'Key' : obj.key})
The issue is with:
s3.Object(dest_bucket, dest_key).copy_from(CopySource= {'Bucket': obj.bucket_name,
'Key': obj.key})
change dest_bucket to dest_bucket.name:
s3.Object(dest_bucket.name, dest_key).copy_from(CopySource= {'Bucket': obj.bucket_name,
'Key': obj.key})
dest_bucket is a resource and name is its identifier.

Resources