I have a serverless code in python. I am using serverless-python-requirements:^4.3.0 to deploy this into AWS lambda.
My code imports another python file in same directory as itself, which is throwing an error.
serverless.yml:
functions:
hello:
handler: functions/pleasework.handle_event
memorySize: 128
tags:
Name: HelloWorld
Environment: Ops
package:
include:
- functions/pleasework
- functions/__init__.py
- functions/config
(venv) ➜ functions git:(master) ✗ ls
__init__.py boto_client_provider.py config.py handler.py sns_publish.py
__pycache__ cloudtrail_handler.py glue_handler.py pleasework.py
As you can see, pleasework.py and config are in same folder, but when I do import config in pleasework I get an error:
{
"errorMessage": "Unable to import module 'functions/pleasework': No module named 'config'",
"errorType": "Runtime.ImportModuleError"
}
I am struggling with this for few days and think I am missing something basic.
import boto3
import config
def handle_event(event, context):
print('lol: ')
ok, so i found out my isssue. Way i was importing the file was wrong
Instead of
import config
I should be doing
import functions.config
#Pranay Sharma's answer worked for me.
An alternate way is creating and setting PYTHONPATH environment variable to the directory where your handler function and config exist.
To set environment variables in the Lambda console
Open the Functions page of the Lambda console.
Choose a function.
Under Environment variables, choose Edit.
Choose Add environment variable.
Enter a key and value.
In our case Key is "PYTHONPATH" and value is "functions"
Related
Some variables had to be moved from the main server.py in to a configuration file called config.py.
Its contents are something like this:
class Config(object):
#All the settings and variables go here
In server.py, in invoke this file with:
app = Flask(__name__)
app.config.from_object("config.py")
However, i get this error:
werkzeug.utils.ImportStringError: import_string() failed for 'config.py'. Possible reasons are:
- missing __init__.py in a package;
- package or module path not included in sys.path;
- duplicated package or module name taking precedence in sys.path;
- missing module, class, function or variable;
Debugged import:
- 'config' found in '/mnt/io_files/config.py'.
- 'config.py' not found.
Original exception:
ImportError: module 'config' has no attribute 'py'
I don't know what to do in order to properly invoke this file. In the same directory, i do have a __init__.py file.
This is what worked eventually (i don't know what was going on previously):
#server.py
app = Flask(__name__)
app.config.from_object("config")
#config.py
class Config(object):
#String variables go here
I am working on a project using serverless framework and python. I've used serverless-python-requirements plugins and it still giving me an error.
The deployment is fine, but every time I trigger the function it gives me this error:
[ERROR] FileNotFoundError: [Errno 2] No such file or directory: '/tmp/_temp-sls-py-req' -> '/tmp/sls-py-req'
Here's a piece of my serverless.yml file:
custom:
pythonRequirements:
dockerizePip: true
zip: true
plugins:
- serverless-offline
- serverless-python-requirements
and here's a piece of my code that causing the error:
try:
import unzip_requirements
except ImportError:
pass
import json
import boto3
import pandas as pd
from sklearn.neighbors import NearestNeighbors
from boto3.dynamodb.types import TypeDeserializer
All I knew was, it was giving an error when importing unzip_requirements (line 2). I followed the documentation and it requires me to do the import.
The cause of the error seems like because it can't find something on lambda /tmp folder.
We ran into a similar issue with serverless-python-requirements recently. For us it was caused by a deployment, where serverless didn't recognize the requirements correctly for one function and tried to unzip an empty .requirements.zip file as a result. The failing line unzip_requirements.py#L22:
#...
zipfile.ZipFile(zip_requirements, 'r').extractall(tempdir)
# the following line fails as the extract command unzips nothing
# and doesn't create any directory that could be renamed
os.rename(tempdir, pkgdir) # Atomic
Therefore I'd recommend trying out the following:
Use pandas as a layer instead, e.g. by using a public pandas layer or creating you own one following the plugin instructions for layers e.g.
# in serverless.yaml
custom:
pythonRequirements:
layer: true
#...
functions:
hello:
handler: handler.hello
layers:
- Ref: PythonRequirementsLambdaLayer
or if the requirements are small enough, then try the unzipped version. This solved it for us with serverless version 2.69.0
I also heard that potentially downgrading to serverless 1.83 addressed a similar issue in the past, but couldn't verify that so far. Good luck
I am using python-decouple 3.4 for setting up environment variables for my django application. My .env file is in the same directory as that of manage.py. Except for SECRET_KEY (in settings.py), loading other environment variables in either settings.py or views.py directly fails stating that they have not been defined. The other environment variables which give error will be used in views.py.
Here is my .env file:-
SECRET_KEY=<django_app_secret_key>
file_path=<path_to_the file>
If I try to define them in settings.py like:-
from decouple import config
FILE_PATH = config('file_path')
and then use them in views.py,
from django.conf.settings import FILE_PATH
print(FILE_PATH)
then also I get the same error. How can I define environment variable for my views.py specifically?
[Edit: This is the error which I get:-
raise UndefinedValueError('{} not found. Declare it as envvar or define a default value.'.format(option))
decouple.UndefinedValueError: file_path not found. Declare it as envvar or define a default value.
whether I used this
from decouple import config
FILE_PATH = config('file_path')
in settings.py directly or views.py directly or first in settings.py and then in views.py like the example shown above]
I reproduced the same code you explained and it is working for me. The .env file should be in the root folder where manage.py exists. Make sure you are referencing the same settings file:
python manage.py runserver --settings=yourproj.settings.production
In the .env file:
file_path='/my/path'
In the settings.py file:
from decouple import config
FILE_PATH = config('file_path')
Also in the views.py file import the should be like this:
from django.conf import settings
print(settings.FILE_PATH)
In the .env file, the values assigned to variables were not enclosed in qoutes and that was why it was giving the error that it was unable to find file_path variable.
The .env file should be like this:-
SECRET_KEY='<django_app_secret_key>'
file_path='<path_to_the file>'
Anyways thanks to #Iain Shelvington and #Prakash S for your help.
Need a little help on aws lambda if ya'll came across this issue with deployment package uploading in aws lambda.
Regards,
xxSoumya----
[find snippet of issue ] [1]: https://i.stack.imgur.com/2QeGe.png
Your deployment package structure should be something like this,
main.py <---------- lambda entry/handler file
(can be name anything, just config your aws lambda to use it)
demo.py
mylib/
__init__.py
foo.py
bar.py
numpy/
...
pandas/
...
If demo.py is in a another folder from where your main lambda handler file is locate, then you will need to put a "__init__".py in that folder.
main.py <---------- lambda entry/handler file
mylib/
__init__.py
demo.py
foo.py
bar.py
numpy/
...
pandas/
...
Now in main, you will need to do, from mylib.demo import .....
How do I load a python module, that is not built in. I'm trying to create a plugin system for a small project im working on. How do I load those "plugins" into python? And, instaed of calling "import module", use a string to reference the module.
Have a look at importlib
Option 1: Import an arbitrary file in an arbiatrary path
Assume there's a module at /path/to/my/custom/module.py containing the following contents:
# /path/to/my/custom/module.py
test_var = 'hello'
def test_func():
print(test_var)
We can import this module using the following code:
import importlib.machinery
myfile = '/path/to/my/custom/module.py'
sfl = importlib.machinery.SourceFileLoader('mymod', myfile)
mymod = sfl.load_module()
The module is imported and assigned to the variable mymod. We can then access the module's contents as:
mymod.test_var
# prints 'hello' to the console
mymod.test_func()
# also prints 'hello' to the console
Option 2: Import a module from a package
Use importlib.import_module
For example, if you want to import settings from a settings.py file in your application root folder, you could use
_settings = importlib.import_module('settings')
The popular task queue package Celery uses this a lot, rather than giving you code examples here, please check out their git repository