Issues with python and the serverless framework - python-3.x

Question

I am trying to understand how to setup multiple python lambdas and a step function within one single serverless.yml with each python lambda having its own dependencies. All of my lambda functions collaborate in the context of a step function for a shared common goal. With this rationale, it makes sense to me to put all of the code under one serverless.yml file.
As part of my MANY hours of trial and error and reading I found about the serverless-python-requirements plugin for The Serverless Framework that helps in packaging python functions that rely on OS-specific python libraries and also allow the separation of multiple requirements.txt in case different lambdas require different dependencies.
So at this point my problem is that the generated package is not including the dependencies that I provide in the requirements.txt whenever each function has its own requirements.txt
These are my artifacts:
package.json
{
"engines": {
"node": ">=10.0.0",
"npm": ">=6.0.0"
},
"name": "example",
"version": "1.0.0",
"description": "",
"main": "index.js",
"dependencies": {
"serverless-python-requirements": "^5.1.0"
},
"devDependencies": {
"serverless": "^1.72.0"
},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"license": "ISC"
}
serverless.yml
service: example
frameworkVersion: ">=1.72.0 <2.0.0"
plugins:
- serverless-python-requirements
custom:
stage: "${opt:stage, env:SLS_STAGE, 'local'}"
log_level: "${env:LOG_LEVEL, 'INFO'}"
pythonRequirements:
dockerizePip: true
provider:
name: aws
# profile: ${self:custom.profile}
stage: ${self:custom.stage}
runtime: python3.8
environment:
LOG_LEVEL: ${self:custom.log_level}
package:
individually: true
exclude:
- ./**
include:
- vendored/**
functions:
function1:
# module: folder1
handler: folder1/function1.handler
package:
include:
- 'folder1/**'
memorySize: 128
timeout: 60
function2:
# module: folder2
handler: folder2/function2.handler
package:
include:
- 'folder2/**'
memorySize: 128
timeout: 60
finally, my 2 python lambda functions are in separate folders and one of them requires specific dependencies:
folder1
function1.py
requirements.txt
folder2
function1.py
function1.py
import json
import logging
import os
import sys
import pyjokes
log_level = os.environ.get('LOG_LEVEL', 'INFO')
logging.root.setLevel(logging.getLevelName(log_level))
_logger = logging.getLogger(__name__)
class HandlerBaseError(Exception):
'''Base error class'''
class ComponentIdentifierBaseError(HandlerBaseError):
'''Base Component Identifier Error'''
def handler(event, context):
'''Function entry'''
_logger.debug('Event received: {}'.format(json.dumps(event)))
body = {
"message": "Go Serverless v1.0! Your function executed successfully!",
"joke":pyjokes.get_joke()
}
resp = {
'status': 'OK',
"body": json.dumps(body)
}
_logger.debug('Response: {}'.format(json.dumps(resp)))
return resp
if __name__ == "__main__":
handler('', '')
requirements.txt
pyjokes==0.6.0
function2.py
import json
import logging
import os
import sys
log_level = os.environ.get('LOG_LEVEL', 'INFO')
logging.root.setLevel(logging.getLevelName(log_level))
_logger = logging.getLogger(__name__)
class HandlerBaseError(Exception):
'''Base error class'''
class ElasticSearchPopulatorBaseError(HandlerBaseError):
'''Base Component Identifier Error'''
def handler(event, context):
'''Function entry'''
_logger.debug('Event received: {}'.format(json.dumps(event)))
resp = {
'status': 'OK'
}
_logger.debug('Response: {}'.format(json.dumps(resp)))
return resp
Note: I did try using the module+handler keywords in the serverless.xml as recommended on this link: ttps://github.com/UnitedIncome/serverless-python-requirements without any success
Something that I noted is that if I use the module+handler as follows:
functions:
function1:
module: folder1
handler: function1.handler
package:
include:
- 'folder1/**'
memorySize: 128
timeout: 60
Then, when I try running the function locally using: serverless invoke local -f function1 --log I get an error saying:
ModuleNotFoundError: No module named 'function1'
Also, if anyone has an example of multiple lambdas with different requirements.txt that works I would be very gratelful, ideally something just different than the typical hello world examples, the hello worlds all work very well for me ;), but in scenarios like this one where I would like to setup common libraries, have different dependencies, etc, and use one common serverless.yml things seem to fall apart. Again, my opinion is that these lambdas will operate together under one step function umbrella so there's strong cohesion here and I think that their build and deployment should happen under one common serverless service.

Answer 1

I recently developed a similar application using the serverless-python-requirements plugin that encapsulates multiple lambdas as part of one stack, and I was receiving ModuleNotFoundError whilst invoking the lambda function locally, yet it would work remotely; however, when I removed the module parameter from my serverless.yml file I was able to invoke locally but then it broke for remote executions.
I've been able to find a workaround by setting a path prefix in my serverless.yml:
functions:
LambdaTest:
handler: ${env:HANDLER_PATH_PREFIX, ""}handler.handler
module: src/test_lambda
package:
include:
- src/test_lambda/**
When I invoke the function locally, I prepend the environment variable to my command:
HANDLER_PATH_PREFIX=src/test_lambda/ ./node_modules/serverless/bin/serverless.js invoke local -f LambdaTest -p ./tests/resources/base_event.yml
I don't include the environment variable when invoking the function in AWS.
In order for this to work, I needed to add an init.py file to the root directory where my lambda function resides with the following code (taken from this solution) so that any modules I'm including in the code that exist in the lambda's directory (e.g. some_module -- see directory tree below):
import os
import sys
sys.path.append(os.path.dirname(os.path.realpath(__file__)))
My lambda's directory structure:
src/
└── test_lambda
├── __init__.py <=== add the code to this one
├── handler.py
├── requirements.txt
└── some_module
├── __init__.py
└── example.py
As for your question regarding lambdas that use different requirements.txt files -- I use the individually parameter, like so:
package:
individually: true
include:
- infra/**
exclude:
- "**/*"
Within each requirements.txt for each lambda I refer to a separate requirements.txt file that resides in the base directory of my project using the -r option -- this file contains libraries that are common to all lambdas, so when Serverless is installing packages for each lambda it'll also include packages included in my ./requirements.txt file too.
I've included this solution in an issue regarding the serverless-python-requirements plugin in GitHub which would be worth keeping an eye on should this behaviour of the module parameter turns out to be a bug.
Hope it helps and let me know if you require clarification on anything.