I am facing an issue with AWS lambda function. I am unable to import pygit2 library into my function. I tried it on 4 different runtime environments and each gives a different error, which makes me think that there is something wrong within the library rather than my code. These are the errors I get:
github_pull_source is the name of my function.
Python 2.7 - Unable to import module 'github_pull_source': No module named _pygit2
Python 3.6 - Unable to import module 'github_pull_source': /var/task/_cffi_backend.so: undefined symbol: _Py_ZeroStruct
Python 3.7 - Unable to import module 'github_pull_source': No module named 'pygit2._pygit2'
Python 3.8 - Unable to import module 'github_pull_source': No module named 'pygit2._pygit2'
I import the library like so:
from pygit2 import Keypair, discover_repository, Repository, clone_repository, RemoteCallbacks
Structure of the deployment package
deployment_package.zip
- source_code.py
- __init__.py
- pygit2
- libgit2-f9f42b17.so.1.0.0
- libgit2.tar.gz
- libgit2.so.26
- _pygit2.so
- and some other libraries like requests etc
Have anyone experienced anything similar?
Thanks
I have now managed to solve the problem by creating a lambda layer. The problem seems to be with the python packages that are in the lambda container by default.
See details in this issue I raised here.
Hopefully this will help others too.
Related
I am trying to implement this AWS Lambda Rest API Handler in my lambda code to handle proper response code. For this I needed to repackage the library aws_lambda_powertools and add as a layer in lambda function.
All the import related to this lib below are working.
from aws_lambda_powertools import Logger, Tracer
from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext
But When I am creating object of above Tracer class below its giving error(Rest two commented object logger and app are working fine.
tracer = Tracer()
# logger = Logger()
# app = APIGatewayRestResolver()
Error I am getting while declaring tracer object is below:
Response
{
"errorMessage": "Unable to import module 'lambda_function': No module named 'aws_xray_sdk'",
"errorType": "Runtime.ImportModuleError",
"stackTrace": []
}
Function Logs
OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
START RequestId: ae8b006b-e7f7-495b-99a0-eb5231c3f81c Version: $LATEST
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'aws_xray_sdk'
Traceback (most recent call last):
I tried to install pip install aws_xray_sdk and repackaged it and re-added to layer still its giving the same error.
Can anyone help me with this? I am new to lambda. Thanks in advance.
Fixed the error by using AWS Arn arn:aws:lambda:{region}:017000801446:layer:AWSLambdaPowertoolsPythonV2:18📋 instead of using my own custom repackaged library layer.
Reference Link:
https://awslabs.github.io/aws-lambda-powertools-python/2.6.0/
Tracing (like validation and parsing) requires additional dependencies. They're not included by default in an attempt to keep the resulting package as small as possible.
When packaging with AWS SAM, I'm using this in my my_code/requirements.txt, which I then pip install in my local virtual env:
aws-lambda-powertools[parser, validation, tracer]==2.6.0
Additionally, I'm including this in tests/requirements.txt, which I also pip install locally but which is not picked up by SAM (keeping the image small again, and it's not required at runtime anyhow).
aws-lambda-powertools[aws-sdk]==2.6.0
pytest==7.2.1
In version 2.0 the xray SDK is not included by default, as this would introduce a size overhead even for those who would not be using Tracer.
To solve this problem, just put in your requirements.txt the tracer dependency using aws-lambda-powertools[tracer] or all dependencies using aws-lambda-powertools[all].
Refer This link:
https://github.com/awslabs/aws-lambda-powertools-python/issues/1872
I tried to upgrade a few packages with pip and also my python from 3.9 to 3.9.13. so now i use RIDE v2.0b1 running on Python 3.9.13. i use robotframework-sudslibrary-aljcalandra 1.1.4. instead of 1.1 and urllib3 1.26.12 instead of 1.26.9 for calls to webservices
I get an error when running a keyword of sudslibrary (Create Soap Client)
FAIL : AttributeError: module 'urllib' has no attribute 'pathname2url'
Since i don't have any knowledge of python and all solutions on google are about python, i'm stuck with this error. could anybody please help me?
In the following site
https://itecnote.com/tecnote/python-3-4-2-urlib-no-attribute-pathname2url/ it says that pathname2url function of urllib is now in another library which is urllib.request.pathname2url.
Should i install this python library independently by pip?
it's strange because with my older installations (ython 3.9) i didn't have urllib.request.pathname2url library when i pip(ed) list it.
Thank you for your precious help i'm badly stucked
This seems a problem on the sudslibrary you are using. A possible workaround is to fix the offending code by replacing with the new module reference.
You could (should?) find all references to the import or use of pathname2urlin that module library and replace with fixing code. For example:
import urllib.request.pathname2url as pathname2url
or
if you have from urllib import pathname2url
from urllib.request import pathname2url
The best would be, after identifying the fix, to propose a Pull Request for the sudslibrary, with something like:
try:
from urllib import pathname2url
except ImportError:
from urllib.request import pathname2url
And similar for the absolut imports.
I'm trying to work with Microsofts Hyperspace application.
In order to make it work with Python I need to install the module called Hyperspace.
When I implement the code from hyperspace import * I get the following error:
ModuleNotFoundError: No module named 'hyperspace'
I tried the following but still no luck
from pyspark hyperspace import *
Can someone let me know what it will take to successfully install the module?
I
Thanks
The module isn't supported on Databricks
I am unable to import numpy 1.19.1 in AWS Lambda with python3.8 on AWS Lambda
I am using the following dependencies:
pandas 1.1.0
pyarrow 1.0.0
numpy 1.19.1
psycopg2 2.8.5
Because I work on a windows environment, I created an EC2 Linux instance installed python3.8 and downloaded all required libraries, then I added them into the project, but the moment I try to import pandas I get the following:
[ERROR] ImportError: Unable to import required dependencies:
numpy:
IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
1. Check that you expected to use Python3.8 from "/var/lang/bin/python3.8",
and that you have no directories in your PATH or PYTHONPATH that can
interfere with the Python and numpy version "1.18.2" you're trying to use.
2. If (1) looks fine, you can open a new issue at
https://github.com/numpy/numpy/issues. Please include details on:
- how you installed Python
- how you installed numpy
- your operating system
- whether or not you have multiple versions of Python installed
- if you built from source, your compiler versions and ideally a build log
- If you're working with a numpy git repository, try `git clean -xdf`
(removes all files not under version control) and rebuild numpy.
Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.
Original error was: No module named 'numpy.core._multiarray_umath'
Traceback (most recent call last):
File "/var/task/src/py38-lib-test.py", line 28, in py38test
import pandas
File "/tmp/lib/pandas/__init__.py", line 16, in <module>
raise ImportError(END RequestId: 07762380-1fc4)
Lastly, I noticed AWS Lambda provides a layer with numpy and sci-kit, I tried removing my numpy version but kept the rest and added the layer to the function, but the same error occurs.
Thanks in advance your comments.
I use the layer provided by Klayers to solve the problem.
Suppose you're running python 3.8 in us-east-1 region, according to this Klayers document, you can use arn:aws:lambda:us-east-1:770693421928:layer:Klayers-p38-numpy:9 as your layer so that you can run import numpy in the lambda function.
AWS Lambda function don't work this way. If you open the Pandas package it'll be having the Numpy package with them but they would not work.
The easy solution is to first download the required packages separately depending upon your python version and work enviroment from this site, unzip them and add them to your project directory. Create a .zip of your project and deploy it on AWS Lambda function. It'll work this way.
You can refer to this site in order to follow the complete procedure.
Is your ec2 instance an amazon linux2 machine? You could also try building and run a docker image for amazon linux 2 and get the python libs compatible to the environment you need in your Lambda, by volume mounting to your host.
Something similat to docker lambda:
https://github.com/lambci/docker-lambda/tree/master/python3.8
I had the same issue, tried packaging all libs with my base code, tried custom lambda layer by separating numpy and pandas libs. Nothing worked.
Used default AWS Layers. In the default layers, AWS provides layers like AWSSDKPandas, CodeGuru, Lambda Insights, etc. AWSSDKPandas layer is packaged with pandas libs and other dependencies like numpy, etc.
So I removed numpy dependency from my base package and added AWSSDKPandas as Lambda layer. Worked well.
I got code file as data_load_util.py from Git hub. I'm following some tutorial where this import is being used. Using Python 3.x and Juypter Notebooks with connection to SAP Hana 2.0 Express Edition.
File location - https://github.com/SAP-samples/hana-ml-samples/blob/master/Python-API/pal/notebooks/data_load_utils.py
Command I'm using for tutorial:
from hana_ml import dataframe
from data_load_utils import DataSets, Settings
Error I'm getting:
ModuleNotFoundError: No module named 'data_load_utils'
Since I found this utility data_load_util.py as code file but not sure how I use this or attach this to python or juypter notebooks so that I can use code and this error will be gone.
Help will be appreciated.
Link to error screen shot
You need to tell Jupyter where to look for modules via sys.path.
From this doc, you can add your module’s sub-directory to Python's path like this:
import os
import sys
sys.path.insert(0, os.path.abspath('../module-subdirectory'))
Then you can simply import it:
from data_load_utils import DataSets, Settings
Note: Here module-subdirectory is the sub-directory that has got data_load_util.py.
For alternate methods, please refer this doc.