I have a python function which didn't work initially on cloud9(Amazon Linux2). After having installed libX11(the code uses headless chrome with selenium) the problem is gone. I just wanted to do the same on lambda so tried to add libX11 as a layer as explained in the link below but that didn't resolve the issue.
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-linux-binary-package/.
My packages are here
I packaged them directly and didn't work, on my 2nd try i packaged them under /lib directory but not joy. I actually don't want docker as an option as it gives you no control on modification of the code. Any alternative guys?
Related
Environment
I do use CI/CD of gitlab to bundle my application.
I do use node:14-alpine as image and do run yarn to build my app.
After build is finished, I do deploy my app via rsync to the target-server, which run's ubuntu 20.04.
On this server, I do use pm2 to start the app and keep it running.
Issue
If I look into the logs, I do see an error like this:
I've searched a bit, and found that the issue might be caused of musl-dev is missing.
I've installed it at my server, and into the docker-container, but with same result.
BUT, if I do delete the node_modules directory from server, and run yarn install right at the Server, the app run like expected
Question
So why does this issue happens here? Must I have the same distribution & version of linux in my docker-container to fit all dependencies?
Don't use an Alpine image if you're deploying on Ubuntu.
So why does this issue happens here?
The fundamental C standard library implementation is different on the two (Alpine uses musl libc; Ubuntu and more or less all other distros use GNU C Library (glibc)).
Trying to move binaries (such as those that might appear in node_modules for native modules) built against one libc implementation to a system using the other will likely be painful or not work at all (as you noticed).
Must I have the same distribution & version of linux in my docker-container to fit all dependencies?
If none of the dependencies use native code, then you should be able to just move things over without issues, but otherwise it'll be easiest (e.g. considering the versions of other libraries your dependencies may link against) to just use the same version as your target OS – or, if you don't want to think about that, just deploy your application as a Docker container.
Even if the suggestion from #AKX is a good answer, I've played a bit around to figure out how to solve this special case.
Here is my solution:
install musl-dev at the server
link it to /lib
apt-get install musl-dev
ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1
In my case it's only this single dependency which cause the trouble. If I got more of this, I will follow AKX's suggestion and choose a debian/ubuntu-like distribution to bundle it.
I'm trying to get a Lambda happy version of XPDF's pdftohtml to work but am having no luck.
So far the following has been tried:
Created Docker container running the latest amazonlinux image
I've copied the source code into this container and ran:
yum install cmake, gcc, gcc-c++, freetype-devel
Compiling the code with cmake produces a binary which executes perfectly in the container which should be the same OS and environment as Lambda.
I've verified the version of libc.so.6 as 2.26 within the container.
I've copied this into my AWS zip folder and included the following dependencies in a lib folder ready to upload:
libfreetype.so.6.10.0, libpng15.so.15, libstdc++.so.6.0.24
These dependencies are copied directly from the container used to compile the code.
Python function then connects these via
os.environ.update(dict(LD_LIBRARY_PATH='/var/task/lib'))
At the end of this, I run the function and get the following error code:
/var/task/pdftohtml: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /var/task/lib/libstdc++.so.6)
I've no idea where the GLIBC_2.18 comes from as this version isn't present in the container used to compile it.
Really stumped but keen to get it finished as it would produce a lightweight binary perfect for a Lambda function!
Where am I going wrong?
EDIT
SOLVED - see the comments below. There are two versions of AWS Linux and Lambda runs this version
I ran in an EC2 instance as one of the commenters suggested. Whilst the libstdc++.so.6.24 looked to be the right version, as it was itself compiled with a different GLIBC version, it throws an error. Compiling in EC2 from the source code worked fine. The other trick was making sure the CXX_FLAGS included -std=c++11. Thanks to those who contributed to help me solve this!
I've no idea where the GLIBC_2.18 comes from as this version isn't present in the container used to compile it.
I think you don't understand symbol version dependencies (see here).
The error message is telling you that your libstdc++.so.6 was built against GLIBC-2.18 or newer, and you are running against GLIBC-2.17 or older.
Where am I going wrong?
Your build environment is targeting something much newer than what your deployment environment contains.
You need to either find a built environment that matches your deployment target, or you need to change your deployment target to be not older than your build environment.
I need libv8-3.14 to run some R packages on linux, but I don't have root access/sudo access on the linux computer I'm using so I'd like to install an external folder instance of libv8-3.14. I've seen R packages reference this as external as CDFLAG="folder/v8-3.14" so I know it is possible.
I'm new(ish) to linux but I've installed external libraries before with tar.gz files which then have a configure file in them, which I set the external folder with ./configure --prefix==/folder/loc, but the only downloads I can find of libv8 are .git (which I can't get to work either).
How can I install an libv8-3.14 to a folder and install so I can set:
export PATH=$PATH:/path/to/install/
and
export `LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/install/`
I had the exact same problem. In case somebody in the future comes across this post, I will leave my suggestions and how it worked out in the end. Also, all credits go to an experienced colleague of mine.
The most sure thing to do is to consult IT, or someone who has already had the same problem, there is usually a workaround these issues.
A way you can do it yourself:
Create an anaconda environment, you can name it 'V8' or something (make sure the environment is based on the latest python version, or recent enough for r-v8).
activate it
install the conda version of the V8 R interface with conda install -c conda-forge r-v8
That's it. Whenever you need V8, fire up your environment beforehand, and it should be A-OK.
Further advice: If you run into errors when installing r-v8, it may be a good idea to update your conda and all the packages. However, depending on your conda version conda update conda and conda upgrade --all MAY BREAK your conda installation, so be careful. (For further information on this problem, see the endless complaints of people in this issue: https://github.com/conda/conda/issues/8920).
V8 doesn't use autotools, so it has no ./configure. In fact, it provides no installation facilities at all, because it is meant for embedding, not installing.
What I would try is to download the Ubuntu package (guessing from your other question, you are on Ubuntu, right?) for the right architecture from https://packages.ubuntu.com/trusty/libv8-3.14.5, and extracting it manually. .deb files are just ZIP archives.
As a side note, there's no point in setting PATH, because libv8, being a library, provides no executables. LD_LIBRARY_PATH is all you need.
I need some advice.
I have a custom module requires version Python3.5 during installation or later versions. (no critical for run my script)
My script will simultaneously use "some_decoder.pyc" file which was compiled under Python3.3 (legacy) and need this version for run. (critical for run)
How I can to solve this problem if I use Linux and virtualenv?
I have one idea in two steps:
I install basic module under global environment (Python3.5).
I will create an environment which inherit basic modules but use python3.3 (in Windows I was able to do so)
But I hope that this problem can be solved in a different way, or redirect me where this problem was solved.
I'm using AWS Lambda, which involves creating an archive of my node.js script, including the node_modules folder and uploading that to their infrastructure to run.
This works fine, except when it comes to node modules with native bindings (using node-gyp). Because the binding was complied and project archived on my local computer (OS X), it is not compatible with AWS's (Amazon Linux) servers.
How can I cross-compile/install a node module (specifically, node-sqlite3) so when I upload it to another server arch it runs?
While not really a solution to your problem, a very easy workaround could be to simply compile the native addons on a Linux machine.
For your particular situation, I would use Vagrant. Vagrant can create virtual machines and configure them within seconds.
Find an OS image that resembles Amazon's Linux distro (Fedora, CentOS, others that use yum as package manager - see Wiki)
Use a simple configuration script that, when run by Vagrant on machine startup, will run npm install (optionally it might also remove the node_modules folder before to ensure a clean installation)
For extra comfort, the script can also create the zip file for deployment
Once the installation finishes, the script will shutdown the VM to avoid unnecessary consumption of system resources
Deploy!
It might require some tuning if the linked libraries are not at the same place on the target machine but generally this seems to me like the best and quickest solution.
While installing the app using Vagrant might be sufficient in some cases, I have found it necessary to build the app on Linux which is as close to Lambda's Amazon Linux AMI as possible.
You can read the original answer here: https://stackoverflow.com/a/34019739/303184
Steps to make it work:
Spawn new EC2 instance. Make sure it is based on exactly the same image as your AWS Lambda runtime. You can review Lambda env details here: http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html. In our case, it was Amazon Linux AMI called amzn-ami-hvm-2015.03.0.x86_64-gp2.
Install nvm and use it to install the same version of Node.js as on the AWS Lambda. At the time of writing this, it was v0.10.36. You can refer to http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html again to find out.
You will probably need to install git & g++ compiler on the EC2. You can do this running
sudo yum install git gcc-c++
Finally, clone your app to your new EC2 and install your app's dependecies:
nvm use 0.10.36
npm install --production
You can then easily download the node_modules using scp or such.
Same lines as Robert's answer, when I had to work on my MAC in a different OS I use vm ware like Oracle's free virtualizer VirtualBox to get a linux on my mac, no cost to me. Or sign up for a new AWS account, you get a micro for a year free. Use that to get your linux box, do whatever you need there.
AWS has a page describing how to deal with native NPM modules: https://aws.amazon.com/blogs/compute/nodejs-packages-in-lambda/