I've been trying to get node gm to work on aws lambda.
I installed the imagemagick and graphicmagick libraries on an EC2 instance created from a lambda execution environment. I pointed the appPath to these libraries from gm.
I still get the following error -
Error: Could not execute GraphicsMagick/ImageMagick: /var/task/graphicsmagick/bin/identify "-ping" "-format" "%wx%h" "./resultant-file.jpg" this most likely means the gm/convert binaries can't be found
Can anyone suggest me the right folder structure for such an app or any pointers as to where I'm going wrong?
Related
I am trying to make my express.js app works on AWS Lambda. The deployment (using serverless framework completes successfully, however the app returns a 500 internal error when I test my requests. My logs shows me this error :
/var/task/node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node: invalid ELF header.
I've tried to replace bcrypt with bcryptjs but the problem persists.
Also, in my serverless.yml, I've added the following line to force lambda to install bcrypt by itself, but apparently it didn't make the trick :
package:
exclude:
- node_modules/**
Any suggestion ?
First, please include node modules. AWS won't install anything into the lambda node env besides the aws-sdk.
Secondly, you're seeing this error because your likely developing on a mac OS machine, so the bycrypt binary from your machine is ending up getting uploaded to lambda.
Please double check that when you tried bcryptjs, you fully removed bcrypt from your project dependencies.
I am attempting to build an AWS Lambda (Node) function that utilizes the Sentry CLI. So far I have something like:
const CLI = require("#sentry/cli");
const cli = new CLI(null, {
org: '...',
authToken: '...',
});
exports.handler = async (event) => {
const response = await cli.execute(["releases", "list"]);
// ...create a release/deploy/etc...
};
This however fails with:
/var/task/node_modules/#sentry/cli/sentry-cli: cannot execute binary file
There seems to have been a similar issue reported and the suggestion is to change the permission of the executable.
How can I ensure that the permissions on the executable are not stripped when zipping/uploading the function to AWS?
TL;DR
chmod 644 $(find . -type f)
chmod 755 $(find . -type d)
chmod +x ./node_modules/#sentry/cli/sentry-cli // Same command for other binaries as well
// Reupload function code using update-function code using steps below.
Deep dive
This answer is a summary of the steps outlined in the docs, with additional explanations for why they are needed and prerequisites/debugging workflows if anything goes wrong. The docs suggest the following steps for uploading NodeJS projects with additional dependencies as follows. I have designed the steps with an existing AWS Lambda instance already running to help limit the scope of the error when debugging (to AWS or Sentry).
(Recommended) Steps with existing project
1.1 Install Node w/NPM locally (I assume you've done this). Make a note of your local node version and check this matches AWS Lambda instance.
$ node -v
1.2 Install the AWS CLI (must be version 2!).
1.3 Configure the AWS CLI with:
$ aws configure
Note: You can configure this manually as well if you need to with different guides for each platform. I will leave these details out since they are straightforward.
1.4 Try deploying a hello-world Lambda first and see if that works without the sentry-cli package. If it does work, you know sentry is probably the issue and NOT AWS.
1.5 Install Sentry CLI:
$ npm install #sentry/cli
1.6 Automatic sentry-cli configuration:
$ sentry-cli login
1.7 Verify your sentry-cli config is valid with $ sentry-cli info. If not, you need to follow the steps recommended in the console output.
$ sentry-cli info
1.8 Install dependencies using aws-xray-sdk:
$ npm install aws-xray-sdk
1.8.1 (Optional) Navigate to your project root folder. This is just for illustration; the current version of the AWS SDK is pre-installed in Lambda, but you could use this technique to load other pre-built JavaScript packages or if you actually needed an earlier version of the AWS SDK for compatibility reasons (not applicable).
$ npm install --prefix=. aws-sdk
1.8.2 (Sanity Check) Check the permissions of all files in the subfolders of root directory have the executable permissions. Try running the project locally, to see if the executable permission exists:
$ ls -l && node function.js
1.9 Zip the project:
$ zip -r function.zip . // The .zip file must be **less than 50 MB**!
1.10 Upload the function code using the aws command-line tool update-function-code (this is important because this will fix the permissions issue.
$ aws lambda update-function-code --function-name my-function --zip-file fileb://function.zip
1.11 If the operation was successful, you will get an output like the following:
{
"FunctionName": "my-function",
"FunctionArn": "arn:aws:lambda:us-east-2:123456789012:function:my-function",
"Runtime": "nodejs12.x",
"Role": "arn:aws:iam::123456789012:role/lambda-role",
"Handler": "index.handler",
"CodeSha256": "Qf0hMc1I2di6YFMi9aXm3JtGTmcDbjniEuiYonYptAk=",
"Version": "$LATEST",
"TracingConfig": {
"Mode": "Active"
},
"RevisionId": "983ed1e3-ca8e-434b-8dc1-7d72ebadd83d",
...
}
1.12 If you get an error with the upload, for example, you can follow the docs here. Check the AWS logs if you need to on AWS CloudWatch.
1.13 Test the running lambda, after you are sure that the update-function-code was successful.
$ aws lambda invoke --function-name my-function --payload '{"key1": "value1", "key2": "value2", "key3": "value3"}' output.txt
Another potential solution (not ideal)
Make the sentry cli executable before you run the CLI config command using child_process.
var exec = require('child_process').exec, child;
child = exec('chmod +x /var/task/node_modules/#sentry/cli/sentry-cli',
function (error, stdout, stderr) {
console.log('stdout: ' + stdout);
console.log('stderr: ' + stderr);
if (error !== null) {
console.log('exec error: ' + error);
}
});
child();
Alternative: You can also try using this package.
Refactoring with Sentry Node NPM package
If in the steps above, you notice that the Sentry CLI is the issue, you can try to refactor your code without this package. Use the Sentry Node NPM package instead since this NPM package was built for NodeJS, and maybe refactor your code. The Sentry Node may be easier to get running but doesn't have functions for deployment/release. From their Usage page:
Sentry's SDK hooks into your runtime environment and automatically reports errors, exceptions, and rejections.
(Note) Sentry with AWS Lambda Docs
The Sentry docs recommend using the #sentry/serverless as a package for integration with AWS Lambda. If you don't want to refactor your code use this guide.
With the AWS Lambda integration enabled, the Node SDK will:
Automatically report all events from your Lambda Functions.
Allows you to modify the transaction sample rate using tracesSampleRate.
Issue reports automatically include:
A link to the cloudwatch logs
Function details
sys.argv for the function
AWS Request ID
Function execution time
Function version
Caveats:
The .zip file must be less than 50 MB. If it's larger than 50 MB, Amazon recommends uploading it to an Amazon Simple Storage Service (Amazon S3) bucket.
The .zip file can't contain libraries written in C or C++. If your .zip file contains C-extension libraries, such as the Pillow (PIL) or numpy libraries, we recommend using the AWS Serverless Application Model (AWS SAM) command line interface (CLI) to build a deployment package.
The .zip file must contain your function's code and any dependencies used to run your function's code (if applicable) on Lambda. If your function depends only on standard libraries, or AWS SDK libraries, you don't need to include these libraries in your .zip file. These libraries are included with the supported Lambda runtime environments.
If any of the libraries use native code, use an Amazon Linux environment to create the deployment package. Also, ensure you package the native code (if you have some) locally on the same platform as the Lambda!
If your deployment package contains native libraries, you can build the deployment package with AWS Serverless Application Model (AWS SAM). You can use the AWS SAM CLI sam build command with the --use-container to create your deployment package. This option builds a deployment package inside a Docker image that is compatible with the Lambda execution environment.
I'm deploying a lambda on AWS that uses the Sharp module to resize an image.
Despite using linux to deploy the following error happens:
"Something went wrong installing the sharp module,
Module parse failed: Unexpected character 'u007f' (1: 0),
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders,
(Source code omitted for this binary file), "
In my research I saw that the problem was associated with the OS. That it was necessary to install sharp using "npm install --arch = x64 --platform = linux sharp".
However, the problem continues to happen.
Even using linux x84_64 this problem happens, does anyone know what it could be?
My question is similar to an existing one, but I'm using webpack and thus don't even have a "node_modules" directory.
I've still tried the following environment variable settings without success:
NODE_ICU_DATA: 'node_modules/full-icu'
NODE_ICU_DATA: '/var/task/node_modules/full-icu/icudt58l.dat'
NODE_ICU_DATA: '/var/task/node_modules/full-icu/icudt60l.dat'
All without success, the lambdas simply refuse to start and a log message is show in CloudWatch:
/var/lang/bin/node: could not initialize ICU (check NODE_ICU_DATA or --icu-data-dir parameters)
I found very little about this problem on Google, which seems curious given that it should concern so many that are building internal apps on AWS lambda.
Make sure you're deploying the correct .dat file version into the directory defined by NODE_ICU_DATA. In my case, the correct file was icudt62l.dat when using the runtime nodejs10.x for my lambdas. If you're not deploying node_modules as part of your lambda, you can grab the correct .dat file and deploy that.
I ran into the same error message when upgrading the runtime version of some old lambdas from nodejs6.10 to nodejs10.x. In my case, the file icudt58l.dat (compatible with nodejs6.10) was deployed into the project root, and the value of NODE_ICU_DATA was . (i.e. the directory where my lambda executes, equivalent to /var/task/).
To get a compatible .dat file, I re-installed the full-icu package using the node version used by the lambdas:
npx -p node#10.x npm i full-icu
It's important that the version of the .dat file is correct for the runtime version your lambdas are using. Initially, I made the mistake of just running npm i full-icu using node version 10.0 but this installed the wrong version of the .dat file and I just got the same error message again.
Finally, I copied the file node_modules/full-icu/icudt62l.dat into the project root and deleted the old icudt58l.dat so that it ends up in /var/task/ where the lambda can find it when NODE_ICU_DATA=..
I am trying to deploy a small python 3.7 code which uses "pdftotext" on AWS-Lambda. I was able to run the code successfully on my local machine (Mac). Using virtual-environment as specified by AWS documentation, I created a deployment package of the code with pdftotext. However still I am getting module not found error in Lambda. If you have been able to create a complete build package for pdftotext with all its dependencies, can you share please. Thanks in advance.
Using virtual-environment as specified by AWS documentation, I created a deployment package of the code with pdftotext.
import pdftotext
Getting module not found error for pdftotext when testing in AWS-Lambda.
A bit late to the party on the answer to this question, but I asked a similar question here: Get pdftotext Python module running on Lambda
My problem was I was compiling the binary on Amazon Linux 1 when the Python 3.8 lambda runtime uses Amazon Linux 2 shared libraries.
Install the library on an Amazon Linux 2 EC2 instance or docker container, and the library will be usable in your lambda.
Posting this in case it helps someone that comes across this question.