I am running some code in AWS Lambda that dynamically creates SageMaker models.
I am locking Sagemaker's API version like so:
const sagemaker = new AWS.SageMaker({apiVersion: '2017-07-24'});
And here's the code to create the model:
await sagemaker.createModel({
ExecutionRoleArn: 'xxxxxx',
ModelName: sageMakerConfigId,
Containers: [{
Image: ecrUrl
}]
}).promise()
This code runs just fine locally with aws-sdk on 2.418.0.
However, when this code is deployed to Lambda, it doesn't work due to some validation errors upon creating the model:
MissingRequiredParameter: Missing required key 'PrimaryContainer' in params
UnexpectedParameter: Unexpected key 'Containers' found in params
Is anyone aware of existing bugs in the aws-sdk for NodeJS using the SDK provided by AWS in the Lambda context? I believe the SDK available inside AWS Lambda is more up-to-date than 2.418.0 but apparently there are compatibility issues.
As you've noticed the 'embedded' lambda version of the aws-sdk lags behind. It's actually on 2.290.0 (you can see the full details on the environment here: https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html)
You can see here: https://github.com/aws/aws-sdk-js/blame/master/clients/sagemaker.d.ts that it is not until 2.366.0 that the params for this method included Containers and did not require PrimaryContainer.
As you've noted, the workaround is to deploy your lambda with the aws-sdk version that you're using. This is sometimes noted as a best practice, as it pins the aws-sdk on the functionality you've built and tested against.
Related
I Currently have a project in AWS with several lambda functions, most of the functions in NodeJS, I want to know if is there a way to create a lambda layer with my own code functions that I use in different lambdas without publish it in npm, I already search in old questions in stack question-1 question-2, but these were not answered
Thanks for help!
create a folder in your local machine called nodejs
put your "shared" logic in that folder like /nodejs/shared.js
you can zip this nodejs folder and upload as a layer
in your lambda code require the shared.js as const shared = require('/opt/nodejs/shared.js')
Links:
Lambda layers: https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html
Detailed guide: https://explainexample.com/computers/aws/aws-lambda-layers-node-app
Using layers with SAM: https://docs.aws.amazon.com/serverlessrepo/latest/devguide/sharing-lambda-layers.html
I'm currently developing a Lambda function using node14 on AWS, but I have an issue when I want to develop it locally.
In my dev environment I have a corporate proxy needed to connect to AWS. In this case, I use aws-sdk-proxy library. However, I don't want to use this package while in production (as Lambda already has the aws-sdk injected in lambda core).
So, I created this snippet to perform the switch between the 2 environments:
// aws.js
import AWSPROD from 'aws-sdk'
import AWSDEV from 'aws-sdk-proxy'
import config from '../lib/Config.js'
import logger from './logger.js'
let AWS = null
if (config.get('ENVIRONMENT') === 'dev') {
logger.debug('[DEV] Using aws-sdk-proxy')
AWS = AWSDEV
} else {
AWS = AWSPROD
}
export default AWS
Not so clean, but it works. With this code, I can perform this:
import AWS from './aws.js'
Now the problem is that this code implies that I must provide aws-sdk-proxy into the "prod" dependencies of my package.json.
I think it breaks performance of the lambda as the code raised 9Mo (2Mo without), but I wish to keep this way of calling "AWS" SDK [the 2d code block].
I tried to use required or dynamic import but none of these solutions work.
Do you have any advice to improve my code?
PS:
The code is transpiled to ES5 using Babel to fit Lambda requirements
aws-sdk-proxy library must stay in devDependencies as it's a dev dependency
I'm trying to set a parameter using putParameter in the AWS SDK for JavaScript in Node.js. In particular, I'd like to take advantage of the "Advanced" Tier, with an Expiration policy and Tags if possible. When I execute my code, I keep getting errors like:
There were 2 validation errors:
* UnexpectedParameter: Unexpected key 'Policies' found in params
* UnexpectedParameter: Unexpected key 'Tier' found in params
I suspected the issue was around the aws-sdk version I was using, so I've tried running the code locally using SAM local, and from Lambda functions using the nodejs8.10 and nodejs10.x environments. The errors do not go away.
const AWS = require('aws-sdk');
AWS.config.update({region: 'us-east-1'});
const ssm = new AWS.SSM({apiVersion: '2014-11-06'});
exports.lambdaHandler = async () => {
const tokenExpiration = new Date();
tokenExpiration.setSeconds(tokenExpiration.getSeconds() + 60);
await ssm.putParameter({
Name: 'SECRET_TOKEN',
Type: 'SecureString',
Value: '12345',
Policies: JSON.stringify([
{
"Type":"Expiration",
"Version":"1.0",
"Attributes":{
"Timestamp": tokenExpiration.toISOString()
}
}
]),
Overwrite: true,
Tier: 'Advanced'
}).promise();
};
I would expect this code to work and set a parameter with the expiration. However, it appears that the sdk doesn't recognize the "Policies" and "Tier" parameters, which are available according to the documentation. I don't know if it's an issue of waiting for the newest AWS SDK for JavaScript, but the runtimes page suggest that nodejs10.x is running AWS SDK for JavaScript 2.437.0.
It might be helpful to know that I can get the code running correctly without the parameters in question (ie, just the "Name", "Type", and "Value" parameters).
Unfortunately both Tier and Policies weren't added until v2.442.0 (see diff)
This means that to use these features you'll have to deploy with the version of the aws-sdk you're developing against.
It should be noted that either developing/testing against the built-in version, or deploying with the aws-sdk you do use, is often cited as good practice. If you're deploying your version you can use explicit client imports (e.g. const SSM = require('aws-sdk/clients/ssm') to keep the deployment size down. This is even more effective if you develop against the preview AWS-SDK Version 3.
Does anyone know how I can connect to a PostgreSQL database through an AWS Lambda function. I searched it up online but I couldn't find anything about it. If you could tell me how to go about it that would be great.
If you can find something wrong with my code (node.js) that would be great otherwise can you tell me how to go about it?
exports.handler = (event, context, callback) => {
"use strict"
const pg = require('pg');
const connectionStr =
"postgres://username:password#host:port/db_name";
var client = new pg.Client(connectionStr);
client.connect(function(err){
if(err) {
callback(err)
}
callback(null, 'Connection established');
});
context.callbackWaitsForEmptyEventLoop = false;
};
The code throws an error:
cannot find module 'pg'
I wrote it directly on AWS Lambda and didn't upload anything if that makes a difference.
I wrote it directly on AWS Lambda and didn't upload anything if that makes a difference.
Yes this makes the difference! Lambda doesnt provide 3rd party libraries out of the box. As soon as you have a dependency on a 3rd party library you need to zip and upload your Lambda code manually or with the use of the API.
Fore more informations: Lambda Execution Environment and Available Libraries
You need to refer Creating a Deployment Package (Node.js)
Simple scenario – If your custom code requires only the AWS SDK library, then you can use the inline editor in the AWS Lambda console. Using the console, you can edit and upload your code to AWS Lambda. The console will zip up your code with the relevant configuration information into a deployment package that the Lambda service can run.
and
Advanced scenario – If you are writing code that uses other resources, such as a graphics library for image processing, or you want to use the AWS CLI instead of the console, you need to first create the Lambda function deployment package, and then use the console or the CLI to upload the package.
Your case like mine falls under Advanced scenario. So we need to create a deployment package and then upload it. Here what I did -
mkdir deployment
cd deployment
vi index.js
write your lambda code in this file. Make sure your handler name is index.handler when you create it.
npm install pg
You should see node_modules directory created in deployment directory which has multiple modules in it
Package the deployment directory into a zip file and upload to Lambda.
You should be good then
NOTE : npm install will install node modules in same directory under node_modules directory unless it sees a node_module directory in parent directory. To be same first do npm init followed by npm install to ensure modules are installed in same directory for deployment.
I am writing services with Serverless Framework & Azure Functions. Examples out there are very simple. But when I try to take a step further, I run into problem. Currently learning from AWS Lambda and then trying to implement it on Azure Functions.
The goal of doing so is:
1) Implement functions as es6 classes and then building the project with webpack.
2) Find a right project structure, which makes more sense.
3) Follow SoC pattern.
I have created a github project https://github.com/GeekOnGadgets/serverless-azure-settings and when I try to build this project serverless package it creates .serverless folder and inside it there is .zip file (the compiled version). Which I understand gets deployed to azure when you run serverless deploy. But when I check on Azure the function is just development code and not the compiled one (please refer to the code below).
Can someone please help with this. Any suggestions is appreciated.
import Settings from './src/Settings/Settings'
module.exports.settings = (event, context, callback) => {
let settings = new Settings();
const response = {
statusCode: 200,
headers: {
"Content-Type": "application/json"
},
body: JSON.stringify(settings.dev()),
};
callback(null, response);
}
Indeed javascript azure functions run on nodejs so commonjs modules are the natural format. Node also natively supports much of ES6, though the Functions version of node might not be the latest.
however, there is a current speed issue with loading all the dependencies in node_modules. This is due to file access so a workaround exists to bundle everything into a single script which package.json -> main points to.
I cant comment on how that fits in with serverless, but perhaps this will help clarify.
As far as I know, Node.js still does not support import/export ES6 syntax for modules. See also here.
Try a new deploy changing from
import Settings from './src/Settings/Settings'
to
const Settings = require('./src/Settings/Settings')