I'm facing one of these AWS Lambda node.js timeout when trying to access DynamoDB issues but the symptoms appear different and the solutions I found don't solve this issue.
Timeout is set to 5min, memory is set to 128MB but doesn't exceed 30MB usage.
IAM policies for the role are:
AWSLambdaFullAccess
AmazonDynamoDBFullAccess
AWSLambdaVPCAccessExecutionRole
The default VPC has 7 security groups and include the default security group with:
Inbound: All Traffic, All protocol, All port range,
Outbound: All Traffic, All protocol, All port range, 0.0.0.0/0
Here is the code:
var aws = require('aws-sdk');
exports.handler = function(event, context) {
var dynamo = new aws.DynamoDB();
dynamo.listTables(function(err, data) {
if (err) {
context.fail('Failed miserably:' + err.stack);
} else {
context.succeed('Function Finished! Data :' + data.TableNames);
}
});
};
And the Outcome:
START RequestId: 5d2a0294-fb6d-11e6-989a-edaa5cb75cba Version: $LATEST
END RequestId: 5d2a0294-fb6d-11e6-989a-edaa5cb75cba
REPORT RequestId: 5d2a0294-fb6d-11e6-989a-edaa5cb75cba Duration: 300000.91 ms Billed Duration: 300000 ms Memory Size: 128 MB Max Memory Used: 21 MB
2017-02-25T15:21:21.778Z 5d2a0294-fb6d-11e6-989a-edaa5cb75cba Task timed out after 300.00 seconds
The related node.js version issue solved here doesn't work for me and returns a "ReferenceError: https is not defined at exports.handler (/var/task/index.js:6:16)". Also AWS has deprecated version 0.10.
Here is the code with the https reference:
var aws = require('aws-sdk');
exports.handler = function(event, context) {
var dynamo = new aws.DynamoDB({
httpOptions: {
agent: new https.Agent({
rejectUnauthorized: true,
secureProtocol: "TLSv1_method",
ciphers: "ALL"
})
}
});
dynamo.listTables(function(err, data) {
if (err) {
context.fail('Failed miserably:' + err.stack);
} else {
context.succeed('Function Finished! Data :' + data.TableNames);
}
});
};
Outcome:
START RequestId: 6dfd3db7-fae0-11e6-ba81-a52f5fc3c3eb Version: $LATEST
2017-02-24T22:27:31.010Z 6dfd3db7-fae0-11e6-ba81-a52f5fc3c3eb ReferenceError: https is not defined
at exports.handler (/var/task/index.js:6:16)
END RequestId: 6dfd3db7-fae0-11e6-ba81-a52f5fc3c3eb
REPORT RequestId: 6dfd3db7-fae0-11e6-ba81-a52f5fc3c3eb Duration: 81.00 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 26 MB
RequestId: 6dfd3db7-fae0-11e6-ba81-a52f5fc3c3eb Process exited before completing request
With a timeout set to 5min I can't believe that AWS wouldn't be able to return the list of tables in the allocated timeframe and permission issues typically appear in the logs.
Thanks for looking into this.
I guess your Lambda is in a private subnet. In this case by default your Lambda will not have outbound internet access. You need to create a NAT Gateway or NAT Instance to let VPC protected resources to access outside Internet. DynamoDB API is outside Internet from VPC point of view.
You no longer need to create a NAT gateway/instance
You can create a VPC Endpoint for Dynamo DB which will open Lambda in the private subnet to access Dynamo. Create an endpoint in your VPC that aligns to the VPC/subnet setup you have for lambda and you will have no issues with access.
You can limit access to specific services or resources.
https://aws.amazon.com/blogs/aws/new-vpc-endpoints-for-dynamodb/
This can be done for any global AWS service, S3 etc
Related
I have an Azure (AZ) Function does two things:
validate submitted info involving 3rd party packages.
when ok call a postgreSQL function at AZ to fetch a small set of data
Testing with Postman, this AF localhost response time < 40 ms. Deployed to Cloud, change URL to AZ, same set of data, took 30 seconds got Status: 500 Internal Server Error.
Did a search, thought this SO might be the case, that I need to bump my subscription to the expensive one to avoid cold start.
But more investigation running part 1 and 2 individually and combined, found:
validation part alone runs perfect at AZ, response time < 40ms, just like local, suggests cold start/npm-installation is not an issue.
pg function call always long and status: 500 regardless it runs alone or succeeding part 1, no data returned.
Application Insight is enabled and added a Diagnostic settings with:
FunctionAppLogs and AllMetrics selected
Send to LogAnalytiscs workspace and Stream to an event hub selected
Following queries found no error/exceptions:
requests | order by timestamp desc |limit 100 // success is "true", time taken 30 seconds, status = 500
traces | order by timestamp desc | limit 30 // success is "true", time taken 30 seconds, status = 500
exceptions | limit 30 // no data returned
How complicated my pg call is? Standard connection, simple and short:
require('dotenv').config({ path: './environment/PostgreSql.env'});
const fs = require("fs");
const pgp = require('pg-promise')(); // () = taking default initOptions
require('dotenv').config({ path: './environment/PostgreSql.env'});
const fs = require("fs");
const pgp = require('pg-promise')(); // () = taking default initOptions
db = pgp(
{
user: process.env.PGuser,
host: process.env.PGhost,
database: process.env.PGdatabase,
password: process.env.PGpassword,
port: process.env.PGport,
ssl:
{
rejectUnauthorized: true,
ca: fs.readFileSync("./environment/DigiCertGlobalRootCA.crt.pem").toString(),
},
}
);
const pgTest = (nothing) =>
{
return new Promise((resolve, reject) =>
{
var sql = 'select * from schema.test()'; // test() does a select from a 2-row narrrow table.
db.any(sql)
.then
(
good => resolve(good),
bad => reject({status: 555, body: bad})
)
}
);
}
module.exports = { pgTest }
AF test1 is a standard httpTrigger anonymous access:
const x1 = require("package1");
...
const xx = require("packagex");
const pgdb = require("db");
module.exports = function(context)
{
try
{
pgdb.pgTest(1)
.then
(
good => {context.res={body: good}; context.done();},
bad => {context.res={body: bad}; context.done();}
)
.catch(err => {console.log(err)})
}
catch(e)
{ context.res={body: bad}; context.done(); }
}
Note:
AZ = Azure.
AZ pg doesn't require SSL.
pg connectivity method: public access (allowed IP addresses)
Postman tests on Local F5 run against the same AZ pg database, all same region.
pgAdmin and psql all running fast against the same.
AF-deploy is zip-file deployment, my understanding it is using the same configuration.
I'm new to Azure but based on my experience, if it's about credential then should come back right away.
Update 1, FunctionAppLogs | where TimeGenerated between ( datetime(2022-01-21 16:33:20) .. datetime(2022-01-21 16:35:46) )
Is it because my pg network access set to Public access?
My AZ pgDB is a flexible server, current Networking is Public access (allowed IP address), and I have added some Firewall rule w/ client IP address. My assumption is access is allowed within AZ, but it's not.
Solution 1, simply check this box: Allow public access from any Azure servcie within Azure to this server at the bottom of the Settings -> Networking.
Solution 2, find out all AF's outbound IP and add them into Firewall rule, under Settings -> Networking. Reason to add them all is Azure select an outbound IP randomly.
I am working on an Azure Cloud Function (runs on node js) that should return a collection of documents from my Azure Cosmos DB for MongoDB API account. It all works fine when I build and start the function locally, but fails when I deploy it to Azure. This is the error: MongoNetworkError: failed to connect to server [++++.mongo.cosmos.azure.com:++++] on first connect ...
I am new to CosmosDB and Azure Cloud Functions, so I am struggling to find the problem. I looked at the Firewall and virtual networks settings in the portal and tried out different variations of the connection string.
As it seems to work locally, I assume it could be a configuration setting in the portal. Can someone help me out?
1.Set up the connection
I used the primary connection string provided by the portal.
import * as mongoClient from 'mongodb';
import { cosmosConnectionStrings } from './credentials';
import { Context } from '#azure/functions';
// The MongoDB Node.js 3.0 driver requires encoding special characters in the Cosmos DB password.
const config = {
url: cosmosConnectionStrings.primary_connection_string_v1,
dbName: "****"
};
export async function createConnection(context: Context): Promise<any> {
let db: mongoClient.Db;
let connection: any;
try {
connection = await mongoClient.connect(config.url, {
useNewUrlParser: true,
ssl: true
});
context.log('Do we have a connection? ', connection.isConnected());
if (connection.isConnected()) {
db = connection.db(config.dbName);
context.log('Connected to: ', db.databaseName);
}
} catch (error) {
context.log(error);
context.log('Something went wrong');
}
return {
connection,
db
};
}
2. The main function
The main function that execute the query and returns the collection.
const httpTrigger: AzureFunction = async function (context: Context, req: HttpRequest): Promise<void> {
context.log('Get all projects function processed a request.');
try {
const { db, connection } = await createConnection(context);
if (db) {
const projects = db.collection('projects')
const res = await projects.find({})
const body = await res.toArray()
context.log('Response projects: ', body);
connection.close()
context.res = {
status: 200,
body
}
} else {
context.res = {
status: 400,
body: 'Could not connect to database'
};
}
} catch (error) {
context.log(error);
context.res = {
status: 400,
body: 'Internal server error'
};
}
};
I had another look at the firewall and private network settings and read the offical documentation on configuring an IP firewall. On default the current IP adddress of your local machine is added to the IP whitelist. That's why the function worked locally.
Based on the documentation I tried all the options described below. They all worked for me. However, it still remains unclear why I had to manually perform an action to make it work. I am also not sure which option is best.
Set Allow access from to All networks
All networks (including the internet) can access the database (obviously not advised)
Add the inbound and outbound IP addresses of the cloud function project to the whitelistThis could be challenging if the IP addresses changes over time. If you are on the consumption plan this will probably happen.
Check the Accept connections from within public Azure datacenters option in the Exceptions section
If you access your Azure Cosmos DB account from services that don’t
provide a static IP (for example, Azure Stream Analytics and Azure
Functions), you can still use the IP firewall to limit access. You can
enable access from other sources within the Azure by selecting the
Accept connections from within Azure datacenters option.
This option configures the firewall to allow all requests from Azure, including requests from the subscriptions of other customers deployed in Azure. The list of IPs allowed by this option is wide, so it limits the effectiveness of a firewall policy. Use this option only if your requests don’t originate from static IPs or subnets in virtual networks. Choosing this option automatically allows access from the Azure portal because the Azure portal is deployed in Azure.
If use this code within a Lambda which complies with everything I read on stackoverflow and on the AWS SDK documentation.
However, it neither returns anything nor throws an error. The code is simply stuck on s3.getObject(params).promise() so the lambda function runs on a timeout, even after more then 30 seconds. The file i try to fetch is actually 25kb.
Any idea why this happens?
var AWS = require('aws-sdk');
var s3 = new AWS.S3({httpOptions: {timeout: 3000}});
async function getObject(bucket, objectKey) {
try {
const params = {
Bucket: bucket,
Key: objectKey
}
console.log("Trying to fetch " + objectKey + " from bucket " + bucket)
const data = await s3.getObject(params).promise()
console.log("Done loading image from S3")
return data.Body.toString('utf-8')
} catch (e) {
console.log("error loading from S3")
throw new Error(`Could not retrieve file from S3: ${e.message}`)
}
}
When testing the function, i receive the following timeout.
START RequestId: 97782eac-019b-4d46-9e1e-3dc36ad87124 Version: $LATEST
2019-03-19T07:51:30.225Z 97782eac-019b-4d46-9e1e-3dc36ad87124 Trying to fetch public-images/low/ZARGES_41137_PROD_TECH_ST_LI.jpg from bucket zarges-pimdata-test
2019-03-19T07:51:54.979Z 97782eac-019b-4d46-9e1e-3dc36ad87124 error loading from S3
2019-03-19T07:51:54.981Z 97782eac-019b-4d46-9e1e-3dc36ad87124 {"errorMessage":"Could not retrieve file from S3: Connection timed out after 3000ms","errorType":"Error","stackTrace":["getObject (/var/task/index.js:430:15)","","process._tickDomainCallback (internal/process/next_tick.js:228:7)"]}
END RequestId: 97782eac-019b-4d46-9e1e-3dc36ad87124
REPORT RequestId: 97782eac-019b-4d46-9e1e-3dc36ad87124 Duration: 24876.90 ms
Billed Duration: 24900 ms Memory Size: 512 MB Max Memory Used: 120 MB
The image I am fetching is actually public available:
https://s3.eu-central-1.amazonaws.com/zarges-pimdata-test/public-images/low/ZARGES_41137_PROD_TECH_ST_LI.jpg
const data = (await (s3.getObject(params).promise())).Body.toString('utf-8')
If your Lambda function is associated with a VPC it loses internet access which is required to access S3. However, instead of following the Lambda warning that says "Associate a NAT" etc, you can create an S3 endpoint in the VPC > Endpoints settings, and your Lambda function will work as expected, with no need to manually set up Internet access for your VPC.
https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/
Default timeout of AWS SDK is 120000 ms. If your lambda's timeout is shorter then that, you will never receive the actual error.
Either extend your AWS timeout
var AWS = require('aws-sdk');
var s3 = new AWS.S3({httpOptions: {timeout: 3000}});
or extend the timout of your lambda.
This issue is definitely related to connection.
Check out your VPC settings as it is likely blocking the Lambda connection to the Internet (AWS managed services as S3 are accessible only via Internet).
If you are using localstack, make sure SSL is false and s3ForcePathStyle is true.
That was my problem
AWS.S3({endpoint: '0.0.0.0:4572', sslEnabled: false, s3ForcePathStyle:true})
More details here
Are you sure you are providing your accessKeyId and secretAccessKey? I was having timeouts with no error message until I added them to the config:
AWS.config.update({ signatureVersion: 'v4', region: "us-east-1",
accessKeyId: secret.accessKeyID,
secretAccessKey: secret.secretAccessKey });
I am using aws-sdk for publishing message on topic below is the code:
var AWS = require('aws-sdk');
AWS.config.region = 'us-east-1';
AWS.config.credentials = {
accessKeyId: 'myaccesskeyid',
secretAccessKey: 'mysecretaccesskey'
}
function LEDOnIntent() {
this.iotdata = new AWS.IotData({
endpoint: 'XXXXXXXXX.iot.us-east-1.amazonaws.com'
});
}
LEDOnIntent.prototype.publishMessage = function() {
console.log('>publishMessage');
var params = {
topic: 'test_topic',
/* required */
payload: new Buffer('{action : "LED on"}') || 'STRING_VALUE',
qos: 1
};
this.iotdata.publish(params, function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else {
console.log("Message published : " + data); // successful response
}
});
}
It works fine in local unit testing but when I deploy this code on AWS lambda then I got very uneven behaviour. For the first few requests it will not publish message then it will work fine when I continuously test it. When I test after some break then again it stop working for some initial requests.
Behind the scene, Lambda runs like a container model. It means to create a container when needs and destroy it if doesn't require the container.
The reason you see a delay in the initial request because It takes time to set up a container and do the necessary bootstrapping, which adds some latency each time the Lambda function is invoked. You typically see this latency when a Lambda function is invoked for the first time or after it has been updated because AWS Lambda tries to reuse the container for subsequent invocations of the Lambda function.
AWS Lambda maintains the container for some time in anticipation of another Lambda function invocation. In effect, the service freezes the container after a Lambda function completes, and thaws the container for reuse if AWS Lambda chooses to reuse the container when the Lambda function is invoked again.
Please read the official documentation here
I have a simple lambda function as follows
var AWS = require("aws-sdk");
exports.handler = (event, context, callback) => {
var ec2 = new AWS.EC2({region:'us-east-1'});
return ec2.describeRegions({}).promise()
.then(function(regionResponse) {
console.log(regionResponse.Regions)
callback(null, regionResponse.Regions);
})
.catch(
function (err) {
console.log({"error" : err});
callback(err, null);
}
)
};
I can run this function outside of a VPC successfully.
I create a VPC using the VPC wizard and create a VPC with a single public subnet and an Internet Gateway. I place the function in the VPC and give it an execution role with Lambda VPC Execution rights.
It now fails with a timeout, which I have set to 10 seconds (normal execution 1 sec)
What am I missing from my config that prevents the function from accessing the AWS SDK inside the VPC?
You are putting callback after return statement. Of course it will never be executed because you returned from the function.
If the subnet you are running the Lambda is not public or does not have NAT Gateway, it won't be able to connect to Internet, thus to AWS API's.