How to use request module in node.js lambda - node.js

I am trying to use the request module in AWS Lambda for Node.js, but I am not having good results.
This is the function I have so far:
const request = require('request');
url = 'https://www.google.com'
exports.handler = async (event) => {
request(url, (error,res,body) => {
console.log(error); //this is not printing
console.log(res); //this is not printing
console.log(body); //this is not printing
console.log('Come on!!'); //this is not printing
});
console.log(url); //This is the only thing printing, you can see in the Function Logs below, second line.
};
and this is the response I am getting:
Response:
null
Request ID:
"3e46f401-f26d-435d-90be-ac848c6c3a39"
Function Logs:
START RequestId: 3e46f401-f26d-435d-90be-ac848c6c3a39 Version: $LATEST
2019-10-14T08:06:23.755Z 3e46f401-f26d-435d-90be-ac848c6c3a39 INFO https://www.google.com
END RequestId: 3e46f401-f26d-435d-90be-ac848c6c3a39
REPORT RequestId: 3e46f401-f26d-435d-90be-ac848c6c3a39 Duration: 368.93 ms Billed Duration: 400 ms Memory Size: 128 MB Max Memory Used: 92 MB Init Duration: 461.54 ms
I am wondering why I am not getting any response from inside the requests method. I tried on my laptop and the part of the code outside the handler works just fine.
I uploaded the project as a zip file, so the node_modules folder with the request module is indeed there.
I noted a couple of solutions using the http module instead of request, but I would like to understand first why this is not working, before moving to a different solution.

Your function is marked as async, yet you are using a callback to handle the response. So nothing is being awaited on.
This means that your function returns an implicit promise that is already fulfilled. This makes the lambda stop. So, either:
A) Write your code using async/await, which probably requires request-promise or similar:
exports.handler = async event => {
const res = await promisifiedRequest(url);
console.log(res.statusCode);
console.log(url);
};
or
B) Convert the function from async/await to use callbacks, like so:
exports.handler = (event, context, callback) => {
const res = request(url, (error, res, body) => {
console.log('Come on!!');
callback(null, res.statusCode);
});
console.log(url);
};

Related

Node.js HTTPS call not working in AWS Lambda

I'm trying to create an AWS Lambda function that will call a DELETE on a schedule.
I'm using Node.js. When running just from Node.js on my local machine the request works fine.
Here is the code:
const https = require('https');
var options = {
host: 'MY_HOST',
path: '/v1/api/orders',
port: 80,
method: 'DELETE'
};
console.info('Do the DELETE call');
var reqDelete = https.request(options, function(res) {
res.on('data', function(d) {
console.info('DELETE result:\n');
console.log(d.toString('utf8'));
console.info('\nCall completed');
});
});
reqDelete.on('error', function(e) {
console.error(e);
});
reqDelete.end();
My output is like this:
Do the DELETE call
DELETE result:
{"message":"Cleanup triggered"}
Call completed
Just as I expect. However when I run from inside an AWS Lambda function I get the result of null and the Log output for the Lambda function is this.
START RequestId: fb2a1969-94e8-4c11-b43e-14ff6a4cc426 Version: $LATEST
2020-06-16T01:42:06.875Z fb2a1969-94e8-4c11-b43e-14ff6a4cc426 INFO Do the DELETE call
END RequestId: fb2a1969-94e8-4c11-b43e-14ff6a4cc426
REPORT RequestId: fb2a1969-94e8-4c11-b43e-14ff6a4cc426 Duration: 483.49 ms Billed Duration: 500 ms
Memory Size: 128 MB Max Memory Used: 67 MB Init Duration: 125.35 ms
Notice that it prints out the "Do the DELETE call" so I know it's getting to my code but nothing else is being printed out.
The body of the Lambda is like this:
exports.handler = async (event) => {
// The exact code from above with the actual host name.
};
Why is my API call not executed from the Lambda function when it is working from my local machine?
You are using an async Lambda function handler, but the HTTP request is not awaited. This causes the function to finish execution before the https.request() call can complete.
If you would like to use callbacks and not promises, then define a non-async handler for the function:
exports.handler = (event) => {
// The exact code from above with the actual host name.
};

how to get the first bytes of a file from a given url node.js

I want to a file from a given url, in a aws lambda function.
I wrote this code:
exports.handler = (event, context, callback) => {
var http = require('http');
var url= "https://mail.google.com/mail/u/0/?ui=2&ik=806f533220&attid=0.1&permmsgid=msg-a:r-8750932957918989452&th=168b03149469bc1f&view=att&disp=safe&realattid=f_jro0gbqh0"
//var client = http.createClient(80, url);
var request = http.request({
port: 80,
host: url
});
request.on('response', function( res ) {
res.on('data', function( data ) {
console.log(data);
});
});
request.end();
const result = {
statusCode: 200,
body: JSON.stringify('Hello from Lambda!'),
};
callback(null, result);
};
but I get an error saying:
"Response: {
"errorMessage": "RequestId: 52baec5e-60bc-47ea-911e-8e6cb1d2f1da Process exited before completing request"
}"
since I only need the first 2 bytes' I thought maybe I should read them, and not the whole file.
any ideas?
thanks a lot!
Did you increase your lambda execution Timeout limit? When you first created, lambda comes setted with only 3 seconds by default. You can change that under Basic settings. Change Timeout to 2 or 3 minutes to allow your lambda to finish execution. Also check if your memory is enough. You may need to increase it a little bit. I have mine with 256 MB.
When you test your lambda, pay attention at the Duration and Memory Size values. Lambda will print that in the last line of the log output. So if you set your lambda execution Timeout to 5 minutes and it only takes 2 minutes or if your lambda Memory Size is to close to your Memory you might want to increase it so your lambda do not fail execution due to a memory problem.
I usually do this with fetch or request. You can make it with request like this:
exports.handler = (event, context, callback) => {
var request = require('request');
var url= "https://mail.google.com/mail/u/0/?ui=2&ik=806f533220&attid=0.1&permmsgid=msg-a:r-8750932957918989452&th=168b03149469bc1f&view=att&disp=safe&realattid=f_jro0gbqh0"
request(url, { json: true, timeout: 1000 }, (err, response, body) => {
if (err) {
console.log(err);
callback(err, null);
} else {
console.log(body);
callback(null, "Hello from Lambda");
}
});
};
Just run npm install request to get module request and you are good to go.
It is always a good idea to start with your local development environment and when everything is ready, you just zip your files and then upload them to lambda. This way you know everything is just fine with your code and you can focus on the lambda configuration details. This way it much easier to test and you do not consume any lambda resource.
This is how to upload your file to lambda:

getting Task timed out after 59.05 seconds aws lambda nodejs

I am trying to write a lambda function in node 8.10 supported by aws lambda
this is with reference to my previous question
I am trying node async/await , same code work when I run it in my local(and is not even taking a second to get the response back) but when I am trying it in lambda getting time out error, here the code control is getting inside promise and printing url urltohit with the correct url but after it is getting stuck, have tried different thing like increasing the time in lambda config and even tried node request instead of http but still getting the same error.
'use strict';
var http = require('http');
var request = require('request');
exports.handler = async (event) => {
const response = await getRequest(urltohit);
console.log(response);
}
const getRequest = async (url) => {
console.log("Test log");
return new Promise((resolve, reject) => {
console.log(`url ${url}`);
http.get(url, function(res) {
console.log("Got response: " + res.statusCode);
resolve(res.statusCode);
}).on('error', function(e) {
console.log("Got error: " + e.message);
reject(e.message);
});
});
}
You need to call callback after function execution.
exports.handler = async (event, context, callback) => {
const response = await getRequest(urltohit);
console.log(response);
callback(null);
}
It's lambda behaviour to keep function running until callback called.
More could be found in official documentation

Lambda Invocation error with callback called: metrics show no errors

I have created a Lambda function in AWS that export logs from Cloudfront to Elasticsearch.
From the AWS console, I still have a warining in front of Invocation error, though the metrics show there is none for more than 24hours.
A typical workflow of logs looks like
START RequestId: 302f0b95-7856-11e8-9486-55b3f10e7d4e Version: $LATEST
Request complete
END RequestId: 302f0b95-7856-11e8-9486-55b3f10e7d4e
REPORT RequestId: 302f0b95-7856-11e8-9486-55b3f10e7d4e Duration: 794.93 ms Billed Duration: 800 ms Memory Size: 128 MB Max Memory Used: 52 MB
There is no error in the logs, and the only thing I guess could trigger this invocation error is that sometimes two request starts at the same time
09:01:47
START RequestId: 63cd00e1-7856-11e8-8f96-1f900def8e65 Version: $LATEST
09:01:47
START RequestId: 63e1e7f3-7856-11e8-97e6-3792244f6ab0 Version: $LATEST
Except from this, I don't understand where this error comes from.
Do I miss something? Or do I have to wait more than 24hours before the satus change? May be there is a way to pinpoint the error with AWS console/API that I did not find about?
Would be happy to hear what you think about this.
Edit: In case you'd like to take a look at the code itself.
var aws = require('aws-sdk');
var zlib = require('zlib');
var async = require('async');
const CloudFrontParser = require('cloudfront-log-parser');
var elasticsearch = require('elasticsearch');
var s3 = new aws.S3();
var client = new elasticsearch.Client({
host: process.env.ES_HOST,
log: 'trace',
keepAlive: false
});
exports.handler = function(event, context, callback) {
var srcBucket = event.Records[0].s3.bucket.name;
var srcKey = event.Records[0].s3.object.key;
async.waterfall([
function fetchLogFromS3(next){
console.log('Fetching compressed log from S3...');
s3.getObject({
Bucket: srcBucket,
Key: srcKey
},
next);
},
function uncompressLog(response, next){
console.log("Uncompressing log...");
zlib.gunzip(response.Body, next);
},
function publishNotifications(jsonBuffer, next) {
console.log('Filtering log...');
var json = jsonBuffer.toString();
var records;
CloudFrontParser.parse(json, { format: 'web' }, function (err, accesses) {
if(err){
console.log(err);
} else {
records = accesses;
}
});
var bulk = [];
records.forEach(function(record) {
bulk.push({"index": {}})
bulk.push(record);
});
client.bulk({
index: process.env.ES_INDEXPREFIX,
type: 'log',
body: bulk
}, function(err, resp, status) {
if(err) {
console.log('Error: ', err);
}
console.log(resp);
next();
});
console.log('CloudFront parsed:', records);
}
], function (err) {
if (err) {
console.error('Failed to send data: ', err);
} else {
console.log('Successfully send data.');
}
callback(null, 'success');
});
};
You need to explicitly return information back to the caller.
Here's the related documentation:
The Node.js runtimes v6.10 and v8.10 support the optional callback
parameter. You can use it to explicitly return information back to the
caller. The general syntax is:
callback(Error error, Object result); Where:
error – is an optional parameter that you can use to provide results
of the failed Lambda function execution. When a Lambda function
succeeds, you can pass null as the first parameter.
result – is an optional parameter that you can use to provide the
result of a successful function execution. The result provided must be
JSON.stringify compatible. If an error is provided, this parameter is
ignored.
If you don't use callback in your code, AWS Lambda will call it
implicitly and the return value is null.
When the callback is called (explicitly or implicitly), AWS Lambda
continues the Lambda function invocation until the event loop is
empty.
The following are example callbacks:
callback(); // Indicates success but no information returned to the caller.
callback(null); // Indicates success but no informatio returned to the caller.
callback(null, "success"); // Indicates success with information returned to the caller.
callback(error); // Indicates error with error information returned to the caller
https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html

Can AWS Lambda reach/interact with S/FTP?

I wrote some basic js to just list the files of a FTP but I get:
"Process exited before completing request"
Is that because Lambda can't interact with FTP?
I'm using jsftp btw.
Here's my setup:
I use Serverless to create the project
For my lambda, I used nodejs and I'm using JSFTP to deal with the ftp stuff.
My code:
// Require Serverless ENV vars
var ServerlessHelpers = require('serverless-helpers-js').loadEnv();
// Require Logic
var lib = require('../lib');
// Lambda Handler
module.exports.handler = function (event, context) {
lib.respond(event, function (error, response) {
return context.done(error, response);
});
};
My ftp lambda code:
var JSFtp = require("jsftp");
module.exports.respond = function (event, cb) {
var ftp = new JSFtp({
host: "host",
user: "user",
password: "password"
});
ftp.auth(ftp.user, ftp.password, function(err, res) {
if (err) console.log(err);
else console.log(res);
ftp.ls(".", function (err, res) {
var results = [];
res.forEach(function (file) {
results.push(file.name);
});
ftp.raw.quit();
return cb(null, results.length);
})
});
};
I added some console.log() all over the place and it seems like it choked once it tried to ftp.auth.
The output I see in cloud watch:
START RequestId: __ID__ Version: $LATEST
END RequestId: __ID__
REPORT RequestId: __ID__ Duration: 526.46 ms Billed Duration: 600 ms Memory Size: 1024 MB Max Memory Used: 33 MB
Process exited before completing request
So it looks like it just choked somewhere...
in short, ftp will not work with lambda since they use ephemeral ports.
sftp will work nicely with lambda. i tested using java code via jsch with no issues; tho i cant see how it wouldnt work with any js sftp lib.
It is possible tested just now.
Make sure ur timeout is set to be long enough and you are calling context.succeed() on process termination
function __main__(event, context) {
var JSFtp = require("jsftp");
var ftp = new JSFtp({
host: "speedtest.tele2.net",
port: 21, // defaults to 21
});
ftp.ls(".", function(err, res) {
var results = []; res.forEach(function(file) {
results.push(file.name);
});
context.succeed(results);
});
};
By default, Lambda functions only have 3 seconds to complete. If it takes longer than that, you'll get the error you are seeing.
You can adjust the timeout to anything up to 5 minutes. To change it using the aws CLI, run:
aws lambda update-function-configuration --function-name my-lambda-function --timeout 300

Resources