I need to publish data from aws lambda through mqtt protocol using aws iot. i have created a lambda function with node.js code. like this
exports.handler = (event, context, callback) => {
var awsIot = require('aws-iot-device-sdk');
var device = awsIot.device({
keyPath: 'samplepath/test.pem.key',
certPath: 'samplepath/test.crt',
caPath: 'samplepath',
clientId: 'sampleId',
region: 'us-east-1'
});
device
.on('connect', function () {
console.log('connected');
device.publish('test_topic', JSON.stringify({ "test_name": "hello", "test_value": 1001 }));
console.log('published successfully');
callback(null, 'item added');
});
}
I got mqtt message on subscriber. but lambda produce error message like this
Task timed out after 10.00 seconds
I have used context.succeed() instead of callback, lambda is exited properly. i cant get any messages on subscriber.
In both cases console prints published successfully message properly.
What is the issue related with my publishing code?
I understand my lambda function is timing out when connecting to AWS
IoT. About the sdk we are using, the aws-iot-device-sdk is designed to
use inside of an embedded device. When we are using a Lambda function
or trying to publish in a computer, the best practice is use the
aws-sdk. Using the aws-sdk we don't need to use the certificates to
publish in the AWS IoT, we just use the AWS credentials to do this.
Also, with aws-sdk we can do administrative tasks in the IoT, we can
create a thing, create a certificate, etc.
Coming to my code, the reason the function does not end and times out
is because the callback must be waiting for an asynchronous call to
finish execution, which I assume is being help up by the connection
being maintained from the function to IoT. The reason
context.succeed() exited properly but we did not get any messages must
be because context.succeed does not wait for our async calls to finish
execution.
Make sure you disconnect from the device after you published the message, otherwise Lambda will wait while the connection stays alive (see http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html, look for callbackWaitsForEmptyEventLoop).
To disconnect when done, simply change callback(null, 'item added'); to
device.end((err) => {
callback(err, "item added");
});
Related
I have a problem with understanding of working SQS from AWS in NodeJS. I'm creating a simple endpoint which receive data sended to my server and add to SQS queue:
export const receiveMessageFromSDK = async (req: Request, res: Response) => {
const payload = req.body;
try {
await sqs.sendMessage({
QueueUrl: process.env.SQS_QUEUE_URL,
MessageBody: payload
}).promise();
} catch (error) {
//something else
}
}
and ok, it working and MessageBody is adding to my SQS queue. And right now I have a problem with processing of the data from this queue... How to do this?
In Google Cloud I'm creating simple request queue with sending (in body payload) also url of endpoint which should process data from queue, then gcloud send to this endpoint this body and my server starting business logic on this data. But how to do this in SQS? How to receive data from this queue and start processing data on my side?
I'm thinking about QueueUrl param but in docs is written that this value should be an url from aws console like https://sqs.eu-central-1.amazonaws.com... so I have no idea how to process this data from queue on my side.
So..., can anybody help me?
Thanks, in advice for any help!
Should I call launch this function on cron or AWS can eq. send request on providing from me endpoint with this data etc
SQS is for pulling data, which means that you have to have a cron job working (or any equivalent system in your app) that iteratively pulls your queue for messages using receiveMessage, , e.g. every 10 seconds.
If you want push type messaging system, then you have to use SNS instead of SQS. SNS can push messages to your HTTP/HTTPS endpoint automatically if your application exposes any such endpoint for the messages.
i'm writing a serverless node rest api, and i have few functions and today i faced an issue on sending responds from lambda function to api gateway, my callback doesn't work as it expected, what i am doing wrong?
module.exports.create = (event, context, callback) => {
client.on('connect', () => {
console.log("connected to redis");
callback(null, {
statusCode: 200,
headers: { 'Content-Type': 'text/plain' },
body: 'connection established.',
});
return;
});
};
A common issue people have with Lambda and NodeJS is timing... I think what's happening here is that the Lambda Function terminates before your response comes back. Lambda does not wait around for an async response, so most of the time doesn't execute the responds events, so never hits your callback.
Try using a Promise, which keeps the code/Lambda running until the async call comes back and the callback is called.
This is a good article on how to achieve that:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
Ok, I've encountered Internal server error a few times before and I suggest you to do this.
First, a little background knowledge you must have:
When you're deploying your serverless application, what is happening under the hood is that serverless framework creates necessary configurations and .zip file (your lambda functions code and dependencies) under .serverless folder.
So if you're missing necessary dependencies in your package.json or forget to include them in the .zip file, your lambda will return Internal server error.
And you should check whether you included dependencies into dev-dependencies in package.json, too. (This will prevent your necessary modules to be included in .zip file).
And secondly, if you're using serverless-webpack plugin, you should include these lines in your serverless.yaml file.
custom:
webpack:
includeModules: true
This worked for my case.
If you don't understand or have anything to ask, feel free to do that :)
Internal errors are when the return isn't hit for some reason, it can be a coding error or a timeout because the default serverless timeout is too low for what you are trying to do.
If you want to alter the timeout you can do something like this in the serverless.yml:
functions:
create:
handler: handler/create
timeout: 30
...
I am currently using nodemailer to send Email from my node.js application.
What i am looking for is "something" that queues and schedules the sending of large number of Emails(500+ per user) by multiple users.
This should run as a separate process in background and must be triggered by my node.js application.
Email sending should not affect my Node.Js application(server) from responding to requests.
I know the above statements seems more like a s/w requirement rather than a problem to fix but i need a robust solution.Please give a brief solution(i.e., what "somethings" to use).
Create a lambda function to send one email only with your nodeMailer script.
In this lambda I would cache the mailer outside the actual exports function like:
const nodemailer = require('nodemailer');
exports.handler = (event, context, callback) => {
// ...NodeMailer stuff
callback(null, event)
};
Then trigger this lambda with an SQS event as seen here:
SQS as an event source to trigger Lambda | Medium
You can add a message to the queue using the JS AWS-SDK from your node js app as seen here:
Calling the sendMessage operation | AWS Docs
I am trying to publish message from AWS Lamba using Nodejs to AWS IoT .
I have zipped the project and uploaded on to the AWS IoT
below is the code snippet
var awsIot = require('aws-iot-device-sdk');
var device = awsIot.device({
keyPath: 'keyfilepath',
certPath: 'pem file path',
caPath: 'root-CA.crt',
clientId: 'iotTest7526532135',
host: 'host id'
});
device
.on('connect', function() {
console.log('connect');
device.subscribe('topic_3');
device.publish('topic_1', JSON.stringify({ message_id:23,Message:'HelloWorld'}));
});
device
.on('message', function(topic, payload) {
console.log('message', topic, payload.toString());
});
I am getting below error
"errorMessage": "Cannot find module 'aws-iot-device-sdk'",
I know that iot sdk is missing, I am not sure how to install it on AWS Lambda.
Any suggestions will be really helpful
I would highly recommend not using the aws-iot-device-sdk to interact with AWS Iot from a Lambda function.
You need to understand there are 2 javascript APIs that you can use to access AWS IoT
The AWS IOT Device SDKs for javascript, using MQTT as a protocol and x509 certificates for authentication. These are typically used in devices running outside of your AWS cloud.
The AWS SDK for javascript, using HTTP as a protocol, and IAM roles (among other things) for authentication. These SDKs are typically run inside your AWS cloud (such as a Lambda)
There are multiple reasons why you should opt for the HTTP based SDK :
The aws-iot-device-sdk is specifically targeted toward devices "living" outside of Aws (IoT gateways / devices in the field) that need to remotely connect.
The Device SDK uses MQTT and x509 certificates to interact with AWS IoT. There is no need to configure x509 securities in your lambda. A Lambda running on your AWS account can easily access AWS IoT through an IAM role, so if your lambda function is configured with the correct role, you can use the standard aws sdks.
A protocol like MQTT (or any pub/sub protocol) doesn't match well with a serverless lambda architecture. In your lambda function you are subscribing to a topic, something that you don't typically do in a short-lived lambda function.
The AWS SDK for NodeJS is provided to your lambda out of the box.There's no need to require or package additional node modules)
Your code can become as simple as this (notice that there are no credentials or extra node modules required) :
var AWS = require('aws-sdk');
var iotdata = new AWS.IotData({endpoint:"yourendpoint.iot.eu-central-1.amazonaws.com"});
exports.handler = function(event, context, callback) {
console.log("found iotdata",iotdata);
var params = {
topic: 'topic/test',
payload: 'blah',
qos: 0
};
iotdata.publish(params, function(err, data){
if(err){
console.log("Error occured : ",err);
}
else{
console.log("success.....");
}
});
callback();
};
When you zip your project, you also zip ./node_modules folder. So as long as aws-iot-device-sdk is there (along with all the dependencies), your Lambda will be fine.
So all you need is:
npm install aws-iot-device-sdk
zip ...
You'll need to make sure you're uploading your package.json file as well, which should have a dependency requirement for aws-iot-device-sdk
you can add the package to your package.json by running
npm -i --save aws-iot-device-sdk
from your project directory.
If you are just going to publish the data to IoT topic, you are better of using http protocol instead of MQTT connection.
And using HTTP connection doesnt require aws-iot-device-sdk package. AWS default SDK has iotdata. And iotdata will provide the http connection to the device.
iotHTTPConnection= AWS.IotData({endpoint: Your_END_POINT});
iotHTTPConnection.publish(params) will publish the data using http without iot specific sdk.
https://docs.aws.amazon.com/iot/latest/apireference/API_iotdata_Publish.html
There is no http subscribe though.
Add something like the below in your package.json file. then run npm install. This will create a node_modules folder. Now zip it and upload it once again.
"aws-iot-device-sdk": "^2.1.0"
There is not a good practice adding iot-sdk library into your Lambda function. Thats also the reason it doesn't come pre-installed into the Lambda services. It is clearly a library to connect devices to AWS cloud.
In your case as they mentioned above, the pre-installed aws-sdk is all you need.
Though you can't subscribe to a topic for the reason that the protocol to publish a message is HTTP.
But usually isn't a reason for that too.
I have an AWS API Gateway method that proxies requests through to AWS Lambda. However, it errors after three seconds with the following in the logs:
Endpoint response body before transformations: {"errorMessage":"2017-09-05T16:30:49.987Z 922186c0-9257-11e7-9db3-51921d5597a2 Task timed out after 3.00 seconds"}
Thus, I went on to check my Node 6.10 AWS Lambda function to see why it was timing out. I added logging statements before and after every function call. Surprisingly, it did everything it's supposed to do: called the API Gateway callback, and run a query against the database after that. All that takes 0.6s, and as far as I'm aware there's no other code left to run. Nevertheless, it appears to keep on running for the rest of the three seconds and then timing out. (This is, I think, because I'm leaving a connection to the database open.)
The logs statements I placed before and after the callback call indicate that the that call is executed in under half a second. Yet, that response doesn't seem to make it to API Gateway, whereas the error after three seconds does.
What could be potential reasons for this, and how can I debug it?
By default calling the callback() function in a NodeJS Lambda function does not end the function execution. It will continue running until the event loop is empty. A common issue with NodeJS Lambda functions continuing to run after callback is called occurs when you are holding on to open database connections. You haven't posted any code, so I can't give specific recommendations, but you would need to determine if you are leaving database connections open in your code or something similar.
Alternatively, you can change the behavior such that the execution ends as soon as the callback function is called by setting callbackWaitsForEmptyEventLoop = false on the context object.
Your API gateway has a fixed timeout of 29 seconds. Most queries do complete within this timeframe.
Increase your lambda execution timeout to anywhere between 30 Sec to 3 Min 00 Sec.
Use context.succeed() instead of callback.
This worked for me.
const mysql = require('mysql');
const connection = mysql.createConnection({
host : 'your_mysql_host',
user : 'your_mysql_user',
password : 'your_mysql_password',
database : 'your_mysql_db'
});
exports.handler = (event, context) => {
var userId = event.params.querystring.userid;
const sql = 'SELECT * FROM users where USER_ID=' + userId;
var response = {
"statusCode": 200,
"body": "body_text_goes_here"
}
if(userId){
connection.query(sql, function (error, results, fields) {
if (error) {
context.succeed(error);
} else {
response.body = results;
context.succeed(response);
}
});
}
}
I had same issue and I've updated the timeout in my code but no luck finally increased lambda execution time that fixed my problem.
How to increase an AWS Lambda timeout?