Upload large files to AWS s3 using express js synchrounsly - node.js

Currently i am using putObject to upload the large file to AWS s3 with REST api call.
var params ={
Bucket:'lambdacushbu',
Key:req.files.image.name,
Body:req.files.image.data
}
s3.putObject(params,function(err, data) {
if (err) console.log(err, err.stack); // an error occurred
else {
console.timeEnd('Uploadtime');
console.log("uploaded",data);
res.json({
'status':'Uploaded',
'url':data.Location
});
} // successful response
});
But its looks like asynchronous i want the above in synchronous mode also a timeout is occurred but the file is being uploaded to the AWS s3.
So how can i increase the timeout value?? tried with connect-timeout package
app.use(timeout('600000'));
But it dosen't worked

Try using upload function instead of putObject. That should solve your timeout problem.
Here is a documentation for that function: http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property
Synchronous call will definitely lower your app's performance. Can you provide me more details about your problem so we can find an async solution?
EDIT:
Here is how you should return response in your controller:
router.post('/your-route',
//additional middlewares
function(req, res, next) {
var params = {
Bucket:'lambdacushbu',
Key:req.files.image.name,
Body:req.files.image.data
}
s3.upload(params,function(err, data) {
if (err) { res.json(err); }
else {
res.json({
'status':'Uploaded',
'url':data.Location
}
});
}
);
And make sure you don't call res.json() or res.send() anywhere else in this route

Related

Angular NodeJS Upload File and parameters together

I have been looking through multiple tutorials and stack overflow questions but for some reason I just cannot make this work. I have issues with uploading a file, so maybe fixing that first would solve the whole issue.
I tried a few options of sending a file from the front end to the back end, but it seems to always "get lost" before reaching the back end.
I have decided to use multer at the NodeJS backend to upload the file. Not sure if I am calling multer upload single right or not. Currently this is the code which I have for it:
const multer = require('multer');
const storage = multer.diskStorage({
destination: './uploadedImages',
filename: function(req,file,cb){
cb(null,file.originalname)
}
}) ;
const upload = multer({storage: storage})
exports.saveDrawing = async(req, res, next) => {
try
{
//save image
//tried a few different logs, but with FormData it seems like everything always empty
console.log("Image:");
console.log(req.body.drawingElement);
console.log(req.file);
upload.single('body.file');
return res.status(200).json({message: element});
}
}
catch (err)
{
console.log("Error at drawing save: " + err)
return res.status(500).json({message: "Error - Could not add/edit Drawing"});
}
}
And this is how it is sent from the Angular front end:
setDrawing(params, image): Observable<any> {
const formData = new FormData();
formData.append('file', image)
formData.append('data', params)
console.log("File: ");
console.log(formData.get('file'));
console.log("Data: ");
console.log(formData.get('data'));
return this.http.post<any>(`api/v1/structure/drawing/save`, formData);
}
At this stage printing out the data shows the right values. And the browser shows the right payload too:
At the back end I cannot see them in the req, req.body is empty, there is no req.form. For this api call before I have tried to include any files without the FromData I have accessed the data from req.body.
Am I looking for the data at the right place?
You're not using multer correctly, it's not doing anything.
To implement it as a middleware which you call from your handler, check the example from the docs
So, your handler should look something like this:
// setup multer middleware, set file field name
const upload = multer({storage: storage}).single('file');
exports.saveDrawing = async(req, res, next) => {
// now use the middleware, handle errors
upload(req, res, function (err) {
if (err instanceof multer.MulterError) {
// A Multer error occurred when uploading.
return res.status(500).json({message: "Error - Could not add/edit Drawing"});
} else if (err) {
// An unknown error occurred when uploading.
return res.status(500).json({message: "Error - Could not add/edit Drawing"});
}
// Everything went fine.
console.log("Image:");
console.log(req.body.drawingElement);
console.log(req.file);
return res.status(200).json({message: element});
});
});

Lambda function only works once

So I'm new to AWS serverless architecture. I deployed my first lambda function using Claudia. I'm not sure whether I did it correctly. I deployed all the APIs to one lambda function using Claudia. The API endpoints works individually when I test it on Insomnia. But when I use it in my application only one specific API works and the lambda dies. For instance, I used this POST request to post some items and I have a useEffect in my React application which has a get request to retrieve all the items from the database. But once I post the item, nothing is returned. Could anyone help me understand what I'm doing wrong. P.S this is my final year project which is due in a few weeks. So, a quick answer would be appreciated.
Here is a sample code.
// Create a new Intake
router.post("/create", async (req, res) => {
const intake = req.body;
const { name, intakeCode, intakeYear } = req.body;
const checkIntake = await Intakes.findOne({
where: {
intakeCode: intakeCode,
},
});
if (checkIntake) {
res.json({ err: `An intake under ${intakeCode} already exists!` });
} else {
try {
await Intakes.create(intake);
res.json({ msg: `Successfully created ${name} ` });
} catch (e) {
if (e.name == "SequelizeDatabaseError") {
res.json({ err: "Year only accepts integer" });
} else {
res.json({ err: e.name });
}
}
}
});
// Find all Intakes
router.get("/findAll", async (req, res) => {
const listOfIntakes = await Intakes.findAll();
res.json(listOfIntakes);
});
Cheers
Looks like you are trying to build a Lambda function using JavaScript - but have encountered problems. I'm not familiar with Claudia. One suggestion that I have is to follow the official AWS SDK for JavaScript DEV Guide here:
https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/scheduled-events-invoking-lambda-example.html
That content will walk you through how to create a Lambda function using JS.

aws-sdk crashing nodejs program

I am using aws-sdk for javascript.
The code below works fine when using in a stand-alone program
//program.js
const AWS = require('aws-sdk');
const firehose = new AWS.Firehose({
accessKeyId: "XXX",
secretAccessKey: "YY"
});
const params = {
DeliveryStreamName: 'demo1',
Record: {
Data: new Buffer("Hello World")
}
};
firehose.putRecord(params, function (err, data){
if (err) {
console.log(err);
return;
}
console.log(data); // successful response
});
Again, the above code works fine as a stand alone file. Data gets pushed into firehose and then further down to Redshift.
so if i execute
node program.js
I am able to see my data in Redshift. Yay!!
=============================
However, what i really want to achieve is to push data to firehose when a certain route gets hit in my express application. So I take the exact same code as above and stick it in my route
// router.js
const AWS = require('aws-sdk');
const firehose = new AWS.Firehose({
accessKeyId: "XXX",
secretAccessKey: "YY"
});
router
.get('/v1/locations/:id?', (req, res) => {
const params = {
DeliveryStreamName: 'demo1',
Record: {
Data: new Buffer("Hello World")
}
};
firehose.putRecord(params, function (err, data){
if (err) {
console.log(err);
return;
}
console.log(data);
});
// do the work that needs to be done for this route and send a response
res.send("some data");
});
The minute firehose.putRecord is executed .. it crashes my program with the following error:
```
TypeError: doneCallback.cal is not a function
at Request.callListeners (/api-project/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
at callNextListener (/api-project/node_modules/aws-sdk/lib/sequential_executor.js:95:12)
at /api-project/node_modules/aws-sdk/lib/event_listeners.js:74:9
at finish (/api-project/node_modules/aws-sdk/lib/config.js:315:7)
at /api-project/node_modules/aws-sdk/lib/config.js:333:9
at Credentials.get (/api-project/node_modules/aws-sdk/lib/credentials.js:126:7)
at getAsyncCredentials (/api-project/node_modules/aws-sdk/lib/config.js:327:24)
at Config.getCredentials (/api-project/node_modules/aws-sdk/lib/config.js:347:9)
at Request.VALIDATE_CREDENTIALS (/api-project/node_modules/aws-sdk/lib/event_listeners.js:69:26)
at Request.callListeners (/api-project/node_modules/aws-sdk/lib/sequential_executor.js:101:18)
I can't understand why this code crashes my express program. is this is bug in the aws-sdk library or am i doing something wrong ?
You should be sending the express response inside your success callback.
firehose.putRecord(params, function (err, data) {
if (err) {
console.log(err);
return;
}
console.log(data);
res.send("some data");
}
);
FYI, your res.send(data) will effectively exit the program and send data. However, your putRecord callback is the time when your exit should occur. In node, things do not happen in a sequence from top to bottom of the code, but instead it executes in order of the callback events. so the execution flow for your code would be like this:
file executes
some operation is performed
callback for operation occurs
then if theres additional code outside of operate, it will continue, otherwise the code will exit in that callback. Hence, put the res.send in your putRecord callback.

How to handle callback error in node.js

I am new to node.js, I have a requirement where i am trying to handle the error that is being returned from the callback method/function. How do i assign the error that is being sent as part of callback to my response payload.
The node module that i am calling to validate swagger supports both callback function as well as Promise.
So how do i assign the err to my response payload. Currently i am just logging to my console, but since we plan to expose this through an API i would like to return the error information in the response payload.
var express = require('express');
var SwaggerParser = require('swagger-parser');
var myParser = require("body-parser");
var app = express();
var fs = require("fs");
app.use(myParser.urlencoded({extended : true}));
app.use(myParser.json());
function errorHandler (err, req, res, next) {
res.status(500)
res.render('error', { error: err })
}
app.post('/v1/swagger/validate',function(request,response){
/**SwaggerParser.validate(request.body, function(err, api) {
if (err) {
console.error(err);
console.log("Inside Error");
}
else {
console.log("API name: %s, Version: %s", api.info.title, api.info.version);
console.log("Inside Success");
}
}); **/
SwaggerParser.validate(request.body)
.then(function(api) {
console.log("API name: %s, Version: %s", api.info.title, api.info.version);
})
.catch(function(err) {
console.error(err);
});
response.end();
});
app.listen(8082);
You can decide how you want to communicate the error back from your API.
If the error is internal to your server and not something caused directly by a poor API request, then you probably return a 500 status code (internal server error).
response.status(500).end();
If there's something meaningful to communicate back to the other end of the API (like nothing found for the query or a specific validation error), then you have to design what you want that to be. For example, you could be sending back some JSON:
response.json({result: null, msg: "Validation Failed"});
So, it's really up to you what you want your API to return for a given situation. The main point is that you decide what you want that response to be and you send it as the response, even in error conditions. You need to make it a design that makes sense to the consumers of your API so they can clearly tell when they have a proper result and clearly tell when they have an error and if the error is their fault they need to be able to tell why it is their fault based on the response (so they more detail you provide on the issue in the response, the better).

Export a function from a function

I have an ExpressJS app where I have api.js in routes that manages connecting to Couchbase and then emits event couchbaseConnected that is awaited by init() function inside api.js.
Inside init() I want to push those exports.someFunction(req, res){return something;}. But when I just put these exports inside init() function, I get an error .get() requires callback functions but got a [object Undefined] so it seems like I am doing it wrong.
The question is how I can export functions from another function in NodeJS?
Here is the code:
//connecting to couchbase and emitting event on connection
couchbase.connect(dbConfiguration, function (err, bucket) {
if (err) {
console.log(err);
}
cb = bucket;
eventEmitter.emit('couchbaseConnected');
});
//listening for the event and fire init() when it's there
eventEmitter.on('couchbaseConnected', function (e) {
console.log('Connected to Couchbase.'.green);
init();
});
function init() {
exports.getUserData = function (req, res) {
if (req.user != undefined && req.user.meta != undefined) {
res.json(200, {result: 'ok'})
}
else {
res.json(401, {error: 'Unauthorized request.'})
}
};
}
Here is the ExpressJS .get() that is located in app.js:
app.get('/api/user/data/:type', api.getUserData);
Here is the ExpressJS .get() that is located in app.js:
app.get('/api/user/data/:type', api.getUserData);
The error message .get() requires callback functions but got a [object Undefined] makes quite clear what happens: You require the API module, it starts to connect to the db, you are defining your express app by passing a non-existent property to .get - which fails, since init has not yet been called and getUserData has not yet been assigned. What you need to do is
var api = require('api');
eventEmitter.on('couchbaseConnected', function () {
app.get('/api/user/data/:type', api.getUserData); // now it is available
});
However, this does not look like good code. Instead of loosely coupling them via that couchbaseConnected event you better should use explicit callbacks that are invoked with the requested values (i.e. the cb bucket, or the getUserData method). At least pass them as parameters to the emitted event.
Also, your setup is unconventional. I don't see why getUserData would need to be asynchronously defined - it should always be available. If the couchbase connection failed, I would not expect the /api/user/data/ service to not exist, but to respond with some 500 internal server error message.
This is an answer that I have made some assumptions as the data provided by you is not sufficient to know what you are doing when you start your app.
I would suggest some change in code:
module.exports.connect = function(callback){
couchbase.connect(dbConfiguration, function (err, bucket) {
if (err) {
console.log(err);
callback(err);
}
cb = bucket;
module.exports.getUserData = getUserData();
callback(err); //just callback with no error as you don't require to send the database
});
}
function getUserData(req, res){
if (req.user != undefined && req.user.meta != undefined) {
res.json(200, {result: 'ok'})
}
else {
res.json(401, {error: 'Unauthorized request.'})
}
};
and in your app.js file where you are starting the app just do this
var api = require('./api');
api.connect(function(error){
if (error) throw error;
app.listen(3000);
console.log('Express started on port 3000');
});
And then you can continue doing exactly what you were doing. It should work.
I have assumed that you are not explicitly calling the connect or its equivalent when you are starting the app.....So your getting that error.

Resources