I am using Formidable with Express in nodeJS in an attempt to have a simple single file upload scheme. I have confirmed that a file is actually sent over from the client-side, but where it seems to run into troubles is on the server-side.
index.js
app.post('/', (req, res) => {
const form = formidale();
form.on('file', (filename, file) => {
fs.rename(file.path, `./data/nodes.csv`, err => {
if (err) {
console.log(`There was an error in downloading a CSV file: ${err}`);
return null;
}
else {
console.log("CSV file has been uploaded correctly.");
}
});
});
form.on('error', err => {
console.log(`There was an error in downloading a CSV file: ${err}`);
return null;
});
form.on('end', () => {
console.log(fs.readFileSync('./data/nodes.csv')); // test to see if file exists
const nodes = assignMetrics();
console.log(nodes);
return nodes;
});
form.parse(req);
});
}
The main trouble I seem to find is that the form.on('end', ...) event does not seem to wait till the file has finished uploading to fire. I have confirmed this by trying to read the file in the event, but by that point it doesn't exist? The documentation though appears to suggest it is only meant to fire "after all files have been flushed [from the APIs pipe it infers]".
There appears to be no other events available that might wait till the file has been uploaded to be called? I also don't want to start throwing in layers of promises and such unless it is the only option, as each new layer of promises I find is a chance for unintended effects to happen.
Related
I have an internal HTTP Post API that generates files, size 5-10mb each. This service is not modifiable.
I want to "proxy" this file download through the public API, which is based on Node.js+Express. However, I can't figure out the best way of doing so.
I guess I can download this file with Axios into a temporary file in the Node.js API container, but that seems to be prone to issues with these temporary files potentially piling up and requiring later cleanup. Is there a way to achieve such file download -> send further to a client without creating a temporary file?
Or what would be the most efficient and "clean" way of doing so if temporary files are unavoidable?
router.post('/route/:someid',
[someRequestVerificationMiddleware],
(req, res, next) => {
const myFileId = req.params.someid;
const downloadRequestParams= {
"id": myFileId
};
let dlPromise = axios.post(`http://myinternalservice:80`,
downloadRequestParams, {responseType: "stream"});
dlPromise.then(response => {
try {
let filename = response.headers["x-result-filename"];
//
// What would be the most efficient way to return the received file
// from response data to the client calling this route without creating
// too much garbage?
//
} catch (e) {
console.error(e);
}
})
.catch(e=>{
console.error(e);
res.status(500);
})
.finally(() => {
next();
})
});
module.exports = router;
res is a stream. You can simply pipe your axios stream to the response.
res.setHeader("content-type", "...");
return dlPromise.then((response) => response.data.pipe(res));
I am unable to understand how the event loop is processing my snippet. What I am trying to achieve is
to read from a csv
download a resource found in the csv
upload it to s3
write it into a new csv file
const readAndUpload = () => {
fs.createReadStream('filename.csv')
.pipe(csv())
.on('data', ((row: any) => {
const file = fs.createWriteStream("file.jpg");
var url = new URL(row.imageURL)
// choose whether to make an http or https request
let client = (url.protocol=="https:") ? https : http
const request = client.get(row.imageURL, function(response:any) {
// file save
response.pipe(file);
console.log('file saved')
let filePath = "file.jpg";
let params = {
Bucket: 'bucket-name',
Body : fs.createReadStream(filePath),
Key : "filename.jpg"
};
// upload to s3
s3.upload(params, function (err: any, data: any) {
//handle error
if (err) {
console.log("Error", err);
}
//success
if (data) {
console.log("Uploaded in:", data.Location);
row.imageURL = data.Location
writeData.push(row)
// console.log(writeData)
}
});
});
}))
.on('end', () => {
console.log("done reading")
const csvWriter = createCsvWriter({
path: 'out.csv',
header: [
{id: 'id', title: 'some title'}
]
});
csvWriter
.writeRecords(writeData)
.then(() => console.log("The CSV file was written successfully"))
})
}
Going by the log statements that I have added, done reading and The CSV file was written successfully is printed by before file saved. My understanding was that the end event is called after the data event, so I am unsure of where I am going wrong.
Thank you for reading!
I'm not sure if this is part of the problem or not, but you've got an extra set of parens in this part of the code. Change this:
.on('data', ((row: any) => {
.....
})).on('end', () => {
to this:
.on('data', (row: any) => {
.....
}).on('end', () => {
And, if the event handlers are set up properly, your .on('data', ...) event handler gets called before the .on('end', ....) for the same stream. If you put this:
console.log('at start of data event handler');
as the first line in that event handler, you will see it get called first.
But, your data event handler uses multiple asynchronous calls and nothing you have in your code makes the end event wait for all your processing to be done in the data event handler. So, since that processing takes awhile, it's natural that the end event would occur before you're done running all that asynchronous code on the data event.
In addition, if you ever can have more than one data event (which one normally would), you're going to have multiple data events in flight at the same time and since you're using a fixed filename, they will probably be overwriting each other.
The usual way to solve something like this is to to stream.pause() to pause the readstream at the start of the data event processing and then when all your asynchronous stuff is done, you can then stream.resume() to let it start going again.
You will need to get the right stream in order to pause and resume. You could do something like this:
let stream = fs.createReadStream('filename.csv')
.pipe(csv());
stream.on('data', ((row: any) => {
stream.pause();
....
});
Then, way inside your s3.upload() callback, you can call stream.resume(). You will also need much, much better error handling that you have or things will just get stuck if you get an error.
It also looks like you have other concurrency issues too where you call:
response.pipe(file);
And you then attempt to use the file without actually waiting for that .pipe() operation to be done (which is also asynchronous). Overall, this whole logic really needs a major cleanup. I don't understand what exactly you're trying to do in all the different steps to know how to write a totally clean and simpler version.
Im struggling to find material on this
I have a rest API, written in node.js, that uses mongoDB.
I want users to be able to upload images (profile pictures) and have them saved on the server (in mongoDB).
A few questions, Ive seen it is recommended to use GridFS, is this the best solution?
How do i send these files? Ive seen res.sendFile, but again is this the best solution?
If anyone has any material they can link me I would be appreciative
thanks
You won't be able to get the file object on the server directly. To get file object on the server, use connect-multiparty middleware. This will allow you to access the file on the server.
var multipart = require('connect-multiparty');
var multipartmiddleware = multipart();
var mv = require('mv');
var path = require('path');
app.post("/URL",multipartmiddleware,function(req,res){
var uploadedImage = req.files.file;
for (var i = 0; i < uploadedImage.length; i++) {
var tempPath = uploadedImage[i].path;
var targetPath = path.join(__dirname ,"../../../img/Ads/" + i + uploadedImage[i].name);
mv(tempPath, targetPath, function (err) {
if (err) { throw err; }
});
}
})
Use file system
Generally in any database you store the image location in the data as a string that tells the application where the image is stored on the file system.
Unless your database needs to be portable as a single unit, the storing of images inside of the database as binary objects generally adds unnecessary size and complexity to your database.
-Michael Stearne
In MongoDB, use GridFS for storing files larger than 16 MB.
- Mongo Documentation
Therefore unless your images will be over 16 MB, you should either store the file on a CDN (preferable) or the server's own file system and save its URL to user's document on the database.
Local file system implementation
This method uses Busboy to parse the photo upload.
in relevant html file:
<input type="file" title="Choose a file to upload" accept="image/*" autofocus="1">
Handler function for your photo upload route in server file (you will need to fill in the variables that apply to you and require the necessary modules):
function photoUploadHandlerFunction (req, res) {
var busboy = new Busboy({ headers: req.headers })
busboy.on('file', function (fieldname, file, filename, encoding, mimetype) {
const saveToDir = path.join(__dirname, uploadsPath, user.id)
const saveToFile = path.join(saveToDir, filename)
const pathToFile = path.join(uploadsPath, user.id, filename)
const writeStream = fs.createWriteStream(saveToFile)
createDirIfNotExist(saveToDir)
.then(pipeUploadToDisk(file, writeStream))
.then(findUserAndUpdateProfilePic(user, pathToFile))
.catch((err) => {
res.writeHead(500)
res.end(`Server broke its promise ${err}`)
})
})
busboy.on('finish', function () {
res.writeHead(200, { 'Connection': 'close' })
res.end("That's all folks!")
})
return req.pipe(busboy)
}
Where the promise functions createDirIfNotExist and pipeUploadToDisk could look like this:
function createDirIfNotExist (directory, callback) {
return new Promise(function (resolve, reject) {
fs.stat(directory, function (err, stats) {
// Check if error defined and the error code is "not exists"
if (err) {
if (err.code === 'ENOENT') {
fs.mkdir(directory, (err) => {
if (err) reject(err)
resolve('made folder')
})
} else {
// just in case there was a different error:
reject(err)
}
} else {
resolve('folder already existed')
}
})
})
}
function pipeUploadToDisk (file, writeStream) {
return new Promise((resolve, reject) => {
const fileWriteStream = file.pipe(writeStream)
fileWriteStream.on('finish', function () {
resolve('file written to file system')
})
fileWriteStream.on('error', function () {
reject('write to file system failed')
})
})
}
To answer your question 'How do I send these files?', I would need to know where to (MongoDB, to the client...). If you mean to the client, you could serve the static folder where they are saved.
If you still want to learn about implementing GridFs tutorialspoint have a good tutorial
More material
Good tutorial on handling form uploads
Tutorial using the node-formidable module
If you're using the mongoose odm you can use the mongoose-crate module and send the file wherever for storage.
Also, this is a good case for shared object storage like AWS S3 or Azure blob storage. If you are running a distributed setup in something like AWS, you usually don't want to store photos on the local server.
Store the url or key name in the database that points to the S3 object. This also integrates with CloudFront CDN pretty easily.
As suggested before. MultiPart for the actual upload.
I am using 'multiparty' parser to read and process multipart-form-data in my application. My app middleware reads the uploaded file contents, parses it on-the-fly, and if successful, sends to next middleware. If not, i have to abort the request and return error.
/**
* Middleware to read n parse uploaded multi-part form data
* for campaign creation.
* After successful parse, sends to next layer.
*
**/
var mp = require('multiparty');
module.exports = function(req, res, next) {
console.log("File processor at work...");
var form = new mp.Form();
form.on('part', function(part) {
if (!part.filename) { part.resume();}
else {
part.setEncoding('utf8');
part.on('readable', function() {
readFile(req, res, part);
});
part.on('error', function (e) {
console.log("Am i reached??");
//return res.status(400).json(e);
form.emit('error', e);
})
}
});
form.on('error', function(err) {
console.log("Error while processing upload.", 3);
res.status(400).json(err);
});
form.on('close', function() {
console.log("Reading finished. Forwarding to next layer...");
return next();
});
form.parse(req);
}
In case of any errors in during parsing (i.e. readFile()), I want to return HTTP error without having to consume the remaining part buffer. I am sure there could be a decent way to do this, but I am not getting it right.
I tried throwing exception from within my readFile(), trying to catch it in form.on('part', ...). Even though I was able to catch the exception, it didn't abort the flow. A return() from form.on(part,..) would would return from this evt handler function, but not from the outer function.
As per the nodejs streams documentation, I tried emitting error event from within readFile() and handle the error in part.on('error', ...). This also gives the same behaviour, and does not end the processing.
What am I missing here? Is there a proper way to tell the stream that I don't want to process it any further..?
I have a function which uploads a file to a server and returns a promise.
I'd like to check when each promise finishes and display a log of "Successfully deployed filename..".
I don't care about the order of the promises, But Q.all() doesn't help me since it returns only when all of the promises finished or failing fast when one failed.
I'd like to write something that checks whenever one of my promises finish and displays the log.
Since each file can be large, I want to user to be alerted what has been uploaded thus far.
When one of my files fails, the user will see something like:
Successfully deployed file1.txt
Successfully deployed file2.txt
Failed deploying file3.txt
Why not simply loop over your files (or promises)?
var files = ['file1.txt', 'file2.txt'/*, ...*/];
files.forEach(function (file) {
uploadFile(file).done(function () {
console.log('Successfully deployed ' + file);
}, function () {
console.log('Failed deployed ' + file);
});
});
#kamituel's answer works perfectly. If you need fail-fast behavior though (as you requested in the comment) you can just set a flag on failure.
var failed = false;
var files = ['file1.txt', 'file2.txt'/*, ...*/];
files.forEach(function (file) {
uploadFile(file).done(function () {
if (failed) return;
console.log('Successfully deployed ' + file);
}, function () {
if (failed) return;
failed = true;
console.log('Failed deployed ' + file);
});
});