I have an express service that's taking a pdf file from my front-end and saving it to an s3 bucket. I'm running into issues trying to take the file and create a stream so that I can then pass that to the s3 upload function. I'm trying to avoid writing the file to disc so I don't think I can use fs.createReadStream() but I can't seem to find an alternative way to do it..
router.post('/upload', upload.single('my-pdf'), async (req, res, next) {
const file = req.file;
// Needs a file path not an actual file
const stream = fs.createReadStream(file);
return s3.upload(file).promise();
}
Any help or advice on how to get around this would be greatly appreciated.
Assuming that req.file.<name_of_upload_field> is a buffer holding the file contents, you can convert that to a readable stream via
var str = new stream.PassThrough();
str.end(req.file.<name_of_upload_field>);
return s3.upload(str).promise();
Related
Is there a better option for uploading a file?
In my case I need to upload a lot of small files (pdf or text) and I have to decide the best option, but anyway are there differences between these two methods?
Here two examples taken directly from the documentation
Save method: (Docs)
const {Storage} = require('#google-cloud/storage');
const storage = new Storage();
const myBucket = storage.bucket('my-bucket');
const file = myBucket.file('my-file');
const contents = 'This is the contents of the file.';
file.save(contents, function(err) {
if (!err) {
// File written successfully.
}
});
//-
// If the callback is omitted, we'll return a Promise.
//-
file.save(contents).then(function() {});
Upload method: (Docs)
const {Storage} = require('#google-cloud/storage');
const storage = new Storage();
const bucket = storage.bucket('albums');
//-
// Upload a file from a local path.
//-
bucket.upload('/local/path/image.png', function(err, file, apiResponse) {
// Your bucket now contains:
// - "image.png" (with the contents of `/local/path/image.png')
// `file` is an instance of a File object that refers to your new file.
});
At its core, both of these functions do the same thing. They upload a file to a bucket. One is just a function on bucket and the other a function on file. They both end up calling file.createWriteStream, so they have the same performance as well.
The functions behave differently in terms of upload type. file.save will default to a resumable upload unless you specify otherwise (you can set the resumable boolean on SaveOptions to false). bucket.upload will perform a multipart upload if the file is smaller than 5MB and a resumable upload otherwise. For bucket.upload, you can force a resumable or multipart upload by modifying the resumable boolean on UploadOptions.
Note that in the upcoming major version release (https://github.com/googleapis/nodejs-storage/pull/1876), this behavior will be unified. Both functions will default to a resumable upload regardless of file size. The behavior will be the same.
For small files, multipart uploads are recommended.
I've set up a small node js BE app, built with express and fastCsv module on top of it. The desired outcome would be to be able to download a csv file to the client side, without storing it anywhere inside the server, since the data is generated depending on user criteria.
So far I've been able to get somewhere it it, Im using streams, since that csv file could be pretty large depending on the user selection. Im pretty sure something is missing inside the code bellow:
const fs = require('fs');
const fastCsv = require('fast-csv');
.....
(inside api request)
.....
router.get('/', async(req, res) => {
const gatheredData ...
const filename = 'sometest.csv'
res.writeHead(200, {
'Content-Type': 'text/csv',
'Content-Disposition': 'attachment; filename=' + filename
})
const csvDataStream = fastCsv.write(data, {headers: true}).pipe(res)
})
The above code 'works' in some way as it does deliver back the response, but not the actual file, but the contents of the csv file, which I can view in the preview tab as a response. To sum up, Im trying to stream in that data, into a csv and push it to download file to client, and not store it on the server. Any tips or pointers are very much appreciated.
Here's what worked for me after created a CSV file on the server using the fast-csv package. You need to specify the full, absolute directory path where the output CSV file was created:
const csv = require("fast-csv");
const csvDir = "abs/path/to/csv/dir";
const filename = "my-data.csv";
const csvOutput = `${csvDir}/${filename}`;
console.log(`csvOutput: ${csvOutput}`); // full path
/*
CREATE YOUR csvOutput FILE USING 'fast-csv' HERE
*/
res.type("text/csv");
res.header("Content-Disposition", `attachment; filename="${filename}"`);
res.header("Content-Type", "text/csv");
res.sendFile(filename, { root: csvDir });
You need to make sure to change the response content-type and headers to "text/csv", and try enclosing the filename=... part in double-quotes, like in the above example.
I am sending audio data stored as a blob to my backend (node/express). When I save the file as .wav and attempt to use in the SpeechRecogition package in python it throws an error saying the "file does not start with RIFF id". So how can I add the headers to my blob file before I save it so that it is a correctly formatted .wav file? I can provide the code if necessary.
node.js file
var multer = require('multer');
var fs = require('fs'); //use the file system so we can save files
var uniqid = require('uniqid');
var spawn = require('child_process').spawn;
const storage = multer.memoryStorage()
var upload = multer({ storage: storage });
router.post('/api/test', upload.single('upl'), function (req, res) {
console.log(req.file);
console.log(req.file.buffer);
var id = uniqid();
fs.writeFileSync(id+".wav", Buffer.from(new Uint8Array(req.file.buffer))); //write file to server as .wav file
const scriptPath = 'handleAudio.py'
const process = spawn('python3', [__dirname+"/../"+scriptPath, "/home/bitnami/projects/sample/"+id+".wav", req.file.originalname, 'True']); //throws error about header in .wav
});
Also I had this same example working with a php endpoint that just saved the blob to a file with .wav extension and the python file accepted it. What could be different in the move_uploaded_file in php and what I am doing above with node?
Every .wav file needs a header specified by the WAVE file format, available here. While it's fine for you to build the header yourself, it's much easier to just use a proper lib to do the work for you.
One example is node-wav, which has a nice API to write WAVE files from raw PCM data (what you have at the moment). Example code is provided by the node-wav documentation.
After sending a file to an Express server which uses multer middleware, we are given the file's path and a buffer to the file contents.
app.post('/api/createProduct', upload.single('file'), (req, res) => {
var path = req.file.path // full path of uploaded file
var buff = req.file.buffer // buffer of entire file
// ... business logic
res.sendStatus(200);
});
After some "business logic", we may want to move the file to a new location or delete it.
Having access to both the path and the buffer, what would be the most efficient way to either delete the file or move it to a new location? Is there any particular situation where access to the buffer is desirable?
I'm using multer to read in multi-part form data. However, I don't actually want to upload it. I want to put its contents into a string. Is there a simple way to do this?
Non-file fields are not stored on disk when you use multer's DiskStorage (the default storage type).
However, if you want files to be stored in memory too, then you need to use multer's MemoryStorage which will store files as Buffers, which you can then convert to string if you like:
var storage = multer.memoryStorage();
var upload = multer({ storage: storage });
// ...
app.post('/profile', upload.single('aboutme'), function(req, res) {
console.log(req.file.buffer);
});