After sending a file to an Express server which uses multer middleware, we are given the file's path and a buffer to the file contents.
app.post('/api/createProduct', upload.single('file'), (req, res) => {
var path = req.file.path // full path of uploaded file
var buff = req.file.buffer // buffer of entire file
// ... business logic
res.sendStatus(200);
});
After some "business logic", we may want to move the file to a new location or delete it.
Having access to both the path and the buffer, what would be the most efficient way to either delete the file or move it to a new location? Is there any particular situation where access to the buffer is desirable?
Related
I'm trying to do a reverse image search using googlethis on an image the user uploads. It supports reverse image searching, but only with a Google-reachable image URL. Currently, I upload the image to file.io, which deletes it after it gets downloaded.
This is the current application flow:
User POSTs file -> Server uploads file to file.io -> Google downloads the file -> Server does things with the reverse image search
However, I want to skip the middleman and have Google download files directly from the server:
User POSTs file -> Server serves file at unique URL -> Google downloads the file -> Server deletes the file -> Server does things with the reverse image search
I've looked at Serving Temporary Files with NodeJs but it just shows how to serve a file at a static endpoint. If I added a route to /unique-url, the route would stay there forever (a very slow memory leak! Probably! I'm not really sure!)
The only way I can think of is to save each file with a UUID and add a parameter: /download?id=1234567890, which would probably work, but if possible, I want to do things in memory.
So:
How do I do this using normal files?
How do I do this in-memory?
Currently working (pseudo) code:
app.post('/', (req, res) => {
const imagePath = saveImageTemporarily(req)
const tempUrl = uploadToFileIo(imagePath)
const reverseImageResults = reverseGoogleSearch(tempUrl)
deleteFile(imagePath)
doThingsWithResults(reverseImageResults).then((result) => { res.send(result) })
}
The other answer is a good one if you are able to use Redis -- it offers lots of helpful features like setting a time-to-live on entries so they're disposed of automatically. But if you can't use Redis...
The basic idea here is that you want to expose a (temporary) URL like example.com/image/123456 from which Google can download an image. You want to store the image in memory until after Google accesses it. So it sounds like there are two (related) parts to this question:
Store the file in memory temporarily
Rather than saving it to a file, why not create a Buffer holding the image data. Once you're done with it, release your reference to the buffer and the Node garbage collector will dispose of it.
let image = Buffer.from(myImageData);
// do something with the image
image = null; // the garbage collector will dispose of it now
Serve the file when Google asks for it
This is a straightforward route which determines which image to serve based on a route parameter. The query parameter you mention will work, and there's nothing wrong with that. Or you could do it as a route parameter:
app.get('/image/:id', (req, res) => {
const id = req.params.id;
res.status(200).send(/* send the image data here */);
});
Putting it all together
It might look something like this:
// store image buffers here
const imageStore = {};
app.post('/image', (req, res) => {
// get your image data here; there are a number of ways to do this,
// so I leave it up to you
const imageData = req.body;
// and generate the ID however you want
const imageId = generateUuid();
// save the image in your store
imageStore[imageId] = imageData;
// return the image ID to the client
res.status(200).send(imageId);
});
app.get('/image/:id', (req, res) => {
const imageId = req.params.id;
// I don't know off the top of my head how to correctly send an image
// like this, so I'll leave it to you to figure out. You'll also need to
// set the appropriate headers so Google recognizes that it's an image
res.status(200).send(imageStore[imageid]);
// done sending? delete it!
delete imageStore[imageId];
});
I would use REDIS for the in-memory DB, and on the server, I would transform the image to base64 to store it in Redis.
In Redis, you can also set TTL on the images.
Check my code below
import {
nanoid
} from 'nanoid'
function base64_encode(file) {
// read binary data
var bitmap = fs.readFileSync(file);
// convert binary data to base64 encoded string
return new Buffer(bitmap).toString('base64');
}
app.post('/', async(req, res) => {
const client = redisClient;
const imagePath = saveImageTemporarily(req)
//const tempUrl = uploadToFileIo(imagePath)
var base64str = base64_encode(imagePath);
const id = nanoid()
await client.set(id, JSON.stringify({
id,
image: base64str
}));
const reverseImageResults = reverseGoogleSearch(JSON.parse(await client.get(id)).image)
await client.del(id);
doThingsWithResults(reverseImageResults).then((result) => {
res.send(result)
})
}
I have an express service that's taking a pdf file from my front-end and saving it to an s3 bucket. I'm running into issues trying to take the file and create a stream so that I can then pass that to the s3 upload function. I'm trying to avoid writing the file to disc so I don't think I can use fs.createReadStream() but I can't seem to find an alternative way to do it..
router.post('/upload', upload.single('my-pdf'), async (req, res, next) {
const file = req.file;
// Needs a file path not an actual file
const stream = fs.createReadStream(file);
return s3.upload(file).promise();
}
Any help or advice on how to get around this would be greatly appreciated.
Assuming that req.file.<name_of_upload_field> is a buffer holding the file contents, you can convert that to a readable stream via
var str = new stream.PassThrough();
str.end(req.file.<name_of_upload_field>);
return s3.upload(str).promise();
I've set up a small node js BE app, built with express and fastCsv module on top of it. The desired outcome would be to be able to download a csv file to the client side, without storing it anywhere inside the server, since the data is generated depending on user criteria.
So far I've been able to get somewhere it it, Im using streams, since that csv file could be pretty large depending on the user selection. Im pretty sure something is missing inside the code bellow:
const fs = require('fs');
const fastCsv = require('fast-csv');
.....
(inside api request)
.....
router.get('/', async(req, res) => {
const gatheredData ...
const filename = 'sometest.csv'
res.writeHead(200, {
'Content-Type': 'text/csv',
'Content-Disposition': 'attachment; filename=' + filename
})
const csvDataStream = fastCsv.write(data, {headers: true}).pipe(res)
})
The above code 'works' in some way as it does deliver back the response, but not the actual file, but the contents of the csv file, which I can view in the preview tab as a response. To sum up, Im trying to stream in that data, into a csv and push it to download file to client, and not store it on the server. Any tips or pointers are very much appreciated.
Here's what worked for me after created a CSV file on the server using the fast-csv package. You need to specify the full, absolute directory path where the output CSV file was created:
const csv = require("fast-csv");
const csvDir = "abs/path/to/csv/dir";
const filename = "my-data.csv";
const csvOutput = `${csvDir}/${filename}`;
console.log(`csvOutput: ${csvOutput}`); // full path
/*
CREATE YOUR csvOutput FILE USING 'fast-csv' HERE
*/
res.type("text/csv");
res.header("Content-Disposition", `attachment; filename="${filename}"`);
res.header("Content-Type", "text/csv");
res.sendFile(filename, { root: csvDir });
You need to make sure to change the response content-type and headers to "text/csv", and try enclosing the filename=... part in double-quotes, like in the above example.
I am trying to upload image to backblaze-b2 using hapi.js. I am using a plugin called easy-backblaze but while using it I need to mention the path of the file. But While importing the file using Hapijs I am not able to understand how to get that path.
This is the Code I have written and here in b2.uploadFile I need to mention the path of the file on local Drive:
var B2 = require('easy-backblaze');
var b2 = new B2('accountId', 'applicationKey');
b2.uploadFile('C:/Users/Lovika/Desktop/addDriver.png', {
name: 'addDriver.png', // Optional, renames file
bucket: 'testBucket', // Optional, defaults to first bucket
}, function(err, res) {
console.log('Done!', err, res);
});
Do I need to upload the file to the server first and then to backblaze or Is there any way to upload the file directly to backblaze
I'm using multer to read in multi-part form data. However, I don't actually want to upload it. I want to put its contents into a string. Is there a simple way to do this?
Non-file fields are not stored on disk when you use multer's DiskStorage (the default storage type).
However, if you want files to be stored in memory too, then you need to use multer's MemoryStorage which will store files as Buffers, which you can then convert to string if you like:
var storage = multer.memoryStorage();
var upload = multer({ storage: storage });
// ...
app.post('/profile', upload.single('aboutme'), function(req, res) {
console.log(req.file.buffer);
});