I've set up a small node js BE app, built with express and fastCsv module on top of it. The desired outcome would be to be able to download a csv file to the client side, without storing it anywhere inside the server, since the data is generated depending on user criteria.
So far I've been able to get somewhere it it, Im using streams, since that csv file could be pretty large depending on the user selection. Im pretty sure something is missing inside the code bellow:
const fs = require('fs');
const fastCsv = require('fast-csv');
.....
(inside api request)
.....
router.get('/', async(req, res) => {
const gatheredData ...
const filename = 'sometest.csv'
res.writeHead(200, {
'Content-Type': 'text/csv',
'Content-Disposition': 'attachment; filename=' + filename
})
const csvDataStream = fastCsv.write(data, {headers: true}).pipe(res)
})
The above code 'works' in some way as it does deliver back the response, but not the actual file, but the contents of the csv file, which I can view in the preview tab as a response. To sum up, Im trying to stream in that data, into a csv and push it to download file to client, and not store it on the server. Any tips or pointers are very much appreciated.
Here's what worked for me after created a CSV file on the server using the fast-csv package. You need to specify the full, absolute directory path where the output CSV file was created:
const csv = require("fast-csv");
const csvDir = "abs/path/to/csv/dir";
const filename = "my-data.csv";
const csvOutput = `${csvDir}/${filename}`;
console.log(`csvOutput: ${csvOutput}`); // full path
/*
CREATE YOUR csvOutput FILE USING 'fast-csv' HERE
*/
res.type("text/csv");
res.header("Content-Disposition", `attachment; filename="${filename}"`);
res.header("Content-Type", "text/csv");
res.sendFile(filename, { root: csvDir });
You need to make sure to change the response content-type and headers to "text/csv", and try enclosing the filename=... part in double-quotes, like in the above example.
Related
I'm trying to do a reverse image search using googlethis on an image the user uploads. It supports reverse image searching, but only with a Google-reachable image URL. Currently, I upload the image to file.io, which deletes it after it gets downloaded.
This is the current application flow:
User POSTs file -> Server uploads file to file.io -> Google downloads the file -> Server does things with the reverse image search
However, I want to skip the middleman and have Google download files directly from the server:
User POSTs file -> Server serves file at unique URL -> Google downloads the file -> Server deletes the file -> Server does things with the reverse image search
I've looked at Serving Temporary Files with NodeJs but it just shows how to serve a file at a static endpoint. If I added a route to /unique-url, the route would stay there forever (a very slow memory leak! Probably! I'm not really sure!)
The only way I can think of is to save each file with a UUID and add a parameter: /download?id=1234567890, which would probably work, but if possible, I want to do things in memory.
So:
How do I do this using normal files?
How do I do this in-memory?
Currently working (pseudo) code:
app.post('/', (req, res) => {
const imagePath = saveImageTemporarily(req)
const tempUrl = uploadToFileIo(imagePath)
const reverseImageResults = reverseGoogleSearch(tempUrl)
deleteFile(imagePath)
doThingsWithResults(reverseImageResults).then((result) => { res.send(result) })
}
The other answer is a good one if you are able to use Redis -- it offers lots of helpful features like setting a time-to-live on entries so they're disposed of automatically. But if you can't use Redis...
The basic idea here is that you want to expose a (temporary) URL like example.com/image/123456 from which Google can download an image. You want to store the image in memory until after Google accesses it. So it sounds like there are two (related) parts to this question:
Store the file in memory temporarily
Rather than saving it to a file, why not create a Buffer holding the image data. Once you're done with it, release your reference to the buffer and the Node garbage collector will dispose of it.
let image = Buffer.from(myImageData);
// do something with the image
image = null; // the garbage collector will dispose of it now
Serve the file when Google asks for it
This is a straightforward route which determines which image to serve based on a route parameter. The query parameter you mention will work, and there's nothing wrong with that. Or you could do it as a route parameter:
app.get('/image/:id', (req, res) => {
const id = req.params.id;
res.status(200).send(/* send the image data here */);
});
Putting it all together
It might look something like this:
// store image buffers here
const imageStore = {};
app.post('/image', (req, res) => {
// get your image data here; there are a number of ways to do this,
// so I leave it up to you
const imageData = req.body;
// and generate the ID however you want
const imageId = generateUuid();
// save the image in your store
imageStore[imageId] = imageData;
// return the image ID to the client
res.status(200).send(imageId);
});
app.get('/image/:id', (req, res) => {
const imageId = req.params.id;
// I don't know off the top of my head how to correctly send an image
// like this, so I'll leave it to you to figure out. You'll also need to
// set the appropriate headers so Google recognizes that it's an image
res.status(200).send(imageStore[imageid]);
// done sending? delete it!
delete imageStore[imageId];
});
I would use REDIS for the in-memory DB, and on the server, I would transform the image to base64 to store it in Redis.
In Redis, you can also set TTL on the images.
Check my code below
import {
nanoid
} from 'nanoid'
function base64_encode(file) {
// read binary data
var bitmap = fs.readFileSync(file);
// convert binary data to base64 encoded string
return new Buffer(bitmap).toString('base64');
}
app.post('/', async(req, res) => {
const client = redisClient;
const imagePath = saveImageTemporarily(req)
//const tempUrl = uploadToFileIo(imagePath)
var base64str = base64_encode(imagePath);
const id = nanoid()
await client.set(id, JSON.stringify({
id,
image: base64str
}));
const reverseImageResults = reverseGoogleSearch(JSON.parse(await client.get(id)).image)
await client.del(id);
doThingsWithResults(reverseImageResults).then((result) => {
res.send(result)
})
}
At my endpoint in my NodeJS server, after retrieving an audio file stored as a Buffer in MongoDB, I want to represent it with a URL (much like how you do with URL.createObjectURL(blob) in the frontend on the browser). I then plan to res.render() the URL in HTML through Handlebars on the client, so that the user can click on it to download it:
<a href={{url}}>Click me to download the file!</a>
In the NodeJs server, I have converted the MongoDB Buffer into a JavaScript ArrayBuffer through:
var buffer = Buffer.from(recordingFiles[0].blobFile);
var arrayBuffer = Uint8Array.from(buffer).buffer;
I am unsure where to proceed from here. I seen solutions using fs or res.download(), but they don't seem applicable to my situation. Thanks in advance for any help!
Hopefully this can help.
var blob = new Blob(BUFFER, {type: "audio mime type"});
var link = document.createElement('a');
link.href = window.URL.createObjectURL(blob);
var fileName = reportName;
link.download = fileName;
link.click();
Do you always need to preload the audio file onto the page?
If not, then I would advise you to add a separate endpoint to download the file on demand. The frontend link can send a get request to the endpoint and download the file only if the user clicked it.
Otherwise you'd always be downloading the buffer behind the scenes, even if the user didn't intend to download it. This is especially problematic on slow connections.
Frontend:
<a href={{`${baseUrl}/download/${audioId}`}}>Click me to download the file!</a>
Backend:
const stream = require('stream');
app.get('/download/:audioId', function (request, response) {
// Retrieve the tag from our URL path
const audioId = request.params.audioId;
const fileData; // TODO: Get file buffer from mongo.
const fileContents = Buffer.from(fileData, 'base64');
const readStream = new stream.PassThrough();
readStream.end(fileContents);
response.set('Content-disposition', 'attachment; filename=' + fileName);
response.set('Content-Type', '<your MIME type here>');
readStream.pipe(response);
});
A list of relevant MIME types can be found here.
I'm trying to analyze a file I'll be uploading from react, I need to know if it can be uploaded based on several factors.
I found https://github.com/TooTallNate/node-wav
It works great on nodejs and I'm trying to use it on react. The sample creates a readable stream and pipes it to the wav reader.
var fs = require('fs');
var wav = require('wav');
var file = fs.createReadStream('track01.wav');
var reader = new wav.Reader();
// the "format" event gets emitted at the end of the WAVE header
reader.on('format', function (format) {
//Format of the file
console.log(format);
});
file.pipe(reader);
Using FilePond controller I'm able to get a base64 string of the file. But I can't figure out how to pass it to the reader
this is what I have so far on ReactJS:
var reader = new wav.Reader();
reader.on('format', function (format) {
//Format of file
console.log('format', format);
});
const buffer = new Buffer(base64String, 'base64')
const readable = new Readable()
readable._read = () => { }
readable.push(buffer)
readable.push(null)
readable.pipe(reader)
But I get Error: bad "chunk id": expected "RIFF" or "RIFX", got "u+Zj"
Since this file works on NodeJS with the same lib is obvious I'm doing something wrong.
EDIT:
this was a problem with my Base64 string, this method works if anyone needs to analyze a wav on the frontend
I want to have persistent memory (store the user's progress) in a .json file in %AppData%. I tried doing this according to this post, but it doesn't work. For testing purposes I'm only working with storing one object.
The code below doesn't work at all. If I use fs.open(filePath, "w", function(err, data) { ... instead of readFile(..., it does create a json file in %AppData%, but then it doesn't write anything to it, it's always 0 bytes.
var nw = require('nw.gui');
var fs = require('fs');
var path = require('path');
var file = "userdata.json";
var filePath = path.join(nw.App.dataPath, file);
console.log(filePath); // <- This shows correct path in Application Data.
fs.readFile(filePath ,function (err, data) {
var idVar = "1";
var json = JSON.parse(data);
json.push("id :" + idVar);
fs.writeFile(filePath, JSON.stringify(json));
});
If anyone has any idea where I'm messing this up, I'd be grateful..
EDIT:
Solved, thanks to kailniris.
I was simply trying to parse an empty file
There is no json in the file you try to read. Before parsing data check if the file is empty. If it is then create an empty json, push the new data into it then write it to the file else parse the json in the file.
I'm working on an app that creates a PDF document on the server, then displays a Download Here button. When I click the button, the process appears to work. When I inspect the Network>Preview and Network>Headers tabs in the Chrome console I can see that the file has definitely been returned.
The problem is, it does not display and it does not offer the option to save. Am I missing a client side step here? My preferred outcome is either to give the user the option to save the file or to automatically begin the download to their default path.
Here is relevant the server code:
exports.show = function(req, res) {
var file = req.params.id;
var filePath = __dirname + '../../../lib/completedforms/';
var thisPath = path.resolve(filePath + file);
res.attachment(thisPath);
res.setHeader('Content-Type', 'application/pdf');
res.setHeader("Content-Disposition", "attachment");
res.download(thisPath);
};
Thanks in advance for any guidance here.
There's no need for both res.attachment() AND res.download(). Just use the latter.
Also, res.download() already sets the Content-Disposition header, so you can remove that too.
You can also simplify your path generation:
var thisPath = path.resolve(__dirname, '../../../lib/completedforms/', file);
Although you should probably sanitize file and/or check that thisPath is not some location where it shouldn't be. This will prevent someone from supplying a potentially malicious req.params.id value like ../../../../../../../etc/passwd.