I use Node JS. (Sails JS framework, but that's less important).
Purpose
Download a CSV file that includes data transformed from MongoDB.
That's the scenario of the server on a response to a large data download request
Read data from MongoDB.
Transform the data format (to CSV).
Send the data to response. (Download).
So, the user is expected to download this large data to their browser.
I'm not sure what would be the best way to handle such request.
MongoDB has built-in support for streaming and Node.js clients can provide a readable stream for the cursor. All HTTP response objects also provide a way to stream the response body through a series of writes or you can pipe to a socket when using WebSockets. Most of the work will be offloaded to the MongoDB server and Node.js was developed for exactly these kinds of requirements.
If you're going to use HTTP on the client side, you can use fetch() and stream the response body into memory. Here is an excellent article that shows how to efficiently process a response for large a CSV file:
const res = await fetch('/big-data.csv')
const csvStream = response.body
.pipeThrough(new TextDecoder())
.pipeThrough(new CSVDecoder())
for (;;) {
const {row, done} = await csvStream.read()
if (done) break
drain(row)
}
Don't forget that both the server and client support encoded content, so make sure you compress the responses to further improve I/O.
It is always a good idea to post some code of what you have tried so far.
First, you would have to retrieve the data from MongoDB using something like mongoose or Waterline if you are using SailsJS
Secondly you can use a library like csv to convert the data to a csv file.
Once you have created the file you can return the file to the user using the sails response like so:
res.attachment('path/to/file.csv');
Related
I want to send to the client in the same request files from the dir and some content from the DB.
DB query -
const derivatives = await Derivative.findAll();
Here is the res -
res.status(200).send({
data: derivatives.map((derivative) => ({
id: derivative.id,
date: derivative.date,
})),
});
Here is the download -
const fileName = derivatives.map((name) => name.wex);
res.status(200).download(__dirname, `/assets/${fileName}`);
How can I add that to my response?
HTTP lets you specify a "content disposition" to indicate whether a response should be treated as a download, but doesn't support sending downloads arbitrarily, it has to be a response to a request. You can't have part of a response be a download and part be not, for a single request.
So if you need a file to be downloaded, and some JSON used to display some UI, you need to handle that in the client somehow. Either:
The client sends a request, and server returns JSON containing a URL for the download as well as the other data you wanted to send, and then the client requests a download of the URL through JavaScript
The client sends two requests, one for the download and one for the other data; this may complicate things on the server if you need to associate the two requests (want to do a database lookup only once for instance), but is simplest on the client.
The client sends a request, and the server returns a response containing the JSON data and the file data, packed in some way (the file data could be inside the JSON but that would be inefficient), and it's unpacked on the client (using JavaScript) and the client then constructs a Blob URL to "download" (in this case the data is already downloaded, so this just entails saving a file)
There are any number of ways you might pack the file and JSON data together, which is what /u/Quentin was alluding to. Sending both as one response may be better for performance, but you probably don't need to.
I am trying to upload a file to my NodeJs Server from My Mobile ReactNative App.
I tried to use FormData with Axios post but it is resulting in a NetworkError. Logging the FormData object before sending it gives me an object with an Array _parts that contains Arrays of my fields.
Also when I console.log the prototypes of FormData I only get two methods that I can use, which are append and getParts. I can't use any method that does exist in the documentation like getHeaders or getBoundary
Now If I want to make a file upload without using FormData, Should I send a fileStream of the picture I want to upload or just send the uri of the picture? I am using multer to capture the Files in my server.
What was causing the Network Error is me using a nested object inside dataForm.
//Other code onTop
const {location, ...other} = payload;
form.append("location", JSON.stringify(location));
...
I hope this might help someone.
Also Files Are Blobs, basically a readabaleStream. Read More About it Here
I am currently working on an application with the current tech stack:
Backend:
Mongoose
Express
Apollo
GraphQL
Frontend:
Vuejs
Apollo
GraphQL
I have succeeded in uploading files to the server using GraphQL, what I am stuck with is how to implement the 'download' feature. With a normal RESTApi endpoint I can use res.download(filePath) and it works. How do I do this with GraphQL since I don't want to use REST.
Or is there any other standard to go by in this scenario?
Thanks!
GraphQL uses JSON format, which represents as text format, not as binary.
If you don't want download files with REST, then you should:
Encode your file content into base64 string in the back end. Related question
Send this string as part of query response.
Save encoded base64 string as a file in the front end. Related question
But right architecture design is add a file link in the GraphQL response and use browser for downloading/rendering the file.
It's better to send a hashed & temporary link to download it
Save the file and hash the name on your static server (to limit access to other users)
The file should be temporary and should expire in a short time
Send the link of the file in response to API
I'm implementing a webserver using Koa. One of my request handlers needs to receive and store a large zip file. The file is NOT uploaded from a web page, but sent from another NodeJs application. Therefore it's not multipart encoded, but just applcation/octet-stream. How can I read such a file from the request?
I've noticed there is a request.socket object but I could not find any documentation on how to use it.
In other words I need the opposite to
this.body = fs.createReadStream(path);
I am trying to build a test app for learning Node.js. I came from wordpress background and Apache has setup most of backend logics for me. But now, I have to build my own. I have a question about how to serve JSON files from server side to client side. What is the workflow -- Backbone.js handle all client side Data manipulation, send/save/get/fetch from couchDB, serve JSON object from NODE.js backend?
I am using Express Microframework for building HTTP server, installed the Cradle middleware for access CouchDB NoSQL database. I successfully posted the data from Client side HTML (Jade template engine) to the CouchDB Database/Document and able to retrieve those data back from Server through Cradle middleware. Things work out great. But it was all done by Backend.
I want to use Backbone.js for my client side Javascript. Backbone.js is looking for JSON object which send back from the HTTP server. Cradle Middleware is able to create JSON object but only send them directly to the Jade Template, I could use Jade syntax for loop to iterate over the data object but it still not meet what I want for Backbone.js handle all the data entry. I realize that I need to get JSON obj via ajax ( either a file generated by HTTP then send back to the client OR send straight object to the client ). Cradle Middleware could not do that.
I did more research on this questions. I tried CouchApp, it does what I need. Using the Backbone.js to handling all the data, send/save/fetch data from CouchDB database. But it is a running in CouchApp, it is not an Express Node.js workflow. ( Maybe I am wrong or just do not how it work )
I tried backbone-couchdb.js. I read through the Details and still do not know it is going to help me to get what I want. ( Maybe need more tutorial or code example ). I am still thinking that I need a Backbone CouchDB driver to connect those two and somehow serving them by NODE.js backend.
Is there anybody who could tell me about how JSON file is being served by Node.js, how backbone.js interact with data save/fetch/get from CouchDB? What is the best practice / workflow? Other good resources, code examples, useful tools?
Cradle Middleware is able to create JSON object but only send them directly to the Jade Template
This is incorrect. You can just send the json back without rendering a template.
function(req, res, next){
db.view('user/byUsername', { key: 'luke' }, function (err, doc) {
res.send(doc); // or res.json(doc);
});
}