I'm trying to implement a file uploading feature. I have watched and read many tutorials, but all of them require one or another library like multer, formidable, axios or something else. As far as I learnt, it can be done without any external library, so it is how I want it - with minimal to none additional libraries.
On frontend I pack the file into FormData and send it via fetch:
if (file) {
const fd = new FormData()
fd.append('file', file)
const res = await fetch('/api/files', {
method: 'POST',
body: fd
})
}
On backend the body of request looks like this:
-----------------------------26088851433443883591868954602
Content-Disposition: form-data; name="file"; filename="1.png"
Content-Type: image/png
�PNG
→
IHDR�♦☻g�V¶☺sRGB��∟�♦gAMA��♂�a♣ pHYs♫�♫�☺�o�d{�IDATx^��⌂��W}▼����č↔;��6�4�6m��Lg�5�;q�&D�ۦ�8Nm�4����n�
-----------------------------312671409112038349881281368983--
I tried so far:
sending file without packing it into FormData does not change anything
fs.writeFile() gives a file with about right size, but it is not valid file
fs.writeFile() with base64 option gives a file with only one line inside it
req.files.mv() which worked in tutorial with axios cannot read files and fails
req.body.mv() also fails with mv is not a function
What can I do about it to make it working? If it helps somehow, my project is written in NextJS.
Related
From the Shopify API, I receive a link to a large amount of JSONL. Using NodeJS, I need to read this data line-by-line, as loading it all at once would use lots of memory. When I hit the JSONL url from the web browser, it automatically downloads the JSONL file to my downloads folder.
Example of JSONL:
{"id":"gid:\/\/shopify\/Customer\/6478758936817","firstName":"Joe"}
{"id":"gid:\/\/shopify\/Order\/5044232028401","name":"#1001","createdAt":"2022-09-16T16:30:50Z","__parentId":"gid:\/\/shopify\/Customer\/6478758936817"}
{"id":"gid:\/\/shopify\/Order\/5044244480241","name":"#1003","createdAt":"2022-09-16T16:37:27Z","__parentId":"gid:\/\/shopify\/Customer\/6478758936817"}
{"id":"gid:\/\/shopify\/Order\/5057425703153","name":"#1006","createdAt":"2022-09-27T17:24:39Z","__parentId":"gid:\/\/shopify\/Customer\/6478758936817"}
{"id":"gid:\/\/shopify\/Customer\/6478771093745","firstName":"John"}
{"id":"gid:\/\/shopify\/Customer\/6478771126513","firstName":"Jane"}
I'm unsure how to process this data in NodeJS. Do I need to hit the url, download all of the data and store it in a temporary file, then process the data line-by-line? Or can I read the data line-by-line directly after hitting the url (via some sort of stream?) and process it without storing it in a temporary file on the server?
(The JSONL comes from https://storage.googleapis.com/ if that helps.)
Thanks.
using axios you can set the response to be a stream, and then using a buildin readline module, you can process your data line by line.
import axios from 'axios'
import { createInterface } from 'node:readline'
const response = await axios.get('https://raw.githubusercontent.com/zaibacu/thesaurus/master/en_thesaurus.jsonl', {
responseType: 'stream'
})
const rl = createInterface({
input: response.data
})
for await (const line of rl) {
// do something with the current line
const { word, synonyms } = JSON.parse(line)
console.log('word, synonyms: ', word, synonyms);
}
testing this there is barely any memory usage
You can easily run a great CLI tool called jq. Magic.
Unlike tying yourself to browser code, this code can be run in any way you need to parse JSONL.
jq -cs '.' doodoo.myshopify.com.export.jsonl > out.json
Would take my nicely just downloaded bulk file from a query and give me a very nice pure JSON data structure to play with, or save.
I'm trying to migrate one of my systems from RequestJS to the new (currently experimental) fetch feature of Node 17.5+ but I'm having issues with the uploading of files, I'm hoping someone can assist.
At the receiving end I have Multer which is doing a simple upload.single('document') which always returns an empty object, so I'm fairly sure the way I'm sending the data is wrong.
In order to send the request, I'm doing the below (this is not the exact code, it's just to give an idea):
var requestOptions = {
headers: {'Content-Type': 'multipart/form-data'},
formData: {
document: {
value: _bufferData,
options: {
filename: _fileName,
contentType: _mimeType
}
}
}
};
Then I'm obviously sending the request using:
fetch('http://myurl.com/fileUpload', requestOptions).then(....);
I'm guessing new fetch API doesn't process the formData property the same way requestJS does. Does anyone know how I should be doing this?
Thanks.
Update: I've investigated further and the issue appears to be due to the content-disposition header missing the filename. I'm not sure how Multer creates this header but I assume its from the formdata, where the filename is clearly being passed in.
I am using the One Drive API to grab a file with a node application using the axios library.
I am simply trying to save the file to the local machine (node is running locally).
I use the One Drive API to get the download document link, which does not require authentication (with https://graph.microsoft.com/v1.0/me/drives/[location]/items/[id]).
Then I make this call with the download document link:
response = await axios.get(url);
I receive a JSON response, which includes, among other things, the content-type, content-length, content-disposition and a data element which is the contents of the file.
When I display the JSON response to the console, the data portion looks like this:
data: 'PK\u0003\u0004\u0014\u0000\u0006\u0000\b\u0000\u0000\u0000!\u...'
If the document is simply text, I can save it easily using:
fs.writeFileSync([path], response.data);
But if the file is binary, like a docx file, I cannot figure out how to write it properly. Every time I try it seems to have the wrong encoding. I tried different encodings.
How do I save the file properly based on the type of file retrieved.
Have you tried using an encoding option of fs.writeFileSync of explicitly null, signifying the data is binary?
fs.writeFileSync([path], response.data, {
encoding: null
});
I am trying to write an app that will allow my users to upload files to my Google Cloud Storage account. In order to prevent overwrites and to do some custom handling and logging on my side, I'm using a Node.js server as a middleman for the upload. So the process is:
User uploads file to Node.js Server
Node.js server parses file, checks file type, stores some data in DB
Node.js server uploads file to GCS
Node.js server response to user's request with a pass/fail remark
I'm getting a little lost on step 3, of exactly how to send that file to GCS. This question gives some helpful insight, as well as a nice example, but I'm still confused.
I understand that I can open a ReadStream for the temporary upload file and pipe that to the http.request() object. What I'm confused about is how do I signify in my POST request that the piped data is the file variable. According to the GCS API Docs, there needs to be a file variable, and it needs to be the last one.
So, how do I specify a POST variable name for the piped data?
Bonus points if you can tell me how to pipe it directly from my user's upload, rather than storing it in a temporary file
I believe that if you want to do POST, you have to use a Content-Type: multipart/form-data;boundary=myboundary header. And then, in the body, write() something like this for each string field (linebreaks should be \r\n):
--myboundary
Content-Disposition: form-data; name="field_name"
field_value
And then for the file itself, write() something like this to the body:
--myboundary
Content-Disposition: form-data; name="file"; filename="urlencoded_filename.jpg"
Content-Type: image/jpeg
Content-Transfer-Encoding: binary
binary_file_data
The binary_file_data is where you use pipe():
var fileStream = fs.createReadStream("path/to/my/file.jpg");
fileStream.pipe(requestToGoogle, {end: false});
fileStream.on('end, function() {
req.end("--myboundary--\r\n\r\n");
});
The {end: false} prevents pipe() from automatically closing the request because you need to write one more boundary after you're finished sending the file. Note the extra -- on the end of the boundary.
The big gotcha is that Google may require a content-length header (very likely). If that is the case, then you cannot stream a POST from your user to a POST to Google because you won't reliably know what what the content-length is until you've received the entire file.
The content-length header's value should be a single number for the entire body. The simple way to do this is to call Buffer.byteLength(body) on the entire body, but that gets ugly quickly if you have large files, and it also kills the streaming. An alternative would be to calculate it like so:
var body_before_file = "..."; // string fields + boundary and metadata for the file
var body_after_file = "--myboundary--\r\n\r\n";
var fs = require('fs');
fs.stat(local_path_to_file, function(err, file_info) {
var content_length = Buffer.byteLength(body_before_file) +
file_info.size +
Buffer.byteLength(body_after_file);
// create request to google, write content-length and other headers
// write() the body_before_file part,
// and then pipe the file and end the request like we did above
But, that still kills your ability to stream from the user to google, the file has to be downloaded to the local disk to determine it's length.
Alternate option
...now, after going through all of that, PUT might be your friend here. According to https://developers.google.com/storage/docs/reference-methods#putobject you can use a transfer-encoding: chunked header so you don't need to find the files length. And, I believe that the entire body of the request is just the file, so you can use pipe() and just let it end the request when it's done. If you're using https://github.com/felixge/node-formidable to handle uploads, then you can do something like this:
incomingForm.onPart = function(part) {
if (part.filename) {
var req = ... // create a PUT request to google and set the headers
part.pipe(req);
} else {
// let formidable handle all non-file parts
incomingForm.handlePart(part);
}
}
Because the file should be generated dynamically, maybe I should use the fs modules's writeStream. But I couldn't find any example codes with my poor googling. Sorry.
More specifically, I want to give a CSV file or a PDF file with my datas in the MongoDB, when someone requests.
Anyone, please give me some hints.
Thanks.
With the express, I can implement like this.
app.get('/down2', function(req, res){
var filename = 'data.csv';
res.attachment(filename);
res.end('hello,world\nkeesun,hi', 'UTF-8'); //Actually, the data will be loaded form db.
});
How simple is it. Thanks.
you don't need fs, just stream your data from db in request handler
you can set file metadata using Content-Disposition header
Content-Type: image/jpeg
Content-Disposition: attachment; filename=genome.jpeg;
modification-date="Wed, 12 Feb 1997 16:29:51 -0500";
Content-Description: a complete map of the human genome