I want send a part of bigFile to serve by pure nodejs+axios+form-data.
nodejs v16.14.0
but I don't know how to send buffer direct to server
import FormData from 'form-data'
const buffer = fs.readFileSync(filePath)
const chunk = buffer.slice(start, end)
// some code ...
// below code can work
// but the way need write file to disk
fs.writeFileSync(chunkPath, chunk)
formData.append('file', fs.createReadStream(chunkPath))
// end below code can work
// below code doesn't work
import { Readable } from 'stream'
formData.append('file', Readable.from(chunk).read())
dont's save buffer to disk, and send to server
Related
I have HTTP call with Buffer image inside from NodeJS Server 1 (run at localhost:5000) to NodeJS Server 2 (run at localhost:5001)
There are the Server 1 axios call:
await axios.post('http://localhost:5001/', {
buffer: fileBuffer,
...otherData //
})
I read that Multer gets the image from req.files, but in my case they arrive as req.body.buffer
How can i do?
Solved
I haven't used Multer
On Server 2 use Buffer.from() for get Buffer from your http call (because is an JSON array) and then fs.writeFile() or fs.writeFileSync() write file on disk
This is the way I used (Server 1):
await axios.post('http://localhost:5001/', {
buffer: fileBuffer,
name: fileName
})
Server 2:
const { body } = req // This is the HTTP object, it contains buffer and name
const data = Buffer.from(body.buffer) // Transform array to buffer value
const path = './public/' + body.name // "./public/sample.jpg"
fs.writeFileSync(path, data) // writeFile and writeFileSync also accept Buffer
I have a Node.js (16.13.1) REST API using Express and one of my endpoints receives one or more uploaded files. The client (web app) uses FormData into which the files are appended. Once they're submitted to my API, the code there uses multer to grab the files from the request object.
Now I'm having trouble trying to send those same files to another API. multer attaches the files to req.files and each file object in that array has several properties one of which is buffer. I tried using the stream package's Duplex object to convert this buffer to a stream so that I could append the file to another FormData object, but when the server the second API is running on receives the request, I get an error from the web server saying that "a potentially dangerous request.form value was detected from the client.".
Any suggestions?
I am working on a nest project I was also facing this issue did some research and found that we need to create a Readable from the Buffer of that file and it's working for me.
// Controller
#UseInterceptors(FileInterceptor('file'))
async uploadFile(#UploadedFile() file: Express.Multer.File) {
return this.apiservice.upload(file);
}
// Service
uploadFile(file: Express.Multer.File) {
const readstream = Readable.from(file.buffer)
console.log(readstream)
const form = new FormData();
form.append('file', file, { filename: extra.filename });
const url = `api_endpoint`;
const config: AxiosRequestConfig = {
headers: {
'Content-Type': 'multipart/form-data'
},
};
return axios.post(url, form, config);
}
I am receiving a base64 encoded image from the client (as a screenshot) and would like to upload it to another server using multipart encoding
var base64Encoded="iVBORw0KGgoAAAANSUhEUgAAAAoAAABkCAYAAAC/zKGXAAAAK0lEQVR42u3KMQ0AAAgDsOFfJwcusMBL0t6tSToHJYqiKIqiKIqiKIri57hqIbTd/KhOmQAAAABJRU5ErkJggg==";
const x =Axios({url:"https://slack.com/api/files.upload",method:"POST", data:{title:"Hello", file: <INSERT HERE>, filetype:"png", channel: "testchannel"},headers:{Authorization: authorization,'Content-Type': 'multipart/form-data'}});
I've tried a lot here
Converting it to a buffer
var buffer = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAoAAABkCAYAAAC/zKGXAAAAK0lEQVR42u3KMQ0AAAgDsOFfJwcusMBL0t6tSToHJYqiKIqiKIqiKIri57hqIbTd/KhOmQAAAABJRU5ErkJggg==", 'base64');
Converting the buffer into a readable stream
var buffer = Buffer.from("iVBORw0KGgoAAAANSUhEUgAAAAoAAABkCAYAAAC/zKGXAAAAK0lEQVR42u3KMQ0AAAgDsOFfJwcusMBL0t6tSToHJYqiKIqiKIqiKIri57hqIbTd/KhOmQAAAABJRU5ErkJggg==", 'base64');
console.log(buffer);
const readable = new Readable();
readable._read = function(){};
readable.push(buffer)
readable.push(null)
None of these seem to work. Slack API simply gives me a invalid form data response
It works if i upload via postman
PS: I have to send axios request using the nodejs server, not browser
I'm developing a Web application to send images, videos, etc. to two monitors from an admin interface. I'm using ws in Node.js for the server side. I've implemented selecting images available on the server and external URLs and sending them to the clients, but I also wanted to be able to directly send images selected from the device with a file input. I managed to do it using base64 but I think it's pretty inefficient.
Currently I send a stringified JSON object containing the client to which the resource has to be sent, the kind of resource and the resource itself, parse it in the server and send it to the appropriate client. I know I can set the Websocket binaryType to blob and just send the File object, but then I'd have no way to tell the server which client it has to send it to. I tried using typeson and BSON to accomplish this, but it didn't work.
Are there any other ways to do it?
You can send raw binary data through the WebSocket.
It's quite easy to manage.
One option is to prepend a "magic byte" (an identifier that marks the message as non-JSON). For example, prepend binary messages with the B character.
All the server has to do is test the first character before collecting the binary data (if the magic byte isn't there, it's probably the normal JSON message).
A more serious implementation will attach a header after the magic byte (i.e., file name, total length, position of data being sent etc').
This allows the upload to be resumed on disconnections (send just the parts that weren't acknowledged as received.
Your server will need to split the data into magic byte, header and binary_data before processing. but it's easy enough to accomplish.
Hope this help someone.
According to socket.io document you can send either string, Buffer or mix both of them
On Client side:
function uploadFile(e, socket, to) {
let file = e.target.files[0];
if (!file) {
return
}
if (file.size > 10000000) {
alert('File should be smaller than 1MB')
return
}
var reader = new FileReader();
var rawData = new ArrayBuffer();
reader.onload = function (e) {
rawData = e.target.result;
socket.emit("send_message", {
type: 'attachment',
data: rawData
} , (result) => {
alert("Server has received file!")
});
alert("the File has been transferred.")
}
reader.readAsArrayBuffer(file);
}
on server side:
socket.on('send_message', async (data, cb) => {
if (data.type == 'attachment') {
console.log('Found binary data')
cb("Received file successfully.")
return
}
// Process other business...
});
I am using pure WebSocket without io, where you cannot mix content - either String or Binary. Then my working solution is like this:
CLIENT:
import { serialize } from 'bson';
import { Buffer } from 'buffer';
const reader = new FileReader();
let rawData = new ArrayBuffer();
ws = new WebSocket(...)
reader.onload = (e) => {
rawData = e.target.result;
const bufferData = Buffer.from(rawData);
const bsonData = serialize({ // whatever js Object you need
file: bufferData,
route: 'TRANSFER',
action: 'FILE_UPLOAD',
});
ws.send(bsonData);
}
Then on Node server side, the message is catched and parsed like this:
const dataFromClient = deserialize(wsMessage, {promoteBuffers: true}) // edited
fs.writeFile(
path.join('../server', 'yourfiles', 'yourfile.txt'),
dataFromClient.file, // edited
'binary',
(err) => {
console.log('ERROR!!!!', err);
}
);
The killer is promoteBuffer option in deserialize function.
I have a client code using form-data module to hit a url that returns a content-type of image/jpeg. Below is my code
var FormData = require('form-data');
var fs = require('fs');
var form = new FormData();
//form.append('POLICE', "hello");
//form.append('PAYSLIP', fs.createReadStream("./Desert.jpg"));
console.log(form);
//https://fbcdn-profile-a.akamaihd.net/hprofile-ak-xfp1/v/t1.0- 1/c8.0.50.50/p50x50/10934065_1389946604648669_2362155902065290483_n.jpg?oh=13640f19512fc3686063a4703494c6c1&oe=55ADC7C8&__gda__=1436921313_bf58cbf91270adcd7b29241838f7d01a
form.submit({
protocol: 'https:',
host: 'fbcdn-profile-a.akamaihd.net',
path: '/hprofile-ak-xfp1/v/t1.0-1/c8.0.50.50/p50x50/10934065_1389946604648669_2362155902065290483_n.jpg?oh=13640f19512fc3686063a3494c6c1&oe=55ADCC8&__gda__=1436921313_bf58cbf91270adcd7b2924183',
method: 'get'
}, function (err, res) {
var data = "";
res.on("data", function (chunks) {
data += chunks;
});
res.on("end", function () {
console.log(data);
console.log("Response Headers - " + JSON.stringify(res.headers));
});
});
I'm getting some chunk data and the response headers i received was
{"last-modified":"Thu, 12 Feb 2015 09:49:26 GMT","content-type":"image/jpeg","timing-allow-origin":"*","access-control-allow-origin":"*","content-length":"1443","cache-control":"no-transform, max-age=1209600","expires":"Thu, 30 Apr 2015 07:05:31 GMT","date":"Thu, 16 Apr 2015 07:05:31 GMT","connection":"keep-alive"}
I am now stuck as how to process the response that i received to a proper image.I tried base64 decoding but it seemed to be a wrong approach any help will be much appreciated.
I expect that data, once the file has been completely downloaded, contains a Buffer.
If that is the case, you should write the buffer as is, without any decoding, to a file:
fs.writeFile('path/to/file.jpg', data, function onFinished (err) {
// Handle possible error
})
See fs.writeFile() documentation - you will see that it accepts either a string or a buffer as data input.
Extra awesomeness by using streams
Since the res object is a readable stream, you can simply pipe the data directly to a file, without keeping it in memory. This has the added benefit that if you download really large file, Node.js will not have to keep the whole file in memory (as it does now), but will write it to the filesystem continuously as it arrives.
form.submit({
// ...
}, function (err, res) {
// res is a readable stream, so let's pipe it to the filesystem
var file = fs.createWriteStream('path/to/file.jpg')
res.on('end', function writeDone (err) {
// File is saved, unless err happened
})
.pipe(file) // Send the incoming file to the filesystem
})
The chunk you got is the raw image. Do whatever it is you want with the image, save it to disk, let the user download it, whatever.
So if I understand your question clearly, you want to download a file from an HTTP endpoint and save it to your computer, right? If so, you should look into using the request module instead of using form-data.
Here's a contrived example for downloading things using request:
var fs = require('fs');
var request = require('request')
request('http://www.example.com/picture.jpg')
.pipe(fs.createWriteStream('picture.jpg'))
Where 'picture.jpg' is the location to save to disk. You can open it up using a normal file browser.