I am trying to receive a file on Node.js Express framework (via express-fileupload middleware), and then POST it directly to another server via the request package without saving it on the first device's disk and then re-reading it.
const streamifier = require('streamifier');
const fileUpload = require('express-fileupload');
app.use(fileUpload());
app.post('/upload', function(req, res) {
var fileBuffer = req.files.upload.data;
var fileReadStream = streamifier.createReadStream(fileBuffer);
//Use 'request' to send fileReadStream to another API for additional processing
}
However, the receiving API does not receive any file when I run this code.
When I run this other code, however, it works fine, but fs is only able to create a ReadStream from a file on the server, so it would involve me saving the file to some temporary hard disk location first, reading it again, and then sending it to the remote API, which I think is a bit wasteful.
var fs = require('fs');
app.post('/upload', function(req, res) {
var fileReadStream = fs.createReadStream('test.txt');
//Use 'request' to send fileReadStream to another API for additional processing
}
Any idea why the 1st code block results in no file received by the external API, while the 2nd one works perfectly fine? It seems that fileReadStream would be the same in both cases.
Turns out I should have just been sending the buffer by itself without transforming it into a ReadStream. However, I did have to add the name to it, or the receiving API wouldn't recognize that I was sending a file.
const fileUpload = require('express-fileupload');
app.use(fileUpload());
app.post('/upload', function(req, res) {
var fileBuffer = req.files.upload.data;
fileBuffer.name = 'test.png';
//Use 'request' to send fileBuffer to another API for additional processing
}
Related
I'm looking for a way to request a csv file from any url to save it then. If possible only using fs, http, request and express modules.
For testing I tried it with
request('http://localhost:3000/data1.csv').pipe(fs.createWriteStream('data2.csv'))
but i always get as a resonse (data2.csv)
<pre>Cannot GET /data1.csv</pre>
Simplified Code
const fs = require('fs')
const request = require('request')
const express = require('express')
const app = express()
app.listen(3000)
app.get('/download', (req, res) => {
request('http://localhost:3000/data1.csv').pipe(fs.createWriteStream('data2.csv'))
})
The file data1.csv is saved in the root of my project folder. But is this also the root of my nodejs server from where I started it?
What I'm doing wrong here?
Thanks!
You need to have an explicit handler that returns data1.csv:
app.get('/data1.csv', (req, res) => {
res.sendFile('data1.csv');
});
I've got google extension, react frontend app and express server.
I use mediaRecorder to record my screen and insert it into frontend page.There is no problem, video works just fine in frontend
const blob = new Blob(chunks, { type: "video/mp4;" });
const savedVideo = document.getElementById("savedVideo");
chunks = [];
const videoURL = window.URL.createObjectURL(blob);
savedVideo.src = videoURL;
var tracks = stream.getTracks();
tracks[0].stop();
let response = await fetch('http://localhost:3001/upload', {
method: 'POST',
headers: {
'Content-Type': 'application/octet-stream',
},
body: blob
});
The problem starts when i send blob to server.I want to save video(Only on server side)I suppose problem is in handling blob on the server side, maybe I doing smth wrong, here my server code:
const express = require("express");
const cors = require('cors');
const fs = require('fs');
const app = express();
const port = 3001;
app.use(cors({
origin: 'http://localhost:3000'
}));
app.post("/upload", (req, res) => {
console.log('req.body', req.body)
req.on('readable', function(){
const data = req.read();
if(data) {
fs.createWriteStream('videeoo.mp4').write(data);
// also i didnt sure about this method to write file
}
console.log('data', data);
});
});
app.listen(port, () => {
console.log(`Server started at http://localhost:${port}`);
});
express logs
I'am waiting your best practices)Grasias!
In order to handle blobs in nodejs app.post() you should introduce express.raw() into it. Then you can create a blob from the buffer:
app.post('/raw/:cmd', express.raw({type: "*/*"}), async (req, res) => {
const buffer = req.body
const blob = new Blob([buffer], {type: "application/octet-stream"})
})
well... this is problematic... the req.read() doesn't normally process binary data. There's also a conceptual issue here: a video can potentially be huge, but in your application you're waiting for the whole file to be uploaded before you start writing it. So if you have 10 users, each uploading 10GB files, this is a problem. So you really want to store the file as it arrives, so that you only keep a few bytes in your memory at a time... but then what if you want to limit the size of the file? probably 10GB files is not something you want to deal with?
So... there are really a lot of corner cases and things to consider. In general, you don't want to handle these things manually. Luckily there are libraries like multer that can handle all these issues for you: https://expressjs.com/en/resources/middleware/multer.html you just define the destination directory, the max file size, etc and the library takes care of everything for you
I am Making http proxy program that check http url and If It is download link(content-type:octet-stream), I would get response and relay that response to other computer by using request.post and other computer download file with response that are given by http proxy.
Let's suppose that web proxy computer is A. And It is part of code of A. 192.168.5.253
if(contentType && (contentType== "application/octet-stream" || contentType == "application/gzip")){
console.log("remoteRes##app",remoteRes);
let filepath = req.url.split('/');
let FileName = getFilename(remoteRes, filepath);
let writeStream = fs.createWriteStream(FileName);
/*remoteRes is octect-stream response.
I can get file buffer If I use remoteRes.on(data, chunk => {...})*/
remoteRes.pipe(writeStream); //It works but I want to send file buffer to B without writing file.
.........
I can download file in A. but I want to send this response to pc B(192.168.5.32:10001) server.
So I want to streaming like this:
remoteRes.pipe(request.post('http://192.168.5.32:10001/upload));
And this is part of Server B(192.168.5.32) code
router.post('/upload', (req, res, next) => {
let wstream = fs.createWriteStream('ffff.txt');
req.pipe(wstream); //It dosen't work, but I want to do like this.
})
I want to get filebuffer in router.post('/upload'). It dosen't matter If It is post or put.
I saw that when I use remoteRes.pipe(request.post('http://192.168.5.32:10001/upload));
,I saw that request from ServerA get to ServerB. But I couldn't get file buffer in ServerB.
In short, I want to pipe response to request.post.
You need to store the incoming buffers using your own middleware, so it will be available in the routers request handler
Here you have a working example (you can save it and test it as a single file) :
//[SERVER B]
const express = require('express'); const app = express()
//:Middleware for the incoming stream
app.use(function(req, res, next) {
console.log("[request middleware] (buffer storing)")
req.rawBody = ''
req.on('data', function(chunk) {
req.rawBody += chunk
console.log(chunk) // here you got the incoming buffers
})
req.on('end', function(){next()})
});
//:Final stream handling inside the request
app.post('/*', function (req, res) {
/* here you got the complete stream */
console.log("[request.rawBody]\n",req.rawBody)
});
app.listen(3000)
//[SERVER A]
const request = require('request')
request('http://google.com/doodle.png').pipe(request.post('http://localhost:3000/'))
I hope you can extrapolate this for your specific use case.
I am serving up my web application using NodeJS server (ExpressJS) currently. One of the new requirements is for the users to be able to upload large videos (potentially in gigs) to the server. They will later then be able to download them again. Could I achieve this with the current stack?
I came across tutorials using Multer JS with "multipart/form-data" specified by the front-end. If I use this, would my Express be able to serve up other HTTP requests while writing a giant single video file to the server's disk?
Having done this myself (~20GB files, without multer) I can recommend the following (Probably most of which you have considered, but for completeness):
Choose an appropriate upload control (or write your own, but basically something that chunks up the data is best. I used plupload I think)
On the server make an API to handle the received chunk data, and write it out to a temp location during upload (You may want to hash and check individual pieces as well).
Once upload complete, check file (I used a hash generated by the client, and checked it against the uploaded data)
Multer should work fine. It'll stream the data as it comes in over HTTP to a file on disk. It won't lock up your server. Then in your endpoint you have access to that file location and can do whatever you need with it.
var express = require("express");
var app = express();
var async = require('async');
var fs = require('fs');
var client = redis.createClient();
var multiparty = require('multiparty');
var util = require('util');
app.post('/',function(req,res){
var form = new multiparty.Form();
form.parse(req, function(err, fields, files) {
client.get(fields.otp, function(err, reply) {
var oldpath = files.file[0].path;
var newpath = path + files.file[0].originalFilename;
fs.rename(oldpath, newpath, function (err) {
if (err) throw err;
res.write('File uploaded and moved!');
res.end();
});
}
}
app.listen(2001,function(){
console.log("Server is running on port 2001");
});
I have a Node server using express.
I was originally using body-parser, but that doesn't allow for file uploads. So, I switched to multer (the easiest integration with express). However, in order to get any of the req (specifically req.body), this is my code:
var multer = require('multer');
var upload = multer({ dest : 'uploads/' });
server.all('/example', function (req, res, next) {
var up = upload.single('photo')
up(req, res, function(err) {
console.log(req.body); // I can finally access req.body
});
}
The problem with this, is that not all of my routes need to upload a file. Do I need to waste the CPU on calling upload.single() for each route in order to get access to the body? upload.single('') ends up not uploading any file, but it's still precious time spent on the main thread.
It appears that upload.single() waits for the callback, so it may not be as big of a deal as I'm making it, but I don't like calling functions when I don't have to.
Is there a way around calling upload.single(), or am I just making a bigger deal out of this than it really is?
For text-only multipart forms, you could use any of the multer methods, which are .single(), .array(), fields()
For instance using .array()
var multer = require('multer');
var upload = multer({ dest : 'uploads/' });
server.all('/example', upload.array(), function (req, res, next) {
console.log(req.body);
});
It doesn't really matter which you use, as long as it's invoked without arguments Multer will only parse the text-fields of the form for you, no files