I have an internal HTTP Post API that generates files, size 5-10mb each. This service is not modifiable.
I want to "proxy" this file download through the public API, which is based on Node.js+Express. However, I can't figure out the best way of doing so.
I guess I can download this file with Axios into a temporary file in the Node.js API container, but that seems to be prone to issues with these temporary files potentially piling up and requiring later cleanup. Is there a way to achieve such file download -> send further to a client without creating a temporary file?
Or what would be the most efficient and "clean" way of doing so if temporary files are unavoidable?
router.post('/route/:someid',
[someRequestVerificationMiddleware],
(req, res, next) => {
const myFileId = req.params.someid;
const downloadRequestParams= {
"id": myFileId
};
let dlPromise = axios.post(`http://myinternalservice:80`,
downloadRequestParams, {responseType: "stream"});
dlPromise.then(response => {
try {
let filename = response.headers["x-result-filename"];
//
// What would be the most efficient way to return the received file
// from response data to the client calling this route without creating
// too much garbage?
//
} catch (e) {
console.error(e);
}
})
.catch(e=>{
console.error(e);
res.status(500);
})
.finally(() => {
next();
})
});
module.exports = router;
res is a stream. You can simply pipe your axios stream to the response.
res.setHeader("content-type", "...");
return dlPromise.then((response) => response.data.pipe(res));
Related
I´ve got a weird problem.
Using Node, React, Express, MongoDB -> MERN Stack.
So my page generates a PDF file which then gets send to the backend (as blob data) and is being stored on there.
The problem I have, now I need to send a payment ID along with that blob data to save the order in the data base. I need both in one post request, to make it as smooth as possible:
await axios
.post(process.env.REACT_APP_SERVER_API + '/payment/cash', {
blobData,
paymentId
})
.then(async (res) => ...
like so.
Before, when I just sent the blob data, I could simply access the data in the backend by writing:
exports.createCashOrder = async (req, res) => {
const { filename } = req.file; // THIS RIGHT HERE
const fileHash = await genFileHash(filename);
try {
await saveOrder(filename, fileHash, "cash", paymentId);
//await sendOrderCreatedEmail(req.body, fileHash);
//await sendOrderReceivedConfirmEmail(req.body, fileHash);
res.send({ filename: filename });
}
But that doesn't work anymore. I dont have access to that file object anymore when sending that request object.
Neither by trying
req.body.blobData
req.body.blobData.file
req.file
Any idea how to achieve that, except from making two seperate post requests?
Glad for any help, cheers!
Send the data as a form
await axios
.postForm(process.env.REACT_APP_SERVER_API + '/payment/cash', {
blobData,
paymentId
})
.then(async (res) => ...
And then use multer middleware to handle the form in express.
I am using Formidable with Express in nodeJS in an attempt to have a simple single file upload scheme. I have confirmed that a file is actually sent over from the client-side, but where it seems to run into troubles is on the server-side.
index.js
app.post('/', (req, res) => {
const form = formidale();
form.on('file', (filename, file) => {
fs.rename(file.path, `./data/nodes.csv`, err => {
if (err) {
console.log(`There was an error in downloading a CSV file: ${err}`);
return null;
}
else {
console.log("CSV file has been uploaded correctly.");
}
});
});
form.on('error', err => {
console.log(`There was an error in downloading a CSV file: ${err}`);
return null;
});
form.on('end', () => {
console.log(fs.readFileSync('./data/nodes.csv')); // test to see if file exists
const nodes = assignMetrics();
console.log(nodes);
return nodes;
});
form.parse(req);
});
}
The main trouble I seem to find is that the form.on('end', ...) event does not seem to wait till the file has finished uploading to fire. I have confirmed this by trying to read the file in the event, but by that point it doesn't exist? The documentation though appears to suggest it is only meant to fire "after all files have been flushed [from the APIs pipe it infers]".
There appears to be no other events available that might wait till the file has been uploaded to be called? I also don't want to start throwing in layers of promises and such unless it is the only option, as each new layer of promises I find is a chance for unintended effects to happen.
I am making a request with a mandatory request and I need to send the response to the immediate user. I mean, I need to send the data just when I receive it.
I want request to make the request and as it obtains the data, it will send it to the user in real time.
I have the following code and all I can do in this way is save the file and then send it, which does not work at all well, because it takes even longer to send the file
const def = (req,res)=> {
const request = request.get("url")
const getFile = fs.createWriteStream("path")
request.on("error", ()=> {
res.send("error")
})
request.pipe(getFile)
getFile.on("finish", ()=>{
const sendFile = fs.createReadStream("path")
sendFile.on("data", (chunk)=>{
res.send(chunk)
})
sendFile.on("finish", ()=> {
res.end()
})
})
}
This works but has several problems.
I need to save the file
Until it is downloaded to the server, it cannot be sent
It is much slower than sending it in real time
I need something maybe like the following (I know that this does not exist in such a way but I try to give an idea)
request.on("data", (chunk)=>{
res.send(chunk)
})
request.on("finish", ()=> res.end())
//or
getFile.on("data", (chunk)=>{
res.send(chunk)
})
getFile.on("finish", () => res.end())
Please help me
Just pipe the response directly to the client:
const def = (req, res) => {
request
.get("url")
.on("error", () => {
res.send("error")
})
.pipe(res)
}
Official documentation.
I am working on NodeJs/Express project and need to show in the browesr pdf file that is stored in
/public/images
Here is relevant router code:
router.post('/show_file', async (req,res)=>{
try {
let path = './public/images/1.pdf'
var data =fs.readFileSync(path);
res.contentType("application/pdf");
res.send(data);
} catch (err) {
res.status(500)
console.log(err)
res.send(err.message)
}
})
I don't get any errors but nothing is happening ie.browser is not opening etc.
Thanks in advance for any guidance.
The first change that I would do is to remove the async. It will just mess out the code with unneeded Promises.
Second, I removed the need to catch the exception, verifying the existence of the file with fs.existsSync(path). Try to not to rise exceptions as often as possible. If you know something can rise an exception, test it.
Last, and most important, I created a reading stream of the file and piped the result to the response with fs.createReadStream(path).pipe(res). This way, the client recieves the file as it is read and your memory is spared. Great for large files.
Reading a file can be memory intensive, so loading it all in memory is a bad practice. You just need a handfull of request to overload your machine.
You can read more on the pipe method here.
In this example, any GET call to /router/show_file will return the pdf.
const express = require('express')
const app = express()
const fs = require('fs')
const router = express.Router()
router.get('/show_file', (req, res) => {
const path = './public/images/1.pdf'
if (fs.existsSync(path)) {
res.contentType("application/pdf");
fs.createReadStream(path).pipe(res)
} else {
res.status(500)
console.log('File not found')
res.send('File not found')
}
})
app.use('/router', router) // Here we pass the router to the app with a path
app.listen(9999, () => console.log('Listening to port 9999'))
I'm trying to set up HTTP2 for an Express app I've built. As I understand, Express does not support the NPM http2 module, so I'm using SPDY. Here's how I'm thinking to go about it-I'd appreciate advice from people who've implemented something similar.
1) Server setup-I want to wrap my existing app with SPDY, to keep existing routes. Options are just an object with a key and a cert for SSL.
const app = express();
...all existing Express stuff, followed by:
spdy
.createServer(options, app)
.listen(CONFIG.port, (error) => {
if (error) {
console.error(error);
return process.exit(1)
} else {
console.log('Listening on port: ' + port + '.')
}
});
2) At this point, I want to enhance some of my existing routes with a conditional PUSH response. I want to check to see if there are any updates for the client making a request to the route (the client is called an endpoint, and the updates are an array of JSON objects called endpoint changes,) and if so, push to the client.
My idea is that I will write a function which takes res as one of its parameters, save the endpoint changes as a file (I haven't found a way to push non-file data,) and then add them to a push stream, then delete the file. Is this the right approach? I also notice that there is a second parameter that the stream takes, which is a req/res object-am I formatting it properly here?
const checkUpdates = async (obj, res) => {
if(res.push){
const endpointChanges = await updateEndpoint(obj).endpointChanges;
if (endpointChanges) {
const changePath = `../../cache/endpoint-updates${new Date().toISOString()}.json`;
const savedChanges = await jsonfile(changePath, endpointChanges);
if (savedChanges) {
let stream = res.push(changePath, {req: {'accept': '**/*'}, res: {'content-type': 'application/json'}});
stream.on('error', function (err) {
console.log(err);
});
stream.end();
res.end();
fs.unlinkSync(changePath);
}
}
}
};
3) Then, within my routes, I want to call the checkUpdates method with the relevant parameters, like this:
router.get('/somePath', async (req, res) => {
await checkUpdates({someInfo}, res);
ReS(res, {
message: 'keepalive succeeded'
}, 200);
}
);
Is this the right way to implement HTTP2?