I am making a request with a mandatory request and I need to send the response to the immediate user. I mean, I need to send the data just when I receive it.
I want request to make the request and as it obtains the data, it will send it to the user in real time.
I have the following code and all I can do in this way is save the file and then send it, which does not work at all well, because it takes even longer to send the file
const def = (req,res)=> {
const request = request.get("url")
const getFile = fs.createWriteStream("path")
request.on("error", ()=> {
res.send("error")
})
request.pipe(getFile)
getFile.on("finish", ()=>{
const sendFile = fs.createReadStream("path")
sendFile.on("data", (chunk)=>{
res.send(chunk)
})
sendFile.on("finish", ()=> {
res.end()
})
})
}
This works but has several problems.
I need to save the file
Until it is downloaded to the server, it cannot be sent
It is much slower than sending it in real time
I need something maybe like the following (I know that this does not exist in such a way but I try to give an idea)
request.on("data", (chunk)=>{
res.send(chunk)
})
request.on("finish", ()=> res.end())
//or
getFile.on("data", (chunk)=>{
res.send(chunk)
})
getFile.on("finish", () => res.end())
Please help me
Just pipe the response directly to the client:
const def = (req, res) => {
request
.get("url")
.on("error", () => {
res.send("error")
})
.pipe(res)
}
Official documentation.
Related
I´ve got a weird problem.
Using Node, React, Express, MongoDB -> MERN Stack.
So my page generates a PDF file which then gets send to the backend (as blob data) and is being stored on there.
The problem I have, now I need to send a payment ID along with that blob data to save the order in the data base. I need both in one post request, to make it as smooth as possible:
await axios
.post(process.env.REACT_APP_SERVER_API + '/payment/cash', {
blobData,
paymentId
})
.then(async (res) => ...
like so.
Before, when I just sent the blob data, I could simply access the data in the backend by writing:
exports.createCashOrder = async (req, res) => {
const { filename } = req.file; // THIS RIGHT HERE
const fileHash = await genFileHash(filename);
try {
await saveOrder(filename, fileHash, "cash", paymentId);
//await sendOrderCreatedEmail(req.body, fileHash);
//await sendOrderReceivedConfirmEmail(req.body, fileHash);
res.send({ filename: filename });
}
But that doesn't work anymore. I dont have access to that file object anymore when sending that request object.
Neither by trying
req.body.blobData
req.body.blobData.file
req.file
Any idea how to achieve that, except from making two seperate post requests?
Glad for any help, cheers!
Send the data as a form
await axios
.postForm(process.env.REACT_APP_SERVER_API + '/payment/cash', {
blobData,
paymentId
})
.then(async (res) => ...
And then use multer middleware to handle the form in express.
I know what is wrong with my code and I have looked into the best way of solving it, however with my lack of experience, I am having a hard time finding a good answer.
I need my first route(/data) to be fully completed before the second(/logo) express route sends the data. In short, I just need the variable symbolUrl to be completed before it goes into the second fetch call. Here is the code down below to explain
app.use(express.static('public'));
const url =
'https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest';
const qString =
'?CMC_PRO_API_KEY=' + process.env.apiKey + '&start=1&limit=10&convert=USD';
let symbol = [];
app.get('/data', async (req, res) => {
const fetch_res = await fetch(url + qString);
const coinData = await fetch_res.json();
for (let i = 0; i < 9; i++) {
symbol.push(coinData.data[i]['symbol']);
};
res.json(coinData);
});
app.get('/logo', async (req, res) => {
const symbolUrl = symbol.join(',');
const url2 = 'https://pro-api.coinmarketcap.com/v1/cryptocurrency/info';
const qString2 = `?CMC_PRO_API_KEY=${apiKey}%symbol=${symbolUrl}`;
const fetch_res2 = await fetch(url2 + qString2);
const coinLogo = await fetch_res2.json();
res.json(coinLogo);
});
The issue I am trying to solve with this project is that I want to send the data(/data) to be sent to the front end first because this API call will load the majority of the page. Then my second call will load images and other larger files afterward. HOWEVER, the API I am working with to get the logos(images) of the specific crypto coins I want, I need a different endpoint as well as use %symbol=${symbolUrl} in the API call to get the correct tokens I want to call.
client code:
fetch('http://localhost:2000/data')
.then(async (response) => {
return response.json();
})
.then(async (data) => {
const parsedData = data['data'];
// console.log(data['data'][0]['name'])
await parsedData.forEach((element) => {
// this just has about 20 lines of code generating the the look of the page. It works as intended
});
fetch('http://localhost:2000/logo')
.then(async (response) => {
return response.json();
})
.then(async (logo) => {
console.log(logo)});
***I have tried putting this in an async function and awaiting the first fetch call
All I need to be done is for app.get(/data) to be fully complete before doing my second app.get. I have done testing and I know that is the issue. I apologize if it is something easy, but I couldn't find anything on making an app.get synchronous and I have tried putting both in a async function, however that did not work.
You cannot send responses in fragments like you're trying to do, it would throw an error saying Can't set headers after they are sent to client
The proper method to implement what you are trying to do is to define the first layer as middleware, and then allow the second layer to return the response. Here layer basically means a function handler.
In order to control when the execution passes to the next layer / next function handler, express has a third parameter (request, response, next). You're only using request and response, researching about next will solve your concern.
Express next function, what is it really for?
First handler
app.get('something_unique', async (req, res, next) => {
// do whatever you want to do first
// save data into res.locals
res.locals.foo = {...}
next()
})
Second Handler
app.get('something_unique', (req, res) => {
const data = res.locals.foo;
// whatever you want
return res.json({ anything })
})
More:
Express next function, what is it really for?
Error: Can't set headers after they are sent to the client
Passing variables to the next middleware using next() in Express.js
I'm not sure what client code you're really running as it sounds like you've bee trying several things, but this should work to sequence the /data request and the /logo request so that the /logo request is not run until the response from the /data request has been received.:
async function run() {
const r1 = await fetch('http://localhost:2000/data');
const data = await r1.json();
const parsedData = data.data;
parsedData.forEach((element) => {
// this just has about 20 lines of code generating
// the the look of the page. It works as intended
});
const r2 = await fetch('http://localhost:2000/logo');
const logo = await r2.json();
return logo;
}
run().then(logo => {
console.log(logo);
}).catch(err => {
// handle errors here
console.log(err);
});
If there is any asynchronous code inside the .forEach(), then we will have to see that also to properly sequence that.
As I've said in my comments, stuffing the data from the first request into a server-side variable is probably the wrong design on the server because two separate clients both issuing /data requests will conflict with one another, creating race conditions. But, you haven't explained what this data is really for or why you're stuffing it into a variable on the server for us to suggest an alternate design.
My end goal is that I want to be able to create a test that satisfies the following statement:
verify that requests to valid URLs return a 200 HTTP status code
A valid URL for example would be /about-page or /jobs, basically any directory that I add in my content folder that contains a file with the extension /index.md.
This is my code so far:
app.js
const readFilePromise = util.promisify(fs.readFile)
app.get('/*', (req, res) => {
readFilePromise(path.join(__dirname, 'content', req.url) + '/index.md', 'utf8')
.then(data => {
convertData(data, res)
})
.catch(err => {
res.status(404).send('Page doesn\'t exist!')
})
})
const convertData = (data, res) => {
const convertedData = md.render(data)
readFilePromise(path.join(__dirname, '/template.html'), 'utf8')
.then(data => {
data = data.replace(/\{\{content\}\}/, convertedData)
res.send(data)
})
.catch(err => {
console.log(err)
})
}
app.listen(3000)
module.exports = app
After reading this article, it mentions that
Requests are asynchronous, which means you must be able to conduct asynchronous tests.
So I wrote the following test:
app.test.js
const app = require('./app.js')
const request = supertest(app)
const supertest = require('supertest')
it('Gets the test endpoint and returns a 200 status', async done => {
const res = await request.get('/*')
expect(res.status).toBe(200)
done()
})
When I run the test, it fails with a 404 status, rather than returning a 200 status. I thought this might be due to my app.js not being in the async/await style, so I changed app.js to:
const readFilePromise = util.promisify(fs.readFile)
app.get('/*', async (req, res) => {
try {
await readFilePromise(path.join(__dirname, 'content', req.url) + '/index.md', 'utf8')
} catch (err) {
res.status(404).send('Page doesn\'t exist!')
}
try {
const convertedData = md.render(data)
await readFilePromise(path.join(__dirname, '/template.html'), 'utf8')
data = data.replace(/\{\{content\}\}/, convertedData)
res.send(data)
} catch (err) {
console.log(err)
}
})
app.listen(3000)
module.exports = app
I tried running the test again, but it still fails with a 404. I think my set up within app.test.js is wrong, but I'm not sure exactly what, as I've tried using the various set ups as the article. How would I fix this?
Separately, when I try going to a URL using the async/await style in app.js, I get a ReferenceError: data is not defined error, but I'm not sure how to define data in the async/await format.
I explained here how to set up app for the test environment: supertest not found error testing express endpoint
You did not mention how you set the database environment, make sure your database is not empty. Then make your get request. but just checking status for get request is not enough because if your db is empty you will still get 200.
const response = await request(app).get("/route").send().expect(200);
expect(response.body.length).toBeGreaterThan(0)
Better approach would be connect to a different database, post your data first and then check the response
const response = await request(app).get("/api/tickets").send().expect(200);
expect(response.body.length).toEqual(2); // if you post two items
Also before you every test make sure you start with empty database inside beforeEach()
I have an internal HTTP Post API that generates files, size 5-10mb each. This service is not modifiable.
I want to "proxy" this file download through the public API, which is based on Node.js+Express. However, I can't figure out the best way of doing so.
I guess I can download this file with Axios into a temporary file in the Node.js API container, but that seems to be prone to issues with these temporary files potentially piling up and requiring later cleanup. Is there a way to achieve such file download -> send further to a client without creating a temporary file?
Or what would be the most efficient and "clean" way of doing so if temporary files are unavoidable?
router.post('/route/:someid',
[someRequestVerificationMiddleware],
(req, res, next) => {
const myFileId = req.params.someid;
const downloadRequestParams= {
"id": myFileId
};
let dlPromise = axios.post(`http://myinternalservice:80`,
downloadRequestParams, {responseType: "stream"});
dlPromise.then(response => {
try {
let filename = response.headers["x-result-filename"];
//
// What would be the most efficient way to return the received file
// from response data to the client calling this route without creating
// too much garbage?
//
} catch (e) {
console.error(e);
}
})
.catch(e=>{
console.error(e);
res.status(500);
})
.finally(() => {
next();
})
});
module.exports = router;
res is a stream. You can simply pipe your axios stream to the response.
res.setHeader("content-type", "...");
return dlPromise.then((response) => response.data.pipe(res));
From inside my expressJS application I have to verify that a cookie token is valid with a back-end server. So the relevant code involved in this is as follows:
app.get('*', (req, res, next) => {
console.log('GET: ' + req.path);
// ...
const payload = JSON.stringify({ authnToken: token });
const opts = { ... authServerOptions };
opts.headers['Content-Length'] = payload.length;
// build request
const restReq = https.request(authServerOptions, result => {
console.log('back-end response' + result.statusCode);
result.on('data', data => {
next(); // token is good now proceed.
});
result.on('error', error => {
res.redirect('somewhere'); // token is bad or timeout
});
});
restReq.write(token);
restReq.end();
}
So the main get function sets the REST request in motion and then just returns without calling next() or anything.
Questions:
Is this the right code for doing this? What happens if the callbacks are never called?
Is the application blocked from processing other requests until the back-end server returns or times out?
If so is there some way of freeing up the thread to process more requests?
Thanks in advance for any help. I haven't found many examples for this code pattern so if there is one a link would be appreciated.
Yes, I think the general idea of your implementation is correct.
I would also suggest, as done in the comments, to use a client such as axios to handle the request in a less verbose and more comprehensive manner, which would leave your code looking something like this:
const axios = require('axios');
app.get('*', (req, res, next) => {
const payload = JSON.stringify({ authnToken: token });
const opts = { ... authServerOptions };
opts.headers['Content-Length'] = payload.length;
axios.post(url, payload, opts)
.then(response => next())
.catch(error => {
console.error(error);
res.redirect('somewhere');
});
});
A bit more to the point, but functionally almost equivalent to your implementation. The one thing you are missing is the onerror callback for your request object, which currently may fail and never return a response as you correctly suspected. You should add:
restReq.on('error', error => {
console.error(error);
res.redirect('somewhere');
});
On the same vein, it would probably be more fitting to call next on result end, instead of doing so while reading response data:
result.on('end', () => {
next();
});
Then you'd be covered to guarantee that a callback would be invoked.
Neither implementation blocks the processing of future requests, as the call to the token validation service is done asynchronously in both cases.