I'm trying to implement an asynchronous method which makes a POST call to a certain API and retrieves its data, and then stores the result in the user's session. The task itself seems to be quite easy, but it gets problematic when I do it the 'fire-and-forget' style.
I do not wish to wait for the result to come back from the external API, but instead let the user load his requested page and continue browsing. Only when the response is ready, I want it saved in the session.
Everything seems to be working fine, but the only problem is the moment I set the session and proceed to the next route, everything gets lost as if I never set it in the first page. Debugging the code also shows that the process completes without any errors whatsoever, therefore I suspect an issue with setting the session after the response headers are already set. Can't seem to understand why that should be an issue, though.
I am using nodejs with express, and mongodb for session & database handling.
router.get('/myroute', function(req, res, next) {
request({
url: endpoint,
method: "POST",
json: true,
body: jsonData
}, function(error, response, body) {
req.session.mykey = 'some response';
});
res.redirect('/otherroute');
});
router.get('/checkresult', function(req, res, next) {
res.json(req.session.mykey);
});
I think you'll need to explicitly call save.
function(error, response, body) {
req.session.mykey = 'some response';
req.session.save();
}
https://expressjs.com/en/resources/middleware/session.html#sessionsavecallback
This method is automatically called at the end of the HTTP response if the session data has been altered
In your case this automatic save will be occurring too soon.
Related
Let's say that I'm using a GET request on https://example.com and the response is this:
This is a response message.
How would I modify it in a way so that in my code, so that it can change the response to say something like this:
This is a MODIFIED response message.
For example, if my Electron app were to navigate to https://example.com, the screen would show me the modified content instead of the original content.
Essentially, I am trying to literally modify the request.
I have based my code off of this question but it only shows a proof of concept with a pre-typed Buffer, as in my situation I'd like modify the response instead of outright replacing it. So, my code looks like this:
protocol.interceptBufferProtocol("http", (req, CALLBACK) => {
if(req.url.includes("special url here")) {
var request = net.request({
method: req.method,
headers: req.headers,
url: req.url
});
request.on("response", (rp) => {
var d = [];
rp.on("data", c => d.push(c));
rp.on("end", () => {
var e = Buffer.concat(d);
console.log(e.toString());
// do SOMETHING with 'e', the response, then callback it.
CALLBACK(e);
});
});
request.end();
} else {
// Is supposedly going to carry out the request without interception
protocol.uninterceptProtocol("http");
}
}
This is supposed to manually request the URL, grab the response and return it. Without the protocol event, it works and gives me a response, but after some debugging, this piece of code consistently calls the same URL over and over with no response.
There is also the WebRequest API, but there is no way of modifying the response body, you can only modify the request headers & related content.
I haven't looked fully into Chromium-based solutions, but after looking at this, I'm not sure if it is possible to modify the response so it appears on my app's end in the first place. Additionally, I'm not familiar with the Chromium/Puppeteer messages that get sent all over the place.
Is there an elegant way to have Electron to get a URL response/request, call the URL using the headers/body/etc., then save & modify the response to appear different in Electron?
Нello! I am looking to call a function which has been passed to an expressRouter.post(...) call.
This expressRouter.post(...) call is occurring in a file which I am unable to modify. The code has already been distributed to many clients and there is no procedure for me to modify their versions of the file. While I have no ability to update this file for remote clients, other developers are able to. I therefore face the issue of this POST endpoint's behaviour changing in the future.
I am also dealing with performance concerns. This POST endpoint expects req.body to be a parsed JSON object, and that JSON object can be excessively large.
My goal is to write a GET endpoint which internally activates this POST endpoint. The GET endpoint will need to call the POST endpoint with a very large JSON value, which has had URL query params inserted into it. The GET's functionality should always mirror the POST's functionality, including if the POST's functionality is updated in the future. For this reason I cannot copy/paste the POST's logic. Note also that the JSON format will never change.
I understand that the issue of calling an expressjs endpoint internally has conventionally been solved by either 1) extracting the router function into an accessible scope, or 2) generating an HTTP request to localhost.
Unfortunately in my case neither of these options are viable:
I can't move the function into an accessible scope as I can't modify the source, nor can I copy-paste the function as the original version may change
Avoiding the HTTP request is a high priority due to performance considerations. The HTTP request will require serializing+deserializing an excessively large JSON body, re-visiting a number of authentication middlewares (which require waiting for further HTTP requests + database queries to complete), etc
Here is my (contrived) POST endpoint:
expressRouter.post('/my/post/endpoint', (req, res) => {
if (!req.body.hasOwnProperty('val'))
return res.status(400).send('Missing "val"');
return res.status(200).send(`Your val: ${req.body.val}`);
});
If I make a POST request to localhost:<port>/my/post/endpoint I get the expected error or response based on whether I included "val" in the JSON body.
Now, I want to have exactly the same functionality available, but via GET, and with "val" supplied in the URL instead of in any JSON body. I have attempted the following:
expressRouter.get('/my/get/endpoint/:val', (req, res) => {
// Make it seem as if "val" occurred inside the JSON body
let fakeReq = {
body: {
val: req.params.val
}
};
// Now call the POST endpoint
// Pass the fake request, and the real response
// This should enable the POST endpoint to write data to the
// response, and it will seem like THIS endpoint wrote to the
// response.
manuallyCallExpressEndpoint(expressRouter, 'POST', '/my/post/endpoint', fakeReq, res);
});
Unfortunately I don't know how to implement manuallyCallExpressEndpoint.
Is there a solution to this problem which excludes both extracting the function into an accessible scope, and generating an HTTP request?
This seems possible, but it may make more sense to modify req and pass it, rather than create a whole new fakeReq object. The thing which enables this looks to be the router.handle(req, res, next) function. I'm not sure this is the smartest way to go about this, but it will certainly avoid the large overhead of a separate http request!
app.get('/my/get/endpoint/:val', (req, res) => {
// Modify `req`, don't create a whole new `fakeReq`
req.body = {
val: req.params.val
};
manuallyCallExpressEndpoint(app, 'POST', '/my/post/endpoint', req, res);
});
let manuallyCallExpressEndpoint = (router, method, url, req, res) => {
req.method = method;
req.url = url;
router.handle(req, res, () => {});
};
How about a simple middleware?
function checkVal(req, res, next) {
const val = req.params.val || req.body.val
if (!val) {
return res.status(400).send('Missing "val"');
}
return res.status(200).send(`Your val: ${val}`);
}
app.get('/my/get/endpoint/:val', checkVal)
app.post('/my/post/endpoint', checkVal)
This code isn't tested but gives you rough idea on how you can have the same code run in both places.
The checkVal function serves as a Express handler, with request, response and next. It checks for params first then the body.
I am developing an API which takes input in XML containing IDs for media and gives output in XMLform with details of given IDs. I am facing a problem while sending the response of second simultaneous request; here the second request goes into loop showing "loading" on postman.
What I am doing is calling a function in app.post which parses the media and gives output in the callback and send it using res.send, but it works only for single request.
While doing parallel request to same API either it goes in loop or it gives can't set the headers after they are sent as I am using res.send but res.send is the only way which I can use to send the response (even the next doesn't work).
var getCompositeData = function(req, res, next){
abc.getData(req.body, function(err, xmlOutput){
if(err){
console.log("error");
} else {
xmlData = xmlOutput
return next()
}
}
app.post(apiUrl, [
rawBodyParser({
type: 'application/xml'
}),
app.oauth.authorise()
], getCompositeData, function (req, res) {
res.setHeader('Content-Type', 'application/xml');
res.send(xmlData);
});
There are several issues with your code:
if (err) {
console.log("error");
}
If an error occurs, you still need to make sure a response will be sent back, otherwise the request will stall until a timeout happens. You can pass an error to next, and Express will handle it:
if (err) {
return next(err);
}
Next problem:
xmlData = xmlOutput
xmlData is an undeclared variable, which gets overwritten with each request. If two requests happens at (almost) the same time, it's likely that one client gets back an incorrect response (remember, Node.js runs JS code in a single thread; there is not thread-local storage so xmlData gets shared between all requests).
A good place to "store" this sort of data is in res.locals:
res.locals.xmlData = xmlOutput;
return next();
// and later:
res.send(res.locals.xmlData);
Trying to understand how to work properly with :
1. Express
2. request
3. middleware
It's a follow up question from here where the discussion wad fruitful and helpfull (thanks #BlazeSahlzen , you are great!) but I realize that I tried at one point to put too much issues (although they are all related) into the same question.
So, this one is a focused question... I hope :-)
Case: I want to build POST() that recives parameter via path (/:param1),
uses it to request() #1 an external API,
gets the result from the external API,
Uses the result to do somwething and send ANOTHER request() #2 to a 2nd external API,
get's the outcome of the 2nd APi request(),
decide if the POST is statusCode = 200 with message="ok" or statusCode = something_else and message = "problem"
and res.send() it properly.
for that, here is my pseudo code -
var middle_1 = function(req, res, next) {
param1 = req.params.param1; //trying to access the param1 from the path, not sure it will work in middleware
req.middle_1_output = {
statusCode: 404,
message: "param1"
}
var options = {
method: 'PUT',
url: `EXTERNAL_API_1`,
headers: {
'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded',
apikey: `KEY`
}
};
request(options, function(error, response, body) {
if (error) throw new Error(error);
// CODE THAT DO SOMETHING AND GET INFORMATION
req.request_1_output.statusCode = 200;
req.request_1_output.message = "hello world";
next(); // not sure what happens here - will it "jump" outside of the middle_1() or go to the next request() down the code??
});
var options = {
method: 'PUT',
url: `EXTERNAL_API_2`,
headers: {
'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded',
apikey: `KEY`
}
};
request(options, function(error, response, body) {
if (error) throw new Error(error);
//Can I use here the req.request_1_output.message ???
//How can I use here ALSO some of the EXTERNAL_API_1 outcome????
// Some more CODE THAT DO SOMETHING AND GET INFORMATION
req.request_2_output.statusCode = 201;
req.request_2_output.message = "hello world";
next(); // not sure what happens here
});
}
//This middleware is only used to send success response
var response_success = function(req, res) {
sum_statusCode = req.request_1_output.statusCode + req.request_2_output.statusCode;
if (req.request_2_output.message == req.request_1_output.message) {
meassge = "hello world";
} else {
message = "goodbye world!";
}
res.json({
"statusCode": sum_statusCode,
"message": message
});
}
app.post('/test', middle_1, response_success);
I am not sure how to connect the different requests (request #1 and request #2) in this case - should they all become middleware? how should I write it? (connect => make them run one only after the other is done.)
How can I get also infomation from the request #1 outcome and use it in the request #2 ?
look at my code at response_success() -> will this work? can I access like this data from req that originated within the request #1 and request #2?
How am I suppose to access inside the response_success() data which is the OUTCOME of the request #1 and request #2?
// EDITED - question #5 and #6 are a late edition of mine but should be a stand alone questions. I leave them here but I will be opening a new thread just for them.
Let's say my middle_1 needs to get information as an outcome from the request_1 , calculate something, and move it forward to a middle_2... how do I take the request_1 information into something that can be transffered into a middle_2? I think I am suppose to create a property inside "req" , something like req.middle_1_outcome = DATA , but I am not sure how to "get the DATA" from the request_1 outcome...
How do I "monitor and wait" for request_1 to be done before my middle_1 moves forward to calculate things? is there a requestSync() funciton for Synced requests?
Thanks in advance to all the helpers :-)
A given middleware function should call next() only once when it is done with all its processing.
A given request handler should only submit one response per request. Anything more will be considered an error by Express.
I am not sure how to connect the different requests (request #1 and
request #2) in this case - should they all become middleware? how
should I write it? (connect => make them run one only after the other
is done.)
If your two request() calls can run in parallel (the second one does not depend upon the first results), then you can run them both and then monitor when they are both done, collect the results, do what you need to do with the request and then once and only once call next().
If they must be run in sequence (use the results from the first in the second), then you can nest them.
How can I get also information from the request #1 outcome and use it
in the request #2 ?
There are a variety of ways to solve that issue. If the requests are being run in sequence, then the usual way is to just put request #2 inside the completion callback for request #1 where the results from #1 are known.
Look at my code at response_success() -> will this work? can I access like this data from req that originated within the request #1 and request #2?
You can't quite do it like that because you can't call next() multiple times from the same middleware.
How am I suppose to access inside the response_success() data which is the OUTCOME of the request #1 and request #2?
If you nest the two operations and run request #2 from inside the completion of
request #1, then inside the completion for request #2, you can access both results. There is no need to a completely separate request handler to process the results. That just makes more complication that is necessary.
If you need to serialize your two requests because you want to use the result from the first request in the second request, then you can use this structure where you nest the second request inside the completion of the first one:
function middle_1(req, res, next) {
var param1 = req.params.param1; //trying to access the param1 from the path, not sure it will work in middleware
var options = {
method: 'PUT',
url: `EXTERNAL_API_1`,
headers: {
'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded',
apikey: `KEY`
}
};
request(options, function (error, response, body) {
if (error) return next(error);
// CODE THAT DO SOMETHING AND GET INFORMATION
var options = {
method: 'PUT',
url: `EXTERNAL_API_2`,
headers: {
'cache-control': 'no-cache',
'content-type': 'application/x-www-form-urlencoded',
apikey: `KEY`
}
};
// this second request is nested inside the completion callback
// of the first request. This allows it to use the results from
// from the first request when sending the second request.
request(options, function (error2, response2, body2) {
if (error2) return next(error2);
// right here, you can access all the results from both requests as
// they are all in scope
// response, body from the first request and
// response2, body2 from the second request
// When you are done with all your processing, then you
// can send the response here
res.json(....);
});
});
}
app.post('/test', middle_1);
Note several things about the structure of this code:
To use the results of the first request in the second one, just nest the two.
When nesting like this, the results from both requests will be available in the completion callback for request #2 as long as you give the arguments unique names so they don't accidentally hide parent scoped variables of the same name.
It does you no good to throw from an async callback, since there's no way for your code to ever catch an exception throw from a plain async callback. The request will likely just sit there forever until it eventually times out if you throw. You need to actually handle the error. Since I didn't know what error handling you wanted, I called next(err) to at least give you default error handling.
I would not suggest using multiple middleware functions that are really just one operation. You may as well just put the one operation in one request handler function as I've shown.
I want to get some data that are available after authentication. I pass through a post login and password on the page http://site.domain.com/auth.html and I want to get html from another page http://site.domain.com/anotherpage.html
request.post({followAllRedirects: true, url:'http://site.domain.com/auth.html', form:{user:'login#domain.com', pass:'password'}},
function (error, response, body) {
if (!error) {
request('http://site.domain.com/anotherpage.html', function(error, response, html){
fs.appendFileSync('log.txt', html, encoding='utf8');
});
}
});
Authentication takes place normally (there is a message in the html with greeting), after request I get the data as if the authentication is not passed.
fixed result:
var j = request.jar(); var request = request.defaults({jar:j});
and then my code
Most often than not, in the web, authentication information is store as cookies in the user's browser. Because this is a server request, I don't think two "unrelated" requests is going to cut it, as no header information pertaining to the first request is being sent along with the second request. Perhaps you could try this strategy or some other similar procedure to mimic that header interaction.
I found a solution, I put in top of my code, these lines
var j = request.jar(); var request = request.defaults({jar:j});
jar - If true, remember cookies for future use (or define your custom cookie jar;