I'm using node-rest-client to make a GET call to an endpoint, and would like to make this call multiple times (in a loop), then expose a parameter of the response.
The code looks like:
// registering remote methods
client.registerMethod("reflect", "<the URL>", "GET");
// call the method
var i = 10;
while (i>0) {
client.methods.reflect(function (data, response) {
console.log("x-forwarded-for: " + data.headers["x-forwarded-for"]);
// raw response
//console.log(response);
});
i--;
}
The error I get is:
TypeError: Cannot read property 'x-forwarded-for' of undefined
If i is equal to 2, then this is OK.
I suppose the issue comes from the fact that this is asynchronous execution and all the calls in the while are fired at once, resulting into some clogging somewhere along the lines.
What is the best way of having a synchronous execution (assuming this is where the problem lies)?
First of all please check x-forwarded-for lies in response and if the same problem persists as you are expecting(asynchronous call) then
just wrap this call inside a anonymous function like this
while(i > 0) {
(function abc(){
client.methods.reflect(function (data, response) {
console.log("x-forwarded-for: " + data.headers["x-forwarded-for"]);
// raw response
//console.log(response);
});
}())
i--;
}
Related
I'm new to Node and the async programming model. I'm having problems dealing with a simple requirement that seems pretty basic in synchronous environments: paging through an API response until the response is empty.
More specifically, the API, on a successful call, will return data and a status of 200 or 206 (partial content). If I see the 206 response, I need to keep making calls to the API (also sending a page query param that I increment each time) until I see the 200 response.
In a synchronous language, the task will be a piece of cake:
// pseudocode
data = []
page = 1
do {
response = api.call(page)
data.append(response.data)
page++
} while (response != 200)
return data
Now, in Node, for a single api call, code like this will work:
// fire when '/' has a GET request
app.get('/', (req, res) => {
axios.get('https://api.com/v1/cats')
.then(response => {
// now what??
});
});
});
See the //now what?? comment? That's the point where I'm wondering how to proceed. I came across this somewhat-relevant post but am not able to convert this to a format that will work for me in Node and Axios.
Is it enough to just wrap the axios code in a separate function? I don't think so, because if I do this:
function getData(pageNum) {
axios.get('https://api.com/v1/cats')
.then(response => {
// now what??
});
});
}
I can't rely on a return value because as soon axios.get() gets executed, the function will be over. I can call getData() again after I get the first response, but then, suppose I want to return all the data from these multiple calls as the HTTP response from my Express server . . . how do I do that?
I hope I will not get downvoted for laziness or something. I've really looked around but not found anything relevant.
First, a counter-question: Is the data set so big that you need to worry about using up all the memory? Because if so then it will take more work to structure your code in a way that streams the data all the way through. (In fact I'm not even sure whether express allows streaming... you are using express aren't you?)
From the axios documentation, it looks like response is a readable stream which provides the response body. So reading it is also an asynchronous task. So you should write a function that does that. See the "Stream" page of the nodejs docs for more details. Or I could be persuaded to help with that too, time permitting. But for now, I'll assume you have a function readResponse, which takes an axios response object as an argument and returns a promise, and the promise resolves to an object such as { statusCode: 206, result: ['thing1', 'thing2'] }. I'll also assume that your goal is to get all the result arrays and concatenate them together to get e.g. ['thing1', 'thing2', 'thing3', 'thing4', 'thing5', 'thing6'].
You could write a self-calling version of your getData function. This will retrieve all data from a given page onwards (not just the page itself):
function getData(pageNum) {
axios.get('https://api.com/v1/cats' + (pageNum ? '?page=' + pageNum) : '')
.then(readResponse)
.then(function(parsedResponse) {
if(parsedResponse.statusCode == 200) {
return parsedResponse.result;
} else if(parsedResponse.statusCode == 206) {
return getData(pageNum + 1).then(function(laterData) {
return parsedResponse.result.concat(laterData);
});
} else {
// error handling here, throw an exception or return a failing promise.
}
});
});
}
Then, to get all data, just call this function with pageNum = 0:
// fire when '/' has a GET request
app.get('/', (req, res) => {
getData(0)
.then(function(results) {
// results is now the array you want.
var response = JSON.stringify(results); // or whatever you're doing to serialise your data
res.send(response);
});
});
I want to be access array outside the function or outside the loop in nodejs. I written following code.
var result = [];
function setid (swfid){
crud.getswift(swfid).then(function (response) {
console.log("response",response);
result = response;
// res.send(response);
}).catch(function (err) {
return ("error:" + err);
});
console.log("result",result);
}
console.log("result",result);
But its returning null. your suggestions please
You wrote a new statement in the function call and therefore you scoped it. This is one of the things wrong there. Apart from that, as the first person commenting to this answer mentioned, you have an async call here. Therefore, you need to return a promise from setid and wait for the response to get the result.
You're mixing your Aysnc logic with Sync. You won't get the response outside the .then function scope because there's no response available at the time you're trying to get the results.
Try using a callback in the promise - You'd need to invoke the function in the promise callback and send the response as function param, then play with the data.
> Promise / API call etc
.then(() => gotDataCallBack(data));
gotDataCallBack(data){
// handle your data and logic here.
// this will make sure you have the data available before you move ahead with
your application/manipulation logic.
}
I work with node-red and develop a custom node at the moment that uses websockets to connect to a device and request data from it.
function query(node, msg, callback) {
var uri = 'ws://' + node.config.host + ':' + node.config.port;
var protocol = 'Lux_WS';
node.ws = new WebSocket(uri, protocol);
var login = "LOGIN;" + node.config.password;
node.ws.on('open', function open() {
node.status({fill:"green",shape:"dot",text:"connected"});
node.ws.send(login);
node.ws.send("REFRESH");
});
node.ws.on('message', function (data, flags) {
processResponse(data, node);
});
node.ws.on('close', function(code, reason) {
node.status({fill:"grey",shape:"dot",text:"disconnected"});
});
node.ws.on('error', function(error) {
node.status({fill:"red",shape:"dot",text:"Error " + error});
});
}
In the processResponse function I need process the first response. It gives me an XML with several ids that I need to request further data.
I plan to set up a structure that holds all the data from the first request, and populate it further with the data that results from the id requests.
And that's where my problem starts, whenever I send a query from within the processResponse function, I trigger an event that results in the same function getting called again, but then my structure is empty.
I know that this is due to the async nature of nodejs and the event system, but I simply don't see how to circumvent this behavior or do my code in the right way.
If anybody can recommend examples on how to deal with situations like this or even better could give an example, that would be great!
So I've been trying to set a global variable from inside a request, but seem to be getting nothing. The code I'm using
A username for testing is test2 after username=
var forSearching = "test2";
var name = "";
console.log(forSearching);
request("http://mercsystem.esy.es/get.php?username=" + forSearching, function(err, res, body)
{
if (err) return console.error(err);
var main = JSON.parse(body);
if (main.success == "false")
{
message.reply("Sorry, invalid user!")
}
else
{
name = main.Username
}
});
If you insert a console.log(name) right after you set the value, you will see that the value is set just fine.
The issue is likely one of timing. request() is an asynchronous operation. That means calling request() starts the asynchronous operation, then the rest of your code continues to run to completion and then, some time LATER, the callback gets called with the final asynchronous results.
Though you don't show where you are trying to use the name variable, you are probably checking the value of this global variable before the callback has been called and thus before the value has been set.
In node.js, what you are doing is not how you use asynchronous results. It will never work reliably (or at all). Instead, the only place to use your asynchronous result is in the callback itself or in some function you call from the callback and pass the result to.
var forSearching = "test2";
console.log("begin");
request("http://mercsystem.esy.es/get.php?username=" + forSearching, function (err, res, body) {
console.log("in request() callback");
if (err) return console.error(err);
var main = JSON.parse(body);
if (main.success == "false") {
message.reply("Sorry, invalid user!")
} else {
var name = main.Username
console.log(name); // value shows fine here
// use the name variable here or call some function and pass
// the name variable to it
}
});
console.log("after request() call");
// You cannot use the name value here (even if it was in a global) because
// the async callback has not yet been called
If you ran this code with the console.log() statement I've added, you would see this sequence of events:
begin
after request() call
in request() callback
From this sequence, you can see that code after your request() call runs BEFORE the async callback runs. Thus, you cannot use your name variable there, even if it is in a global.
I am writing a node js application. I am using request and cheerio to load a set of URLs and get a bunch of information for the site, now let's assume all I am trying to get is the title:
var urls = {"url_1", "url_2", "url_3",...,"url_n"};
for(var i=0; i<urls.length; i++)
{
getDOMTitle(urls[i],function(error,title){
if(error)
console.log("Error while getting title for " + urls[i]);
else
console.log("The title for " + urls[i] + " is " + title);
});
}
This is how my getDOMTitle method looks:
function getDOMTitle(urlReq,callback)
{
var request = require('request');
var cheerio = require('cheerio');
request({url:urlReq},function(error, response, doc){
var $ = cheerio.load(doc);
if(error)
{
callback(true,null);
}
else
{
$('title', 'head').each(function (i, elem) {
var title = $(this).text();
callback(false,title);
});
}
}
}
In the case where the module throws an uncaught exception, how do I handle that situation?
I have tried adding the following:
process.on('uncaughtException', function (err) {
console.error(err);
console.log("Node NOT Exiting...");
callback(true,null);
});
When I do that, I get an error saying I cannot set the headers once they have been sent. If I remove the callback from the process error handling, I do not see that error but the client spins for a long time because I assume we are never calling the callback.
How can I solve this?
Also, I have read somewhere that you can catch uncaught exceptions at the application level so you don't have to replicate the code to catch it in every method, is that possible? and if it is and the method that threw the exception is expected to callback with some information, how can that be achieved?
Thank you,
To answer your stated question, using an uncaught exception handler as a general error-trapping mechanism is commonly regarded as poor design. It's a false economy to use it to handle anything other than non-recoverable situations where you just need to do some cleanup before exiting.
You've got some problems in your example code. In your for loop, all the callbacks are going to report that they were working with the very last URL in your array because they're all referring to the same copy of i, which will be at its highest value by the time any of them execute. You need to use a helper function or an immediate function invocation to give each callback a private copy of i.
In getDOMTitle the error callback should be callback(error) and the code in your loop should include the returned value in the error message. The success callback should use null as its first parameter, though this is just a matter of convention.