Sorry for the inconvenience, I am a newbie in Node. I am trying to store a json in "usersData" variable so I want to use it later in another functions. The problem is that if I test the variable with console.log inside the "if" it returns to me results, but when trying to show the variable outside the request subfunction, it comes out 'undefined'. I have declared the global usersData variable as shown below. Thank you.
var usersData;
function getAllUsers(){
request({url, json: true}, function (error, response, body) {
if (!error && response.statusCode == 200) {
usersData = body
//console.log(usersData) //Here returns a value
}
});
console.log(usersData) //here returns undefined
}
request is asynchronous method, so if you want to use its result later in another functions, should handle that in second parameter callback. i.e
var usersData;
var handleUserData = function() {};
function getAllUsers(){
request({url, json: true}, function (error, response, body) {
if (!error && response.statusCode == 200) {
usersData = body
//console.log(usersData) //Here returns a value
// use results in another function
handleUserData(body);
}
});
}
or use Promise
function getAllUsers() {
return new Promise(function(resolve, reject) {
request({url, json: true}, function (error, response, body) {
if (!error && response.statusCode == 200) {
usersData = body
//console.log(usersData) //Here returns a value
resolve(body);
} else {
reject(error);
}
});
});
}
// handle `usersData`
getAllUsers().then(body => {
handleUserData(body);
});
Here are some stuffs you need to know,
request is a asynchronous function which means it runs on background without blocking main thread, so the callback function is executed after the request completes, that is why value of body is assigned to userData and when printed gives some value.
But when printing outside of the callback function, it gives undefined because of request being async takes time to complete which run on background. So, consoling statement runs before the body is assigned to userData and so when printed gives undefined.
Related
I have an endpoint in my node backend in which will need to retrieve for each item in my Adhoc collection from my local database the _id along with a number value which I need to calculate from the body of a request() function in an array of objects. The objects will be like this
{id: "id", sum: 3}
To do this I need to iterate through the Adhocs with a for loop and make a request for each to get the sum value and I need to be able to store these values before I have all of them and res.send() the array to the front end. I am having trouble storing the sum value in a variable. I have provided below the code of the request.
let theSum = request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(
'Response: ' + response.statusCode + ' ' + response.statusMessage
);
let bodyy = JSON.parse(body);
let sum = bodyy.fields.timetracking.originalEstimateSeconds / 3600 * theRate;
return sum;
});
I know this is wrong as the return statement is for the function inside the request function so it won't return the sum to my variable. And adding another callback function will basically be the same scenario. Anyone has any suggestions of how I can store the value from the request function so I can make further calls?
I found an answer that works for me pretty well. I didn't try Terry's answer above but I suppose that works as well since it's using Promise as well as me. What I've done is in a function I wrapped the request call in a Promise with a callback which is returned. Code below:
function asyncRequest (url) {
return new Promise (function (resolve, reject) {
var options = {
url: 'http://localhost:8080/rest/' + url,
auth: { username: 'username', password: 'password' },
headers: {
'Accept': 'application/json'
}
}
request(options, function (err, response, body) {
if (err) reject(err);
resolve(JSON.parse(body))
});
})
}
When I want to retrieve something I just have something like this:
let json = await asyncRequest('agile/1.0/issue/'+ adhoc[u].jIssue);
And that variable has the body of the request function inside and I can use it.
You can use async and await along with request-promise-native to loop over your objects and get the list of results you wish to have.
You can call the readEstimates function in a express.get( ).. handler as long as the handler is asynchronous (or you can use readEstimates().then(..)).
Now, we will wrap an error handler around the readEstimates call since, this could potentially throw an error.
For example:
const rp = require('request-promise-native');
async function readEstimates() {
const sumList = [];
for(const adhoc of adhocList) {
// Set your options here, e.g. url for each request.. by setting json to true we don't need to JSON.parse the body.
let options = { url: SOME_URL, json: true, resolveWithFullResponse: true };
let response = await rp(options);
console.log('Response: ' + response.statusCode + ' ' + response.statusMessage);
const sum = response.body.fields.timetracking.originalEstimateSeconds / 3600 * theRate;
sumList.push(sum);
}
return sumList;
}
async function testReadEstimates() {
try {
const sumList = await readEstimates();
console.log("Sumlist:", sumList);
} catch (error) {
console.error("testReadEstimates: An error has occurred:", error);
}
}
testReadEstimates();
You can also use readEstimates in an Express route:
app.get('/', async (req, res) => {
try {
const sumList = await readEstimates();
res.json({sumList}); // Send the list to the client.
} catch (error) {
console.error("/: An error has occurred:", error);
res.status(500).send("an error has occurred");
}
})
I'm working with callbacks to improve my program's efficiency. I would like to wait for my 'a' variable to get the value from callback if it hasn't already at some point. My code looks like this:
function myFunction(callback){
request("url", function(error, response, body) {
if (!error && response.statusCode == 200)
{
result = JSON.stringify(JSON.parse(body));
return callback(null, result);
}
else
{
return callback(error, null);
}
});
}
var a = myFunction(function(err, data){
if(!err)
{
return(data);
}
else
{
return(err);
}
});
//A CHUNK OF CODE EXECUTES HERE
//I'D LIKE TO CHECK IF THE a VARIABLE GOT THE VALUE (if the callback was executed), note that I don't want to nest the chunk inside the callback, but rather I'd like to check that outside function callback via IF statement/loop or some similar alternative
//MORE CODE CONNECTED TO THE CALLBACK
Any ideas on how I can wait for the variable to get the value if it didn't get it already by the time that the chunk of code is executed? I know that the sole intention of using callbacks is to not wait but in this instance it is necessary for me so just don't bother being annoying please :)
I'd suggest trying async/await and the request-promise-native module, this keeps the syntax very simple. If the call fails for some reason an error will be thrown, this is easy to catch:
For example:
const rp = require('request-promise-native');
async function testAsyncRequest() {
try {
let promise = rp('https://my-json-server.typicode.com/typicode/demo/posts');
// Do some stuff here...
let result = await promise;
console.log("Result: ", result);
} catch (err) {
console.error(`Something went wrong: ${err.message}`);
}
}
testAsyncRequest();
So I've ended up making a little compromise (had to nest), but the code works as intended:
var something;
function myFunction(callback){
request("url", function(error, response, body) {
if (!error && response.statusCode == 200)
{
result = JSON.stringify(JSON.parse(body));
return callback(null, result);
}
else
{
return callback(error, null);
}
});
}
var a = myFunction(function(err, data){
if(!err)
{
something = data;
return(data);
}
else
{
return(err);
}
});
//A CHUNK OF CODE EXECUTES HERE
var something_cachedValue=something;
doStuff();
function doStuff() {
if(something===something_cachedValue) {
setTimeout(doStuff, 10);
return;
}
something_cachedValue = something;
//MORE CODE CONNECTED TO THE CALLBACK
}
So basically I'm checking if the callback has completed every 10ms by re-calling the function. When it's completed, the "//MORE CODE CONNECTED TO THE CALLBACK" executes. Not the cleanest and most fancy way to do it but it does the job.
I have a loop, which iterates over array and in every iteration I have to do a http request, like this:
var httpsRequest = require('request')
var getData = function(id) {
var result;
httpsRequest({
url: 'https://link/'+id,
}, (error, resp, body) => {
if(resp.statusCode == 200) {
result = JSON.parse(body);
}
});
//here I would like to wait for a result
}
var data = [];
for(row in rows) {
data.push(getData(row.ID))
}
resp.send(JSON.stringify(data)) //I send data back to the client
I cannot do the rest of the for loop in callback, I have to wait for a result which will be returned from a function getData and move to the next iteration.
How to handle this?
PS I know I could use callback function but what if after the last iteration program will send the response (last line above) before the last getData execution finish?
Regards
As stated in the answer by Johannes, the use of promises is a good idea. Since you're using request I'd like to propose an alternative method by using request-promise which is a promisified version of 'request' using bluebird.
The requests will in this case return a promise, and by using .map() you can create an array of promises that you can await using Promise.all(). When all promises are resolved, the response can be sent! This also differs from the use of .reduce(), which only will start to execute the next request as soon as the previous one is done. By using an array of promises, you can start all the requests at the same time.
var httpsRequest = require('request-promise')
var getData = function(id) {
return httpsRequest({
url: 'https://link/' + id,
}, (error, resp, body) => {
if(resp.statusCode == 200) {
return JSON.parse(body);
} else {
//Throw error, this will be caught in the .catch()
throw error;
}
});
}
var promises = rows.map(function(row){
return getData(row.ID)
});
Promise.all(promises)
.then(function(results){
//All requests are done!
//The variable results will be an array of all the results in the same order as they were requested
resp.send(JSON.stringify(results));
})
.catch(function(error){
//Handle the error thrown in the 'getData' function
});
If you need to wait for each iteration to be done before starting another one, you can use Promises and reduce. If you only want to wait for all requests to be finished it's better to use map + Promise.all as explained in Daniel Bs answer.
// i asume rows is an array as you wrote you iterate over one.
const results = [];
rows.reduce((previous, row) => {
return previous.then(() => getData(row.ID).then(result => results.push(result)) // do whatever you want with the result
);
}, Promise.resolve())
.then(() => resp.send(JSON.stringify(results)));
const getData = (id) => {
return new Promise((resolve, reject)=> {
httpsRequest({
url: 'https://link/'+id,
}, (error, resp, body) => {
if(error) return reject(error);
if(resp.statusCode == 200) {
return resolve(JSON.parse(body));
}
return resolve(); // if you want to pass non 200 through. You may want to do sth different here
});
});
};
In second function in the async waterfall, the eachSeries callback (urlCallback) in my code executes after the waterfall callback (waterfallCallback), for reasons I cannot suss out.
async.waterfall([
function(callback) {
request(website, function (error, response, html) {
if (!error && response.statusCode == 200) {
pageUrls = getPageUrls(html)
callback(null, pageUrls)
}
})
},
function (pageUrls, waterfallCallback) {
async.eachSeries(pageUrls, function (url, urlCallback) {
console.log('SET ' + url)
request(url, function (err, response, body) {
var $ = cheerio.load(body)
$('#div').children().each(function(){
console.log($(this).children("a").attr("href"));
itemUrl = $(this).children("a").attr("href")
itemUrls.push(itemUrl)
})
urlCallback(null,itemUrls)
})
},
waterfallCallback(null, itemUrls))
}
],
function(err, results) {
console.log("results: " + results)
})
AFAIK, the async.eachSeries takes three arguments (array,functionToBeExecuteOnEachItem,callback) and execute them in that order. Somehow not here.
The parameters for async.eachSeries are function definitions. Something like waterfallCallback or function(err,result){}.
When you call waterfallCallback(null, itemUrls), that is not a function definition, that is running the function itself!
Changing to simply waterfallCallback should do the trick.
Update: Also, .eachSeries does not return the values as an array, its final callback is just function(err). Checkout .mapSeries link instead, which will return the resulting array in the final callback function(err,finalArray). (Be aware that each return of a .map will be an element in the array, so if you return an array, you'll get data structures like [ [], [], [] ])
My goal is to make a simple JSON parser which runs continuously.
As I imagined it - an infinity loop makes a request and stores data in my DB. Simple.
But there always were memory leaks.
What is the proper way to make things run repeatedly over and over again in Node.js? What is actually memory leak? Why and how is it happening? I have tried it to do with setInterval, setTimeOut, processNextTick, setImmediate, promises but there is always the same result! I am obviously something missing.
What I get now:
function getItems(callback) {
request({
url: 'http://foo.com',
json: true
}, function (error, response, body) {
if (!error && response.statusCode == 200 && body) {
var total_count = body.total_count;
var body = body.results_html;
...
setTimeout(function() {callback(body}, 1000);
}
});
}
function series(item) {
if (item) {
getItems(function(result) {
console.log(result);
return series(true);
});
} else {
return;
}
}
series(true);