Ok so i am using a method to make a request and pull some tables from another URL
Meteor.methods({
gimmetitle: function () {
var url = 'http://wiki.warthunder.com/index.php?title=B-17G_Flying_Fortress';
request(url, function(err, response, body) {
$ = cheerio.load(body);
var text = $('.flight-parameters td').text();
console.log(text);
return text;
});
}
});
When called the td's in the table succesfully print to the server console: http://prntscr.com/721pjh
Buuut, when that text is returned from that method to this client code, undefined is printed to the console:
Template.title.events({
'click #thebutton': function () {
Meteor.call('gimmetitle', function(error, result){
Session.set('gogle', result);
});
var avar = Session.get('gogle');
console.log(avar);
}
});
Ideas?
You need to understand two different things here :
On the client side, making some calls to the server is always asynchronous, because we have to deal with network latency. That's why we use callbacks to fetch the result of Meteor methods : this code is executed some time in the future, not right away.
This is why Session.set('gogle', result); is actually executed AFTER var avar = Session.get('gogle'); even though it appears before in your event handler code flow.
Contrary to template helpers, event handlers are NOT reactive, so it means that when you set the Session variable to the result of the method, the event handler code is not automatically reexecuted with the new value of Session.get('gogle').
You'll need to either do something with the result right in the Meteor method callback, or use a reactive computation (template helpers or Tracker.autorun) depending on Session.get('gogle') to rerun whenever the reactive data source is modified, and use the new value fetched from the server and assigned to the Session variable.
Quick update..Was able to fix this with just 1 line of code lol.
instead of request(url, function(err, response, body) i used the froatsnook:request package and used var result = request.getSync(url, {encoding: null}); and then just replaced $ = cheerio.load(body); with $ = cheerio.load(result.body);.
Related
So I'm fairly new to node js, and am having trouble wrapping my head around asynchronous programming. I'm trying to get a JSON from a website and pass it to a variable for use later, to test I have been using this code:
var https = require("https");
var a;
function getter(url){
var request = https.get(url, function(response){
var body = "";
response.on("data", function(chunk){
body += chunk;
});
response.on("end", function(){
if(response.statusCode === 200){
try{
a = JSON.parse(body);
}catch(err){
console.log(err);
}
}
})
})
};
getter('https://api.nasa.gov/planetary/apod?api_key=DEMO_KEY');
console.log(a);
When I run this I get a as undefined, which seems to make sense from what I've read. But I'm unclear as to what to do from here. How would I go about passing this JSON into a variable?
http.get is asynchronous and executes the event handlers when the events occur. When you call getter() this function immediately returns, ie it does not wait for the events and the next statement console.log(a) is executed.
Furthermore, js is single threaded, and the current execution stack is never interrupted for any other event/callback or whatsoever. So the event handlers can only run if the current execution has come to an end, ie contains noch more statements. Thus, your console.log() will always be executed before any eventhandler of the request, thus a is still undefined.
If you want to continue after the request finished, you have to do it from the eventhandler.
See this excellent presentation for some more details https://youtu.be/8aGhZQkoFbQ
I'm just starting off with Node.js and struggling with some of the finer points of non-blocking (asynchronous?) code. I know there are lots of questions about blocking vs non-blocking code already, but after reading through some of them, I still couldn't sort out this issue.
As a learning exercise, I made a simple script that loads URLs from a file, queries them using the request module, and notifies me if a URL is the New York Times homepage.
Here is a MWE:
// CSV Parse test
'use strict';
var request = require('request');
var fs = require('fs');
var parse = require('csv-parse');
var text = fs.readFileSync('input.txt','utf8');
var r_isnyt = /New York Times/;
var data = [];
parse(text, {skip_empty_lines: true}, function(err, data){
for (var r = 0; r < data.length; r++) {
console.log ('Logging from within parse function:');
console.log ('URL: '+data[r][0]+'\n');
var url = data[r][0];
request(url, function(error, response, body) {
console.log ('Logging from within request function:');
console.log('Loading URL: '+url+'\n');
if (!error && response.statusCode == 200) {
if (r_isnyt.exec(body)){
console.log('This is the NYT site! ');
}
console.log ('');
}
});
}
});
And here is my input.txt:
http://www.nytimes.com/
www.google.com
From what I understood of non-blocking code, this program's flow would be:
parse(text, {skip_empty_lines: true}, function(err, data){ loads the data and returns the lines of the input file in a 2D array, which is complete and available right after that line.
For Loop iterates through it, loading URLs with the line request(url, function(error, response, body) {, which is non-blocking (right?), so the For loop continues without waiting for the previous URL to finish loading.
As a result, you could have multiple URLs being loaded at once, and the console.log calls within request will print in the order the responses are received, not the order of the input file.
Within request, which has access to the results of the request to url, we print the URL, check whether it's the New York Times, and print the result of that check (all blocking steps I thought).
That's a long-winded way of getting around to my question. I just wanted to clarify that I thought I understood the basic concepts of non-blocking code. So what's baffling me is that my output is as follows:
>node parsecsv.js
Logging from within parse function:
URL: http://www.nytimes.com/
Logging from within parse function:
URL: www.google.com
Logging from within request function:
Loading URL: www.google.com
Logging from within request function:
Loading URL: www.google.com
This is the NYT site!
>
I understand why the request printouts all happen together at the end, but why do they both print Google, and much more baffling, why does the last one say it's the NYT site, when the log line right before it (from within the same request call) has just printed Google? It's like the request calls are getting the correct URLs, but the console.log calls are lagging, and just print everything at the end with the ending values.
Interestingly, if I reverse the order of the URLs, everything looks correct in the output, I guess because of differences in response times from the sites:
node parsecsv.js
Logging from within parse function:
URL: www.google.com
Logging from within request function:
Loading URL: www.google.com
Logging from within parse function:
URL: http://www.nytimes.com/
Logging from within request function:
Loading URL: http://www.nytimes.com/
This is the NYT site!
>
Thanks in advance.
Update
Based on the answer from jfriend00 below, I've changed my code to use a .forEach loop instead as follows. This appears to fix the issue.
// CSV Parse test
'use strict';
var request = require('request');
var fs = require('fs');
var parse = require('csv-parse');
var text = fs.readFileSync('input.txt','utf8');
var r_isnyt = /New York Times/;
var data = [];
parse(text, {skip_empty_lines: true}, function(err, data){
data.forEach( function(row) {
console.log ('Logging from within parse function:');
console.log ('URL: '+row[0]+'\n');
let url = row[0];
request(url, function(error, response, body) {
console.log ('Logging from within request function:');
console.log('Loading URL: '+url+'\n');
if (!error && response.statusCode == 200) {
if (r_isnyt.exec(body)){
console.log('This is the NYT site! ');
}
console.log ('');
}
});
});
});
I understand why the request printouts all happen together at the end,
but why do they both print Google, and much more baffling, why does
the last one say it's the NYT site, when the log line right before it
(from within the same request call) has just printed Google? It's like
the request calls are getting the correct URLs, but the console.log
calls are lagging, and just print everything at the end with the
ending values.
You correctly understand that the for loop initiates all the request() calls and then they finish sometime later in whatever order the responses come back in.
But, your logging statement:
console.log('Loading URL: '+url+'\n');
refers to a variable in your for loop which is shared by all the iterations of your for loop. So, since the for loop runs to completion and THEN sometime later all the responses arrive and get processed, your for loop will have finished by the time any of the responses get processed and thus the variable url will have whatever value it has in it when the for loop finishes which will be the value from the last iteration of the for loop.
In ES6, you can define the variable with let instead of var and it will be block scopes so there will be a unique variable url for each iteration of the loop.
So, change:
var url = data[r][0];
to
let url = data[r][0];
Prior to ES6, a common way to avoid this issue is to use .forEach() to iterate since it takes a callback function so all your loop code is in its own scope by nature of how .forEach() works and thus each iteration has its own local variables rather than shared local variables.
FYI, though let solves this issue and is one of the things it was designed for, I think your code would probably be a bit cleaner if you just used .forEach() for your iteration since it would replace multiple references to data[r] with a single reference to the current array iteration value.
parse(text, {skip_empty_lines: true}, function(err, data){
data.forEach( function(row) {
console.log ('Logging from within parse function:');
console.log ('URL: '+row[0]+'\n');
let url = row[0];
request(url, function(error, response, body) {
console.log ('Logging from within request function:');
console.log('Loading URL: '+url+'\n');
if (!error && response.statusCode == 200) {
if (r_isnyt.exec(body)){
console.log('This is the NYT site! ');
}
console.log ('');
}
});
});
});
Your code is fine and you're correct about how it works (including that differences in response times are what's making everything seem good when you switch the order around), but your logging has fallen victim to an unexpected closure: url is declared and updated in the scope of the parse() callback, and in the case where www.google.com is logged both times, it is being updated to its final value by the loop before your request() callbacks start executing.
So I've been trying to set a global variable from inside a request, but seem to be getting nothing. The code I'm using
A username for testing is test2 after username=
var forSearching = "test2";
var name = "";
console.log(forSearching);
request("http://mercsystem.esy.es/get.php?username=" + forSearching, function(err, res, body)
{
if (err) return console.error(err);
var main = JSON.parse(body);
if (main.success == "false")
{
message.reply("Sorry, invalid user!")
}
else
{
name = main.Username
}
});
If you insert a console.log(name) right after you set the value, you will see that the value is set just fine.
The issue is likely one of timing. request() is an asynchronous operation. That means calling request() starts the asynchronous operation, then the rest of your code continues to run to completion and then, some time LATER, the callback gets called with the final asynchronous results.
Though you don't show where you are trying to use the name variable, you are probably checking the value of this global variable before the callback has been called and thus before the value has been set.
In node.js, what you are doing is not how you use asynchronous results. It will never work reliably (or at all). Instead, the only place to use your asynchronous result is in the callback itself or in some function you call from the callback and pass the result to.
var forSearching = "test2";
console.log("begin");
request("http://mercsystem.esy.es/get.php?username=" + forSearching, function (err, res, body) {
console.log("in request() callback");
if (err) return console.error(err);
var main = JSON.parse(body);
if (main.success == "false") {
message.reply("Sorry, invalid user!")
} else {
var name = main.Username
console.log(name); // value shows fine here
// use the name variable here or call some function and pass
// the name variable to it
}
});
console.log("after request() call");
// You cannot use the name value here (even if it was in a global) because
// the async callback has not yet been called
If you ran this code with the console.log() statement I've added, you would see this sequence of events:
begin
after request() call
in request() callback
From this sequence, you can see that code after your request() call runs BEFORE the async callback runs. Thus, you cannot use your name variable there, even if it is in a global.
I've tried to download this image a million ways and I always get an undefined value in NodeJS, can someone please help me!! The image does download, but I cannot do anything with it, so it essentailly becomes useless!
var rawImg = urlHtml('img').attr('src'); // got the image needed!
var download = function (rawImg, filename, callback) {
request.head(rawImg, function (err, res, body) {
console.log('content-type:', res.headers['content-type']);
console.log('content-length:', res.headers['content-length']);
request(rawImg).pipe(fs.createWriteStream(__dirname + '/public/img/test.jpg')).on('close', function(callback) {
console.log(callback); // always getting undefined
});
});
};
download(rawImg, function () {
console.log('done'); // doesn't even launch because gets undefined before
});
Below is some sample code that downloads and saves an image to your hard drive. Here are some of the reasons why it is different from your code:
I am not sure what function urlHtml is, but my guess is that the rawImg variable is actually just the src attribute on an img html element. If that is the case, then rawImg is actually just a string with the image's URL in it. This is why, in my example code, I have renamed that variable in my code to be url.
Your function download expects three parameters (rawImg, filename, and callback), but you only call the function with two parameters (rawImg and a callback function). Since no filename is provided, your download function thinks that the function you intended to be a callback function is actually the filename. And since there is no third argument, callback is undefined.
Using the request.head function will only retrieve the HTTP headers without the body. So you won't actually get the image bytes with an HTTP HEAD request. See [https://ochronus.com/http-head-request-good-uses/ this article] for more details on HTTP HEAD. Since we are interested in downloading the image itself, we should perform an HTTP GET insted of an HTTP Head. This is why my sample code calls request.get.
.on('close', function(callback) {...}) is incorrect because the on-close function will not be called with any parameters when the close event is triggered. So callback is undefined because it is not passed into the function when the close event fires. This is why, in my example, there is no callback parameter for the on-close function.
Note 1: My example code adds some require calls for completeness, since this question is tagged as a nodejs question. It looks like you already have the fs and request variables in scope, so you may not need to require them again.
Note 2: If you are getting the URL from your urlHtml('img').attr('src'); call, then replace the hardcoded url in my code with urlHtml('img').attr('src');.
Working, example code (tested and all!!):
var fs = require('fs')
var request = require('request')
var url = 'https://avatars0.githubusercontent.com/u/5980429?v=2&s=200'
var download = function (url, filename, callback) {
request.get(url).pipe(fs.createWriteStream(__dirname + '/' + filename)).on('close', function() {
callback()
})
};
download(url, 'image.png', function () {
console.log('Image downloaded successfully. It is now saved in a file called "image.png" in the current working directory!');
});
request(rawImg).pipe(
fs.createWriteStream(__dirname + '/public/img/test.jpg')).on('close',function(callback) {
console.log(callback); // always getting undefined
});
Replace the bolded line with callback();
The 'undefined' value you are seeing is the return value of console.log.
For anyone wondering, I figured out what my problem was. I was trying to download what was in a nested object. Ended up using underscore.js !!
I'm writing a notifier for 3 deal of the day websites in node. I go and parse the body of the webpage to grab the details. In the details there is a timer for the how long the deal will last. I'm reading that timer and trying to use setTimeout/setInterval to set when the function should execute again. However the function calls are continuous instead of waiting.
Pseudo code of what I'm doing:
var getData = function(url) {
request(url, function(err, resp, body){
if(err) throw err;
//process the body getting the deal information and timer
setTimeout(getData(url),timer*1000);
}
getData(url1);
getData(url2);
getData(url3);
Full code here.
I want the program to run, continually calling itself with the new timeouts for the webpages.
I'm a Node.js newbie so I'm guessing I'm getting tripped up with the async nature of things.
Any help is greatly appreciated.
EDIT:
more simply :
var hello = function(){
console.log("hello");
setTimeout(hello(),25000);
}
hello();
prints out hello continuously instead of hello every 2.5s. What am I doing wrong?
The problem is evident in your hello example, so lets take a look at that:
var hello = function(){
console.log("hello");
setTimeout(hello(),25000);
}
hello();
In particular this line: setTimeout(hello(),25000);. Perhaps you are expecting that to call hello after a 25 second timeout? Well it doesn't, it calls hello immediately, (that's what hello() does in Javascript, and there is nothing special about setTimeout), and then it passes the return value of hello() to setTimeout, which would only make sense if hello() returned another function. Since hello recursively calls itself unconditionally, it doesn't ever return, and setTimeout will never be called. It's similar to doing the following:
function hello() {
return doSomething(hello());
}
Is it clear why doSomething will never be called?
If you want to pass a function to setTimeout, just pass the function itself, don't call it and pass the return value: setTimeout(hello, 25000);.
Your fixed code:
var getData = function(url) {
request(url, function(err, resp, body){
if(err) throw err;
//process the body getting the deal information and timer
setTimeout(getData, timer*1000, url);
});
};
getData(url1);
getData(url2);
getData(url3);
Noticed that I passed the argument for getData as a third argument to setTimeout.
What's happening is 'request' is being run as soon as getData is called. Do you want getData to be the function you call to start the timer, or the one that loads the data?
var getData = function(url) {
function doRequest(url) {
request(url, function(err, resp, body) {
if(err) throw err;
//process the body getting the deal information and timer
}
setTimeout(doRequest(url),timer*1000);
}
getData(url1);
getData(url2);
getData(url3);
What you want is 'setTimeout' to point to a function (or anonymous function/callback) that you run after the timer expires. As you originally wrote, getData was immediately calling request (and then calling getData again after your timer)