Web scraping with node.js/cheerio - cannot get <span> text - node.js

I need to make a simple web scraper to grab some basic info about the Athens Stock Exchange in real time. My weapon of choice is Node.js and more specifically the 'cheerio' module.
The info I want to grab is represented in the website as the text inside some elements. These elements are nested inside another one. An example is this:
<span id="tickerGeneralIndex" class="style3red">
<span class="percentagedelta">
-0,50%
</span>
</span>
In this case, the data I want to extract is '-0,50%'.
The code I have written is this:
var request = require('request'),
cheerio = require('cheerio');
request('http://www.euro2day.gr/AseRealTime.aspx', function (error, response, html) {
if (!error && response.statusCode == 200) {
var $ = cheerio.load(html);
var span = $('span.percentagedelta').text();
console.log(span);
}
});
This code does not produce the desired output. When run it logs a single empty line in the console.
I have tried to modify my code like this for testing purposes:
var request = require('request'),
cheerio = require('cheerio');
request('http://www.euro2day.gr/AseRealTime.aspx', function (error, response, html) {
if (!error && response.statusCode == 200) {
var $ = cheerio.load(html);
var span = $('span.percentagedelta').attr('class');
console.log(span);
}
});
This way I get 'percentagedelta' in the console. This is correct, as I have asked to get the class of the element. Of course this is not what I wanted. I merely did this to find out if the 'span' variable is loaded correctly.
I am beginning to suspect this has something to do with the characters in the text. Is it possible that some encoding issue is to blame? And if yes, how can I fix that?

The original html of http://www.euro2day.gr/AseRealTime.aspx has no data in 'percentagedelta'
You can look throw you html variable.
Data is setting synchronically by javascript on the page
$("#tickerGeneralIndex .percentagedelta").html(data.percentageDelta);
Maybe it would be more simple to fetch http://www.euro2day.gr/handlers/data.ashx?type=3 that page loads with ajax

Related

parsing by cookie result sequence of weird things

var request = require('request'),
cheerio = require('cheerio');
var i
for(i=1; i<908; i++){
var options = {
url:k,
var address='http://gallog.dcinside.com/inc/_mylog.php?
gid=chermy018&oneview=Y&cid=59&page=';
var k = address+'i';
request(options, function (err, response, body) {
console.log(body);
});
}
error
i selected parsing method by using cookie. if i run this code for one page, it work well, but for multiple pages it result sequence of weird symbols or letters like above. what cause this and how can i correct error? and because i only searched and modified nodejs codes for parsing, i don't know much well about cheerio. i want to parse texts but i don't know how designate under 'id' part.

Node.js Web Scraping with Jsdom

I would like to scrape the http://www.euromillones.com.es/ website to get the last winning 5 numbers and two stars. It can be seen on the left column of the website. I've been reading tutorials, but I am not capable of achieving this.
This is the code I wrote so far:
app.get('/winnernumbers', function(req, res){
//Tell the request that we want to fetch youtube.com, send the results to a callback function
request({uri: 'http://www.euromillones.com.es/ '}, function(err, response, body){
var self = this;
self.items = new Array();//I feel like I want to save my results in an array
//Just a basic error check
if(err && response.statusCode !== 200){console.log('Request error.');}
//Send the body param as the HTML code we will parse in jsdom
//also tell jsdom to attach jQuery in the scripts and loaded from jQuery.com
jsdom.env({
html: body,
scripts: ['http://code.jquery.com/jquery-1.6.min.js ']
}, function(err, window){
//Use jQuery just as in a regular HTML page
var $ = window.jQuery;
res.send($('title').text());
});
});
});
I am getting the following error:
Must pass a "created", "loaded", "done" option or a callback to jsdom.env.
It looks to me that you've just used a combination of arguments that jsdom does not know how to handle. The documentation shows this signature:
jsdom.env(string, [scripts], [config], callback);
The two middle arguments are optional but you'll note that all possible combinations here start with a string and end with a callback. The documentation mentions one more way to call jsdom.env, and that's by passing a single config argument. What you are doing amounts to:
jsdom.env(config, callback);
which does not correspond to any of the documented methods. I would suggest changing your code to pass a single config argument. You can move your current callback to the done field of that config object. Something like this:
jsdom.env({
html: body,
scripts: ['http://code.jquery.com/jquery-1.6.min.js'],
done: function (err, window) {
//Use jQuery just as in a regular HTML page
var $ = window.jQuery;
res.send($('title').text());
}
});

Insert var into url and follow in node js with cheerio/request

I am trying to take a variable from the original url, insert it into a second url, visit that url and then access variable from it. I have two problems:
Problem 1: 'myurl' variable returns value of
"http://api.trove.nla.gov.au/work/undefined?key=6k6oagt6ott4ohno&reclevel=full"
That is, it is not taking the 'myid' variable.
Problem 2: How do I then follow the 'myurl' url as I want to access the DOM? Do I make another 'request' for 'myurl'?
Here is my code so far:
var request = require('request'),
cheerio = require('cheerio');
request('http://api.trove.nla.gov.au/result?key=6k6oagt6ott4ohno&zone=book&l-advformat=Thesis&sortby=dateDesc&q=+date%3A[2000+TO+2014]&l-availability=y&l-australian=y&n=0&s=0', function(error, response, html) {
if (!error && response.statusCode == 200) {
var $ = cheerio.load(html, {
xmlMode: true
});
var myid = ($('work').attr('id'))
var myurl = "http://api.trove.nla.gov.au/work/" +(myid)+ "?key=6k6oagt6ott4ohno&reclevel=full"
console.log(myurl)
}
});
Your selector is probably wrong, did you mean '#work' or '.work' perhaps?
Yes, you'll have to make another request to myurl using request() (since that's what you're using)

What is the best way to get all image urls from an external web page in node.js

From node, I'd like to get all image urls (src attribute from img tags) from an external web page.
I started by considering phantonjs, but didn't like that it's not really integrated into node (i.e. it runs in an external process).
Next, I tried to use the request module and cheerio. This works great, except I have to deal with relative image urls. E.g.
<img src='http//example.com/i.jpg'>
<img src='/i.jpg'>
<img src='i.jpg'>
<img src='../images/i.jpg'>
I can deal with that, but I'm wondering if there's an easier way?
I ended up using the request node module along with cheerio and url. Here's what I ended up doing (please note, this is mvp code, not production quality):
app.get('/scrape-images', function(req, res) {
request(req.query.url, function (error, response, body) {
if (!error && response.statusCode == 200) {
var $ = cheerio.load(body);
var reqUrl = url.parse(req.query.url);
res.send($('img').map(function(i, e) {
var srcUrl = url.parse($(e).attr('src'));
if (!srcUrl.host) {
return url.resolve(reqUrl, srcUrl);
} else {
return url.format(srcUrl);
}
}));
}
});
});

jsdom form submission?

I'm trying to use the Node.js packages request and jsdom to scrape web pages, and I want to know how I can submit forms and get their responses. I'm not sure if this is possible with jsdom or another module, but I do know that request supports cookies.
The following code demonstrates how I'm using jsdom (along with request and jQuery) to retrieve and parse a web page (in this case, the Wikipedia home page). (Note that this code is adapted from the jquery-request.js code from this tutorial http://blog.nodejitsu.com/jsdom-jquery-in-5-lines-on-nodejs)
var request = require('request'),
jsdom = require('jsdom'),
url = 'http://www.wikipedia.org';
request({ uri:url }, function (error, response, body) {
if (error && response.statusCode !== 200) {
console.log('Error when contacting '+url);
}
jsdom.env({
html: body,
scripts: [
'http://code.jquery.com/jquery-1.5.min.js'
]
}, function (err, window) {
var $ = window.jQuery,
// jQuery is now loaded on the jsdom window created from 'agent.body'
$searchform = $('#searchform'); //search form jQuery object
$('#searchInput').val('Wood');
console.log('form HTML is ' + $searchform.html(),
'search value is ' + $('#searchInput').val()
//how I'd like to submit the search form
$('#searchform .searchButton').click();
);
});
});
The above code prints the HTML from Wikipedia's search form, then "Wood", the value I set the searchInput field to contain. Of course, here the click() method doesn't really do anything, because jQuery isn't operating in a browser; I don't even know if jsdom supports any kind of event handling.
Is there any module that can help me to interact with web pages in this way, or in a similar non-jQuery way? Can this be done in jsdom?
Thanks in advance!
If you don't want to handle the POST request yourself like in the other answer, you can use an alternative to jsdom that does support more things in a browser.
http://www.phantomjs.org/
I'm not familiar with a nodejs library that will let you get a fully interactive client-side view of a web-page, but you can get the results of a form submission without too much worry.
HTML forms are essentially just a way of sending HTTP requests to a specific URL (which can be found as the action attribute of the form tag). With access to the DOM, you can just pull out these values and create your own request for the specified URL.
Something like this as the callback from requesting the wikipedia home page will get you the result of doing a search for "keyboard cat" in english:
var $ = window.jQuery;
var search_term = "keyboard cat";
var search_term_safe = encodeURIComponent(search_term).replace("%20", "+");
var lang = "en";
var lang_safe = encodeURIComponent(lang).replace("%20", "+");
var search_submit_url = $("#searchform").attr("action");
var search_input_name = $("#searchInput").attr("name");
var search_language_name = $("#language").attr("name");
var search_string = search_input_name + "=" + search_term_safe + "&" + search_language_name + "=" + lang_safe;
// Note the wikipedia specific hack by prepending "http:".
var full_search_uri = "http:" + search_submit_url + "?" + search_string;
request({ uri: full_search_uri }, function(error, response) {
if (error && response.statusCode != 200) {
console.log("Got an error from the search page: " + error);
} else {
// Do some stuff with the response page here.
}
});
Basically the important stuff is:
"Submitting a search" really just means sending either a HTTP GET or POST request to the URL specified at the action attribute of the form tag.
Create the string to use for form submission using the name attributes of each of the form's input tags, combined with the value that they are actually submitting, in this format: name1=value1&name2=value2
For GET requests, just append that string to the URL as a query string (URL?query-string)
For POST requests, post that string as the body of the request.
Note that the string used for form submission must be escaped and have spaces represented as +.

Resources