So the basic server JS scraper in Meteor.
The pattern is kinda simple. Script finds certain links, then loads content from them and stores the content in variable.
Script keeps crashing when loading cheerio inside loop.
Where's the catch ? What's the best implementation for this purpose ?
Meteor.methods({
loadPage: function () {
result = Meteor.http.get("http://url.com");
$ = cheerio.load(result.content);
$('.class').each(function(i,elem){
var link = $(this).attr('href');
var title = $(this).text();
var $ = cheerio.load(Meteor.http.get(link).content);
var postContent = $('.classOnLoadedPage');
Images.insert(
{
link: link,
title: title,
postContent: postContent
});
});
}
});
I got exactly the same problem today. Turns out it is problem with cheerio itself. Rather old version of it has this bug. You have to use newer version and then it works.
the most downloaded cheerio package in atmospherejs mrt:cheerio wraps cheerio 0.12.3, while current version in npm is cheerio 0.19.0
add rclai89:cheerio instead of mrt:cheerio and it will deliver cheerio 0.18.0, and with this version load within loop works perfectly.
Related
I have to write some markdown text in my Node.js script which will be rendered on UI with respective formatting.
Here are the two options I have explored so far:
Option 1:
module.exports = {
status: "ENABLED"
remediation: "**Adding markdown remediation here** ",
}
Option 2:
const remediation = "## this is another way which seems a bit better"
module.exports = {
status: "ENABLED"
remediation,
}
The second option renders a bit better but I am still struggling with a few things like defining new lines, adding code body etc.
Wanted to check if there can be a better way to include markdown text in the script.
I would suggest that instead of creating your own wheels, please check markdown-it.
npm install markdown-it --save
// node.js, "classic" way:
var MarkdownIt = require('markdown-it'),
md = new MarkdownIt();
var result = md.render('# markdown-it rulezz!');
// node.js, the same, but with sugar:
var md = require('markdown-it')();
var result = md.render('# markdown-it rulezz!');
// browser without AMD, added to "window" on script load
// Note, there is no dash in "markdownit".
var md = window.markdownit();
var result = md.render('# markdown-it rulezz!');
I am new in node js, and I want to use different header and footer for first and last pages in node js,
I am using html-pdf module which is not working when I use the same code as they have provided.
please have a look into the code
var fs = require('fs');
var pdf = require('html-pdf');
var html = fs.readFileSync('test.html', 'utf8');
var options = { format: 'Letter' };
pdf.create(html, options).toFile('businesscard.pdf', function(err, res) {
if (err) return console.log(err);
console.log(res); // { filename: '/app/businesscard.pdf' }
});
Reference Link: https://www.npmjs.com/package/html-pdf
When I am using below code It will not work on the last page only It also print for middles pages.
<div id="pageHeader-last">Header on last page</div>
...
<div id="pageFooter-last">Footer on last page</div>
you can use html-pdf for node https://www.npmjs.com/package/html-pdf
It is working in the html-pdf version 2.1.0
you can install it via npm install html-pdf#2.1.0
There is some issue with latest version (2.2.0)
html-pdf is basically a wrapper for phantomJS that is very convenient for certain pdf generation. It's worth looking under the hood if you're going to use it much. There are limits on how much you can adjust the header and footer from the javascript side, for example, without modifying the script that node-html-pdf uses or calling phantomJS on your own.
Sometimes it is easier to assemble a PDF from smaller PDF files using node-pdftk than it is to coax node-html-pdf or phantomJS into doing what you want in one pass.
html-pdf sets the header and footer in pdf_a4_portrait.js
function createSection (section, content, options) {
...
if (pageNumFinal === 1 && !html) html = o.first || c.first
if (numPagesFinal === numPages && !html) html = o.last || c.last
return (html || o.default || c.default || '')
.replace(/{{page}}/g, pageNumFinal)
.replace(/{{pages}}/g, numPagesFinal) + content.styles
...
}
I need the jquery source in my npm module that I built.
I originally did this:
fs.readFile("./node_modules/jrender/node_modules/jquery/dist/jquery.js", "utf-8", function(err, src){
});
However this doesn't work if you already installed jquery in your main project, it wont download and install it to this path.
How can I fix this?
Either I check if this file exists in either place, then use the one that exists.
Include a fixed version of jquery in the module
Does require() allow you to view the source of the included javascript file
You can find the location of the jquery file by looking in module.children
var s = require("jquery");
var jqfile = null;
for (var i = 0;i < module.children.length;i++) {
if (module.children[i].filename.match(/jquery.js$/))
jqfile = module.children[i].filename
}
fs.readFile(jqfile, "utf-8", function(err, src){
});
(Edited to greatly simplify)
On node I have the following server.js file.
var Backbone = require('backbone');
var Tweet = Backbone.Model.extend({});
var Tweets = Backbone.Collection.extend({
model : Tweet,
url: function () {
return 'http://search.twitter.com/search.json?q=backbone'
}
});
var myTweets = new Tweets();
myTweets.fetch();
When I run this, I get an error that says. "Cannot call method 'ajax' of undefined" (1359:14)
basically that is the result of $ being undefined. Why is it undefined? Well there are a number of intermediate steps but when the file is loaded, it is expecting "this" to be "window" in browser or "global" on server. executed on node "this" = {}.
So the question, "How do I set 'this' to global" inside the backbone.js file?
On Backbone >= 1.x, you can simply assign Backbone.$ rather than using Backbone.setDomLibrary.
Solution for Backbone < 0.9.9
The first issue you need to address is how you are running this on Node anyway. Nodejs is a server-side JS environment, but it does not include any logic for controlling a DOM. For that you need to load something like JSDom.
When you have some DOM environment set up, you can load jQuery and your code into it and it should work just like a browser.
To answer your question specifically though, loading jQuery into the global is a bit of an ugly way to do it. You should use Backbone's setDomLibrary function to set $ to what you want.
Try something like this:
if (typeof exports !== 'undefined') {
MyModels = exports;
Backbone.setDomLibrary(require('jquery'));
server = true;
} else {
MyModels = this.MyModels = {};
}
This will fail if you try to do any DOM functions though.
For testing purposes I am trying to remove some amd modules and reload updated versions from the server - with the goal of not refreshing the browser.
I am currently doing the following but the browser still doesn't reload the items from the network.
var scripts = document.getElementsByTagName('script');
var context = require.s.contexts['_'];
for (var key in context.defined) {
if(key.indexOf("tests")>-1){
requirejs.undef(key);
for (var i = scripts.length - 1; i >= 0; i--) {
var script = scripts[i];
var attr = script.getAttribute('data-requiremodule')
if (attr === key){
script.parentNode.removeChild(script);
}
}}
It deletes the references from the context and removes the script tags successfully.
But alas...
Does anyone know the mechanism to clear all the references from requirejs?
Any help much appreciated
We are currently trying out this implementation:
require.onResourceLoad = function(context, map)
{
require.undef(map.name);
};
No issues has surfaced so far.
Edit: IE doesn't care much about this fix. Chrome and FF is fine however.
Also, you should try live-edit in PhpStorm. Works like a charm. A demo can be seen here. Chrome only though.
this answer is similar to the user1903890 but i think its easy to follow making other implementation.
Basically we have to encapsulate in an init function the main.js requirejs controller specified in our index.html. Once it's defined, then we call to this init function to init requirejs normally
function init_requirejs(){
console.log("--------------------------- INIT requirejs:");
require([ "helpers/util"], function(util) {
var count=0;
$('#content').empty();
$('#content').append("<input type='button' id='increment_button' value='click to increment the counter'>");
$('#content').append("<h1 id='the_counter'>0</h1>");
$('#content').append("<br><br><input type='button' id='init_button' value='click to initialize requirejs files'>");
$('#increment_button').on('click', function(){
count++;
$('#the_counter').text(count);
});
$('#init_button').on('click', function(){
end();
init_requirejs();
});
util();
});
};
init_requirejs();
Also we need and use the require.onResourceLoad function to store all the files that participate in the requirejs app
var all=[];
require.onResourceLoad = function (context, map, depArray) {
all.push(map.name);
};
And we need a reset requirejs configuration function to delete the actual instance of requirejs, we will do it with the require.undef function:
function end(){
console.log("--------------------------- END requirejs:");
all.map(function(item){
require.undef(item);
});
};
That's all!
Later from our code we can force the reload app without reload the browser only calling to end() function and init_rquirejs() function. For example inside a jquery click event:
$('#init_button').on('click', function(){
end();
init_requirejs();
});
The code of the demo is in
https://github.com/juanantonioruz/requirejs-force-reload
And an online version
https://dl.dropboxusercontent.com/u/8688858/requirejs-force-reload/project.html
I hope this solution work for you!
Juan