CasperJS, parallel browsing WITH the testing framework - node.js

Question : I would like to know if it's possible to do parallel browsing with the testing framework in one script file, so with the tester module and casperjs test command.
I've seen some people create two casper instances :
CasperJS simultaneous requests and https://groups.google.com/forum/#!topic/casperjs/Scx4Cjqp7hE , but as said in the doc, we can't create new casper instance in a test script.
So i tried doing something similar-simple example- with a casper testing script (just copy and execute this it will work):
var url1 = "http://casperjs.readthedocs.org/en/latest/testing.html"
,url2 = "http://casperjs.readthedocs.org/en/latest/testing.html"
;
var casperActions = {
process1: function () {
casper.test.begin('\n********* First processus with our test suite : ***********\n', function suite(test) {
"use strict";
casper.start()
.thenOpen(url1,function(){
this.echo("1","INFO");
});
casper.wait(10000,function(){
casper.test.comment("If parallel, it won't be printed before comment of the second processus !");
})
.run(function() {
this.test.comment('----------------------- First processus over ------------------------\n');
test.done();
});
});
},
process2: function () {
casper.test.begin('\n********* Second processus with our test suite : ***********\n', function suite(test) {
"use strict";
casper.start()
.thenOpen(url1,function(){
this.echo("2","INFO");
});
casper.test.comment("Hi, if parallel, i'm first !");
casper.run(function() {
this.test.comment('----------------------- Second processus over ------------------------\n');
test.done();
});
});
}
};
['process1', 'process2'].forEach(function(href) {
casperActions[href]();
});
But it's not parallel, they are executed one by one.
Currently, i do some parallel browsing but with node so not in the file itself, using child process. So if you split my previous code in two files -proc1.js,proc2.js- (just the two scenarios->casper.test.begin{...}), and launch the code below via node, something like that will work-with Linux, i have to search the equivalent syntax for windows- :
var exec = require("child_process").exec
;
exec('casperjs test proc1.js',function(err,stdout,stderr){
console.log('stdout: ' + stdout);
console.log('endprocess1');
});
exec('casperjs test proc2.js',function(err,stdout,stderr){
console.log('stdout: ' + stdout);
console.log('endprocess2');
});
My problem is that the redirections and open new urls is quite long, so i want for some of them being execute in parallel. I could do XXX files and launch them in parallel with node, but i don't want XXX files with 5 lines of code, so if someone succeeded (if it's possible) to open urls in parrallel in the same testing file without node (so without multiple processes), please teach me!
And i would like to know what is the difference between chaining instructions, or re-use the casper object each time :
so between that :
casper.test.begin('\n********* First processus with our test suite : ***********\n', function suite(test) {
"use strict";
casper.start()
.thenOpen(url1,function(){
this.echo("1","INFO");
})
.wait(10000,function(){
casper.test.comment("If parallel, it won't be print before comment of the second processus !");
})
.run(function() {
this.test.comment('----------------------- First processus over ------------------------\n');
test.done();
});
});
And that :
casper.test.begin('\n********* First processus with our test suite : ***********\n', function suite(test) {
"use strict";
casper.start();
casper.thenOpen(url1,function(){
this.echo("1","INFO");
});
casper.wait(10000,function(){
casper.test.comment("If parallel, it won't be print before comment of the second processus !");
})
casper.run(function() {
this.test.comment('----------------------- First processus over ------------------------\n');
test.done();
});
});
Chaining my instructions, will it block all the chain if one of my step fail (promise rejected) instead of executing every casper steps?
So it would be better to chaining instructions with dependant steps [like thenClick(selector)] and use the casper object with independant steps (like open a new url), wouldn't it?
Edit : i tried and if a step fail, chained or not, it will stop all the next steps, so i don't see the difference using chained steps or not...

Well, chaining or using the casper object each time is just a matter of taste, it does the same, and we can't launch several instances of casper in a testing script. If you have a loop which opens some links, you'll have to wait for each page to be loaded sequentially.
To launch parallel browsing with the testing framework, you have to do multiple processes, so using node does the trick.
After digging, I finally split files with too many redirections to be not longer than my main scenario which can't be split. A folder with 15 files is executed -parallel- in 2/4 min, on local machine.

There's no official support for parallel browsing right now in casperjs, There are multiple work arounds I've set up several different environments and am about to test witch one is the best
I see one person is working with multiple casper instances this way.
var google = require('casper').create();
var yahoo = require('casper').create();
google.start('http://google.com/');
yahoo.start('http://yahoo.com/', function() {
this.echo(google.getTitle());
});
google.run(function() {});
yahoo.run(function() {});
setTimeout(function() {
yahoo.exit();
}, 5000);
Currently I am running multiple caspers in node using 'child_process'. It is very heave on both CPU and memory

Related

Node.JS: Make module runnable through require or from command line

I have a script setupDB.js that runs asynchronously and is intended to be called from command line. Recently, I added test cases to my project, some of which require a database to be set up (and thus the execution of aforementioned script).
Now, I would like to know when the script has finished doing its thing. At the moment I'm simply waiting for a few seconds after requiring setupDB.js before I start my tests, which is obviously a bad idea.
The problem with simply exporting a function with a callback parameter is that it is important that the script can be run without any overhead, meaning no command line arguments, no additional function calls etc., since it is part of a bigger build process.
Do you have any suggestions for a better approach?
I was also looking for this recently, and came across a somewhat-related question: "Node.JS: Detect if called through require or directly by command line
" which has an answer that helped me build something like the following just a few minutes ago where the export is only run if it's used as a module, and the CLI library is only required if ran as a script.
function doSomething (opts) {
}
/*
* Based on
* https://stackoverflow.com/a/46962952/7665043
*/
function isScript () {
return require.main && require.main.filename === /\((.*):\d+:\d+\)$/.exec((new Error()).stack.split('\n')[ 2 ])[ 1 ]
}
if (isScript) {
const cli = require('some CLI library')
opts = cli.parseCLISomehow()
doSomething(opts)
} else {
module.exports = {
doSomething
}
}
There may be some reason that this is not a good idea, but I am not an expert.
I have now handled it this way: I export a function that does the setup. At the beginning I check if the script has been called from command line, and if so, I simply call the function. At the same time, I can also call it directly from another module and pass a callback.
if (require.main === module) {
// Called from command line
runSetup(function (err, res) {
// do callback handling
});
}
function runSetup(callback) {
// do the setup
}
exports.runSetup = runSetup;
make-runnable npm module can help with this.

How to run remote external commands on meteor server from MeteorJS Application?

I create some CasperJS scripts that login into Duolingo, do click on a module and open as If I were playing there.
I create a simple meteorJS application and I want that when I click a button be able to execute that casperjs script. I am looking for someone with that experience to help me or oriented me in the right way because I don't have much idea of what can I use to achieve this little personal game.
I have read about RPC - Remote Procedure Call of MeteorJS, and I have read that with PHP and NodeJS you can run a function that executes the script as if I were typing the commands to run the script.
I have found these resources:
ShellJS: https://github.com/shelljs/shelljs
and NodeJS child process: https://nodejs.org/api/child_process.html#child_process_child_process_exec_command_options_callback.
but I don't have much experience, I am doing this to learn more about CasperJS, MeteorJS.
What I need is to be able to run this command -> "casperjs duolingo.js --engine=slimerjs --disk-cache=no" using my Meteorjs app so I can continue creating my little automation bot to play Duolingo totality.
Thank you very much for your help.
it is a "simple" if you know what to do :-)
Just to know what will happen:
1.) You create a method on server side which can run external processes
2.) You create a meteor remote method which can be called by client
3.) You create the action on client and call remote meteor method
4.) You bind the click event to call the action on client
Method to call external processes
process_exec_sync = function (command) {
// Load future from fibers
var Future = Npm.require("fibers/future");
// Load exec
var child = Npm.require("child_process");
// Create new future
var future = new Future();
// Run command synchronous
child.exec(command, function(error, stdout, stderr) {
// return an onbject to identify error and success
var result = {};
// test for error
if (error) {
result.error = error;
}
// return stdout
result.stdout = stdout;
future.return(result);
});
// wait for future
return future.wait();
}
Meteor remote server method
// define server methods so that the clients will have access to server components
Meteor.methods({
runCasperJS: function() {
// This method call won't return immediately, it will wait for the
// asynchronous code to finish, so we call unblock to allow this client
// to queue other method calls (see Meteor docs)
this.unblock();
// run synchonous system command
var result = process_exec_sync('casperjs duolingo.js --engine=slimerjs --disk-cache=no');
// check for error
if (result.error) {
throw new Meteor.Error("exec-fail", "Error running CasperJS: " + result.error.message);
}
// success
return true;
}
})
Client event and remote method call
Template.mytemplate.events({
'click #run-casper': function(e) {
// try to run remote system call
Meteor.call('runCasperJS', function(err, res) {
// check result
if (err) {
// Do some error notification
} else {
// Do some success action
}
});
}
});
Resume
You need to place the server side methods into files on directory "yourproject/server" (e.g.) main.js and the client part into your template with the Button you wish to press (rename mytemplate to your defined one).
Hope you get out what you need.
Cheers
Tom

Analyse code coverage of a nodejs app during a crash recover story

I am actually trying to reach the highest score of code coverage for my nodeJS app, I am using Mocha + Chai + Istanbul to ensure this code coverage.
Part of my code's app is dedicated to recover a potential service interruption, to do so most of app objects are serialized real time in Redis.
So when the app start for the first time, I am exclusively loading data from mongoDB. After a crash recovery, I am also loading data from Redis. Obviously my app is not using the same part of the code to perform this two kind of initialization.
For this reason Istanbul dead-code report is not accurate and differ if I am running my test in the 1st or the 2nd case.
I am looking for a way to simulate a process interruption without killing Istanbul process to be able to get a complete Istanbul report.
I tried to solved it by using child_process :
var masterLogger = require("./storage/file-logger").master;
const exec = require("child_process").exec;
var redis_flush = exec("redis-cli -n 6 flushall");
var srv1;
var srv2;
redis_flush.on("close", function() {
masterLogger.fatal("Redis clear");
redis_flush = null;
masterLogger.fatal("Starting SRV1");
srv1 = exec("npm test");
srv1.on("close", function() {
masterLogger.fatal("SRV1 killed");
srv1 = null;
masterLogger.fatal("Starting SRV2");
srv2 = exec("npm test");
srv2.on("close", function() {
masterLogger.fatal("SRV2 killed");
srv2 = null;
});
});
});
process.on("SIGINT", function() {
if (redis_flush)
redis_flush.kill();
else if (srv1)
srv1.kill();
else if (srv2)
srv2.kill();
else
process.exit(0);
});
But Istanbul is providing a code-coverage report only for this part of code (Where I reached 100% ^^)...
NB: npm test -> Start my app with a test env
Best regards
After more research on this issue, I realized it was probably easier to "concatenate" the result of two distinct execution.
And apparently the Istanbul team thought about it ->
Look at the last answer of this github ticket
This solution gives me a way to generate multiple coverage files and then generate a single report.

Selenium Webdriver JS Scraping Parallely [nodejs]

I'm trying to create a pool of Phantom Webdrivers [using webdriverjs] like
var driver = new Webdriver.Builder().withCapabilities(Webdriver.Capabilities.phantomjs()).build();
Once the pool gets populated [I see n-number of phantom processes spawned], I try to do driver.get [using different drivers in the pool] of different urls expecting them to work parallely [as driver.get is async].
But I always see them being done sequentially.
Can't we load different urls parallely via different web driver instances?
If not possible in this way how else could I solve this issue?
Very Basic Impl of my question would look like below
var Webdriver = require('selenium-webdriver'),
function getInstance() {
return new Webdriver.Builder().withCapabilities(Webdriver.Capabilities.phantomjs()).build();
}
var pool = [];
for (var i = 0; i < 3; i++) {
pool.push(getInstance());
}
pool[0].get("http://mashable.com/2014/01/14/outdated-web-features/").then(function () {
console.log(0);
});
pool[1].get("http://google.com").then(function () {
console.log(1);
});
pool[2].get("http://techcrunch.com").then(function () {
console.log(2);
});
PS: Have already posted it here
Update:
I tried with selenium grid with the following setup; as it was mentioned that it can run tests parallely
Hub:
java -jar selenium/selenium-server-standale-2.39.0.jar -hosost 127.0.0.1 -port 4444 -role hub -nodeTimeout 600
Phantom:
phantomjs --webdriver=7777 --webdriver-selium-grid-hub=http://127.0.0.1:4444 --debug=true
phantomjs --webdriver=7877 --webdriver-selium-grid-hub=http://127.0.0.1:4444 --debug=true
phantomjs --webdriver=6777 --webdriver-selium-grid-hub=http://127.0.0.1:4444 --debug=true
Still I see the get command getting queued and executed sequentially instead being parall. [But gets properly distributed across 3 instances]
Am I still missing something out?
Why is it mentioned "scale by distributing tests on several machines ( parallel execution )" in the doc?
What is parallel as per the hub? I'm getting clueless
I guess I got the issue..
Basically https://code.google.com/p/selenium/source/browse/javascript/node/selenium-webdriver/executors.js#39 Is synchronous and blocking operation [atleast the get].
Whenever the get command is issued node's main thread get's stuck there. No further code execution.
A little late but for me it worked with webdriver.promise.createFlow.
You just have to wrap your code in webdriver.promise.createFlow() { ... }); and it works for me! Here's an example from Make parallel requests to a Selenium Webdriver grid. All thanks to the answerer there...
var flows = [0,1,2,3].map(function(index) {
return webdriver.promise.createFlow(function() {
var driver = new webdriver.Builder().forBrowser('firefox').usingServer('http://someurl:44111/wd/hub/').build();
console.log('Get');
driver.get('http://www.somepage.com').then(function() {
console.log('Screenshot');
driver.takeScreenshot().then(function(data){
console.log('foo/test' + index + '.png');
//var decodedImage = new Buffer(data, 'base64')
driver.quit();
});
});
});
});
I had the same issues, I finally got around the problem using child_process.
The way my app is setup is that I have many tasks that does different things, and that needs to run simultaneously (each of those use a different driver instance), obviously it was not working.
I now start those tasks in a child_process (which will run a new V8 process) and it does run everything in parallel.

NODE fs.readFile, JSON.parse and fs.writeFile

I'm writing an app in Node and have been running into a rare but detrimental occurrence.
So I have a schedule.txt and I write to it when the user makes a change but then also read it every second and then parse it for use throughout the program.
Rarely what happens is as a user is writing to the file (asynchronously) the app (based on the timer) reads the same file and attempts to parse it and fails.
I know from a design stand-point maybe this is just bound to happen... but I'm wondering if there is a quick fix I can do now. Would using writeFileSync help my situation? (make it more 'atomic'?) I just want to make sure that the app doesn't read the file while another process is still writing to the file.
TIA!
Niko
Seems like you'd want to serialize your read/writes. If it were me, I might try having a "manager" object which encapsulates the serialization, which you'd use like:
var fileManager = require('./file-manager');
// somewhere in the program
fileManager.scheduleWrite(data, function(err){
// now the write is done
});
// somewhere else in the program
fileManager.scheduleRead(function(err, data){
// `data` contains the data
});
Then implement it using Q or a similar promises lib, like:
// in file-manager.js
var wait = Q();
module.exports = {
scheduleWrite: function(data, cb){
wait = wait.then(function(){
// write data and call cb()
});
},
scheduleRead: function(){
wait = wait.then(function(){
// read data and call cb(data)
});
}
};
The wait var will "stack up" into a serialized chain of tasks where the next one won't start until the previous one completes.

Resources