casperJS - grunt-casper: Running multiple Test-Suites in a loop - node.js

Question

Preparation
Hi i am using CasperJS in combination with grunt-casper (github.com/iamchrismiller/grunt-casper) for running automated functional and regression tests in our GUI Development process for verification.
We use it like this, casper runner in gruntfile.js:
casper: {
componentTests: {
options: {
args: ['--ssl-protocol=any', '--ignore-ssl-errors=true', '--web-security=no'],
test: true,
includes: ['tests/testutils/testutils.js']
},
files: {
'tests/testruns/logfiles/<%= grunt.template.today("yyyy-mm-dd-hhMMss") %>/componenttests/concat-testresults.xml': [
'tests/functionaltests/componenttests/componentTestController.js']
}
},
so as it can be seen here we just normally run casper tests with SSL params and calling only ONE Controllerclass here instead of listing the single tests (this is one of the roots of my problem). grunt-casper delivers the object which is in charge for testing and inside every single Controllerclass the tests are included and concatenated....
...now the componentTestController.js looks like the following:
var config = require('../../../testconfiguration');
var urls = config.test_styleguide_components.urls;
var viewportSizes = config.test_styleguide_components.viewportSizes;
var testfiles = config.test_styleguide_components.testfiles;
var tempCaptureFolder = 'tests/testruns/temprun/';
var testutils = new testutils();
var x = require('casper').selectXPath;
casper.test.begin('COMPONENT TEST CONTROLLER', function(test) {
casper.start();
/* Run tests for all given URLs */
casper.each(urls, function(self, url, i) {
casper.thenOpen(url, function() {
/* Test different viewport resolutions for every URL */
casper.each(viewportSizes, function(self, actViewport, j) {
/* Reset the viewport */
casper.then(function() {
casper.viewport(actViewport[0], actViewport[1]);
});
/* Run the respective tests */
casper.then(function() {
/* Single tests for every resolution and link */
casper.each(testfiles, function(self, actTest, k) {
casper.then(function() {
require('.'+actTest);
});
});
});
});
});
});
casper.run(function() {
test.done();
});
});
Here you can see that we running a 3 level loop for testing
ALL URLs given in a JSON config file which are contained in an ARRAY of String ["url1.com","url2.com"....."urln.com"]
ALL VIEWPORT SIZES so that every URL is tested in our desired Viewport resolutions to test the correct Responsibility behaviour of the components
ALL TESTFILES, all testfiles only include a TEST STUB what means, no start, begin or something else, its all in a large Testsourrounding.
MAYBE this is already mocky and can be done in a bette way, so if this is the case i would glad if someone has proposals here, but don't forget that grunt-casper is involved as runner.
Question
So far, so good, the tool in general works fine and the construction we built works as we desired. But the problem is, because all testfiles are ran in a large single context, one failing component fails the whole suite.
In normal cases this is a behaviour i would support, but in our circumstances i do not see any proper solution than log the error / fail the single testcomponent and run on.
Example:
I run a test, which is setUp like described above and in this part:
/* Single tests for every resolution and link */
casper.each(testfiles, function(self, actTest, k) {
casper.then(function() {
require('.'+actTest);
});
});
we include 2 testfiles looking like the following:
Included testfile1.js
casper.then(function () {
casper.waitForSelector(x("//a[normalize-space(text())='Atoms']"),
function success() {
casper.test.assertExists(x("//a[normalize-space(text())='Atoms']"));
casper.click(x("//a[normalize-space(text())='Atoms']"));
},
function fail() {
casper.test.assertExists(x("//a[normalize-space(text())='Atoms']"));
});
});
Included testfile2.js
casper.then(function () {
casper.waitForSelector(x("//a[normalize-space(text())='Buttons']"),
function success() {
casper.test.assertExists(x("//a[normalize-space(text())='Buttons']"));
casper.click(x("//a[normalize-space(text())='Buttons']"));
},
function fail() {
testutils.createErrorScreenshot('#menu > li.active > ul > li:nth-child(7)', tempCaptureFolder, casper, 'BUTTONGROUPS#2-buttons-menu-does-not-exist.png');
casper.test.assertExists(x("//a[normalize-space(text())='Buttons']"));
});
});
So if the assert in testfile1.js fails, everthing failes. So how can i move on to testfile2.js, even if the first fails? Is this possible to configure? Can it be encapsulated somehow?
FYI, this did not work:
https://groups.google.com/d/msg/casperjs/3jlBIx96Tb8/RRPA9X8v6w4J
Almost similar problems
My problem is almost the same like this here:
https://stackoverflow.com/a/27755205/4353553
And this guy here has almost another approach i tried but got his problems too because multiple testsuites ran in a loop occuring problems:
groups.google.com/forum/#!topic/casperjs/VrtkdGQl3FA
MUCH THANKS IN ADVICE

Answer 1

Hopefully I understood what you ware asking - is there a way to suppress the failed assertion's exception throwing behavior?
The Tester's assert method actually allows for overriding the default behavior of throwing an exception on a failed assertion:
var message = "This test will always fail, but never throw an exception and fail the whole suite.";
test.assert(false, message, { doThrow: false });
Not all assert helpers have this option though and the only other solution I can think of is catching the exception:
var message = "This test will always fail, but never throw an exception and fail the whole suite.";
try {
test.assertEquals(true, false, message);
} catch (e) { /* Ignore thrown exception. */ }
Both of these approaches are far from ideal since the requirement in our cases is to not throw for all assertions.
Another short term solution that requires overriding the Tester instance's core assert method is (but is still quite hacky):
// Override the default assert method. Hopefully the test
// casper property doesn't change between running suites.
casper.test.assert =
casper.test.assertTrue = (function () {
// Save original assert.
var __assert = casper.test.assert;
return function (subject, message, context) {
casper.log('Custom assert called!', 'debug');
try {
return __assert.apply(casper.test, arguments);
}
catch (error) {
casper.test.processAssertionResult(error.result);
return false;
}
};
})();
That said, I'm currently looking for a non-intrusive solution to this "issue" (not being able to set a default value for doThrow) as well. The above is relative to Casper JS 1.1-beta3 which I'm currently using.