I am using cypress-io and cypress-axe to run browser tests and check if a page has accessibility violations.
I have this task:
function reportA11yViolations(violations: Result[]) {
const errors: string[] = [];
violations.forEach((v: Result) => v.nodes.forEach((node: NodeResult) => {
errors.push(node.failureSummary);
}));
cy.log(errors.join("\n"));
}
Cypress.Commands.add("checkA11yAndReportViolations", (context, options) => {
cy.checkA11y(context, options, reportA11yViolations);
});
I can see all the browser log messages when running through the nice browser app.
But what about when I am running in headless in the ci environment?
Is there anyway I can display these errors anywhere or write them to a file?
You can output the cypress logs to a file
cy.writeFile('accessibilityReport.txt', `${node.failureSummary} \n`, { flag: 'a+' });
a+ will append the result to the file. The build artifacts of the jenkins pipeline should have the report thus created.
Related
I'm trying to run a pretty simple test through cypress , with chrome , through module-api. I don't want to run the tests through their launcher but from node as I'll start the tests through mocha framework.
The tests at least can start with electron , but can't find the elements I'm trying to click. It works with chrome through the launcher - with npx cypress run --browser chrome. Though when I use the module-api I get this error:
Can't run because you've entered an invalid browser name.
Browser: electron was not found on your system or is not supported by Cypress.
Cypress supports the following browsers:
Here's my config:
const { defineConfig } = require('cypress')
module.exports = defineConfig({
fixturesFolder: false,
e2e: {
experimentalSessionAndOrigin: true, // multi-domain tests as there could be an Auth0 authentication
setupNodeEvents(on, config) {
return {
browsers: config.browsers.filter(
(b) => {
if (b.name == 'chrome') {
console.log(b); // print info about the browser
return b;
}
}
),
}
}
},
})
The file that starts the test (e.g. example-test.js):
const cypress = require('cypress')
return cypress.run({
spec: './cypress/e2e/simple-login.cy.js'
}).then((result) => {
console.log(result);
process.exit(0);
})
I'm starting it with:
node example-test.js
(the simple-login just enters a password , email and clicks a button and it works when I use cypress run ..).
The console log prints correctly the information about the browser ,though still there's an error. Any idea how this can be fixed?
Your return value in setupNodeEvents() is only returning chrome type browsers, so the error
Browser: electron was not found on your system or is not supported by Cypress
is caused by that.
The other side-effect is that other config like experimentalSessionAndOrigin are being removed.
This is how you can log chrome browsers without changing the config
const { defineConfig } = require('cypress')
module.exports = defineConfig({
fixturesFolder: false,
e2e: {
experimentalSessionAndOrigin: true,
setupNodeEvents(on, config) {
config.browsers.forEach((b) => {
if (b.name == 'chrome') {
console.log(b); // print info about the browser
}
})
return config // NOTE unchanged config
}
},
})
Note npx cypress run --browser chrome will work because your filter is allowing chrome browsers to pass, but npx cypress run --browser electron will fail.
I have a really basic playwright (v1.18.1) test on Windows (11):
import { test, expect } from '#playwright/test';
test('test', async ({ page }) => {
await page.goto('http://whatsmyuseragent.org/');
//await page.screenshot({ path: `./tests/tmp/screenshots/whats.my.ua.${project:name}.png` });
await page.screenshot();
});
And when I run this - it says it worked
Running : cd C:\playwright\tests && npx playwright test whats_my_user_agent.spec.ts --reporter=list
Running 5 tests using 3 workers
ok [webkit] › whats_my_user_agent.spec.ts:3:1 › test (3s)
ok [firefox] › whats_my_user_agent.spec.ts:3:1 › test (4s)
ok [chromium] › whats_my_user_agent.spec.ts:3:1 › test (5s)
ok [Mobile Chrome] › whats_my_user_agent.spec.ts:3:1 › test (2s)
ok [Mobile Safari] › whats_my_user_agent.spec.ts:3:1 › test (2s)
5 passed (8s)
but I can find no sign of the screenshot file - where are they [there is no feedback giving any hints]. Also - is there a way to control the saved filename (including the project name so that I don't save 5 png files over each other).
As I see it, if you uncomment the line in which you specify the path for your screenshot to be stored, you should find it there, for example, this:
import { test, expect } from '#playwright/test';
test('testing screenshot path', async ({ page }) => {
await page.goto('https://playwright.dev/');
await page.screenshot({ path:`./screenshots/screenshot.png` });
});
should result in you having a screenshot of the website named 'screenshot.png' in a folder named 'screenshots' in the root of your project. If there's no folder by that name, it will be created.
As for your question, since this would run the same step (saving a screenshot with a certain name in a certain path) for every project that you're running, your screenshot would be overwritten, leaving you only with the screenshot your last project run. If you'd like all screenshots to be stored, or screenshots for one project in particular (although running several projects), you can play with testInfo. Say you only want to store the screenshots for the project you're running in chromium, though checking the steps pass in every other project; you can do something like this:
test('testing chromium screenshots', async ({ page }, testInfo) => {
await page.goto('https://playwright.dev/');
if (testInfo.project.name === 'chromium') {
await page.screenshot({ path:`./screenshots/${testInfo.title}.png` });
}
});
This would store a screenshot named 'testing chromium screenshots' with the extension .png in the folder 'screenshots' in the root of your project, tested with chromium, while checking the test pass in every other project you might be running.
Hope all this helps.
Ok - I found out how - I needed to include workerInfo in the async declaration, and then I can reference it in the screenshot call:
import { test, expect } from '#playwright/test';
test('test', async ({ page }, workerInfo) => { <----------- added workerInfo
await page.goto('http://whatsmyuseragent.org/');
await page.screenshot({ path: "./tests/tmp/screenshots/whats.my.ua."+workerInfo.project.name+".png" });
// ^^ referenced it with this
});
The screenshot is always returned as a buffer by the screenshot function. If you don't pass a path the only way to get that screenshot is through the buffer.
We run our app in two "modes":
Our REST API (express js)
Our background processor (amqplib)
Our REST API that starts using nodemon runs completely fine with debug, however our background processor does not work with debug.
We declare DEBUG=app:* in our .env file and yes, we do see it when we console log but for some reason when we do the following, nothing reports when running our background processor.
We do see that one of amqp's dependencies called bitsyntax uses debug. I am wondering if it does anything to turn off debug but cannot find anything in their code that does that.
Is there anything I can do to resolve this problem?
import amqp from 'amqplib/callback_api'
import dotenv from 'dotenv'
import debug from 'debug'
const testLog = debug('app:worker')
const rabbitmq = `amqp://${process.env.RABBITMQ_USER}:${process.env.RABBITMQ_PASS}#${process.env.RABBITMQ_HOST}`
console.log(process.env.DEBUG) // app:*
const worker = () => {
try {
amqp.connect(rabbitmq, (err, conn) => {
// ...
testLog('this does not show up')
console.log('this does show up')
// ...
})
} catch (err) {
// ...
}
}
worker()
We run our background processor using the following command:
NODE_ENV=development nodemon ./src/workers/index.ts
As per the following github issue response,
on workers the following code needs to be enabled:
debug.enable(process.env.DEBUG)
I am using Cypress to test my Electron application.
Since Cypress uses the browser mode, FS is not supported.
So I am getting this error:
Error in mounted hook: "TypeError: fs.existsSync is not a function"
And I found this on the documentation:
https://docs.cypress.io/api/commands/task.html#Event
So I added this on my test:
it('Sample test', () => {
cy.task('readSettingsJson', settingsFolder).then((content) => {
// This can print the JSON file contents correctly
console.log('content = ' + content)
})
})
And on my plugins/index.js:
on('task', {
readSettingsJson(foldername) {
if (!fs.existsSync(foldername)) {
fs.mkdirSync(foldername, { recursive: true })
// some command to copy the file
} else {
// This is what I am testing at this moment
return fs.readFileSync(path.join(filename, '/settings.json'), 'utf8')
}
return null
}
})
However, it doesnt seem to work. I still get the error:
Error in mounted hook: "TypeError: fs.existsSync is not a function"
And despite the test printing the json file correctly, my app still can't load the JSON file.
Am I missing anything? Help please!
Cypress support for Electron apps are in very early alpha release:
https://www.cypress.io/blog/2019/09/26/testing-electron-js-applications-using-cypress-alpha-release/
As an alternative, try using Spectron, which is the testing framework that is currently recommended by Electron team:
https://www.electronjs.org/spectron
It's not actually a problem, but I do not fully understand, what happened and why.
I have this runner for my test. I test a React app.
let testcafe = null
const isCiEnv = process.env.CI === 'true'
const exit = async err => {
console.log('Exiting...')
if (testcafe) {
console.log('Closing TestCafe...')
testcafe.close()
}
console.log('Exiting process...')
process.exit(err ? 1 : 0)
}
console.log('Is CI ENV: ', isCiEnv)
console.log('Creating TestCafe...')
createTestCafe('localhost', 1337, 1338)
.then(tc => {
testcafe = tc
})
.then(() => {
console.log('Starting server...')
return startServer()
})
.then(() => {
console.log('Starting client...')
return startClient()
})
.then(() => {
console.log('Creating TestCafe Runner...')
return testcafe.createRunner()
})
.then(runner => {
console.log('About to start TestCafe Runner...')
return runner
.src([
'test/e2e/fixtures/auth.js'
])
.browsers({
path: isCiEnv
? '/usr/bin/chromium-browser'
: 'Chrome',
cmd: isCiEnv
? '--no-sandbox --disable-gpu'
: undefined
})
.screenshots('screenshots', true)
.run({
skipJsErrors: true,
selectorTimeout: 25000,
assertionTimeout: 25000
})
})
.then(failedCount => {
console.log('failed count:', failedCount)
return exit(failedCount)
})
.catch(err => {
console.error('ERR', err)
return exit(err)
})
In package.json i have this command for run test
"test:e2e": "HOST=0.0.0.0 NODE_ENV=test NODE_PATH=server babel-node test/e2e/index.js --presets stage-2"
But in the local environment, I run a test with this command
sudo REDIS_HOST=127.0.0.1 PORT=80 yarn test:e2e
That because on my local machine I have different config and I don't want to change it for everyone else.
Usually, test runs in a different, clear version of the browser, without any account data, plugins and etc. But in this case, tests run in a new browser window, but with all plugins and my account name. But, it's doesn't have cookie and session auth data from the browser window, in which I usually work (because I authorized on-site in the working browser and doesn't auth in test browser).
And if I change "Chrome" to "chrome" it stops run completely. Same behavior for Firefox and Safari.
Earlier, without passing REDIS_HOST and HOST, it works as usual and runs in a clean new browser window.
It's not a big problem, for now at least, but it's unexpected behavior and I don't understand, why it works this way.
I'm not very familiar with Node and React, and maybe this related to them.
Spec: macOS 10.12.5, Testcafe 0.20.3, Chrome 67
Specifying browsers using { path, cmd } is a legacy low-level option, you shouldn't use it. When a browser is specified in this way, TestCafe doesn't try to guess browser's type (Chrome, Firefox) and doesn't perform advanced initialization steps like creating a clean profile (because profile structure depends on browser's type). So it's better to use the following runner code:
.browsers(isCiEnv ? 'chromium --no-sandbox --disable-gpu' : 'chrome')