Firefox doesn't wait for a page load Webdriverio - node.js

I am trying to run my test using Selenium and just encountered the problem. I have my test written for the Chrome browser. Now I have been trying to run the same tests in the Firefox browser, but they failed.
I've started to investigate the problem and found out that Firefox doesn't wait until page is fully loaded. Chrome works perfectly.
I am running Selenium in docker containers.
Here is my script
storeSearch(info) {
let that = this;
return new Promise(function (resolve, reject) {
browserClient.init()
.url("http://somewhere.com")
.selectByVisibleText("#store","Tech")
// Redirect to a new page
.setValue("input[name='search']", info.searchCriteria)
.selectByValue(".featured", 'MacBook')
.click("button[name='info']")
.element('.popup')
.then(function (element) {
if (element.state === 'success') {
}
});
});
}
It doesn't try even to select a store type from the select .selectByVisibleText("#store","Tech") and just throws an exception.
"An element could not be located on the page using the given search
parameters (\"input[name='search']\").",
I have tried to add timeouts but it doesn't work as well and gives me an error.
browserClient.init()
.url("http://somewhere.com")
.timeouts('pageLoad', 100000)
.selectByVisibleText("#store","Tech")
The following error is thrown.
"Unknown wait type: pageLoad\nBuild info: version: '3.4.0', revision:
'unknown', time: 'unknown'\nSystem info: host: 'ef7581676ebb', ip:
'172.17.0.3', os.name: 'Linux', os.arch: 'amd64', os.version:
'4.9.27-moby', java.version: '1.8.0_121'\nDriver info: driver.version:
unknown
I have been trying to solve this problem for two days, but no luck so far.
Could someone help, maybe you have some ideas what can cause the problem ?
Thanks.
UPDATE
.url("http://somewhere.com")
.pause(2000)
.selectByVisibleText("#store","Tech")
If I put some pause statements it works, but this is really bad and not what I expect from this framework. Chrome works perfectly. It waits until loading bar is fully loaded and only then performs actions.
The problem is in geckodriver I guess, I have tested it the same flow in Python, Java and the behavior is exactly the same.

I am experiencing the exact behavior you detailed above, all-green/passing test cases in Chrome, but on Firefox, a different story.
First off, never use timeouts, or pause in your test cases unless you are debugging. In which case, chaining a .debug() previously to your failing step/command will actually do more good.
I wrapped all my WDIO commands in waitUntill() and afterwards, I saw green in Firefox as well. See your code bellow:
storeSearch(info) {
let that = this;
return new Promise(function (resolve, reject) {
browserClient.init()
.url("http://somewhere.com")
.waitUntil(function() {
return browser
.isExisting("#store");
}, yourTimeout, "Your custom error msg for this step")
.selectByVisibleText("#store","Tech")
// Redirect to a new page
.waitUntil(function() {
return browser
.setValue("input[name='search']", info.searchCriteria);
}, yourTimeout, "Your custom error msg for this step")
.waitUntil(function() {
return browser
.selectByValue(".featured", 'MacBook');
}, yourTimeout, "Your custom error msg for this step")
.waitUntil(function() {
return browser
.click("button[name='info']");
}, yourTimeout, "Your custom error msg for this step")
.waitUntil(function() {
return browser
.isExisting(".popup");
}, yourTimeout, "Your custom error msg for this step")
.element('.popup')
.then(function (element) {
assert.equal(element.state,'success');
});
});
}
It's not pretty, but it did the job for me. Hopefully for you as well.
Enhancement: If you plan on actually building & maintaining a strong automation harness using WDIO, then you should consider creating custom commands that package the waiting & make your test cases more readable. See bellow an example for .click():
commands.js:
module.exports = (function() {
browser.addCommand('cwClick', function(element) {
return browser
.waitUntil(function() {
return browser.isExisting(element);
}, timeout, "Oups! An error occured.\nReason: element(" + element + ") does not exist")
.waitUntil(function() {
return browser.isVisible(element);
}, timeout, "Oups! An error occured.\nReason: element(" + element + ") is not visible")
.waitUntil(function() {
return browser.click(element);
}, timeout, "Oups! An error occured.\nReason: element(" + element + ") could not be clicked")
});
})();
All that's left to do, is import your module via require() in your test case file: var commands = require('./<pathToCommandsFile>/commands.js');

You can use code in javascript which will be waiting for state of website.
In C# looks like this:
public void WaitForPage(IWebDriver driver, int timeout = 30)
{
IWait<IWebDriver> wait = new WebDriverWait(driver, TimeSpan.FromSeconds(timeout));
wait.Until(driver1 => ((IJavaScriptExecutor)driver).ExecuteScript("return document.readyState").Equals("complete"));
}

I've run into quite a few issues like this with Selenium in Python and C#, unfortunately both in the Chrome and Firefox webdrivers. The problem seems to be that the code gets ahead of itself and tries to reference elements before they even exist/are visible on the page. The solution I found in Python at least was to use the Wait functions like this: http://selenium-python.readthedocs.io/waits.html
If there isn't an equivalent in node, you might have to code a custom method to check the source every so often for the presence of the element in x intervals over time.

Related

Chrome-Extension : restart automatically extension through code

I'm using Chrome Extension to scrape web sites and once in a while the Chrome Extension crashes. To reload the crashed extension, I have created another extension that reloads the crashed extension every X minutes.
It all worked fine until recently. The extension that I was using to restart the scraper did not restart the extension anymore while I haven't changed anything to it.
Here is a sample for the alarm:
chrome.alarms.get(alarmName, function(Alarm)
{
if (typeof Alarm === 'undefined')
{
console.log('is not defined');
chrome.alarms.create(alarmName,
{
periodInMinutes: 1
});
}
else
{
console.log('is defined');
}
});
and then the code to restart the extension:
chrome.alarms.onAlarm.addListener(async function( alarm )
{
var child = null;
chrome.management.getAll(function (info) {
for (var i=0; i < info.length; i++) {
if (info[i].name == 'MyScraper') {
child = info[i];
console.log(child);
chrome.management.setEnabled(child.id, true, function ()
{
console.log('Enabled '+Date.now());
});
break;
}
}
});
});
The code finds the extension correctly and display the child object, the content of this object is ok but once I try to reload it with setEnabled, the console outputs "Enable + time" but the scraping extension is not restarted. I don't get any error message.
The extension is running on a Raspberry Pi 4 with Chromium, it used to run on Windows before but I don't see why the code would not work anymore.
Any idea?
Thanks!
As I couldn't find a way to solve this through a Chrome Extension, I have installed xdotool and created a script that clicks on the reload button every 10 minutes. It's not super clean but at least it solves my issue.
Thanks

chrome.tabs.executeScript is unreliable

I'm trying to inject content script dynamically so I'm using the documented chrome.tabs.executeScript method for this.
However, unlike embedded content scripts (defined in the manifest) the dynamic script runs on a random basis while I need to be sure it runs everytime.
Basically I listen for tab update events in background script and execute dynamic content script on every "loading" event
What I've noticed is that the behavior seems to be connected with page/script loading timing - if page loading completes before the script execution, the script won't run, otherwise it seems to work as expected.
Though this is just a guess based on the observation and if you have any other ides of what is going on here feel free to share your thoughts.
Is there any ways to ensure dynamic script executes 100% of the time despite of any circumstances?
Here is the log sequence in which script doesn't run:
loading: changeInfo = {status: "loading"}
loading: start. sender.tab.id = 1454
loading: start script loading
loading: code size = 4190306 bytes
loading: changeInfo = {status: "complete"} <- page loading completed
loading: time to execute script = 945 <- script execution completed
And here is the code snippet:
chrome.tabs.onUpdated.addListener(function (tabId, changeInfo, tab) {
console.log("loading: changeInfo = ", changeInfo)
if (changeInfo.status !== "loading") {
console.log("loading: skip");
return;
}
console.log("loading: start. sender.tab.id = ", tabId)
console.log("loading: start script loading")
var timerStart = Date.now();
console.log("loading: code size = ", byteCount(scriptCode), " bytes")
chrome.tabs.executeScript(tabId,
{
code: scriptCode,
runAt: "document_idle"
}, function (response) {
console.log("loading: time to execute script = ", Date.now() - timerStart)
var err = chrome.runtime.lastError;
console.log("loading: response = ", response, ", err = ", err)
});
});
There are also no errors in the log
One more thing to add - if I wrap chrome.tabs.executeScript in setTimeout with something like 2000ms delay this guarantees script will never run which suggests timing issue.
Ok, that was dumb - the problem appeared to be in the dynamic script code itself: the main logic was executed in the onload callback and was never run if the script loads after the onload:
window.addEventListener("load", mainLogic, false);
removing the callback and running mainLogic() directly fixed the issue.
I'll leave it here in case someone did the same mistake as I did.

handling out of memory in chromium

I am running a web-app on a Raspberry Pi in chromium that should be running 24/7. The main issue is that it runs out of memory and displays "He's dead Jim". I am wondering if someone can help me to:
Direct me to a chromium extension that will reload/ reboot the browser if memory runs out
Supply a possible cron job to detect when memory is running out and reboot the browser if that's the case
The aim is to keep chromium running everyday without human intervention. So any additional methods/ideas would be appreciated.
Thanks in advance!
I actually found the culprit to be a few ajax request that each ran every few seconds to check if the server is still up or not(it's a long story but has to be done).
Then I found a small memory-saving solution online: to put all the ajax requests in a variable and then clear the variable after use(I also cleared unused java-script variables application-wide). Here is an example below:
function getData(){
var request = $.ajax({
url : "/someurl",
type : "HEAD",
dataType : "json",
success : function(data) {
//use your data
}
error: function(){
//doSomething
},
cache : false
});
//HERE IS THE HACK! :)
data = null;
request.onreadystatechange = null;
request.abort = null;
request = null;
}
setTimeout(function(){
getData();
}, 0.05 * 60 * 1000)
}
P.S I found the code online.

Selenium Webdriver JS Scraping Parallely [nodejs]

I'm trying to create a pool of Phantom Webdrivers [using webdriverjs] like
var driver = new Webdriver.Builder().withCapabilities(Webdriver.Capabilities.phantomjs()).build();
Once the pool gets populated [I see n-number of phantom processes spawned], I try to do driver.get [using different drivers in the pool] of different urls expecting them to work parallely [as driver.get is async].
But I always see them being done sequentially.
Can't we load different urls parallely via different web driver instances?
If not possible in this way how else could I solve this issue?
Very Basic Impl of my question would look like below
var Webdriver = require('selenium-webdriver'),
function getInstance() {
return new Webdriver.Builder().withCapabilities(Webdriver.Capabilities.phantomjs()).build();
}
var pool = [];
for (var i = 0; i < 3; i++) {
pool.push(getInstance());
}
pool[0].get("http://mashable.com/2014/01/14/outdated-web-features/").then(function () {
console.log(0);
});
pool[1].get("http://google.com").then(function () {
console.log(1);
});
pool[2].get("http://techcrunch.com").then(function () {
console.log(2);
});
PS: Have already posted it here
Update:
I tried with selenium grid with the following setup; as it was mentioned that it can run tests parallely
Hub:
java -jar selenium/selenium-server-standale-2.39.0.jar -hosost 127.0.0.1 -port 4444 -role hub -nodeTimeout 600
Phantom:
phantomjs --webdriver=7777 --webdriver-selium-grid-hub=http://127.0.0.1:4444 --debug=true
phantomjs --webdriver=7877 --webdriver-selium-grid-hub=http://127.0.0.1:4444 --debug=true
phantomjs --webdriver=6777 --webdriver-selium-grid-hub=http://127.0.0.1:4444 --debug=true
Still I see the get command getting queued and executed sequentially instead being parall. [But gets properly distributed across 3 instances]
Am I still missing something out?
Why is it mentioned "scale by distributing tests on several machines ( parallel execution )" in the doc?
What is parallel as per the hub? I'm getting clueless
I guess I got the issue..
Basically https://code.google.com/p/selenium/source/browse/javascript/node/selenium-webdriver/executors.js#39 Is synchronous and blocking operation [atleast the get].
Whenever the get command is issued node's main thread get's stuck there. No further code execution.
A little late but for me it worked with webdriver.promise.createFlow.
You just have to wrap your code in webdriver.promise.createFlow() { ... }); and it works for me! Here's an example from Make parallel requests to a Selenium Webdriver grid. All thanks to the answerer there...
var flows = [0,1,2,3].map(function(index) {
return webdriver.promise.createFlow(function() {
var driver = new webdriver.Builder().forBrowser('firefox').usingServer('http://someurl:44111/wd/hub/').build();
console.log('Get');
driver.get('http://www.somepage.com').then(function() {
console.log('Screenshot');
driver.takeScreenshot().then(function(data){
console.log('foo/test' + index + '.png');
//var decodedImage = new Buffer(data, 'base64')
driver.quit();
});
});
});
});
I had the same issues, I finally got around the problem using child_process.
The way my app is setup is that I have many tasks that does different things, and that needs to run simultaneously (each of those use a different driver instance), obviously it was not working.
I now start those tasks in a child_process (which will run a new V8 process) and it does run everything in parallel.

Chrome extension delay condition

I have created a chrome extension which does something after its button is clicked.
However I dont want it be abused so I need the its code to be executed after some time.
How can I surround this code with a timeout in order to achieve this?
Thank you for reading me!
chrome.tabs.getSelected(null,function(tab) {
var Mp=tab.url.substring(0,23);
if(Mp=='https://www.example.com')
{
onWindowLoad();
chrome.extension.onMessage.addListener(function(request, sender) {
if (request.action == "getSource")
{
...Working here
}
});
}
else
{
message.innerHTML='<span style="color: #f00">This is not a valid page</span>';
}
});
function onWindowLoad()
{
var message = document.querySelector('#message');
chrome.tabs.executeScript(null, {file: "getPagesSource.js"});
}
I had to make a compromise so just after the getSelected I added the following line:
chrome.browserAction.disable(tab.Id);
It disables the action button thus it cant be clicked while the script sends the data to the server, as with this extension I grab the tab url in order to store it in my database.
After the ajax response and adding the following
if(xhr.readyState==4)
{
message.innerHTML=xhr.responseText;
chrome.browserAction.enable(tab.Id); /////<---THAT LINE
}
the button gets ready again to be clicked.
I couldnt find the way to add a specific delay in seconds, this way seems stupid but its working, as the response's delay from my server is enough for switching the 2 states of the action button.
With this, however another problem came up, which Ill write in different question.

Resources