I have the following code to run tests in various browsers. Chrome of course works correctly on the machine I am wanting to run these tests on, however, Firefox, IE, and Edge do not. Is this the right way to go about this? I would prefer not to have to have a file I have to download and change every couple of months when the browsers update.
def __init__(self, browser: str = TESTING_BROWSER, home: str = BASE_URL):
"""Hooray for inits."""
if browser.lower() == "ie":
webdriver.Ie.__init__(self, IEDriverManager().install())
elif browser.lower() == 'edge':
webdriver.Edge().__init__(self, EdgeDriverManager().install())
elif browser.lower() == "firefox":
webdriver.Firefox.__init__(self, GeckoDriverManager().install())
else:
chrome_options = Options()
if os.environ.get('RUN_HEADLESS') == 'True':
chrome_options.add_argument('--headless')
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-gpu")
webdriver.Chrome.__init__(self, ChromeDriverManager().install(), chrome_options=chrome_options)
else:
webdriver.Chrome.__init__(self, ChromeDriverManager().install())
As a side note when I try and run Firefox, I am getting the following error:
NotADirectoryError: [Errno 20] Not a directory: '/Users/me/.wdm/geckodriver/v0.24.0/macos/geckodriver'
I have tried adding executable_path=path to geckodriver and that is not working either.
So RTD wins again:
webdriver.Firefox.__init__(self, executable_path=GeckoDriverManager().install())
For those interested, the documentation to webdriver_manager is here
Related
I am using python selenium (last version) with geckodriver 0.31.0 and Firefox 103 to login a website and download a file, but after downloading the file, browser is stuck and browser.quit() is never invoked.
This is the relevant code:
s = Service(DRIVER_PATH)
firefox_options = Options()
firefox_options.set_preference("browser.download.folderList", 2) # to not use the default directory for downloading the file
firefox_options.set_preference("browser.download.manager.showWhenStarting", False) # turns off the showing of download progress
firefox_options.set_preference("browser.download.dir", "/home/<myuser>/")
firefox_options.set_preference("browser.download.directory_upgrade", True)
firefox_options.set_preference("browser.download.prompt_for_download", False)
firefox_options.set_preference("browser.download.manager.showWhenStarting", False)
firefox_options.set_preference("browser.download.manager.alertOnEXEOpen", False)
firefox_options.set_preference("browser.download.manager.focusWhenStarting", False)
firefox_options.set_preference("browser.helperApps.alwaysAsk.force", False)
firefox_options.set_preference("browser.download.manager.alertOnEXEOpen", False)
firefox_options.set_preference("browser.download.manager.closeWhenDone", True)
firefox_options.set_preference("browser.download.manager.showAlertOnComplete", False)
firefox_options.set_preference("browser.download.manager.useWindow", False)
firefox_options.set_preference("services.sync.prefs.sync.browser.download.manager.showWhenStarting", False)
firefox_options.set_preference("pdfjs.disabled", True)
firefox_options.add_argument("--disable-infobars")
firefox_options.add_argument("--disable-extensions")
firefox_options.set_preference("network.proxy.socks_remote_dns", True)
browser = webdriver.Firefox(service=s, options=firefox_options)
browser.get(URL)
browser.find_element(By.NAME, "login").send_keys(USER)
browser.find_element(By.NAME, "password").send_keys(PASSWORD)
browser.find_element(By.CLASS_NAME, CLASS).click()
# download file
browser.get(URL + "feed.rss")
time.sleep(3)
browser.quit()
I know I can download the file with python requests by passing selenium cookies but I need to download the file with Selenium.
Firefox 102.3 shows the download window with this profile (at least in java). I needed to add the following config line:
"browser.download.alwaysOpenPanel" = false
I am not sure if this helps with python or if this is the problem. In selenium-java, I loose the focus with certain code if there is a popup window ("Show all downloads").
I'm using Python 3.9 on macOS. I'm trying to start using webbot, but every time I try, I get this error:
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
exception: Missing or invalid capabilities
(Driver info: chromedriver=2.39.562713
(dd642283e958a93ebf6891600db055f1f1b4f3b2),platform=Mac OS X 10.14.6 x86_64)
I'm using macOS version 10.4 because I use 32 bit software. The part that really puzzles me is why is says chromedriver=2.39.562713. According to the pip, the driver's version is 103.0.5060.53. If I import selenium and try the command help(selenium), towards the end of the output, I get:
VERSION
4.3.0
Where does this lower version come from? I'm pretty sure that's why I have "missing or invalid capabilities." If I start selenium with:
from selenium import webdriver
driver = webdriver.Chrome()
It starts Chrome as expected. Obviously I'm missing something.
I used to start webbot with:
from webbot import Browser
driver = Browser()
But then, just to be sure, I changed it to:
from webbot import Browser
driver = Browser(True, None, '/usr/local/bin/')
'/usr/local/bin/' being the location of a chrome webdriver installed by brew that is explicitly version 103. No difference.
Solution
The approved response was not the solution, but it led me to the solution.
My version of webbot is the latest, but it has a very different __init__ method:
def __init__(self, showWindow=True, proxy=None , downloadPath:str=None):
Upon further inspection, I saw that the driverPath attribute (that I had tried to use earlier) was completely gone by design. So I decided to print the value of the inner variable driverpath inside the __init__ method. This returned the following:
project_root/virtualenvironment/lib/python3.9/site-packages/webbot/drivers/chrome_mac
There was my guilty party! I renamed that executable and in its place put a symbolic link to the correct binary. That worked.
driver = Browser(True, None, '/usr/local/bin/')
actually sets the downloadPath, not the driverPath. Use the parameter name explicitly
driver = Browser(driverPath='/usr/local/bin/')
From webbot.py
class Browser:
def __init__(self, showWindow=True, proxy=None , downloadPath:str=None, driverPath:str=None, arguments=["--disable-dev-shm-usage","--no-sandbox"]):
if driverPath is not None and isinstance(driverPath,str):
driverPath = os.path.abspath(driverPath)
if(not os.path.isdir(driverPath)):
raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), driverPath)
if driverPath is None:
driverfilename = ''
if sys.platform == 'linux' or sys.platform == 'linux2':
driverfilename = 'chrome_linux'
elif sys.platform == 'win32':
driverfilename = 'chrome_windows.exe'
elif sys.platform == 'darwin':
driverfilename = 'chrome_mac'
driverPath = os.path.join(os.path.split(__file__)[0], 'drivers{0}{1}'.format(os.path.sep, driverfilename))
self.driver = webdriver.Chrome(executable_path=driverPath, options=options)
If driverPath is None it will set to
/{parent_folder_abs_path}/drivers/chrome_mac or /{parent_folder_abs_path}/drivers/, I'm guessing you have an older chromedriver version there.
I'm using selenium and python via chromewebdriver (windows) in order to automate a task of downloading large amount of files from different pages.
My code works, but the solution is far from ideal: the function below clicks on the website button that initiating a java script function that generating a PDF file and then downloading it.
I had to use a static wait in order to wait for the download to be completed (ugly) I cannot check the file system in order to verify when the download is completed since i'm using multi threading (downloading lot's of files from different pages at once) and also the the name of the files is generated dynamically in the website itself.
My code:
def file_download(num, drivervar):
Counter += 1
try:
drivervar.get(url[num])
download_button = WebDriverWait(drivervar, 20).until(EC.element_to_be_clickable((By.ID, 'download button ID')))
download_button.click()
time.sleep(10)
except TimeoutException: # Retry once
print('Timeout in thread number: ' + str(num) + ', retrying...')
.....
Is it possible to determine download completion in webdriver? I want to avoid using time.sleep(x).
Thanks a lot.
You can get the status of each download by visiting chrome://downloads/ with the driver.
To wait for all the downloads to finish and to list all the paths:
def every_downloads_chrome(driver):
if not driver.current_url.startswith("chrome://downloads"):
driver.get("chrome://downloads/")
return driver.execute_script("""
var items = document.querySelector('downloads-manager')
.shadowRoot.getElementById('downloadsList').items;
if (items.every(e => e.state === "COMPLETE"))
return items.map(e => e.fileUrl || e.file_url);
""")
# waits for all the files to be completed and returns the paths
paths = WebDriverWait(driver, 120, 1).until(every_downloads_chrome)
print(paths)
Was updated to support changes till version 81.
I have had the same problem and found a solution. You can check weither or not a .crdownload is in your download folder. If there are 0 instances of a file with .crdownload extension in the download folder then all your downloads are completed. This only works for chrome and chromium i think.
def downloads_done():
while True:
for filename in os.listdir("/downloads"):
if ".crdownload" in i:
time.sleep(0.5)
downloads_done()
Whenever you call downloads_done() it will loop itself untill all downloads are completed. If you are downloading massive files like 80 gigabytes then i don't recommend this because then the function can reach maximum recursion depth.
2020 edit:
def wait_for_downloads():
print("Waiting for downloads", end="")
while any([filename.endswith(".crdownload") for filename in
os.listdir("/downloads")]):
time.sleep(2)
print(".", end="")
print("done!")
The "end" keyword argument in print() usually holds a newline but we replace it.
While there are no filenames in the /downloads folder that end with .crdownload
sleep for 2 seconds and print one dot without newline to console
I don't really recommend using selenium anymore after finding out about requests but if it's a very heavily guarded site with cloudflare and captchas etc then you might have to resort to selenium.
With Chrome 80, I had to change the answer from #florent-b by the code below:
def every_downloads_chrome(driver):
if not driver.current_url.startswith("chrome://downloads"):
driver.get("chrome://downloads/")
return driver.execute_script("""
return document.querySelector('downloads-manager')
.shadowRoot.querySelector('#downloadsList')
.items.filter(e => e.state === 'COMPLETE')
.map(e => e.filePath || e.file_path || e.fileUrl || e.file_url);
""")
I believe this is retro-compatible, I mean this shall be working with older versions of Chrome.
There are issues with opening chrome://downloads/ when running Chrome in headless mode.
The following function uses a composite approach that works whether the mode is headless or not, choosing the better approach available in each mode.
It assumes that the caller clears all files downloaded at file_download_path after each call to this function.
import os
import logging
from selenium.webdriver.support.ui import WebDriverWait
def wait_for_downloads(driver, file_download_path, headless=False, num_files=1):
max_delay = 60
interval_delay = 0.5
if headless:
total_delay = 0
done = False
while not done and total_delay < max_delay:
files = os.listdir(file_download_path)
# Remove system files if present: Mac adds the .DS_Store file
if '.DS_Store' in files:
files.remove('.DS_Store')
if len(files) == num_files and not [f for f in files if f.endswith('.crdownload')]:
done = True
else:
total_delay += interval_delay
time.sleep(interval_delay)
if not done:
logging.error("File(s) couldn't be downloaded")
else:
def all_downloads_completed(driver, num_files):
return driver.execute_script("""
var items = document.querySelector('downloads-manager').shadowRoot.querySelector('#downloadsList').items;
var i;
var done = false;
var count = 0;
for (i = 0; i < items.length; i++) {
if (items[i].state === 'COMPLETE') {count++;}
}
if (count === %d) {done = true;}
return done;
""" % (num_files))
driver.execute_script("window.open();")
driver.switch_to_window(driver.window_handles[1])
driver.get('chrome://downloads/')
# Wait for downloads to complete
WebDriverWait(driver, max_delay, interval_delay).until(lambda d: all_downloads_completed(d, num_files))
# Clear all downloads from chrome://downloads/
driver.execute_script("""
document.querySelector('downloads-manager').shadowRoot
.querySelector('#toolbar').shadowRoot
.querySelector('#moreActionsMenu')
.querySelector('button.clear-all').click()
""")
driver.close()
driver.switch_to_window(driver.window_handles[0])
import os
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
class MySeleniumTests(unittest.TestCase):
selenium = None
#classmethod
def setUpClass(cls):
cls.selenium = webdriver.Firefox(...)
...
def test_download(self):
os.chdir(self.download_path) # default download directory
# click the button
self.selenium.get(...)
self.selenium.find_element_by_xpath(...).click()
# waiting server for finishing inner task
def download_begin(driver):
if len(os.listdir()) == 0:
time.sleep(0.5)
return False
else:
return True
WebDriverWait(self.selenium, 120).until(download_begin) # the max wating time is 120s
# waiting server for finishing sending.
# if size of directory is changing,wait
def download_complete(driver):
sum_before=-1
sum_after=sum([os.stat(file).st_size for file in os.listdir()])
while sum_before != sum_after:
time.sleep(0.2)
sum_before = sum_after
sum_after = sum([os.stat(file).st_size for file in os.listdir()])
return True
WebDriverWait(self.selenium, 120).until(download_complete) # the max wating time is 120s
You must do these thing
Wait for server to finish inner business( for example, query from database).
Wait for server to finish sending the files.
(my English is not very well)
To obtain the return of more than one item, I had to change the answer of #thdox by the code below:
def every_downloads_chrome(driver):
if not driver.current_url.startswith("chrome://downloads"):
driver.get("chrome://downloads/")
return driver.execute_script("""
var elements = document.querySelector('downloads-manager')
.shadowRoot.querySelector('#downloadsList')
.items
if (elements.every(e => e.state === 'COMPLETE'))
return elements.map(e => e.filePath || e.file_path || e.fileUrl || e.file_url);
""")
This may not work for all usecases but for my simple need to wait for one pdf to download it works great. Based off of Walter's comment above.
def get_non_temp_len(download_dir):
non_temp_files = [i for i in os.listdir(download_dir) if not (i.endswith('.tmp') or i.endswith('.crdownload'))]
return len(non_temp_files)
download_dir = 'your/download/dir'
original_count = get_non_temp_len(download_dir) # get the file count at the start
# do your selenium stuff
while original_count == get_non_temp_len(download_dir):
time.sleep(.5) # wait for file count to change
driver.quit()
I had the same problem and this method worked for me.
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import ElementClickInterceptedException
from threading import Thread
import os
import datetime
def checkFilePresence(downloadPath, numberOfFilesInitially, artistName,
songTitle):
timeNow = datetime.datetime.now()
found = False
while not found:
numberOfFilesNow = len(os.listdir(downloadPath))
if numberOfFilesNow > numberOfFilesInitially:
for folders, subfolders, files in os.walk(downloadPath):
for file in files:
modificationTime = datetime.datetime.fromtimestamp\
(os.path.getctime(os.path.join(folders, file)))
if modificationTime > timeNow:
if file.endswith('.mp3'):
return
This code work in headless mode and return downloaded file name (based on
#protonum code):
def wait_for_downloads(download_path):
max_delay = 30
interval_delay = 0.5
total_delay = 0
file = ''
done = False
while not done and total_delay < max_delay:
files = [f for f in os.listdir(download_path) if f.endswith('.crdownload')]
if not files and len(file) > 1:
done = True
if files:
file = files[0]
time.sleep(interval_delay)
total_delay += interval_delay
if not done:
logging.error("File(s) couldn't be downloaded")
return download_path + '/' + file.replace(".crdownload", "")
def wait_for_download_to_be_don(self, path_to_folder, file_name):
max_time = 60
counter = 0
while not os.path.exists(path_to_folder + file_name) and time_counter < max_time:
sleep(0.5)
time_counter += 0.5
if time_counter == max_time:
assert os.path.exists(path_to_folder + file_name), "The file wasn't downloaded"
When using test automation, its crucial that developers make the software testable. It is your job to check the software combined with the testability, meaning that you need to request a spinner or a simple HTML tag which indicates when the download is done successfully.
In a case as yours, where you cannot check it in the UI and you cannot check in system, this is the best way to solve it.
Running python 3 with cherrypy 3.2, and have been having a host of problems. First of all, to get cookies to work, i had to fake a fqdn in /etc/hosts.
e.g.
http://test:8080 [no cookies]
http://test.local:8080 [cookies work]
After this, I tried to get sessions to work, but I am getting a new session id each time, and no session_id value is being set in a cookie anywhere in the browser.
class HelloWorld:
#cherrypy.expose
def index(self, *args):
print("\n\n")
### test cookies (works fine, no problems)
print(cherrypy.request.cookie)
cherrypy.response.cookie['c1'] = 'val1'
cherrypy.response.cookie['c1']['max-age'] = '3600'
cherrypy.response.cookie['d1'] = 'val2'
cherrypy.response.cookie['d1']['max-age'] = '3600'
### test sessions (doesn't work)
print(cherrypy.session.load()) # always returns None
print(cherrypy.session.id) # different every refresh
print(cherrypy.session.get('foo')) # always returns None
cherrypy.session['foo'] = 'bar'
cherrypy.session.save() # apparently has no effect
return "Hello world!"
Can anyone offer some advice or suggestions? I see that no no cookie with the session id is being set in chrome, even though my other values are.
My config looks like:
'/': {'tools.sessions.on': True,
'tools.sessions.timeout': 7200}}
Any ideas?
I was facing the same problem. I added tools.sessions.name to the cherrypy config and now it works
I'm in the process of switching my Watir / FireWatir scripts over to use watir-webdriver and need a means in watir-webdriver to determine which type of browser the test is currently being executed against, (IE, FF, Chrome).
With Watir / FireWatir looking at the class of the browser would return either "Watir::IE" or "FireWatir:Firefox". Using that the code could be branched to execute browser specific code.
In watir-webdriver, the class of the browser is always "Watir::Browser", it doesn't vary when running IE, Firefox, or Chrome.
Does anyone know of a way in Ruby with watir-web-driver to identify the browser's type (i.e. IE, Firefox, Chrome)?
For example: With Watir / Firewatir define methods:
def is_ie?()
return self.class.to_s == "Watir::IE"
end
def is_firefox?()
return self.class.to_s == "FireWatir::Firefox"
end
Then invoke them like this...
if(browser.is_ie?)
# run the IE specific code
end
if(browser.is_firefox?)
# run the firefox specific code
end
Thanks in advance,
Joe
Try
browser.driver.browser #=> :firefox
Thanks that is just what I needed!
As I'm in a transition with some scripts ported over to Watir-WebDriver and some still needing to run under Watir / Firewatir I've updated mt method as follows, posting them in case someone else is in the same situation.
def is_chrome?()
if(is_webdriver? == true)
return (self.driver.browser.to_s.downcase == "chrome")
else
return (self.class.to_s == "ChromeWatir::Browser")
end
end
def is_firefox?()
if(is_webdriver? == true)
return (self.driver.browser.to_s.downcase == "firefox")
else
return (self.class.to_s == "FireWatir::Firefox")
end
end
def is_ie?()
if(is_webdriver? == true)
return (self.driver.browser.to_s.downcase == "internet_explorer")
else
return (self.class.to_s == "Watir::IE")
end
end
def is_webdriver?()
if($LOADED_FEATURES.to_s =~/watir-webdriver/)
return true
else
return false
end
end