selenium - Can't work out how to get FireFox WebDriver to work - python-3.x

This should be easy, but I'm missing something. I'm just trying to get some selenium python tests running on firefox, which work perfectly in chrome.
The problem is just trying to get the ff webdriver up and running!
I have the following code, all the paths are correct:
import selenium
from selenium.webdriver.firefox import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary('C:\\Program Files\\Mozilla Firefox\\firefox.exe')
profile = webdriver.FirefoxProfile()
geckopath = 'C:\source\web_deploy_tests\geckodriver.exe'
browser = selenium.webdriver.Firefox(
capabilities={},
executable_path=geckopath,
firefox_profile=profile,
firefox_binary=binary
)
browser.get("http://google.com")
I'm using Python 3.6.2, selenium 3.6.0 and have v0.19.0 of geckodriver.exe and FF is v56.0.1.
When I run the above code, firefox appears but just sits there for about 30 secs then crashes with:
selenium.common.exceptions.WebDriverException: Message: Can't load the
profile. Possible firefox version mismatch. You must use GeckoDriver
instead for Firefox 48+. Profile Dir:
C:\Users\ADMINI~1\AppData\Local\Temp\3\tmpkx5dau8h If you specified a
log_file in the FirefoxBinary constructor, check it for details.
I've tried various combinations of args but I am failing.
Any ideas?
TIA

I got it working by passing in DesiredCapabilities.FIREFOX
import selenium
from selenium.webdriver import DesiredCapabilities
from selenium.webdriver.firefox import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary('C:\\Program Files\\Mozilla Firefox\\firefox.exe')
profile = webdriver.FirefoxProfile()
geckopath = 'C:\source\web_deploy_tests\geckodriver.exe'
browser = selenium.webdriver.Firefox(
capabilities=**DesiredCapabilities.FIREFOX**,
executable_path=geckopath,
firefox_profile=profile,
firefox_binary=binary
)
browser.get("http://google.com")
Now I can at least bring up the browser window!

Related

selenium.common.exceptions.WebDriverException: Message: Service /usr/bin/geckodriver unexpectedly exited. Status code was: 64

I am trying to run a selenium test but getting the above error. I think this is a compatibility issue but I am not sure as I have very basic knowledge of selenium. My geckodriver version is 0.31.0 and firefox browser version is 100 and selenium version is selenium == 4.1.0. Any idea how I can fix this?
This is my conftest.py
import pytest
from selenium import webdriver
from selenium.webdriver.firefox.service import Service as ff_service
options = webdriver.FirefoxOptions()
options.add_argument("--headless")
options.add_argument("--disable-gpu")
driver_path = ff_service(r"/usr/bin/geckodriver")
driver = webdriver.Firefox(service=driver_path,options = options) #getting error at this line
driver.maximize_window()

Why does opt.add_argument('--user-data-dir='+r'path') work but opt.add_argument('--user-data-dir='+fr'"{path}"') doesn't as an option to Selenium?

I have realized of something very weird when trying to deploy a chrome driver using --user-data-dir and --profile-directory from the user on Python 3.9.7, see below:
If you compile the following code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
opt = Options() #the variable that will store the selenium options
opt.add_argument('--user-data-dir='+r'C:\Users\ResetStoreX\AppData\Local\Google\Chrome\User Data') #Add the user data path as an argument in selenium Options
opt.add_argument('--profile-directory=Default') #Add the profile directory as an argument in selenium Options
s = Service('C:/Users/ResetStoreX/AppData/Local/Programs/Python/Python39/Scripts/chromedriver.exe')
driver = webdriver.Chrome(service=s, options=opt)
driver.get('https://opensea.io/login?referrer=%2Faccount')
You get successfully a chrome driver instance using the corresponding --user-data-dir and --profile-directory:
Now, after killing all chrome driver instances using the following code on cmd:
taskkill /F /IM chromedriver.exe
And then compiling this other code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
opt = Options() #the variable that will store the selenium options
path = input('Introduce YOUR profile path:')
opt.add_argument('--user-data-dir='+fr'"{path}"') #Add the user data path as an argument in selenium Options
opt.add_argument('--profile-directory=Default') #Add the profile directory as an argument in selenium Options
s = Service('C:/Users/ResetStoreX/AppData/Local/Programs/Python/Python39/Scripts/chromedriver.exe')
driver = webdriver.Chrome(service=s, options=opt)
driver.get('https://opensea.io/login?referrer=%2Faccount')
For finally typing: C:\Users\ResetStoreX\AppData\Local\Google\Chrome\User Data as input
You get this error:
WebDriverException: unknown error: Could not remove old devtools port
file. Perhaps the given user-data-dir at
"C:\Users\ResetStoreX\AppData\Local\Google\Chrome\User Data"
is still attached to a running Chrome or Chromium process
Why does that happen?
Isn't opt.add_argument('--user-data-dir='+fr'"{path}"') a valid way of passing this user data path:
path = C:\Users\ResetStoreX\AppData\Local\Google\Chrome\User Data ?
I figured it out, I was creating a syntax error with opt.add_argument('--user-data-dir='+fr'"{path}"'), so I changed it for opt.add_argument('--user-data-dir='+fr'{path}'), the improved code would be the following:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
opt = Options() #the variable that will store the selenium options
path = input('Introduce YOUR profile path:')
opt.add_argument('--user-data-dir='+fr'{path}') #Add the user data path as an argument in selenium Options
opt.add_argument('--profile-directory=Default') #Add the profile directory as an argument in selenium Options
s = Service('C:/Users/ResetStoreX/AppData/Local/Programs/Python/Python39/Scripts/chromedriver.exe')
driver = webdriver.Chrome(service=s, options=opt)
driver.get('https://opensea.io/login?referrer=%2Faccount')
After compiling this code, the program will run without throwing any errors and get the same result as the first code shown in this post.

Can't open TOR with Selenium-Python [duplicate]

This question already has answers here:
How to connect to Tor browser using Python
(3 answers)
How to use Tor with Chrome browser through Selenium
(2 answers)
Closed 3 years ago.
Just as the question says, I have been unable to use TOR browser with Selenium on Python. I have read this and this answer, and created a MWE for each of them. Both of them give me the error 'Tor failed to start', as shown in the first link. Here's both examples:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary(r"C:\Users\Asus\Desktop\Tor
Browser\Browser\firefox.exe")
profile = FirefoxProfile(r"C:\Users\Asus\Desktop\Tor
Browser\Browser\TorBrowser\Data\Browser\profile.default")
browser = webdriver.Firefox(profile, binary)
page='https://www.google.com/'
browser.get(page)
Otherwise, I tried this too
from selenium import webdriver
import os
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
torexe = os.popen(r'C:\Users\Asus\Desktop\Tor
Browser\Browser\TorBrowser\Tor\tor.exe')
binary = FirefoxBinary(r"C:\Users\Asus\Desktop\Tor Browser\Browser\firefox.exe")
profile = FirefoxProfile(r"C:\Users\Asus\Desktop\Tor
Browser\Browser\TorBrowser\Data\Browser\profile.default")
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
browser = webdriver.Firefox(firefox_profile = profile, firefox_binary = binary, executable_path=r'C:\Users\Asus\Desktop\ome\geckodriver.exe')
page='https://www.google.com/'
browser.get(page)
I am currently using Python 3.7.3. Geckodriver is ver. 0.24.0. Firefox used by TOR is 60ESR. Selenium version is 3.141.0.

--headless is not an option in Chrome WebDriver for Selenium

I would like to have Selenium run a headless instance of Google Chrome to mine data from certain websites without the UI overhead. I downloaded the ChromeDriver executable from here and copied it to my current scripting directory.
The driver appears to work fine with Selenium and is able to browse automatically, however I cannot seem to find the headless option. Most online examples of using Selenium with headless Chrome go something along the lines of:
import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.binary_location = '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary'`
driver = webdriver.Chrome(executable_path=os.path.abspath(“chromedriver"), chrome_options=chrome_options)
driver.get("http://www.duo.com")`
However when I inspect the possible arguments for the Selenium WebDriver using the command chromedriver -h this is what I get:
D:\Jobs\scripts>chromedriver -h
Usage: chromedriver [OPTIONS]
Options
--port=PORT port to listen on
--adb-port=PORT adb server port
--log-path=FILE write server log to file instead of stderr, increases log level to INFO
--log-level=LEVEL set log level: ALL, DEBUG, INFO, WARNING, SEVERE, OFF
--verbose log verbosely (equivalent to --log-level=ALL)
--silent log nothing (equivalent to --log-level=OFF)
--append-log append log file instead of rewriting
--replayable (experimental) log verbosely and don't truncate long strings so that the log can be replayed.
--version print the version number and exit
--url-base base URL path prefix for commands, e.g. wd/url
--whitelisted-ips comma-separated whitelist of remote IP addresses which are allowed to connect to ChromeDriver
No --headless option is available.
Does the ChromeDriver obtained from the link above allow for headless browsing?
--headless is not argument for chromedriver but for Chrome. --headless Run chrome in headless mode, i.e., without a UI or display server dependencies. ChromeDriver is a separate executable that WebDriver uses to control Chrome and Webdriver is a a collection of language specific bindings to drive a browser.
I am able to run in headless mode with this set of options. I hope this will help:
from bs4 import BeautifulSoup, NavigableString
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
import requests
import re
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')
browser = webdriver.Chrome(chrome_options=options) # see edit for recent code change.
browser.implicitly_wait(20)
Update 12 Aug 2019:
old : browser = webdriver.Chrome(chrome_options=options)
new : browser = webdriver.Chrome(options=options)
Try
options.headless=True
The following is how I set up my headless chrome
options = webdriver.ChromeOptions()
options.headless=True
options.add_argument('window-size=1920x1080')
prefs = {
"download.default_directory": r"C:\FilePath\Download",
"download.prompt_for_download": False,
"download.directory_upgrade": True}
options.add_experimental_option('prefs', prefs)
chromedriver = (r"C:\Filepath\chromedriver.exe")
--headless is not argument for chromedriver but Chrome, you can see more arguments or Command Line Switches for chrome here

Chrome driver not working headlessly with Python and Selenium?

I am new to Selenium programming with Python. The code works fine without the --headless argument, but does not execute at all when I try to run it headlessly. Could someone help me out with it?
Below is my sample code, I am using Python 3.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import time
import requests
from selenium.webdriver.chrome.options import Options
options = Options()
options.set_headless(headless=True)
driver = webdriver.Chrome(chrome_options=options, executable_path=r'E:/Python/[FreeTutorials.Us] Udemy - python-master-web-scraping-course-doing-20-real-projects/03 Step _ Download HTML Content/chromedriver.exe')
driver.get("https://www.tirerack.com/survey/ValidationServlet?autoYear=2006&autoMake=Porsche&autoModel=911%20Carrera%20S%20Cabriolet&newDesktop=true")
print ("Headless Chrome Initialized")
html_doc=driver.page_source
soup= BeautifulSoup(html_doc,'lxml')
print(soup)
driver.quit()
I have added headless as an arguement instead of option and it worked for me. Give a try.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path='path to your chromedriver')
driver.get('https://stackoverflow.com/')
Note: Always keep your chromedriver in homepath of python i.e. C:\Python34.

Resources