Why does opt.add_argument('--user-data-dir='+r'path') work but opt.add_argument('--user-data-dir='+fr'"{path}"') doesn't as an option to Selenium? - python-3.x

I have realized of something very weird when trying to deploy a chrome driver using --user-data-dir and --profile-directory from the user on Python 3.9.7, see below:
If you compile the following code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
opt = Options() #the variable that will store the selenium options
opt.add_argument('--user-data-dir='+r'C:\Users\ResetStoreX\AppData\Local\Google\Chrome\User Data') #Add the user data path as an argument in selenium Options
opt.add_argument('--profile-directory=Default') #Add the profile directory as an argument in selenium Options
s = Service('C:/Users/ResetStoreX/AppData/Local/Programs/Python/Python39/Scripts/chromedriver.exe')
driver = webdriver.Chrome(service=s, options=opt)
driver.get('https://opensea.io/login?referrer=%2Faccount')
You get successfully a chrome driver instance using the corresponding --user-data-dir and --profile-directory:
Now, after killing all chrome driver instances using the following code on cmd:
taskkill /F /IM chromedriver.exe
And then compiling this other code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
opt = Options() #the variable that will store the selenium options
path = input('Introduce YOUR profile path:')
opt.add_argument('--user-data-dir='+fr'"{path}"') #Add the user data path as an argument in selenium Options
opt.add_argument('--profile-directory=Default') #Add the profile directory as an argument in selenium Options
s = Service('C:/Users/ResetStoreX/AppData/Local/Programs/Python/Python39/Scripts/chromedriver.exe')
driver = webdriver.Chrome(service=s, options=opt)
driver.get('https://opensea.io/login?referrer=%2Faccount')
For finally typing: C:\Users\ResetStoreX\AppData\Local\Google\Chrome\User Data as input
You get this error:
WebDriverException: unknown error: Could not remove old devtools port
file. Perhaps the given user-data-dir at
"C:\Users\ResetStoreX\AppData\Local\Google\Chrome\User Data"
is still attached to a running Chrome or Chromium process
Why does that happen?
Isn't opt.add_argument('--user-data-dir='+fr'"{path}"') a valid way of passing this user data path:
path = C:\Users\ResetStoreX\AppData\Local\Google\Chrome\User Data ?

I figured it out, I was creating a syntax error with opt.add_argument('--user-data-dir='+fr'"{path}"'), so I changed it for opt.add_argument('--user-data-dir='+fr'{path}'), the improved code would be the following:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
opt = Options() #the variable that will store the selenium options
path = input('Introduce YOUR profile path:')
opt.add_argument('--user-data-dir='+fr'{path}') #Add the user data path as an argument in selenium Options
opt.add_argument('--profile-directory=Default') #Add the profile directory as an argument in selenium Options
s = Service('C:/Users/ResetStoreX/AppData/Local/Programs/Python/Python39/Scripts/chromedriver.exe')
driver = webdriver.Chrome(service=s, options=opt)
driver.get('https://opensea.io/login?referrer=%2Faccount')
After compiling this code, the program will run without throwing any errors and get the same result as the first code shown in this post.

Related

selenium.common.exceptions.WebDriverException: Message: Service /usr/bin/geckodriver unexpectedly exited. Status code was: 64

I am trying to run a selenium test but getting the above error. I think this is a compatibility issue but I am not sure as I have very basic knowledge of selenium. My geckodriver version is 0.31.0 and firefox browser version is 100 and selenium version is selenium == 4.1.0. Any idea how I can fix this?
This is my conftest.py
import pytest
from selenium import webdriver
from selenium.webdriver.firefox.service import Service as ff_service
options = webdriver.FirefoxOptions()
options.add_argument("--headless")
options.add_argument("--disable-gpu")
driver_path = ff_service(r"/usr/bin/geckodriver")
driver = webdriver.Firefox(service=driver_path,options = options) #getting error at this line
driver.maximize_window()

save document.cookie output in a file

skillshare-downloader says:
grab your cookie by typing:
document.cookie
Then it says:
Copy-paste cookie from developer console (without " if present) into example script.
Example:
from downloader import Downloader
cookie = """
ADD YOUR COOKIE HERE
"""
It adds an extra step.
Is there any way we can save document.cookie output to a file so that we can just read the cookie from the file instead of going to the console and type document.cookie and copy-paste the output?
I checked How to write console.log to a file instead. I also checked Python open browser and run javascript function. It suggests using Selenium or webbroser module. However, I am not sure how to approach this problem.
What can I do?
Assuming that you are using chrome:
Install selenium by running in a terminal pip install selenium
Install a chromedriver via a manager (It allows you to control chrome) by running in a terminal pip install webdriver-manager
Create a file example.py and paste this inside
#import the selenium webdriver and the chromedriver
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from time import sleep
#trying to stop skillshare from detecting we a are a bot
options = webdriver.ChromeOptions()
options.add_argument('--disable-blink-features=AutomationControlled')
#create the instance of chrome
driver = webdriver.Chrome(ChromeDriverManager().install(),options=options)
#get command used to open an url
driver.get('https://www.skillshare.com/')
#login part
#press on sign in
driver.find_element_by_css_selector("a.button.alt-white-ghost.transparent.initialized").click()
#change email here
driver.find_element_by_name("email").send_keys("testmail#mail.com")
#change password here
driver.find_element_by_name("password").send_keys("fakepassword")
#wait a second
sleep(1)
#click on sign in
driver.find_element_by_xpath("//span[text()='Sign In']/parent::button").click()
#wait 3 seconds for the login
sleep(3)
#execute_script is used to execute the command in the browser console, using return here to store it in a variable
cookie = driver.execute_script('return document.cookie')
#python way of creating a file on the given path and write the cookie inside it
f = open("D:\cookie.txt", "w")
f.write(cookie)
f.close()
#closing the chrome instance
driver.close()
open a terminal and type python example.py and it will run the script

--headless is not an option in Chrome WebDriver for Selenium

I would like to have Selenium run a headless instance of Google Chrome to mine data from certain websites without the UI overhead. I downloaded the ChromeDriver executable from here and copied it to my current scripting directory.
The driver appears to work fine with Selenium and is able to browse automatically, however I cannot seem to find the headless option. Most online examples of using Selenium with headless Chrome go something along the lines of:
import os
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.binary_location = '/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary'`
driver = webdriver.Chrome(executable_path=os.path.abspath(“chromedriver"), chrome_options=chrome_options)
driver.get("http://www.duo.com")`
However when I inspect the possible arguments for the Selenium WebDriver using the command chromedriver -h this is what I get:
D:\Jobs\scripts>chromedriver -h
Usage: chromedriver [OPTIONS]
Options
--port=PORT port to listen on
--adb-port=PORT adb server port
--log-path=FILE write server log to file instead of stderr, increases log level to INFO
--log-level=LEVEL set log level: ALL, DEBUG, INFO, WARNING, SEVERE, OFF
--verbose log verbosely (equivalent to --log-level=ALL)
--silent log nothing (equivalent to --log-level=OFF)
--append-log append log file instead of rewriting
--replayable (experimental) log verbosely and don't truncate long strings so that the log can be replayed.
--version print the version number and exit
--url-base base URL path prefix for commands, e.g. wd/url
--whitelisted-ips comma-separated whitelist of remote IP addresses which are allowed to connect to ChromeDriver
No --headless option is available.
Does the ChromeDriver obtained from the link above allow for headless browsing?
--headless is not argument for chromedriver but for Chrome. --headless Run chrome in headless mode, i.e., without a UI or display server dependencies. ChromeDriver is a separate executable that WebDriver uses to control Chrome and Webdriver is a a collection of language specific bindings to drive a browser.
I am able to run in headless mode with this set of options. I hope this will help:
from bs4 import BeautifulSoup, NavigableString
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
import requests
import re
options = Options()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')
browser = webdriver.Chrome(chrome_options=options) # see edit for recent code change.
browser.implicitly_wait(20)
Update 12 Aug 2019:
old : browser = webdriver.Chrome(chrome_options=options)
new : browser = webdriver.Chrome(options=options)
Try
options.headless=True
The following is how I set up my headless chrome
options = webdriver.ChromeOptions()
options.headless=True
options.add_argument('window-size=1920x1080')
prefs = {
"download.default_directory": r"C:\FilePath\Download",
"download.prompt_for_download": False,
"download.directory_upgrade": True}
options.add_experimental_option('prefs', prefs)
chromedriver = (r"C:\Filepath\chromedriver.exe")
--headless is not argument for chromedriver but Chrome, you can see more arguments or Command Line Switches for chrome here

Chrome driver not working headlessly with Python and Selenium?

I am new to Selenium programming with Python. The code works fine without the --headless argument, but does not execute at all when I try to run it headlessly. Could someone help me out with it?
Below is my sample code, I am using Python 3.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import time
import requests
from selenium.webdriver.chrome.options import Options
options = Options()
options.set_headless(headless=True)
driver = webdriver.Chrome(chrome_options=options, executable_path=r'E:/Python/[FreeTutorials.Us] Udemy - python-master-web-scraping-course-doing-20-real-projects/03 Step _ Download HTML Content/chromedriver.exe')
driver.get("https://www.tirerack.com/survey/ValidationServlet?autoYear=2006&autoMake=Porsche&autoModel=911%20Carrera%20S%20Cabriolet&newDesktop=true")
print ("Headless Chrome Initialized")
html_doc=driver.page_source
soup= BeautifulSoup(html_doc,'lxml')
print(soup)
driver.quit()
I have added headless as an arguement instead of option and it worked for me. Give a try.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path='path to your chromedriver')
driver.get('https://stackoverflow.com/')
Note: Always keep your chromedriver in homepath of python i.e. C:\Python34.

selenium - Can't work out how to get FireFox WebDriver to work

This should be easy, but I'm missing something. I'm just trying to get some selenium python tests running on firefox, which work perfectly in chrome.
The problem is just trying to get the ff webdriver up and running!
I have the following code, all the paths are correct:
import selenium
from selenium.webdriver.firefox import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary('C:\\Program Files\\Mozilla Firefox\\firefox.exe')
profile = webdriver.FirefoxProfile()
geckopath = 'C:\source\web_deploy_tests\geckodriver.exe'
browser = selenium.webdriver.Firefox(
capabilities={},
executable_path=geckopath,
firefox_profile=profile,
firefox_binary=binary
)
browser.get("http://google.com")
I'm using Python 3.6.2, selenium 3.6.0 and have v0.19.0 of geckodriver.exe and FF is v56.0.1.
When I run the above code, firefox appears but just sits there for about 30 secs then crashes with:
selenium.common.exceptions.WebDriverException: Message: Can't load the
profile. Possible firefox version mismatch. You must use GeckoDriver
instead for Firefox 48+. Profile Dir:
C:\Users\ADMINI~1\AppData\Local\Temp\3\tmpkx5dau8h If you specified a
log_file in the FirefoxBinary constructor, check it for details.
I've tried various combinations of args but I am failing.
Any ideas?
TIA
I got it working by passing in DesiredCapabilities.FIREFOX
import selenium
from selenium.webdriver import DesiredCapabilities
from selenium.webdriver.firefox import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary('C:\\Program Files\\Mozilla Firefox\\firefox.exe')
profile = webdriver.FirefoxProfile()
geckopath = 'C:\source\web_deploy_tests\geckodriver.exe'
browser = selenium.webdriver.Firefox(
capabilities=**DesiredCapabilities.FIREFOX**,
executable_path=geckopath,
firefox_profile=profile,
firefox_binary=binary
)
browser.get("http://google.com")
Now I can at least bring up the browser window!

Resources