Selenium using Chrome browser - Timeout during HTTP basic authentication [Python] - python-3.x

As a part of automation, I am trying to login to a locally hosted website using selenium. The page serves a HTTP Basic authentication popup and I use the below code to send the credentials. However, upon using a debugger and executing the code step-wise, I deciphered that a TimeOut exception occurs repeatedly (at line marked with a comment beside it).
Details
I tried on Chrome browser and its corresponding chrome WebDriver for all versions from 79.0 till the latest 84.0, but this exception seems to occur in all the cases.
OS - Windows Server W2k12 VM. [Tried on Windows 10 as well]
Python version 3.8
Code
import time
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
driver = webdriver.Chrome(chrome_options=options)
driver.maximize_window()
driver.get(url)
wait = WebDriverWait(driver, 10)
alert = wait.until(EC.alert_is_present()) # This line causes the time out exception
alert = driver.switch_to.alert
alert.send_keys('domain\username' + Keys.TAB + 'password')
alert.accept()
time.sleep(5)
Note:
Due to a bug in the internal page, I cannot send the credentials through the URL using the https://username:password#ipaddress:port format, so this has forced me to resort to making the above selenium method as my only choice.
The corresponding code for firefox works well (for the same target internal website)
Probable Hunch
I wonder, if I am missing any packages on the newly created VM which is crucial for chrome WebDriver to work. For example, Firefox Gecko driver required Visual studio redistributables in order to work. Does chrome WebDriver require any such equivalent packages?

I don't believe that the Basic Auth popup is exposed as an "alert" in ChromeDriver, so AFAIK your only option is https://username:password#ipaddress:port. Interesting that you say that you can program to the popup in Firefox.
Until Chrome 78, the auth popup would display and block the test script, and you had to "manually" enter the credentials (or use a more general "desktop window manipulation" API), but I don't think that works anymore.

What worked here finally was to discard using the Selenium way of using send_keys() approach but use other packages to do the job. The following two snippets worked.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import keyboard
​
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
driver = webdriver.Chrome(r"path_to_chromedriver.exe", options=options)
driver.maximize_window()
driver.get("<replace with url>")
​
keyboard.write(r"<replace with username>")
keyboard.press_and_release("tab")
keyboard.write("<replace with password>")
keyboard.press_and_release("tab")
keyboard.press_and_release("enter")
Or this one (pip install pywin32 before)
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import win32com.client
​
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
driver = webdriver.Chrome(r"path_to_chromedriver.exe", options=options)
driver.maximize_window()
driver.get("<replace with url>")
​
shell = win32com.client.Dispatch("WScript.Shell")
shell.SendKeys(r"<replace with username>")
shell.SendKeys("{TAB}")
shell.SendKeys("<replace with password>")
shell.SendKeys("{TAB}")

Related

Trying to close portable browser via selenium

I am trying to close portable browser via selenium
I passed --remote-debugging-port=9222 because if I do not pass it then the program is stuck in object creation of webdriver.Chrome(). It will open the portable browser but not load the URL.
But after the URL is open I want to close the browser but driver.quit() it is not working for me. I have tried some other methods for close the browser but they do not work as well.
I want to close the specific instance of the browser which is opened by this program not other opened instances of the browser.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.binary_location = 'C:/Portable/GoogleChromePortable/GoogleChromePortable.exe'
chrome_options.add_argument("--remote-debugging-port=9222")
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--profile-directory=Person 1")
driver = webdriver.Chrome(options=chrome_options,executable_path='C:/Portabl/chromedriver_win32/chromedriver.exe')
url = "https://www.google.com/"
driver.get(url)
driver.quit()
I am using:
selenium 3.141.0, windows 10, python 3.8.0, portable chrome version 93.0.4577.63 (32-bit)
Your this statement
I passed --remote-debugging-port=9222 because if I do not pass it then
the program is stuck in object creation of webdriver.Chrome()
is not correct. --remote-debugging-port=9222 looks like a port number where your application is deployed and you have used chrome option to send them to browser object.
driver.quit()
this typically should have worked, what is the error when it did not work ?
also, for closing a single instance you could do
driver.close()
see if that helps.

Selenium/Python/Chrome: Browser Chrome can`t create directory 'Selenium'

Application that normally works on the PC under this User (Administrator) suddenly begins to launch Windows alarm-window: "Browser Google Chrome can't execute read and write operations in directory 'selenium'."
(But after 'Cancel' this alarm - app works Ok - it get the page in Chrome, and handles it normally).
But the app opens different pages in the new Chrome-window many-many times, and i can`t to sit and press 'Cancel' all the time )))
Aditionaly, this code normally work on enother PC in the remote workplace.
Code:
```python
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
...
options = webdriver.ChromeOptions()
options.add_argument("user-data-dir=selenium")
options.add_argument("--remote-debugging-port=9222")
driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)
try:
driver.get(url)
...
```

scraping selenium protected site

I'm running into an issue being able to automate (website (clickhere))
It appears that the site is protected in someway for chromedriver. When I visit the website normally I have no problem, but when selenium attempts to automate the site, the url redirects to some other home page.
Here is my sample code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
chrome_options = Options()
#chrome_options.add_argument("--headless")
EXE_PATH = 'chromedriver.exe'
driver = webdriver.Chrome(executable_path=EXE_PATH)#, options=chrome_options)
driver.get(SEE URL ABOVE)
time.sleep(5)
print(driver.current_url)
driver.quit()
Please use the link in the hyperlinked text. I removed it from my code here.
Wondering if anyone has run into similar issues with websites picking up that the browser is being automated with selenium, and if there is any possible way around this. If not, maybe you have a suggestion that you could share to tackle from another angle.
A bit more about your usecase and why you felt ...that the site is protected... would have helped us to further analyze the issue. However through Selenium to access the site you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
#options.add_argument("--headless")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://publicindex.sccourts.org/horry/publicindex/")
WebDriverWait(driver, 10).until(EC.title_contains("Index"))
print(driver.current_url)
driver.quit()
Console Output:
https://publicindex.sccourts.org/horry/publicindex/
Outro
You can find a couple of relevant discussions in:
Chrome browser initiated through ChromeDriver gets detected
Selenium and non-headless browser keeps asking for Captcha

How can I change remote controller of FireFox in Selenium Python?

When I FireFox is loaded through Selenium, my browser is under remote controller and a bot image show in URL section in my browser. For deal with this problem I changed User-Agent by this code:
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "whatever you want")
driver = webdriver.Firefox(profile)
User-Agent was changed successfully but, bot image in URL section of my browser was remained.
Would you help me, please?
I used this URL for changing User-Agent:
Change user agent for selenium driver
My whole code is:
MainLink="https://blog.feedspot.com/iot_blogs/"
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities.PHANTOMJS
caps["phantomjs.page.settings.userAgent"] = "whatever you want"
driver = webdriver.Firefox()
from selenium import webdriver
profile = webdriver.FirefoxProfile()
profile.set_preference("general.useragent.override", "whatever you want")
driver = webdriver.Firefox(profile)
agent = driver.execute_script("return navigator.userAgent")
print(agent)
driver.get(MainLink)
Your code is a little confusing. You don't need to use both phantomjs and firefox as the driver for selenium. Which one are you going to use?
As I understood, you would like to avoid being detected from the page you're interacting with. This is usually a greater concern if you're operating with a headless browser, which is the case when using phantomjs but not when using firefox without explicitly telling it to run in this mode, which apparently is your case.
If you are in fact having issues of this nature, there are many ways to try to mitigate this, starting from changing the user agent, as mentioned by you. Assuming you want to go with firefox, A code example would be:
from selenium import webdriver
profile = webdriver.FirefoxProfile()
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.47 Safari/537.36'
profile.set_preference("general.useragent.override", user_agent)
driver = webdriver.Firefox(profile)
MainLink="https://blog.feedspot.com/iot_blogs/"
driver.get(MainLink)
In addition, you could set a different user agent each time you make a request, combining with changing the ip address from where the requests are originated, of course, but this is besides the point...
Hope this helps...
Depending on how "deep" you want to go, you could also add cookies copied from a trusted client:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("http://www.example.com")
# Adds the cookie into current browser context
driver.add_cookie({"name": "key", "value": "value"})
See also: https://www.selenium.dev/documentation/en/support_packages/working_with_cookies/
I could run my code without loading page or in headless way (headless browser).
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(firefox_options=options)
print("Firefox Headless Browser Invoked")
driver.get('https://blog.feedspot.com/iot_blogs/')
s=driver.find_element_by_xpath("""/html/body/div[1]/div[2]/div/div/div[1]/article/div[1]/h1""")
print(s.text)

Selenium web driver - login without hard-coding the password in the code

how can I login to an application using Selenium webdriver without hard-coding the password in the code?
Plus I want to hide the 'geckodriver.exe window' during execution.
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import keys
import getpass
driver = webdriver.Firefox()
driver.maxmimize_window()
driver.get("https://example.com")
time.sleep(10)
username=driver.find_element_by_id("txtUserID")
username.clear()
username.send_keys("abc#xyz.com")
.....the rest of the code will continue....
xvfb is a common way of doing this. Searching for "selenium xvfb" should find lots, such as:
How do I run Selenium in Xvfb?
Is it possible to run Selenium scripts without having an X server running, too?
These will help you for headless firefox.
how can I login to an application using Selenium webdriver without
hard-coding the password in the code?
For this you need to create one GUI or one screen from where user add the password and then you will get that value which was added by user and then use as a password in a specific website.
Add an option for firefox headless . Than it will not be seen in the view.
binary.add_command_line_options('-headless')
You will get the total procedure here
For not hardcoding the password you can create a config file and put the password or other configurations there. Then you can parse the password from that config file.
If you want to minimize the browser just apply below code
driver.manage().window().setPosition(new Point(-2000, 0));
Another option is to use the open source Blinq.io, that copies the cookies of the login and then injects them in the automation test . Effectively it enables you to bypass the login of the application when doing test automation. The code is in github: https://github.com/blinq-io/selenium-session-manager/

Resources