I am new to selenium am having trouble authenticating to a website. I was provided a link that automatically authenticates me and redirects me to a homepage of a the site. When I try running this code below I am just redirected to the login page and it says authentication failed(Changed the link for obvious reasons). I use the same link on my browser and it logs me in. Am I missing an option that I have to specify somewhere?
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument('--headless')
browser = webdriver.Chrome('/home/ec2-user/test/chromedriver',options=chrome_options)
browser.get('https://examplelink.com/#/login?token=aewoijfadsklnfwjeojwfqoj234ihg')
Related
I was just wondering, how can i hide the fact that i'm using selenium to access epic games store site? It heavily kicks me even if i'm just browsing the site myself using webdriver. I saw similar posts about this topic on stack, but they didn't help me. Here's the settings I have now. What's wrong with them?
from fake_useragent import UserAgent
from selenium import webdriver
ua = UserAgent()
us_ag = ua.random
options = webdriver.ChromeOptions()
options.add_argument(f'user-agent={us_ag}')
options.add_argument('--ignore-ssl-errors')
options.add_argument('window-size=1280,800')
options.add_argument('--ignore-certificate-errors-spki-list')
options.add_argument('--disable-blink-features=AutomationControlled')
driver = webdriver.Chrome(options=options)
The site keeps telling me that i've done captcha wrong. Even if I try for another 10 times - he still tells me the same. There are no problems when I'm browsing with my regular browser.
As a part of automation, I am trying to login to a locally hosted website using selenium. The page serves a HTTP Basic authentication popup and I use the below code to send the credentials. However, upon using a debugger and executing the code step-wise, I deciphered that a TimeOut exception occurs repeatedly (at line marked with a comment beside it).
Details
I tried on Chrome browser and its corresponding chrome WebDriver for all versions from 79.0 till the latest 84.0, but this exception seems to occur in all the cases.
OS - Windows Server W2k12 VM. [Tried on Windows 10 as well]
Python version 3.8
Code
import time
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
driver = webdriver.Chrome(chrome_options=options)
driver.maximize_window()
driver.get(url)
wait = WebDriverWait(driver, 10)
alert = wait.until(EC.alert_is_present()) # This line causes the time out exception
alert = driver.switch_to.alert
alert.send_keys('domain\username' + Keys.TAB + 'password')
alert.accept()
time.sleep(5)
Note:
Due to a bug in the internal page, I cannot send the credentials through the URL using the https://username:password#ipaddress:port format, so this has forced me to resort to making the above selenium method as my only choice.
The corresponding code for firefox works well (for the same target internal website)
Probable Hunch
I wonder, if I am missing any packages on the newly created VM which is crucial for chrome WebDriver to work. For example, Firefox Gecko driver required Visual studio redistributables in order to work. Does chrome WebDriver require any such equivalent packages?
I don't believe that the Basic Auth popup is exposed as an "alert" in ChromeDriver, so AFAIK your only option is https://username:password#ipaddress:port. Interesting that you say that you can program to the popup in Firefox.
Until Chrome 78, the auth popup would display and block the test script, and you had to "manually" enter the credentials (or use a more general "desktop window manipulation" API), but I don't think that works anymore.
What worked here finally was to discard using the Selenium way of using send_keys() approach but use other packages to do the job. The following two snippets worked.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import keyboard
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
driver = webdriver.Chrome(r"path_to_chromedriver.exe", options=options)
driver.maximize_window()
driver.get("<replace with url>")
keyboard.write(r"<replace with username>")
keyboard.press_and_release("tab")
keyboard.write("<replace with password>")
keyboard.press_and_release("tab")
keyboard.press_and_release("enter")
Or this one (pip install pywin32 before)
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import win32com.client
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
driver = webdriver.Chrome(r"path_to_chromedriver.exe", options=options)
driver.maximize_window()
driver.get("<replace with url>")
shell = win32com.client.Dispatch("WScript.Shell")
shell.SendKeys(r"<replace with username>")
shell.SendKeys("{TAB}")
shell.SendKeys("<replace with password>")
shell.SendKeys("{TAB}")
I'm running into an issue being able to automate (website (clickhere))
It appears that the site is protected in someway for chromedriver. When I visit the website normally I have no problem, but when selenium attempts to automate the site, the url redirects to some other home page.
Here is my sample code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
chrome_options = Options()
#chrome_options.add_argument("--headless")
EXE_PATH = 'chromedriver.exe'
driver = webdriver.Chrome(executable_path=EXE_PATH)#, options=chrome_options)
driver.get(SEE URL ABOVE)
time.sleep(5)
print(driver.current_url)
driver.quit()
Please use the link in the hyperlinked text. I removed it from my code here.
Wondering if anyone has run into similar issues with websites picking up that the browser is being automated with selenium, and if there is any possible way around this. If not, maybe you have a suggestion that you could share to tackle from another angle.
A bit more about your usecase and why you felt ...that the site is protected... would have helped us to further analyze the issue. However through Selenium to access the site you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
#options.add_argument("--headless")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get("https://publicindex.sccourts.org/horry/publicindex/")
WebDriverWait(driver, 10).until(EC.title_contains("Index"))
print(driver.current_url)
driver.quit()
Console Output:
https://publicindex.sccourts.org/horry/publicindex/
Outro
You can find a couple of relevant discussions in:
Chrome browser initiated through ChromeDriver gets detected
Selenium and non-headless browser keeps asking for Captcha
Is there any whatsapp or webdriver API to access WhatsApp web without scanning QR code everytime while accessing it using selenium and chrome webdriver in python?
This is What you need. This code Read QR and store it in cookies
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
jokes = ["You don't need a parachute to go skydiving. You need a parachute to go skydiving twice.",
"This is Test Message."]
options = Options()
options.add_argument("--user-data-dir=chrome-data")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome('/usr/local/bin/chromedriver', options=options)
driver.maximize_window()
driver.get('https://web.whatsapp.com') # Already authenticated
time.sleep(20)
##################### Provide Recepient Name Here ###############################
driver.find_element_by_xpath("//*[#title='MyJakartaNumber']").click()
for joke in jokes:
driver.find_element_by_xpath('//*[#id="main"]/footer/div[1]/div[2]/div/div[2]').send_keys(joke)
driver.find_element_by_xpath('//*[#id="main"]/footer/div[1]/div[3]/button/span').click()
time.sleep(10)
time.sleep(30)
driver.close()
Your "WhatsApp" and "QR Code" don't tell anything to me, however if you're testing an application which requires an extra action to sign in I don't think you will be able to perform it using Selenium as it's browser automation framework.
Web applications identify users via Cookies - special HTTP Headers containing client-side information. When you start a web browser via Selenium bindings - it kicks off a clean browser session which is not authenticated in "WhatsApp" (whatever it is)
The solutions would be in:
Authenticate in WhatsApp manually, store your browser profile somewhere and start Selenium by pointing it to the earlier storied profile folder
Authenticate in WhatsApp manually, store your browser cookies and use WebDriver.add_cookie() function to read the stored cookies into the current clean session
You can use "pywhatkit". pywhatkit is used to send messages using whatssapp web. Run:
pip install pywhatkit
and you are good to go.
I am trying to scrape https://www.hyatt.com and It not for illegal use I just want to make a simple script to find Hotel which matches my search.
But the problem is I am unable to even load the webpage using any bot. It simply does not load.
here are some ways I already tried.
1 - Used selenium
2 - used scrapy frame-work to get the data
3 - used python requests library
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("https://www.hyatt.com")
driver.close()
I just want that the page loads itself. I will take care of the rest.
I took your code added a few tweaks and ran the same test at my end:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
# options.add_argument('disable-infobars')
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.hyatt.com")
WebDriverWait(driver, 20).until(EC.title_contains("Hyatt"))
print(driver.title)
driver.quit()
Eventually I ran into the same issue. Using Selenium I was also unable to even load the webpage. But when I inspected the Console Errors within google-chrome-devtools it clearly showed that:
Failed to load resource: the server responded with a status of 404 () https://www.hyatt.com/149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/fingerprint
Snapshot:
404 Not Found
The HTTP 404 Not Found client error response code indicates that the server can't find the requested resource. Links which lead to a 404 page are often called broken or dead links, and can be subject to link rot.
A 404 status code does not indicate whether the resource is temporarily or permanently missing. But if a resource is permanently removed, ideally a 410 (Gone) should be used instead of a 404 status.
Moving ahead, while inspecting the HTML DOM of https://www.hyatt.com/ it was observed that some of the <script> and <noscript> tags refers to akam:
<script type="text/javascript" src="https://www.hyatt.com/akam/10/28f56097" defer=""></script>
<noscript><img src="https://www.hyatt.com/akam/10/pixel_28f56097?a=dD02NDllZTZmNzg1NmNmYmIyYjVmOGFiOGYwMWI5YWMwZmM4MzcyZGY5JmpzPW9mZg==" style="visibility: hidden; position: absolute; left: -999px; top: -999px;" /></noscript>
Which is a clear indication that the website is protected by Bot Management service provider Akamai Bot Manager and the navigation by WebDriver driven Browser Client gets detected and subsequently gets blocked.
Outro
You can find some more relevant discussions in:
Unable to use Selenium to automate Chase site login
How does recaptcha 3 know I'm using selenium/chromedriver?
Selenium and non-headless browser keeps asking for Captcha