My goal is to automate online bill pay using Selenium for Python.
Login was successful using Webdriver with this code:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('https://website.com/Home')
emailElem = browser.find_element_by_id('UserName') #finds login username field
emailElem.send_keys('username') #enter the username
passwordElem = browser.find_element_by_id('UserPassword') #finds pw field
passwordElem.send_keys('password') #enters pw
passwordElem.submit() #presses submit button
After login, a new page loads, and my next step is to click on a link. The code:
browser.implicitly_wait(3) #allow new page to load (also tried 5 seconds)
click_link = browser.find_element_by_link_text("Bill & Payment")
click_link.click()
And nothing happens. No navigation to the Bill & Payment page. The actual link has a <BR> tag in it, so I also tried including the tag:
click_link = browser.find_element_by_link_text("Bill &<BR>Payment")
But still nothing. What are some other things I should try?
Errors:
Traceback (most recent call last):
File "/home/captain/.PyCharmEdu30/config/scratches/scratch_1.py", line 12, in
click_link = browser.find_element_by_link_text("Bill & Payment")#clicks link on next page
File "/home/captain/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 317, in find_element_by_link_text
return self.find_element(by=By.LINK_TEXT, value=link_text)
File "/home/captain/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 752, in find_element
'value': value})['value']
File "/home/captain/.local/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "/home/captain/.local/lib/python3.5/site-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: {"method":"link text","selector":"Bill & Payment"}
Stacktrace:
at FirefoxDriver.prototype.findElementInternal_ (file:///tmp/tmps7uj9u0l/extensions/fxdriver#googlecode.com/components/driver-component.js:10770)
at fxdriver.Timer.prototype.setTimeout/<.notify (file:///tmp/tmps7uj9u0l/extensions/fxdriver#googlecode.com/components/driver-component.js:625)
The error that you are experiencing is that the element that you are looking for is not in the page. In my experience with Selenium, I have found that css selectors often work the best for interactive with a website. You can also run python from the command line to test if you have a good hook value for an element by:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('https://website.com/Home')
element = "what you want to test here"
driver.find_element_by_id(element).click()
And you can just keep changing the value of element and running the lines as long as you keep the python interpreter open.
If the problem seems to be that Selenium doesn't wait long enough for the page to load, you can always try a wait method like:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
time = 10 # Wait for 10 seconds
by = By.CSS_SELECTOR # The type of hook the element is
hook = "css.selector.here" # The value for the hook (could also be xpath, name, id, etc.)
# Waits until either the element specified by variables by and hook is
# In the page, or the value of time seconds has passed.
WebDriverWait(self.driver, time).until(ec.presence_of_element_located((by, hook)))
driver.find_element(by, hook).click()
Documentation is usually too technical for me to follow, but it was pretty straightforward for Selenium Python Bindings.
Because there was some formatting in the link text, I used the partial link text method and it worked.
example from the documentation:
continue_link = driver.find_element_by_partial_link_text('Conti')
Try to use
click_link=driver.find_element_by_xpath(eg.xpath of 'online bill pay' link)
click_link.click()
Related
I'm trying to download all of the zip files from this page that don't end with the word 'CHECKSUM', so far I have managed to write a code that it's supposed to do that but it's not working as expected, here it is:
import time
import numpy as np
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
opt = Options() #the variable that will store the selenium options
opt.add_experimental_option("debuggerAddress", "localhost:9222") #this allows bulk-dozer to take control of your Chrome Browser in DevTools mode.
s = Service(r'C:\Users\ResetStoreX\AppData\Local\Programs\Python\Python39\Scripts\chromedriver.exe') #Use the chrome driver located at the corresponding path
driver = webdriver.Chrome(service=s, options=opt) #execute the chromedriver.exe with the previous conditions
#Why using MarkPrices: https://support.btse.com/en/support/solutions/articles/43000557589-index-price-and-mark-price#:~:text=Index%20Price%20is%20an%20important,of%20cryptocurrencies%20on%20major%20exchanges.&text=Mark%20Price%20is%20the%20price,be%20fair%20and%20manipulation%20resistant.
time.sleep(2)
if driver.current_url == 'https://data.binance.vision/?prefix=data/futures/um/daily/markPriceKlines/ALICEUSDT/1h/' :
number = 2 #initialize an int variable to 2 because the desired web elements in this page starts from 2
counter = 0
while number <= np.size(driver.find_elements(By.XPATH, '//*[#id="listing"]/tr')): #iterate over the tbody array
data_file_name = driver.find_element(By.XPATH, f'//*[#id="listing"]/tr[{number}]/td[1]/a').text
if data_file_name.endswith('CHECKSUM') == False:
current_data_file = driver.find_element(By.XPATH, f'//*[#id="listing"]/tr[{number}]/td[1]/a')
element_position = current_data_file.location
y_position = str(element_position.get('y'))
driver.execute_script(f"window.scrollBy(0,{y_position})", "") #scroll down the page to know what's being added
current_data_file.click()
print(f'saving {data_file_name}')
time.sleep(0.5)
counter += 1
number += 1
print(counter)
My problem occurs at the 20th element (ALICEUSDT-1h-2022-02-04.zip.CHECKSUM), the program stops and throws errors like the one down below:
ElementClickInterceptedException: element click intercepted: Element
is not clickable at point (418, 1294)
Or this other one with a negative position:
ElementClickInterceptedException: element click intercepted: Element
is not clickable at point (418, -1221)
So, I would like to know how could I improve the code above to handle the errors shown, I know it has everything to do with the scrollbar, but I ran out of ideas after using this other line y_position = str(element_position.get('y')+100) and keep getting the same errors.
I am new to webscraping & not a developer nor have any html exp. and was trying to pull some details from my account after logging in into the website but getting errors in find_element_by_class_name()
this is the code I have tried:
from selenium import webdriver
driver = webdriver.Chrome('path/chromedriver.exe')
driver.get("https://www.URL.COM")
# logged into account manually & maneuvered to the page manually
driver.find_element_by_class_name('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')
Error
---------------------------------------------------------------------------
NoSuchElementException Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_14516/1270707966.py in <module>
----> 1 driver.find_element_by_class_name('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')
C:\ProgramData\Miniconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in find_element_by_class_name(self, name)
562 element = driver.find_element_by_class_name('foo')
563 """
--> 564 return self.find_element(by=By.CLASS_NAME, value=name)
565
566 def find_elements_by_class_name(self, name):
Also tried
driver.find_element_by_css_selector('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')
From inspect I was able to view this & tried to extract the one highlighted in the image:
class name expect a single class name. where as you are passing multiple class name here
css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt
It won't work.
Instead
remove the spaces and make a CSS selector out of it.
driver.find_element(By.CSS_SELECTOR, ".css-901oao.css-cens5h.r-1khnkhu.r-13awgt0.r-1oke55r.r-1enofrn.r-1wzrnnt")
Also, Please remember find_element_by_class_name have been deprecated in newest selenium. You should use this instead
find_element(By.CLASS_NAME, "class name")
having said this, the locator that you've right now looks brittle in nature. Please use static attribute values.
You could try this xpath
//div[starts-with(#class,'css')]//div[#dir='auto' and contains(#style,'-webkit-line-clamp')]
Please check in the dev tools (Google chrome) if we have unique entry in HTML DOM or not.
xpath that you should check :
//div[starts-with(#class,'css')]//div[#dir='auto' and contains(#style,'-webkit-line-clamp')]
Steps to check:
Press F12 in Chrome -> go to element section -> do a CTRL + F -> then paste the xpath and see, if your desired element is getting highlighted with 1/1 matching node.
please I need some help to run this code (https://github.com/PlayingNumbers/ds_salary_proj/blob/master/glassdoor_scraper.py)
In order to scrape job offer data from Glassdoor
Here's the code snippet:
from selenium.common.exceptions import NoSuchElementException, ElementClickInterceptedException
from selenium import webdriver
import time
import pandas as pd
options = webdriver.ChromeOptions()
#Uncomment the line below if you'd like to scrape without a new Chrome window every time.
#options.add_argument('headless')
#Change the path to where chromedriver is in your home folder.
driver = webdriver.Chrome(executable_path=path, options=options)
driver.set_window_size(1120, 1000)
url = "https://www.glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=false&clickSource=searchBtn&typedKeyword="+'data scientist'+"&sc.keyword="+'data scientist'+"&locT=&locId=&jobType="
#url = 'https://www.glassdoor.com/Job/jobs.htm?sc.keyword="' + keyword + '"&locT=C&locId=1147401&locKeyword=San%20Francisco,%20CA&jobType=all&fromAge=-1&minSalary=0&includeNoSalaryJobs=true&radius=100&cityId=-1&minRating=0.0&industryId=-1&sgocId=-1&seniorityType=all&companyId=-1&employerSizes=0&applicationType=0&remoteWorkType=0'
driver.get(url)
#Let the page load. Change this number based on your internet speed.
#Or, wait until the webpage is loaded, instead of hardcoding it.
time.sleep(5)
#Test for the "Sign Up" prompt and get rid of it.
try:
driver.find_element_by_class_name("selected").click()
except NoSuchElementException:
pass
time.sleep(.1)
try:
driver.find_element_by_css_selector('[alt="Close"]').click() #clicking to the X.
print(' x out worked')
except NoSuchElementException:
print(' x out failed')
pass
#Going through each job in this page
job_buttons = driver.find_elements_by_class_name("jl")
I'm getting an empty list
job_buttons
[]
Your problem is with wrong except argument.
With driver.find_element_by_class_name("selected").click() you are trying to click non-existing element. There is no element matching "selected" class name on that page. This causes NoSuchElementException exception as you can see yourself while you are trying to catch ElementClickInterceptedException exception.
To fix this you should use the correct locator or at least the correct argument in except.
Like this:
try:
driver.find_element_by_class_name("selected").click()
except NoSuchElementException:
pass
Or even
try:
driver.find_element_by_class_name("selected").click()
except:
pass
I'm not sure what elements do you want to get into job_buttons.
The search results containing all the details per each job can be found by this:
job_buttons = driver.find_elements_by_css_selector("li.react-job-listing")
So I have an issue where, I am trying to automate an Import on an application that has no API.
As a result, I have to do like 30 navigation clicks just to get to what I want (Exaggeration).
However, I am trying to basically automate the clicks that will allow me to upload a specific file.
As a result, I almost get to the part where I have to select the specific test build I want to import the file with. There is a field that I need to do a send_keys to find the correct import build I have to upload. The Field element looks like this
<input class="lookupInput" type="text" name="brTestScoreImportLookupInput" id="brTestScoreImportLookupInput" style="width: 100px;" tabindex="1" onkeydown="return lookupKeyPressed(event,"","simptbrws000.w")" origvalue="" det="true" aria-labelledby="" autocomplete="off">
However I don't think my code is properly handling the window as it pops-up from the prior selection.
The field I need to update can be found in the picture I uploaded:
Furthermore the XPATH for the Field is //*[#id='brTestScoreImportLookupInput']
You can find the full code here.
The main aspect is I have to Enter TSI into that File ID field and then hit enter on my keyboard to populate the correct import utility I need. Once I do that the import utilities filter out and I need to select a specific File ID:
.
The main code that should be controlling this:
# Click on Test Score Import Wizard - TW
# Test Wizard XPATH = //a[#id='tree1-3-link']/span
element = WebDriverWait(browser, 20).until(
EC.element_to_be_clickable((By.XPATH, "//a[#id='tree1-3-link']/span")))
element.click();
# Send test_upload and Send Keys
# Field XPATH = //*[#id='brTestScoreImportLookupInput']
test_lookup = browser.find_element_by_id("brTestScoreImportLookupInput")
test_lookup.send_keys(test_upload)
If you want to visit the link toe repository code click on here above this.
Any help would be greatly appreciated.
Traceback (most recent call last): File ".\skyward_collegeboard_TSI_import.py", line 115, in
<module> test_lookup = browser.find_element_by_id("brTestScoreImportLookupInput") File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 360, in find_element_by_id return self.find_element(by=By.ID, value=id_) File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py",
line 976, in find_element return self.execute(Command.FIND_ELEMENT, { File "C:\Python38\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute self.error_handler.check_response(response) File "C:\Python38\lib\site-packages\selenium\webdriver\remote\errorhandler.py",
line 242, in check_response raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="brTestScoreImportLookupInput"]"}
(Session info: chrome=80.0.3987.122)
So I was able to accomplish this by using the following method using both selenium and pynput.
# Browser Switches to Window
WebDriverWait(browser,10).until(EC.number_of_windows_to_be(2))
browser.switch_to.window(browser.window_handles[-1])
# Send test_upload and oend Keys
# Field XPATH = //*[#id='brTestScoreImportLookupInput']
test_lookup = browser.find_element_by_id("brTestScoreImportLookupInput")
test_lookup.send_keys(test_upload)
# Press and Release Enter Key
keyboard.press(Key.enter)
keyboard.release(Key.enter)
Essentially I had to switch to that popup window.
I still have the same problem i deleted the code because i forgot to remove my password and so on.
Collect each line from the file as a list of strings with
with open('file.txt') as f:
lines = f.readlines()
then use a for to handle each string into variables or whatever you want to do with them
for l in lines:
//handle line values here
You can also find your anwser in this other SO post:
How to read a file line-by-line into a list?
Edited S Ahmed answer
from selenium import webdriver
browser = webdriver.Chrome('C:\\Users\klime\Desktop\Python Palai Bot\chromedriver')
browser.get('https://www.palai.org/u/sign_in')
# "acc.txt" is your filepath/name
with open('acc.txt') as f:
#this will put your each line in your text file in a list of strings, to use in the loop
lines = f.readlines()
#this will make a loop for each string in your list, using its value in a variable called "line".
for line in lines:
email = browser.find_element_by_id('user_email')
email.send_keys(line)
password = browser.find_element_by_id('user_password')
password.send_keys('mypassword')
commit = browser.find_element_by_name('commit')
commit.click()
collect = browser.find_element_by_link_text('Abholen')
collect.click()
collectTwo = browser.find_element_by_xpath('//*[#id=\"app\"]/article/div[1]/div/div/div/div[1]/div/div/form/input[1]')
collectTwo.click()
browser.get('https://palai.org/a/kliment+843fb7f2d-f17f-4f86-8365-0aeaf61f566e/pay?transfer_amount=176PALAI')
submit = browser.find_element_by_id('submit_transfer')
submit.click()
logout = browser.find_element_xpath('//*[#id=\"app\"]/header/div[2]/form/button')
logout.click()
logout = browser.find_element_by_xpath('//*[#id=\"app\"]/header/div[2]/form/button')
logout.click()
You are almost there.
You have to log in and perform all the task you need to do and log out inside the loop where you are traversing the lines.
In the for loop you have to fix the indention of the send_keys statement. Python is space sensitive.
Also, you are passing the lines list in the send_keys which would through another error. Pass the l variable as it contains the item from the list.
Try This:
from selenium import webdriver
browser = webdriver.Chrome('C:\\Users\klime\Desktop\Python Palai Bot\chromedriver')
with open('acc.txt') as f:
lines = f.readlines()
for l in lines:
#if logout brings you to the login page then you could do this before the loop
browser.get('https://www.palai.org/u/sign_in')
#here l is the object from the list. so use l instead of lines
browser.find_element_by_id('user_email').send_keys(l)
browser.find_element_by_id('user_password').send_keys('yourpassword')
#Do your task
#logout
-----------------
Edit:
As I have said before python is space sensitive. You have to put indents to make your code to work.
Also, the readlines() returns the line with a newline. It causes your app to submit the form on send_keys the line. I've used the splitlines() to remove the newline. And put the code in a try block to handle any exception. With traceback you can handle the exception but still see the errors
Edit 2:
You have to wait for the elements to appear before trying to access the element. That is why it was throwing NoSuchElement exception.
The brower.quit() closes the browser and destroys the webdriver instance. Otherwise it would still be running in the background. If you don't want your browser to close after running or on any exception, comment out it.
Try the following:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import traceback
browser = webdriver.Chrome(executable_path="D:\\DEV SOFT\\chromedriver_win32\\chromedriver.exe")
wait = WebDriverWait(browser, 30)
lines = open('acc.txt', 'r').read().split('\n')
for l in lines:
try:
browser.get('https://www.palai.org/u/sign_in')
email = wait.until(EC.visibility_of(browser.find_element_by_id("user_email")))
email.send_keys(l)
# browser.find_element_by_id('user_email').send_keys(l)
password = wait.until(EC.visibility_of_element_located((By.ID,"user_password")))
password.send_keys("Hard49Pierburg49")
commit = wait.until(EC.visibility_of(browser.find_element(By.NAME,'commit')))
commit.click()
collect = wait.until(EC.visibility_of(browser.find_element_by_link_text('Abholen')))
collect.click()
collectTwo = wait.until(EC.visibility_of(browser.find_element_by_xpath('//*[#id=\"app\"]/article/div[1]/div/div/div/div[1]/div/div/form/input[1]')))
collectTwo.click()
browser.get('https://palai.org/a/kliment+843fb7f2d-f17f-4f86-8365-0aeaf61f566e/pay?transfer_amount=176PALAI')
submit = wait.until(EC.visibility_of(browser.find_element_by_id('submit_transfer')))
submit.click()
logout = wait.until(EC.visibility_of(browser.find_element_by_css_selector("form.button_to > button")))
logout.click()
# browser.save_screenshot("palai.png")
except Exception as e:
traceback.print_exc()
browser.quit()
browser.quit()
To read/write to a file, you must first open it using file = open('fileName.txt','x') where x can be either a for append, w for write, r for read, and r+ for read and write. If you use write, it will delete any contents on file before writing. Append will, well, append it to the end of file. To read a line, you just use the function line = file.readline() in a while loop, breaking when line == null. To write to a file, you just go file.write("Blahblahblah \n'). Note the \n is only necessary to create new lines. Otherwise it would just keep writing at the end of the last character.
For more info just look at the docs, they're fairly basic:
https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files