Unable to figure out where to add wait statement in python selenium - python-3.x

I am searching elements in my list(one by one) by inputing into searchbar of a website and get apple products name that appeared in search result and printed. However I am getting following exception
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
I know its because of changing of element very fast so I need to add wait like
wait(driver, 10).until(EC.visibility_of_element_located((By.ID, "submitbutton")))
or explictly
Q1. But I don't understand where should I add it? Here is my code. Please help!
Q2. I want to go to all the next pages using but that's not working.
driver.find_element_by_xpath( '//div[#class="no-hover"]/a' ).click()
Earlier exception was raised on submitton button and now at if statement.

That's not what implicit wait is for. Since the page change regularly you can't be sure when the current object inside the variable is still valid.
My suggestion is to run the above code in loop using try except. Something like the following:
for element in mylist:
ok = False
while True:
try:
do_something_useful_while_the_page_can_change(element)
except StaleElementReferenceException:
# retry
continue
else:
# go to next element
break
Where:
def do_something_useful_while_the_page_can_change(element):
searchElement = driver.find_element_by_id("searchbar")
searchElement.send_keys(element)
driver.find_element_by_id("searchbutton").click()
items_count = 0
items = driver.find_elements_by_class_name( 'searchresult' )
for i, item in enumerate( items ):
if 'apple' in item.text:
print ('item.text')
items_count += len( items )

I think what you had was doing too much and can be simplified. You basically need to loop through a list of search terms, myList. Inside that loop you send the search term to the searchbox and click search. Still inside that loop you want to grab all the elements off the page that consist of search results, class='search-result-product-url' but also the text of the element contains 'apple'. The XPath locator I provided should do both so that the collection that is returned all are ones you want to print... so print each. End loop... back to next search term.
for element in mylist:
driver.find_element_by_id("search-input").send_keys(element)
driver.find_element_by_id("button-search").click()
# may need a wait here?
for item in driver.find_elements_by_xpath( "//a[#class='search-result-product-url'][contains(., 'apple')]" ):
print item.text

Related

Refresh a page if an element is not visible and click, else click python selenium

I'm dealing with a very non responsive scroll bar in a window on a tableau dashboard. I cannot scroll to find the element using any code, python, or java execution, so I need to write a condition inside my for loop that refreshes the page, and clicks a location (the scroll bar) if the element isn't present, then checks again for the element's presence, but I can't seem to get it.
for key, value in series_dict.items():
xpath = driver.find_element_by_xpath('//*[#id="' + value + '"]') #how do i make this a boolean
while True:
if xpath == False:
driver.refresh()
#click the scroll bar here then break and start the whole for loop over
else:
try:
series_click = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="' + value + '"]')))
series_click.click()
time.sleep(5)
download_click = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="download"]/span[3]')))
download_click.click()
time.sleep(1)
except TimeoutError as e:
print('Error: {}'.format(str(e)))
continue
break
I've never nested so many conditionals before and I think I've gotten my continues, breaks, and order mixed up. I also don't know how to make my xpath object a boolean so that I can test it.
Any help with this would be appreciated.
Thank you!

Selecting values from drop down of website through selenium Python

I am trying web scraping using Python selenium. I am getting the following error: Message: element not interactable (Session info: chrome=78.0.3904.108) I am trying to access option element by id, value or text. It is giving me this error. I am using Python-3. Can someone help to explain where am I going wrong. I used the select tag using xpath and also tried css_selector. The select tag is selected and I can get the output of select tag selected. Here is my code for a better understanding:
Code-1:-
path = r'D:\chromedriver_win32\chromedriver.exe'
browser = webdriver.Chrome(executable_path = path)
website = browser.get("https://publicplansdata.org/resources/download-avs-cafrs/")
el = browser.find_element_by_xpath('//*[#id="ppd-download-state"]/select')
for option in el.find_elements_by_tag_name('option'):
if option.text != None:
option.click()
break
Blockquote
Code-2:-
select_element = Select(browser.find_element_by_xpath('//*[#id="ppd-download-state"]/select'))
# this will print out strings available for selection on select_element, used in visible text below
print(o.value for o in select_element.options)
select_element.select_by_value('AK')
Both codes give the same error how can I select values from drop down from website
Same as question:
Python selenium select element from drop down. Element Not Visible Exception
But the error is different. Tried the methods in comments
State, Plan, and Year:
browser.find_element_by_xpath("//span[text()='State']").click()
browser.find_element_by_xpath("//a[text()='West Virginia']").click()
time.sleep(2) # wait for Plan list to be populated
browser.find_element_by_xpath("//span[text()='Plan']").click()
browser.find_element_by_xpath("//a[text()='West Virginia PERS']").click()
time.sleep(2) # wait for Year list to be populated
browser.find_element_by_xpath("//span[text()='Year']").click()
browser.find_element_by_xpath("//a[text()='2007']").click()
Don't forget to import time

how to use "if" to test the presence of an element in the webpage with python?

I created a loop (while True) to automate a task on the site with python. This code clicks on two fields until an element appears on the page
(browser.find_element_by_id ('formComp: buttonBack').
When this element appears, I want the loop to stop and go to the next block of code.
I tested it that way, but it made a mistake. Python reported that the element "formComp: buttonback" was not found. But that's just it, if not found continue the loop:
while (browser.find_element_by_id('formComp:repeatCompromissoLista:0:tableRealizacao:0:subtableVinculacoes:0:vinculacao_input')):
vinc = wait.until(EC.presence_of_element_located((By.ID, 'formComp:repeatCompromissoLista:0:tableRealizacao:0:subtableVinculacoes:0:vinculacao_input')))
vinc = browser.find_element_by_id('formComp:repeatCompromissoLista:0:tableRealizacao:0:subtableVinculacoes:0:vinculacao_input')
vinc.send_keys('400')
enterElem5 = wait.until(EC.element_to_be_clickable((By.ID, 'formComp:buttonConfirmar')))
enterElem5 = browser.find_element_by_id('formComp:buttonConfirmar')
enterElem5.send_keys(Keys.ENTER)
time.sleep(int(segundosv))
if (browser.find_element_by_id('formComp:buttonRetornar')== True):
break
else:
continue
Try like this hope this helps.Check the length count of the button more than 0.
if (len(browser.find_elements_by_id('formComp:buttonRetornar'))>0):
break
else:
continue
find_element_by_id() does not return False when an element is not found. Instead, it raises selenium.common.exceptions.NoSuchElementException. You can handle the exception to get the flow control you are looking for:
try:
browser.find_element_by_id('formComp:buttonRetornar')
break
except NoSuchElementException:
continue

selenium python onclick() gives StaleElementReferenceException

The developer changed the code to use an onclick() DOM element instead of an url. So Now I need to reload page all the time to prevent it from getting stale. How can I do this with a single find_elements_by_xpath?
I assume it is the document.*.submit() that needs the DOM?
https://localhost:4778/ruleengine/AlarmTest?category=Alert#,
text:(Force),
onclick():document.Forceee0deabfba2341d2a0988779499f5530.submit()
Old code now fails:
driver.get(alarmurl)
elems = driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")
for el in elems:
el.click()
My current workaround is to reload the page after every click, but
we may have 3000 events to remove, making it horrendously slow.
driver.get(alarmurl)
elems = driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")
while len(elems) > 0:
driver.get(alarmurl)
elems = driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")
elems[0].click()
Thanks
I dont think you have to reload the entire page if you encounter StaleElementReferenceException but I may be wrong too. It happens When the element is no longer attached to the DOM, search for the element again to reference the element
The below code may not clear your issue, but should help you start implementing a better solution
driver.get(alarmurl)
elems = driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")
for el in elems:
try:
el.click()
except StaleElementReferenceException:
# find the element again and click
You can fix your code as below:
driver.get(alarmurl)
# get initial events number
elems_count = len(driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]"))
# execute click() for each event
for _ in range(elems_count):
driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")[0].click()
P.S. You also might need to wait until events number decreased by 1 after each click in case events removes too slow... or apply another approach:
driver.get(alarmurl)
# get initial events number
elems_count = len(driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]"))
# execute click() for each event. Starting from the last one
for index in range(elems_count):
driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")[elems_count - index - 1].click()

selenium - clicking a button

I am trying to pull out the names of all courses offered by Lynda.com together with the subject so that it appears on my list as '2D Drawing -- Project Soane: Recover a Lost Monument with BIM with Paul F. Aubin'. So I am trying to write a script that will go to each subject on http://www.lynda.com/sitemap/categories and pull out the list of courses. I already managed to get Selenium to go from one subject to another and pull the courses. My only problem is that there is a button 'See X more courses' to see the rest of the courses. Sometimes you have to click it couple of times that´s why I used while loop. But selenium doesn´t seem to execute this click. Does anyone know why?
This is my code:
from selenium import webdriver
url = 'http://www.lynda.com/sitemap/categories'
mydriver = webdriver.Chrome()
mydriver.get(url)
course_list = []
for a in [1,2,3]:
for b in range(1,73):
mydriver.find_element_by_xpath('//*[#id="main-content"]/div[2]/div[3]/div[%d]/ul/li[%d]/a' % (a,b)).click()
while True:
#click the button 'See more results' as long as it´s available
try:
mydriver.find_element_by_xpath('//*[#id="main-content"]/div[1]/div[3]/button').click()
except:
break
subject = mydriver.find_element_by_tag_name('h1') # pull out the subject
courses = mydriver.find_elements_by_tag_name('h3') # pull out the courses
for course in courses:
course_list.append(str(subject.text)+" -- " + str(course.text))
# go back to the initial site
mydriver.get(url)
Scroll to element before clicking:
see_more_results = browser.find_element_by_css_selector('button[class*=see-more-results]')
browser.execute_script('return arguments[0].scrollIntoView()', see_more_results)
see_more_results.click()
One solution how to repeat these actions could be:
def get_number_of_courses():
return len(browser.find_elements_by_css_selector('.course-list > li'))
number_of_courses = get_number_of_courses()
while True:
try:
button = browser.find_element_by_css_selector(CSS_SELECTOR)
browser.execute_script('return arguments[0].scrollIntoView()', button)
button.click()
while True:
new_number_of_courses = get_number_of_courses()
if (new_number_of_courses > number_of_courses):
number_of_courses = new_number_of_courses
break
except:
break
Caveat: it's always better to use build-in explicit wait than while True:
http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp#explicit-waits
The problem is that you're calling a method to find element by class name, but you're passing a xpath. if you're sure this is the correct xpath you'll simply need to change to method to 'find_element_by_xpath'.
A recommendation if you allow: Try to stay away from these long xpaths and go through some tutorials on how to write efficient xpath for example.

Resources