Hi I have this code which is generating me:
selenium.common.exceptions.StaleElementReferenceException:
Message: stale element reference: element is not attached to the page document
What can be the root cause for it? This code works ok in 10 cases but in 11 not. Is there any chance to improve it to prevent such errors?
Lines which generates the fails are:
compare_announcement_text(context, option_name, element_offer_type)
And:
if element_offer_type.text == option_name:
def select_announcement(context, option_name):
"""Select announcement from 'Offer Type' drop down
:param context:
:param option_name:
:return: Announcement page
"""
offer_type = context.driver.find_element_by_xpath(Locators.offer_type)
offer_type.click()
offer_type_list = context.driver.find_elements_by_xpath(Locators.all_elements_buttons_offer_type)
offer_type_text = []
for element_offer_type in offer_type_list:
compare_announcement_text(context, option_name, element_offer_type)
offer_type_text.append(element_offer_type.text)
if option_name not in offer_type_text:
time.sleep(0.5) # menu not fully rendered - need to wait
offer_type_list = context.driver.find_elements_by_xpath(Locators.all_elements_buttons_offer_type)
for element_offer_type in offer_type_list:
compare_announcement_text(context, option_name, element_offer_type)
from features.pages.announcement_page import AnnouncementPage
return AnnouncementPage(context)
def compare_announcement_text(context, option_name, element_offer_type):
if element_offer_type.text == option_name:
WebDriverWait(context.driver, 20).until(
expected_conditions.visibility_of_all_elements_located(
(By.XPATH, Locators.all_elements_buttons_offer_type)))
for check in range(4):
try:
ActionChains(context.driver).move_to_element(element_offer_type).click(element_offer_type).perform()
return
except selenium.common.exceptions.ElementClickInterceptedException:
time.sleep(0.5)
else:
print(f"Tried click {option_name} for 2 seconds - making final click")
ActionChains(context.driver).move_to_element(element_offer_type).click(element_offer_type).perform()
else:
print(f'Element selected in offer type is not {option_name} and is {element_offer_type.text}')
It's easily cause this problem if you use find_elements_by_xpath to get a set of elements first.
I suggest you find element again in for loop like this:
offer_type_list = context.driver.find_elements_by_xpath(Locators.all_elements_buttons_offer_type)
offer_type_text = []
for i in range(1, len(offer_type_list)+1):
# xpath example: (//div[#class='abc'])[1]
element_offer_type = context.driver.find_element_by_xpath("("+Locators.all_elements_buttons_offer_type+")["+str(i)+"]")
compare_announcement_text(context, option_name, element_offer_type)
offer_type_text.append(element_offer_type.text)
I faced same exception a time ago. Its caused because the element you are interacting is not loaded in the DOM at that moment. You can solve it with a webDriverWait waiting for the element to be displayed.
#Jonx -> It's not the only reason. Question is also about app, #ranger is using. If it's dynamic (with reloading elements e.g. cause new live data) it's also raises this issue. depending on time when data comes in (e.g. after getting reference to element but before performing action). This can be pain in the ass -,-
#Yun - like this idea, but also (if in testing), using Page Object Pattern, good idea is to use views, and then perform actions within try-except blocks, or actually defining Page elements as property methods, so You'll get clean code, and WebDriver will look for element each time call. That eases up issue.
Yeah, I know that this is not so "well-performance" solution, but - as always - trade offs for stable behaviour with live-dynamic apps.
Related
Why is the StaleElementReferenceException still raised even though I ignored the exception? The code will eventually run successfully after a few tries but how can I make it wait until the element is clickable to avoid any exception?
import sys
from selenium.common.exceptions import StaleElementReferenceException
while True:
try:
WebDriverWait(driver, 10, ignored_exceptions=[StaleElementReferenceException]).until(ec.element_to_be_clickable((By.XPATH,"//button[contains(text(), 'Performance Summary')]"))).click()
break
except Exception as e:
exc_type, exc_obj, exc_tb = sys.exc_info()
print(e, exc_type, exc_tb.tb_lineno)
print('Retrying...')
From the source code of WebDriverWait:
class WebDriverWait:
def __init__(
self,
driver,
timeout: float,
poll_frequency: float = POLL_FREQUENCY,
ignored_exceptions: typing.Optional[WaitExcTypes] = None,
):
"""Constructor, takes a WebDriver instance and timeout in seconds.
:Args:
- driver - Instance of WebDriver (Ie, Firefox, Chrome or Remote)
- timeout - Number of seconds before timing out
- poll_frequency - sleep interval between calls
By default, it is 0.5 second.
- ignored_exceptions - iterable structure of exception classes ignored during calls.
By default, it contains NoSuchElementException only.
Example::
from selenium.webdriver.support.wait import WebDriverWait \n
element = WebDriverWait(driver, 10).until(lambda x: x.find_element(By.ID, "someId")) \n
is_disappeared = WebDriverWait(driver, 30, 1, (ElementNotVisibleException)).\\ \n
until_not(lambda x: x.find_element(By.ID, "someId").is_displayed())
"""
self._driver = driver
self._timeout = float(timeout)
self._poll = poll_frequency
# avoid the divide by zero
if self._poll == 0:
self._poll = POLL_FREQUENCY
exceptions = list(IGNORED_EXCEPTIONS)
if ignored_exceptions:
try:
exceptions.extend(iter(ignored_exceptions))
except TypeError: # ignored_exceptions is not iterable
exceptions.append(ignored_exceptions)
self._ignored_exceptions = tuple(exceptions)
It is worth to notice that ignored_exceptions is not iterable.
So NoSuchElementException being the default exception and StaleElementReferenceException being purposely added, can be ignored only once. Hence the second time StaleElementReferenceException is no more handled.
The problem is that once you get a StaleElementReferenceException, you can't just wait it out. Here's how stale elements work.
element = driver.find_element(By.ID, "someId")
driver.refresh() # or anything that changes the portion of the page that 'element' is on
element.click() # throws StaleElementReferenceException
At this point a StaleElementReferenceException is thrown because you had a reference to the element and then lost it (the element reference pointer points to nothing). No amount of waiting is going to restore that reference.
The way to "fix" this is to grab the reference again after the page has changed,
element = driver.find_element(By.ID, "someId")
driver.refresh()
element = driver.find_element(By.ID, "someId") # refetch the reference after the page refreshes
element.click()
Now the .click() will work without error.
Most people that run into this issue are looping through a collection of elements and in the middle of the loop they click a link that navigates to a new page or something else that reloads or changes the page. They later return to the original page and click on the next link but they get a StaleElementReferenceException.
elements = driver.find_elements(locator)
for element in elements
element.click() # navigates to new page
# do other stuff and return to first page
The first loop works fine but in the second loop the element reference is dead because of the page change. You can change this loop to force the elements collection to be refetched at the start of each loop
for element in driver.find_elements(locator)
element.click() # navigates to new page
# do other stuff and return to first page
Now the loop will work. This is just an example but hopefully it will point you in the right direction to fix your code.
I'm using Selenium to scrape Linkedin for jobs but I'm getting the stale reference error.
I've tried refresh, wait, webdriverwait, a try catch block.
It always fails on page 2.
I'm aware it could be a DOM issue and have run through a few of the answers to that but none of them seem to work for me.
def scroll_to(self, job_list_item):
"""Just a function that will scroll to the list item in the column
"""
self.driver.execute_script("arguments[0].scrollIntoView();", job_list_item)
job_list_item.click()
time.sleep(self.delay)
def get_position_data(self, job):
"""Gets the position data for a posting.
Parameters
----------
job : Selenium webelement
Returns
-------
list of strings : [position, company, location, details]
"""
# This is where the error is!
[position, company, location] = job.text.split('\n')[:3]
details = self.driver.find_element_by_id("job-details").text
return [position, company, location, details]
def wait_for_element_ready(self, by, text):
try:
WebDriverWait(self.driver, self.delay).until(EC.presence_of_element_located((by, text)))
except TimeoutException:
logging.debug("wait_for_element_ready TimeoutException")
pass
logging.info("Begin linkedin keyword search")
self.search_linkedin(keywords, location)
self.wait()
# scrape pages,only do first 8 pages since after that the data isn't
# well suited for me anyways:
for page in range(2, 3):
jobs = self.driver.find_elements_by_class_name("occludable-update")
#jobs = self.driver.find_elements_by_css_selector(".occludable-update.ember-view")
#WebDriverWait(self.driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'occludable-update')))
for job in jobs:
self.scroll_to(job)
#job.click
[position, company, location, details] = self.get_position_data(job)
# do something with the data...
data = (position, company, location, details)
#logging.info(f"Added to DB: {position}, {company}, {location}")
writer.writerow(data)
# go to next page:
bot.driver.find_element_by_xpath(f"//button[#aria-label='Page {page}']").click()
bot.wait()
logging.info("Done scraping.")
logging.info("Closing DB connection.")
f.close()
bot.close_session()
I expect that when job_list_item.click() is performed the page is loaded, in this case since you are looping jobs which is a list of WebDriverElement will become stale. You are returning back to the page but your jobs is already stale.
Usually to prevent a stale element, I always prevent the use of the element in a loop or store an element to a variable, especially if the element may change.
The developer changed the code to use an onclick() DOM element instead of an url. So Now I need to reload page all the time to prevent it from getting stale. How can I do this with a single find_elements_by_xpath?
I assume it is the document.*.submit() that needs the DOM?
https://localhost:4778/ruleengine/AlarmTest?category=Alert#,
text:(Force),
onclick():document.Forceee0deabfba2341d2a0988779499f5530.submit()
Old code now fails:
driver.get(alarmurl)
elems = driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")
for el in elems:
el.click()
My current workaround is to reload the page after every click, but
we may have 3000 events to remove, making it horrendously slow.
driver.get(alarmurl)
elems = driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")
while len(elems) > 0:
driver.get(alarmurl)
elems = driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")
elems[0].click()
Thanks
I dont think you have to reload the entire page if you encounter StaleElementReferenceException but I may be wrong too. It happens When the element is no longer attached to the DOM, search for the element again to reference the element
The below code may not clear your issue, but should help you start implementing a better solution
driver.get(alarmurl)
elems = driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")
for el in elems:
try:
el.click()
except StaleElementReferenceException:
# find the element again and click
You can fix your code as below:
driver.get(alarmurl)
# get initial events number
elems_count = len(driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]"))
# execute click() for each event
for _ in range(elems_count):
driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")[0].click()
P.S. You also might need to wait until events number decreased by 1 after each click in case events removes too slow... or apply another approach:
driver.get(alarmurl)
# get initial events number
elems_count = len(driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]"))
# execute click() for each event. Starting from the last one
for index in range(elems_count):
driver.find_elements_by_xpath("//a[contains(text(), '(Force)')]")[elems_count - index - 1].click()
I am searching elements in my list(one by one) by inputing into searchbar of a website and get apple products name that appeared in search result and printed. However I am getting following exception
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
I know its because of changing of element very fast so I need to add wait like
wait(driver, 10).until(EC.visibility_of_element_located((By.ID, "submitbutton")))
or explictly
Q1. But I don't understand where should I add it? Here is my code. Please help!
Q2. I want to go to all the next pages using but that's not working.
driver.find_element_by_xpath( '//div[#class="no-hover"]/a' ).click()
Earlier exception was raised on submitton button and now at if statement.
That's not what implicit wait is for. Since the page change regularly you can't be sure when the current object inside the variable is still valid.
My suggestion is to run the above code in loop using try except. Something like the following:
for element in mylist:
ok = False
while True:
try:
do_something_useful_while_the_page_can_change(element)
except StaleElementReferenceException:
# retry
continue
else:
# go to next element
break
Where:
def do_something_useful_while_the_page_can_change(element):
searchElement = driver.find_element_by_id("searchbar")
searchElement.send_keys(element)
driver.find_element_by_id("searchbutton").click()
items_count = 0
items = driver.find_elements_by_class_name( 'searchresult' )
for i, item in enumerate( items ):
if 'apple' in item.text:
print ('item.text')
items_count += len( items )
I think what you had was doing too much and can be simplified. You basically need to loop through a list of search terms, myList. Inside that loop you send the search term to the searchbox and click search. Still inside that loop you want to grab all the elements off the page that consist of search results, class='search-result-product-url' but also the text of the element contains 'apple'. The XPath locator I provided should do both so that the collection that is returned all are ones you want to print... so print each. End loop... back to next search term.
for element in mylist:
driver.find_element_by_id("search-input").send_keys(element)
driver.find_element_by_id("button-search").click()
# may need a wait here?
for item in driver.find_elements_by_xpath( "//a[#class='search-result-product-url'][contains(., 'apple')]" ):
print item.text
I am trying to pull out the names of all courses offered by Lynda.com together with the subject so that it appears on my list as '2D Drawing -- Project Soane: Recover a Lost Monument with BIM with Paul F. Aubin'. So I am trying to write a script that will go to each subject on http://www.lynda.com/sitemap/categories and pull out the list of courses. I already managed to get Selenium to go from one subject to another and pull the courses. My only problem is that there is a button 'See X more courses' to see the rest of the courses. Sometimes you have to click it couple of times that´s why I used while loop. But selenium doesn´t seem to execute this click. Does anyone know why?
This is my code:
from selenium import webdriver
url = 'http://www.lynda.com/sitemap/categories'
mydriver = webdriver.Chrome()
mydriver.get(url)
course_list = []
for a in [1,2,3]:
for b in range(1,73):
mydriver.find_element_by_xpath('//*[#id="main-content"]/div[2]/div[3]/div[%d]/ul/li[%d]/a' % (a,b)).click()
while True:
#click the button 'See more results' as long as it´s available
try:
mydriver.find_element_by_xpath('//*[#id="main-content"]/div[1]/div[3]/button').click()
except:
break
subject = mydriver.find_element_by_tag_name('h1') # pull out the subject
courses = mydriver.find_elements_by_tag_name('h3') # pull out the courses
for course in courses:
course_list.append(str(subject.text)+" -- " + str(course.text))
# go back to the initial site
mydriver.get(url)
Scroll to element before clicking:
see_more_results = browser.find_element_by_css_selector('button[class*=see-more-results]')
browser.execute_script('return arguments[0].scrollIntoView()', see_more_results)
see_more_results.click()
One solution how to repeat these actions could be:
def get_number_of_courses():
return len(browser.find_elements_by_css_selector('.course-list > li'))
number_of_courses = get_number_of_courses()
while True:
try:
button = browser.find_element_by_css_selector(CSS_SELECTOR)
browser.execute_script('return arguments[0].scrollIntoView()', button)
button.click()
while True:
new_number_of_courses = get_number_of_courses()
if (new_number_of_courses > number_of_courses):
number_of_courses = new_number_of_courses
break
except:
break
Caveat: it's always better to use build-in explicit wait than while True:
http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp#explicit-waits
The problem is that you're calling a method to find element by class name, but you're passing a xpath. if you're sure this is the correct xpath you'll simply need to change to method to 'find_element_by_xpath'.
A recommendation if you allow: Try to stay away from these long xpaths and go through some tutorials on how to write efficient xpath for example.