I know there have been several questions asked regarding stale elements, but I can't seem to resolve these.
My site is private so unfortunately can't share, but seems to always throw the error somewhere within the below for-loop. This loop is meant to get the text of each row in a table (number of rows varies). I've assigned WebDriverWait commands and have a very similar for-loop earlier in my code to do the same thing in another table on the website which works perfectly. I've also tried including the link click command and table, body, and tableText definition inside the loop to redefine at every iteration.
Once the code stops and the error message displays (stale element reference: element is not attached to the page document (Session info: chrome=89.0.4389.128)), if I manually run everything line-by-line, it all seems to work and correctly grabs the text.
Any ideas? Thanks!
link = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, "*link address*")))
link.click()
table = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.ID, "TableId")))
body = tableSig.find_element(By.CLASS_NAME, "*table body class*")
tableText = body.find_elements(By.TAG_NAME, "tr")
rows = len(tableText)
approvedSigs = [None]*rows
for i in range(1, rows+1):
approvedSigs[i-1] = (tableText[i-1].text)
approvedSigs[i-1] = approvedSigs[i-1].lstrip()
approvedSigs[i-1] = approvedSigs[i-1][9:]
approvedSigs[i-1] = approvedSigs[i-1].replace("\n"," ")
Related
[from selenium import webdriver
driver = webdriver.Firefox()
driver.get('https://youtube.com')
searchbox = driver.find_element("xpath",'//*[#id="search"]')
searchbox.send_keys("Akram Khan")
searchButton = driver.find_element("xpath",'//*[#id="search-icon-legacy"]')
searchButton.click()]
(https://i.stack.imgur.com/egHm1.png)
The function [searchbox.send_keys("Akram Khan")] have to be write the mentioned text("Akram Khan") inside the search box of YouTUbe.
So the problem is that when you first try to get the searchbox element with the current version that you had it was searching for the first occurrence of an element with id='search' which there is an element that is before not connected to the search box (therefore send_keys() to that element doesn't work). You can get the specific element that you want by specifying the tag that it is with your current code by doing (because the input element is what you want to be sending your response to):
searchbox = driver.find_element("xpath",'//input[#id="search"]')
#id='search' is reflecting to list of elements so use unique element like #name='search-query'
I am trying to loop through all the pages of a website. but I am getting a stale element reference: element is not attached to the page document error. This happens when the script try to click the third page. The script got the error when it runs to page.click(). Any suggestions?
while driver.find_element_by_id('jsGrid_vgAllCases').find_elements_by_tag_name('a')[-1].text=='...':
links=driver.find_element_by_id('jsGrid_vgAllCases').find_elements_by_tag_name('a')
for link in links:
if ((link.text !='...') and (link.text !='ADD DOCUMENTS')):
print('Page Number: '+ link.text)
print('Page Position: '+str(links.index(link)))
position=links.index(link)
page=driver.find_element_by_id('jsGrid_vgAllCases').find_elements_by_tag_name('a')[position]
page.click()
time.sleep(5)
driver.find_element_by_id('jsGrid_vgAllCases').find_elements_by_tag_name('a')[-1].click()
You can locate the link element each time again according to the index, not to use elements found initially.
Something like this:
amount = len(driver.find_element_by_id('jsGrid_vgAllCases').find_elements_by_tag_name('a'))
for i in range(1,amount+1):
link = driver.find_element_by_xpath("(//*[#id='jsGrid_vgAllCases']//a)["+str(i) +"]")
from now you can continue within your for loop with this link like this:
amount = len(driver.find_element_by_id('jsGrid_vgAllCases').find_elements_by_tag_name('a'))
for i in range(1,amount+1):
link = driver.find_element_by_xpath("(//*[#id='jsGrid_vgAllCases']//a)["+str(i) +"]")
if ((link.text !='...') and (link.text !='ADD DOCUMENTS')):
print('Page Number: '+ link.text)
print('Page Position: '+str(links.index(link)))
position=links.index(link)
page=driver.find_element_by_id('jsGrid_vgAllCases').find_elements_by_tag_name('a')[position]
page.click()
time.sleep(5)
(I'm not sure about the correctness of all the rest your code, just copy-pasted it)
I'm running into an issue with the Stale Element Exception too. Interesting with Firefox no problem, Chrome && Edge both fail randomly. In general i have two generic find method with retry logic, these find methods would look like:
// Yes C# but should be relevant for any WebDriver...
public static IWebElement( this IWebDriver driver, By locator)
public static IWebElement( this IWebElement element, By locator)
The WebDriver variant seems to work fine for my othe fetches as the search is always "fresh"... But the WebElement search is the one causing grief. Unfortunately the app forces me to need the WebElement version. Why he page/html will be something like:
<node id='Best closest ID Possible'>
<span>
<div>text i want</div>
<div>meh ignore this </div>
<div>More text i want</div>
</span>
<span>
<!-- same pattern ... -->
So the code get the closest element possible by id and child spans i.e. "//*[#id='...']/span" will give all the nodes of interest. This is now where i run into issues, enumerating all element, will do two XPath select i.e. "./div[1]" and "./div[3]" for pulling out the text desired. It is only in fetching the text nodes under the elements where randomly a StaleElement will be thrown. Sometimes the very first XPath fails, sometimes i'll go through a few pages, as the pages being might have 10,000's or more pages, while the structure is the same i'll spot check random pages as they all the same format. At most i've gotten through 20 consecutive pages with Chrome (ver 92.0.4515.107) or Edge (ver 94.0.986), both seem to be the latest as of now.
One solution that should work, get all the the span elements first, i.e. '//*[#id='x']/span' get my list then query from the driver like:
var nodeList = driver.FindElements(By.XPath('//*[#id='x']/span' ));
for( int idx = 0 ; idx < nodeList.Count; idx++)
{
string str1 = driver.FindElements(By.XPath("//*[#id='x']/span[idx+1]/div[1]")).GetAttribute("innerText");
string str2 = driver.FindElements(By.XPath("//*[#id='x']/span[idx+1]/div[3]")).GetAttribute("innerText");
}
```
Think it would work but, YUK! This is kind of simplified and being able to do an XPath from the respective "ID" located node would be preferable..
here is my predicament:
What I wanna do is to get newly posts in a table named "theme".
this is what i have for new posts, new posts has a sticker:
new_posts = soup.select('tbody:has(.threadpages [src="images/new.gif"]), '
'tbody:has(.threadpages [src="images/new1.gif"]), '
'tbody:has(.threadpages [src="images/new2.gif"])')
this is what I have for the "theme" table:
theme_posts = soup.select('table:has(font:contains("theme")), '
is there a way I can combine both? I tried do table first, and then do new posts, it gives me Error saying
ResultSet object has no attribute '%s'. You're probably treating a list of elements like a single element. Did you call find_all() when you meant to call find()?" % key
can anyone help me out with it?
I worked around yesterday, and I found a way, which is working pretty well. Here is the codes:
theme = soup.select('table:has(font:contains("there"))')
for item in theme:
for new in item.select('tbody:has(.threadpages [src="images/new.gif"]), '
'tbody:has(.threadpages [src="images/new1.gif"]), '
'tbody:has(.threadpages [src="images/new2.gif"])'):
and then whatever codes follow...
Been trying to get this to work for a couple of hours now.
its my first real python project so yeh, Would love some help.
HTML
<input type="button" id="lyca_cart_newsim_button1" value="FORTSÆT" class="et_pb_more_button et_pb_button lyca_cart_topup_summary" onclick="nc_newsim_open_tab2('payment','sid','tid')">
xpath
//*[#id='lyca_cart_newsim_button1']
This produces error element "not interactable"
driver.find_element_by_xpath("//*[#id='lyca_cart_newsim_button1']").click()
This produces no errors but does not click the button
element = driver.find_element_by_xpath("//*[#id='lyca_cart_newsim_button1']")
driver.execute_script("arguments[0].click();", element)
This times out
WebDriverWait(driver, 10).until(EC.element_to_be_clickable
First one give element not interactable
and the second one gives no errors.
I am using this at a earlier point in the code and its working fine there,
element = driver.find_element_by_xpath("//*[#id='lyca_cart_newsim_button1']")
driver.execute_script("arguments[0].click();", element)
I've never had good luck using the expected conditions from selenium especially waiting for an element to be clickable. What I've done whether or not it's best, but it has worked is to have a loop attempt to click and to keep trying within a certain amount of time. Here is what I use in C#:
int timeToTryMilliseconds = 5000;
bool timeNotExpired = true;
Stopwatch sw = new Stopwatch();
sw.Start();
while (timeNotExpired)
{
try
{
driver.FindElement(By.XPath("//*[#id='lyca_cart_newsim_button1']").click()
break;
}
catch
{
// Half second wait, so it's not polling constantly
System.Threading.Thread.Sleep(500);
timeNotExpired = timeToTryMilliseconds > sw.ElapsedMilliseconds;
}
}
If there is a better way, I'd love to use it.
Please confirm either the button is in iframe tag. if it is in iframe you must to switch to iframe
if it is not in iframe then try using below code it might work
driver.execute_script("$('#lyca_cart_newsim_button1').click()");
I am new to selenium. Searching a web site, I get 10 results for each page. Those results are shown as lists (li tags) on the page and each list contains the same attributes. When my conditions are met, I go to another related web page and get desired content. However, when my code keeps looping for the lists, it fails to find the same attributes for the others. Here is my code:
p_url = "https://www.linkedin.com/vsearch/f?keywords=BARCO%2BNV%2Bkortrijk&pt=people&page_num=5"
driver.get(p_url)
time.sleep(5)
results = driver.find_element_by_id("results-container")
employees = results.find_elements_by_tag_name('li')
#emp_list = []
#for i in range(len(employees)):
# emp_list.append(employees[i])
for emp in employees:
try:
main_emp = emp.find_element_by_css_selector("a.title.main-headline")
name = emp.find_element_by_css_selector("a.title.main-headline").text
href = main_emp.get_attribute("href")
if name != "LinkedIn Member":
location = emp.find_element_by_class_name("demographic").text
href = main_emp.get_attribute("href")
print(href)
print(location)
driver.get(href)
exp = driver.find_element_by_id("background-experience")
amkk = exp.find_elements_by_class_name("editable-item")
for amk in amkk:
him = amk.find_element_by_tag_name("header").text
him2 = amk.find_element_by_class_name("experience-date-locale").text
if '\n' in him:
a = him.split('\n')
print(a[0])
print(a[1])
print(him2)
except Exception as exc:
print(exc)
continue
In this code the line main_emp = emp.find_element_by_css_selector("a.title.main-headline") stop working after it works for the first time. As a result I got an error of Message: stale element reference: element is not attached to the page document
From stackoverflow questions I saw that some say the content is removed from DOM structure and from another post someone suggested to fill a list with the results. Here what I have tried emp_list = []
for i in range(len(employees)):
emp_list.append(employees[i]) , however, it also did not work out.
How can I overcome this?
The selector you are using is wrong. You are getting the results using the results-container id. This works fine, but the collecting the elements form this is not working. It is returning more elements than just the employees (I'm not quite sure why).
If you change you selectors to this single selector you will get just the employees and no other unwanted elements.
employees = results.find_elements_by_css_selector("ol[id='results']>li")
Edit
Since you are opening the employees and losing the list of elements you might want to try opening the employee in a new tab, perform your actions here and close the tab afterwards.
Example:
for emp in employees:
try:
main_emp = emp.find_element_by_css_selector("a.title.main-headline")
# Do stuff you need...
# Open employee in new tab (make sure Keys is imported)
main_emp.send_keys(Keys.CONTROL + 't')
# Focus on new tab
driver.switch_to_window(d.window_handles[1])
# Do stuff inside the employee page
# Close the tab you opened
driver.close()
# Switch back to the first tab
driver.switch_to_window(d.window_handles[0])
Note: For OSX you should use main_emp.send_keys(Keys.COMMAND + 't')