I am trying web scraping using Python selenium. I am getting the following error: Message: element not interactable (Session info: chrome=78.0.3904.108) I am trying to access option element by id, value or text. It is giving me this error. I am using Python-3. Can someone help to explain where am I going wrong. I used the select tag using xpath and also tried css_selector. The select tag is selected and I can get the output of select tag selected. Here is my code for a better understanding:
Code-1:-
path = r'D:\chromedriver_win32\chromedriver.exe'
browser = webdriver.Chrome(executable_path = path)
website = browser.get("https://publicplansdata.org/resources/download-avs-cafrs/")
el = browser.find_element_by_xpath('//*[#id="ppd-download-state"]/select')
for option in el.find_elements_by_tag_name('option'):
if option.text != None:
option.click()
break
Blockquote
Code-2:-
select_element = Select(browser.find_element_by_xpath('//*[#id="ppd-download-state"]/select'))
# this will print out strings available for selection on select_element, used in visible text below
print(o.value for o in select_element.options)
select_element.select_by_value('AK')
Both codes give the same error how can I select values from drop down from website
Same as question:
Python selenium select element from drop down. Element Not Visible Exception
But the error is different. Tried the methods in comments
State, Plan, and Year:
browser.find_element_by_xpath("//span[text()='State']").click()
browser.find_element_by_xpath("//a[text()='West Virginia']").click()
time.sleep(2) # wait for Plan list to be populated
browser.find_element_by_xpath("//span[text()='Plan']").click()
browser.find_element_by_xpath("//a[text()='West Virginia PERS']").click()
time.sleep(2) # wait for Year list to be populated
browser.find_element_by_xpath("//span[text()='Year']").click()
browser.find_element_by_xpath("//a[text()='2007']").click()
Don't forget to import time
Related
I am new to webscraping & not a developer nor have any html exp. and was trying to pull some details from my account after logging in into the website but getting errors in find_element_by_class_name()
this is the code I have tried:
from selenium import webdriver
driver = webdriver.Chrome('path/chromedriver.exe')
driver.get("https://www.URL.COM")
# logged into account manually & maneuvered to the page manually
driver.find_element_by_class_name('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')
Error
---------------------------------------------------------------------------
NoSuchElementException Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_14516/1270707966.py in <module>
----> 1 driver.find_element_by_class_name('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')
C:\ProgramData\Miniconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in find_element_by_class_name(self, name)
562 element = driver.find_element_by_class_name('foo')
563 """
--> 564 return self.find_element(by=By.CLASS_NAME, value=name)
565
566 def find_elements_by_class_name(self, name):
Also tried
driver.find_element_by_css_selector('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')
From inspect I was able to view this & tried to extract the one highlighted in the image:
class name expect a single class name. where as you are passing multiple class name here
css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt
It won't work.
Instead
remove the spaces and make a CSS selector out of it.
driver.find_element(By.CSS_SELECTOR, ".css-901oao.css-cens5h.r-1khnkhu.r-13awgt0.r-1oke55r.r-1enofrn.r-1wzrnnt")
Also, Please remember find_element_by_class_name have been deprecated in newest selenium. You should use this instead
find_element(By.CLASS_NAME, "class name")
having said this, the locator that you've right now looks brittle in nature. Please use static attribute values.
You could try this xpath
//div[starts-with(#class,'css')]//div[#dir='auto' and contains(#style,'-webkit-line-clamp')]
Please check in the dev tools (Google chrome) if we have unique entry in HTML DOM or not.
xpath that you should check :
//div[starts-with(#class,'css')]//div[#dir='auto' and contains(#style,'-webkit-line-clamp')]
Steps to check:
Press F12 in Chrome -> go to element section -> do a CTRL + F -> then paste the xpath and see, if your desired element is getting highlighted with 1/1 matching node.
I am trying to fill in the forms in the Gmail creation page. But for some reason, my driver can't find the code. I've used all the options, path, id, name, class name, but nothing works
chrome_driver = './chromedriver'
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=chrome_driver)
driver.get('https://www.google.com/intl/nl/gmail/about/#')
try:
print('locating create account button')
create_account_button = driver.find_element_by_class_name('h-c-button')
except:
print("error, couldn't find create account button")
try:
create_account_button.click()
print('navigating to creation page')
except:
print('error navigating to creation page')
time.sleep(15)
first_name_form = driver.find_element_by_class_name('whsOnd zHQkBf')
(the sleep is just temporary, to make sure it loads completely, I know it's not efficient)
here's the link to the Gmail page:
https://accounts.google.com/signup/v2/webcreateaccount?service=mail&continue=https%3A%2F%2Fmail.google.com%2Fmail%2F%3Fpc%3Dtopnav-about-n-en&flowName=GlifWebSignIn&flowEntry=SignUp
This is the error I'm getting:
Exception has occurred: NoSuchElementException
Message: no such element: Unable to locate element: {"method":"css
selector","selector":".whsOnd zHQkBf"}
(Session info: chrome=81.0.4044.129)
Thank you for your help
I found your error and I have the solution for you. Let me first determine to you the problem. When you click on the 'Create New Account' a new window is opening. BUT your bot is still undestanding that you are located in the first window (in the one that you click on the first button in order to create the account). Thus, the bot is trying to see if there is a First Name input. That's why is failing. So, the solution is that you have to change the window that you want to specify. The way you can do it is written inside the code block.
CODE
from selenium import webdriver
import time
path = '/home/avionerman/Documents/stack'
driver = webdriver.Firefox(path)
driver.get('https://www.google.com/intl/nl/gmail/about/#')
try:
print('locating create account button')
create_account_button = driver.find_element_by_class_name('h-c-button')
except:
print("error, couldn't find create account button")
try:
create_account_button.click()
print('navigating to creation page')
except:
print('error navigating to creation page')
time.sleep(15)
# Keeping all the windows into the array named as handles
handles = driver.window_handles
# Keeping the size of the array in order to know how many windows are open
size = len(handles)
# Switch to the second opened window (id:1)
driver.switch_to.window(handles[1])
# Print the title of the current page in order to validate if it's the proper one
print(driver.title)
time.sleep(10)
first_name_input = driver.find_element_by_id('firstName')
first_name_input.click()
first_name_input.send_keys("WhateverYouWant")
last_name_input = driver.find_element_by_id('lastName')
last_name_input.click()
last_name_input.send_keys("WhateverYouWant2")
username_input = driver.find_element_by_id('username')
username_input.click()
username_input.send_keys('somethingAsAUsername')
pswd_input = driver.find_element_by_name('Passwd')
pswd_input.click()
pswd_input.send_keys('whateveryouwant')
pswd_conf_input = driver.find_element_by_name('ConfirmPasswd')
pswd_conf_input.click()
pswd_conf_input.send_keys('whateveryouwant')
time.sleep(20)
So, if you will go to line 21 you will see that I have some comments in order to tell you what these lines (from 21 until 31) are doing.
Also, I inserted all the needed code for you (first name, last name, et.c). You only have to locate the creation button (last one).
Note: Try to use ids in such cases and not class names (when the ids are clear and unique) as I already did for you.
I write parser using Selenium and Python 3.7 for next site - https://www.oddsportal.com/soccer/germany/bundesliga/nurnberg-dortmund-fNa2KmU4/
I'm interested, to get the url, that is generated by JavaScript, using Selenium in Python 3?
I need to get the url for events from the sites from which the data is taken in the table.
For example. It seems to me that the data in the first line (10Bet) is obtained from this page - https://www.10bet.com/sports/football/germany-1-bundesliga/20190218/nurnberg-vs-dortmund/
How can get url to this page?
To get the URL of all the links on a page, you can store all the elements with tagname a in a WebElement list and then you can fetch the href attribute to get the link of each WebElement.
you can refer to the following code for all the link present in hole page :
List<WebElement> links = driver.findElements(By.tagName("a")); //This will store all the link WebElements into a list
for(WebElement ele: links) // This way you can take the Url of each link
{
String url = ele.getAttribute("href"); //To get the link you can use getAttribute() method with "href" as an argument
System.out.println(url);
}
In case you need the particilar link explicitly you need to pass the xpath of the element
WebElement ele = driver.findElements(By.xpath("Pass the xapth of the element"));
and after that you need to do this
String url = ele.getAttribute("href") //to get the url of the particular element
I have also shared the link with you so you can go and check i have highlighted elements in that
let us know if that helped or not
Try the below code, which will prints the required URL's as per your requirement :
from selenium import webdriver
driver = webdriver.Chrome('C:\\NotBackedUp\\chromedriver.exe')
driver.maximize_window()
driver.get('https://www.oddsportal.com/soccer/germany/bundesliga/nurnberg-dortmund-fNa2KmU4/')
# Locators for locating the required URL's
xpath = "//div[#id='odds-data-table']//tbody//tr"
rows = "//div[#id='odds-data-table']//tbody//tr/td[1]"
print("=> Required URL's is/are : ")
# Fetching & Printing the required URL's
for i in range(1, len(driver.find_elements_by_xpath(rows)), 1):
anchor = driver.find_elements_by_xpath(xpath+"["+str(i)+"]/td[2]/a")
if len(anchor) > 0:
print(anchor[0].get_attribute('href'))
else:
a = driver.find_elements_by_xpath("//div[#id='odds-data-table']//tbody//tr["+(str(i+1))+"]/td[1]//a[2]")
if len(a) > 0:
print(a[0].get_attribute('href'))
print('Done...')
I hope it helps...
I am searching elements in my list(one by one) by inputing into searchbar of a website and get apple products name that appeared in search result and printed. However I am getting following exception
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
I know its because of changing of element very fast so I need to add wait like
wait(driver, 10).until(EC.visibility_of_element_located((By.ID, "submitbutton")))
or explictly
Q1. But I don't understand where should I add it? Here is my code. Please help!
Q2. I want to go to all the next pages using but that's not working.
driver.find_element_by_xpath( '//div[#class="no-hover"]/a' ).click()
Earlier exception was raised on submitton button and now at if statement.
That's not what implicit wait is for. Since the page change regularly you can't be sure when the current object inside the variable is still valid.
My suggestion is to run the above code in loop using try except. Something like the following:
for element in mylist:
ok = False
while True:
try:
do_something_useful_while_the_page_can_change(element)
except StaleElementReferenceException:
# retry
continue
else:
# go to next element
break
Where:
def do_something_useful_while_the_page_can_change(element):
searchElement = driver.find_element_by_id("searchbar")
searchElement.send_keys(element)
driver.find_element_by_id("searchbutton").click()
items_count = 0
items = driver.find_elements_by_class_name( 'searchresult' )
for i, item in enumerate( items ):
if 'apple' in item.text:
print ('item.text')
items_count += len( items )
I think what you had was doing too much and can be simplified. You basically need to loop through a list of search terms, myList. Inside that loop you send the search term to the searchbox and click search. Still inside that loop you want to grab all the elements off the page that consist of search results, class='search-result-product-url' but also the text of the element contains 'apple'. The XPath locator I provided should do both so that the collection that is returned all are ones you want to print... so print each. End loop... back to next search term.
for element in mylist:
driver.find_element_by_id("search-input").send_keys(element)
driver.find_element_by_id("button-search").click()
# may need a wait here?
for item in driver.find_elements_by_xpath( "//a[#class='search-result-product-url'][contains(., 'apple')]" ):
print item.text
I am trying to pull out the names of all courses offered by Lynda.com together with the subject so that it appears on my list as '2D Drawing -- Project Soane: Recover a Lost Monument with BIM with Paul F. Aubin'. So I am trying to write a script that will go to each subject on http://www.lynda.com/sitemap/categories and pull out the list of courses. I already managed to get Selenium to go from one subject to another and pull the courses. My only problem is that there is a button 'See X more courses' to see the rest of the courses. Sometimes you have to click it couple of times that´s why I used while loop. But selenium doesn´t seem to execute this click. Does anyone know why?
This is my code:
from selenium import webdriver
url = 'http://www.lynda.com/sitemap/categories'
mydriver = webdriver.Chrome()
mydriver.get(url)
course_list = []
for a in [1,2,3]:
for b in range(1,73):
mydriver.find_element_by_xpath('//*[#id="main-content"]/div[2]/div[3]/div[%d]/ul/li[%d]/a' % (a,b)).click()
while True:
#click the button 'See more results' as long as it´s available
try:
mydriver.find_element_by_xpath('//*[#id="main-content"]/div[1]/div[3]/button').click()
except:
break
subject = mydriver.find_element_by_tag_name('h1') # pull out the subject
courses = mydriver.find_elements_by_tag_name('h3') # pull out the courses
for course in courses:
course_list.append(str(subject.text)+" -- " + str(course.text))
# go back to the initial site
mydriver.get(url)
Scroll to element before clicking:
see_more_results = browser.find_element_by_css_selector('button[class*=see-more-results]')
browser.execute_script('return arguments[0].scrollIntoView()', see_more_results)
see_more_results.click()
One solution how to repeat these actions could be:
def get_number_of_courses():
return len(browser.find_elements_by_css_selector('.course-list > li'))
number_of_courses = get_number_of_courses()
while True:
try:
button = browser.find_element_by_css_selector(CSS_SELECTOR)
browser.execute_script('return arguments[0].scrollIntoView()', button)
button.click()
while True:
new_number_of_courses = get_number_of_courses()
if (new_number_of_courses > number_of_courses):
number_of_courses = new_number_of_courses
break
except:
break
Caveat: it's always better to use build-in explicit wait than while True:
http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp#explicit-waits
The problem is that you're calling a method to find element by class name, but you're passing a xpath. if you're sure this is the correct xpath you'll simply need to change to method to 'find_element_by_xpath'.
A recommendation if you allow: Try to stay away from these long xpaths and go through some tutorials on how to write efficient xpath for example.