I'm recently trying to learn Selenium and found a website that just ignores my attempts to find particular element by ID, name or xpath. The website is here:
https://www.creditview.pl/PL/Creditview.htm
I am trying to select first text window, the one labeled Uzytkownik, the code for it goes like that:
I am trying to find it using several methods:
from selenium import webdriver
browser = webdriver.Chrome()
site = "https://www.creditview.pl/pl/creditview.htm"
browser.get(site)
login_txt = browser.find_element_by_xpath(r"/html//input[#id='ud_username']")
login_txt2 = browser.find_element_by_id("ud_username")
login_txt3 = browser.find_element_by_name("ud_username")
No matter what I try I keep getting:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element:
as if the element wasn't there at all.
I have suspected that the little frame containing the field might be an iframe and tried to switch to various elements with no luck. Also tried to check if the element isn't somehow obscured to my code (hidden element). Nothing seems to work, or I am making some newbie mistake and the answer is right in front of me. Finally I was able to select other element on the site and used several TAB keys to move cursor to desired position, but is feels like cheating.
Can someone please point show me how to find the element ? I literally can't sleep because of this issue :)
Given that your element is there, you still need to wait for your element to be loaded/visible/clickable etc. You can do that using selenium's expected conditions (EC).
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
my_XPATH = r"/html//input[#id='ud_username']"
wait_time = 10 # Define maximum time to wait in seconds
driver = webdriver.Chrome()
site = "https://www.creditview.pl/pl/creditview.htm"
driver.get(site)
try:
my_element = driver.WebDriverWait(driver, wait_time).until(EC.presence_of_element_located(By.XPATH,my_XPATH))
except:
print ("element not found after %d seconds" % (wait_time))
Related
So I get the messages from this line:
<pre class="_3Gy8WZD53wWAE41lr57by3 ">Sleep</pre>
My code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
PATH = 'C:\\Users\\User\\Desktop\\chromedriver.exe'
driver = webdriver.Chrome(PATH)
driver.get('https://www.reddit.com')
time.sleep(80) # TIME TO LOGIN IN
search = driver.find_element_by_class_name('_3Gy8WZD53wWAE41lr57by3 ')
print(driver.find_element_by_xpath(".//pre").text) # *LET'S CALL THIS 'S'*
And everything works, kinda. When I print: 's' it prints out the last message from that chat.
Note that whenever someone enters a message, it will be under the variable(class): '_3Gy8WZD53wWAE41lr57by3 '
My goal is to print out the first message from the that chat.
I had to edit it twice because of some mistakes that I had made
I would suggest 2 changes to your code which'll save you major frustration:
Avoid explicit sleep calls, instead, wait for presence of elements. This will allow your program to wait as little time as possible for the page you're trying to load.
Utilize css selectors instead of xpath --> you have much finer control over accessing elements, plus, your code becomes more robust and flexible.
In terms of execution, here's how that looks:
Wait up to 80 seconds for login:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
# Get the page, now the user will need to log in
driver.get('https://www.reddit.com')
# Wait until the page is loaded, up to 80 seconds
try:
element = WebDriverWait(driver, 80).until(
EC.presence_of_element_located((By. CSS_SELECTOR, "pre. _3Gy8WZD53wWAE41lr57by3"))
)
except TimeoutException:
print("You didn't log in, shutting down program")
driver.quit()
# continue as normal here
Utilize css selectors to find your messages:
# I personally like to always use the plural form of this function
# since, if it fails, it returns an empty list. The single form of
# this function results in an error if no results are found
# NOTE: utilize reddit's class for comments, you may need to change the css selector
all_messages = driver.find_elements_by_css_selector('pre._3Gy8WZD53wWAE41lr57by3')
# You can now access the first and last elements from this:
first_message = all_messages[0].text
last_message = all_messages[-1].text
# Alternatively, if you are concerned about memory usage from potentially large
# lists of messages, use css selector 'nth-of-type'
# NOTE: accessing first instance of list of the list exists allows None
# if no results are found
first_message = driver.find_elements_by_css_selector('pre._3Gy8WZD53wWAE41lr57by3:first-of-type')
first_message = first_message[0] if first_message else None
last_message = driver.find_elements_by_css_selector('pre._3Gy8WZD53wWAE41lr57by3:last-of-type')
last_message = last_message[0] if last_message else None
I hope this provides an immediate solution but also some fundamentals how to optimize your web scraping moving forward.
I'm facing a situation where a modal sometimes has a desired button for logging into Facebook and sometimes I have to click 'More Options' to get to the Facebook button. Another hurdle is that the xpath of the more_options_btn is the same as one called 'Trouble logging in?' so I added a [contains(text(),"More Options")], but am unsure of the syntax here and cannot find the right approach in the docs. In any case it is throwing an error, which might be caused by the fact that the button element does not have text, instead it is nested like this button > span > text
Please keep in mind that I want to make my solution as dynamic and robust as possible, but don't have a lot of experience with Selenium yet.
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get('http://localhost:3000/')
try:
fb_btn = driver.find_element_by_xpath(
'//*[#id="modal-manager"]/div/div/div/div/div[3]/span/div[2]/button')
fb_btn.click()
except:
more_options_btn = driver.find_element_by_xpath(
'//*[#id="modal-manager"]/div/div/div/div/div[3]/span/button[contains(text(),"More Options")]')
more_options_btn.click()
fb_btn = driver.find_element_by_xpath(
'//*[#id="modal-manager"]/div/div/div/div/div[3]/span/div[2]/button')
fb_btn.click()
Try to change xpath to:
//*[#id="modal-manager"]/div/div/div/div/div[3]/span/button[contains(.,"More Options")]
It can find the children node of 'button' contains has the text.
You can use OR operator in xpath | to match your xpath with multiple condition. I would recommend to use Relative xpath instead of Absolute one.
As you xpath are too lengthy so it would look like this with OR condition
//*[#id="modal-manager"]/div/div/div/div/div[3]/span/div[2]/button | //*[#id="modal-manager"]/div/div/div/div/div[3]/span/button[contains(text(),"More Options")]
With a combination from the input above I managed to get it to work. I got rid of a lot of the long xpath. This has the advantage that my code will still work if the order of the buttons change. It will however still break if the text of the buttons changes, so it is a compromise. Here is the result:
from time import sleep
from selenium import webdriver
from selenium.common.exceptions import NoSuchElementException
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get('http://localhost:3000/')
sleep(5)
try:
fb_btn = driver.find_element_by_xpath(
"//*/button[contains(.,'Log in with Facebook')]").click()
except NoSuchElementException:
more_options_btn = driver.find_element_by_xpath(
"//*[contains(text(), 'More Options')]")
more_options_btn.click()
fb_btn = driver.find_element_by_xpath(
'//*/button[contains(.,"Log in with Facebook")]')
fb_btn.click()
I am unable to click on the 'Search photos' button on flickr (image below including the html).
I have tried the following:
sp = browser.find_element_by_partial_link_text('/search/?text=tennis%20shoes')
sp.click()
sp = browser.find_element_by_name('Select photos')
sp.click()
searchPhotos = browser.find_element_by_class_name('Search photos')
searchPhotos.click()
browser.find_element_by_xpath("//class[#name='Search photos']").click()
But none of them seem to work. I am learning how to do this, including how to use xpath, so maybe I am not using it correctly. Any advice to point me in the right direction?
EDIT: full section of code to answer comment below:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
profile = webdriver.FirefoxProfile()
profile.set_preference("browser.download.folderList", 2)
profile.set_preference("browser.download.manager.showWhenStarting", False)
profile.set_preference("browser.download.dir", '/Users/home/Box/Temp-to delete')
profile.set_preference("browser.helperApps.neverAsk.saveToDisk", 'png/jpg')
browser = webdriver.Firefox(firefox_profile=profile, executable_path='/usr/local/bin/geckodriver')
browser.get('https://www.flickr.com/')
searchBar = browser.find_element_by_css_selector('#search-field')
searchBar.send_keys(searchTerm)
browser.find_element_by_xpath(".//*[#data-track='autosuggestNavigate_searchPhotos']").click()
Using firefox 72.0.2 (64-bit), python3, geckodriver v0.26.0
The path used in your XPath won't work. Try this one .//*[#data-track='autosuggestNavigate_searchPhotos'].
The .// tells Selenium so search anywhere in the DOM. The asterisk (*) will make Selenium to look for any element (no matter if it is div, li or any other HTML tag). Then it will check which element has the data-track attribute, with value autosuggestNavigate_searchPhotos. Since there is only one element like this, we are fine.
I advise to read more about XPath and train a bit, you may start here
Solved it Just had to hit ENTER for the photos results page to show. Here is the single line of code I changed:
searchBar.send_keys(searchTerm, Keys.ENTER)
I cannot seem to find a way to click on the right element in order to get the url I am looking for. In essence I am trying to click on the top video in a youtube search (the most highly ranked returned video).
How to resolve ElementNotInteractableException: Element is not visible in Selenium webdriver?
-> This is for Java but it let me in the right direction (knowing I needed to execute JavaScript)
http://www.teachmeselenium.com/2018/04/17/python-selenium-interacting-with-the-browser-executing-javascript-through-javascriptexecutor/
-> This shows me how I should try to execute the javascript in python.
I have also seen countless articles about waits but they do not solve my problem.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
wrds = ["Vivaldi four seasons", "The Beatles twist and shout", "50
cent heat"] #Random list of songs
driver = webdriver.Chrome()
for i in wrds:
driver.get("http://www.youtube.com")
elem = driver.find_element_by_id("search")
elem.send_keys(i)
elem.send_keys(Keys.RETURN)
time.sleep(5)
driver.execute_script("arguments[0].click()",driver.find_element_by_id('video-title')) #THIS CLICKS ON WRONG VIDEO
#elem = driver.find_element_by_id("video-title").click() #THIS FAILS
time.sleep(5)
url = driver.current_url
driver.close()
I get a ElementNotVisibleException: Message: element not interactable error when I do not execute any javascript (even though it has actually worked before it is just no way near robust). When I do execute the javascript it clicks on the wrong videos.
I have tried all types of waits "Explicit" and "Implicit" this did now work.
I am quite sure I need to execute some JavaScript but I don't know how.
You were almost there. You need to induce WebDriverWait for the element to be clickable and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wrds = ["Vivaldi four seasons", "The Beatles twist and shout", "50 cent heat"]
kwrd = ["Vivaldi", "Beatles", "50"]
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\\chromedriver.exe')
for i, j in zip(wrds, kwrd):
driver.get("https://www.youtube.com/")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#search"))).send_keys(i)
driver.find_element_by_css_selector("button.style-scope.ytd-searchbox#search-icon-legacy").click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "h3.title-and-badge.style-scope.ytd-video-renderer a"))).click()
WebDriverWait(driver, 10).until(EC.title_contains(j))
print(driver.current_url)
driver.quit()
That's one of the reasons you should never use JavaScript click, Selenium Webdriver is designed to stimulate as if a real user is able to click. Real user can't click an invisible element in the page but you can click through Javascript. If you search the element by that id video-title, it matches totally 53 videos. But I don't know which one you want to click. You may match that element by some other way(not by id).
I will give you an idea how to click that element but you need to find out the index first before you click.
driver.find_element_by_xpath("(//*[#id='video-title'])[1]").click
If the first one is invisible, then pass 2, [2] then three, figure out which one it's clicking the desired element. Or you may specify the exact element, we may try to locate that element by some other way.
I am trying to figure out how to use selenium to perform a next page click on a news release page. Here is my code that will go to the correct website and perform a search to obtain the correct news release articles topic page. This site is configured so that to see every news release on page 1 after the search has executed you must also select the More News Results button at the bottom of the page. I am able to get to the full page 1 news section without problem. Here is my code that performs the search and page clicks.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
browser = webdriver.Chrome()
browser.get('http://www.businesswire.com/portal/site/home/')
search_box_element = browser.find_element_by_id('bw-search-input')
search_box_element.clear()
search_box_element.send_keys('biotechnology')
search_box_element.send_keys(Keys.ENTER)
search_box_element_two = browser.find_element_by_id('more-news-results')
search_box_element_two.click()
That part of the code works fine but I want to be able to click on the next button to move to page 2 and then page 3 and so on. Here is the code that I thought would work but it doesn't:
next_page_click_element = browser.find_element_by_class_name("bw-paging-next")
next_page_click_element.click()
This portion of the code throws an error:
selenium.common.exceptions.ElementNotVisibleException: Message:
element not visible
I have also tried using
next_page_click_element = browser.find_element_by_xpath('//*[#id="more-news-pagination"]/div/div[1]/div/a')
but got the same error message. I have also tried using a wait by adding these lines of codes just before the next_page_click_element section.
element_present = EC.presence_of_element_located((By.ID, "bw-paging-next"))
WebDriverWait(browser, 10).until(element_present)
While this does cause the program to wait it returns this error message:
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Any suggestions on how to resolve this issue would be greatly appreciated.
Here is the Answer to your Question:
A few words about the solution-
As the Page 2,Page 3,Next etc elements are at the bottom of the page, you have to scroll down to bring those elements within the Viewport.
Once you scroll down you may consider to induce some ExplicitWait for the elements to be located properly on the HTML DOM.
Here is your own code with some simple tweaks which will finally scroll to the bottom of the page and click on Next link:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
browser=webdriver.Chrome("C:\\Utility\\BrowserDrivers\\chromedriver.exe")
browser.get('http://www.businesswire.com/portal/site/home/')
search_box_element = browser.find_element_by_id('bw-search-input')
search_box_element.clear()
search_box_element.send_keys('biotechnology')
search_box_element.send_keys(Keys.ENTER)
search_box_element_two = browser.find_element_by_id('more-news-results')
search_box_element_two.click()
last_height = browser.execute_script("return document.body.scrollHeight")
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
next_page_click_element = WebDriverWait(browser, 10).until(
EC.presence_of_element_located((By.XPATH, "//div[#id='more-news-pagination']/div/div/div/a[text()='Next']"))
)
next_page_click_element.click()
Let me know if this Answers your Question.