I am trying to figure out how to use selenium to perform a next page click on a news release page. Here is my code that will go to the correct website and perform a search to obtain the correct news release articles topic page. This site is configured so that to see every news release on page 1 after the search has executed you must also select the More News Results button at the bottom of the page. I am able to get to the full page 1 news section without problem. Here is my code that performs the search and page clicks.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
browser = webdriver.Chrome()
browser.get('http://www.businesswire.com/portal/site/home/')
search_box_element = browser.find_element_by_id('bw-search-input')
search_box_element.clear()
search_box_element.send_keys('biotechnology')
search_box_element.send_keys(Keys.ENTER)
search_box_element_two = browser.find_element_by_id('more-news-results')
search_box_element_two.click()
That part of the code works fine but I want to be able to click on the next button to move to page 2 and then page 3 and so on. Here is the code that I thought would work but it doesn't:
next_page_click_element = browser.find_element_by_class_name("bw-paging-next")
next_page_click_element.click()
This portion of the code throws an error:
selenium.common.exceptions.ElementNotVisibleException: Message:
element not visible
I have also tried using
next_page_click_element = browser.find_element_by_xpath('//*[#id="more-news-pagination"]/div/div[1]/div/a')
but got the same error message. I have also tried using a wait by adding these lines of codes just before the next_page_click_element section.
element_present = EC.presence_of_element_located((By.ID, "bw-paging-next"))
WebDriverWait(browser, 10).until(element_present)
While this does cause the program to wait it returns this error message:
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Any suggestions on how to resolve this issue would be greatly appreciated.
Here is the Answer to your Question:
A few words about the solution-
As the Page 2,Page 3,Next etc elements are at the bottom of the page, you have to scroll down to bring those elements within the Viewport.
Once you scroll down you may consider to induce some ExplicitWait for the elements to be located properly on the HTML DOM.
Here is your own code with some simple tweaks which will finally scroll to the bottom of the page and click on Next link:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
browser=webdriver.Chrome("C:\\Utility\\BrowserDrivers\\chromedriver.exe")
browser.get('http://www.businesswire.com/portal/site/home/')
search_box_element = browser.find_element_by_id('bw-search-input')
search_box_element.clear()
search_box_element.send_keys('biotechnology')
search_box_element.send_keys(Keys.ENTER)
search_box_element_two = browser.find_element_by_id('more-news-results')
search_box_element_two.click()
last_height = browser.execute_script("return document.body.scrollHeight")
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
next_page_click_element = WebDriverWait(browser, 10).until(
EC.presence_of_element_located((By.XPATH, "//div[#id='more-news-pagination']/div/div/div/a[text()='Next']"))
)
next_page_click_element.click()
Let me know if this Answers your Question.
Related
I am trying to locate the input “to” field on this webpage: Link here
(sorry about having to login to see it)
My code:
name_field = driver.find_element_by_xpath("//form[#id='compose-message']/div[6]/div/div/div[2]/div/div/textarea")
But that gives the following error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//form[#id='compose-message']/div[6]/div/div/div[2]/div/div/textarea"}
(Session info: chrome=93.0.4577.63)
For some reason this page is the only page that has given me any issue.
Most of the textarea web elements are wrapped inside iframes.
Using explicit waits, does give your script a stability that it need.
Also prefer to use css selector over xpath.
In Selenium, to access elements which are inside of iframe, we need to switch the focus of webdriver to iframe. Like below
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, "iframe[src^='/message/compose/']")))
and once you are inside this iframe you can interact with to input web element. Like below
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.NAME, "to"))).send_keys('something here')
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[text()='message']/following-sibling::div/descendant::textarea"))).send_keys('Some message')
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Also, remember to switch to default content when you are done with iframe.
driver.switch_to.default_content()
In order to work with elements which are outside of this iframe.
The Element you are trying to find is in an iframe. Was able to find and send text to the elements with below code.
from selenium import webdriver
from selenium.webdriver import ChromeOptions
import time
opt = ChromeOptions()
opt.add_argument("--user-data-dir=C:\\Users\\*****\\AppData\\Local\\Google\\Chrome\\User Data")
driver = webdriver.Chrome(executable_path="path to chromedriver.exe",options=opt)
driver.maximize_window()
driver.implicitly_wait(10)
driver.get("https://www.reddit.com/message/compose/")
driver.switch_to.frame(driver.find_element_by_xpath("//iframe[#class='saPujbGMyXRwqISHcmJH9']"))
driver.find_element_by_xpath("//input[#name='to']").send_keys("Sample Text")
driver.find_element_by_xpath("//div[#class='usertext']//textarea").send_keys("Sample Text")
time.sleep(5)
driver.quit()
First of all that element is inside an iframe. To access it you need to switch to that iframe. Also, your locator are not looking good.
Please try the following code:
wait = WebDriverWait(driver, 20)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(#src,'message')]")))
name_field = wait.until(EC.visibility_of_element_located((By.XPATH, "//input[#name='to']")))
To use the above expected conditions you will need the following imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
use command CSS selector instead of XPath such as use below code.
yourwebsriver.find_elements_by_css_selector("textarea[name=text]").send_keys('something here')
This is the website "https://agta.org/directory/", no matter what i tried i was not able to click the search button, i gave a delay clicked the button manually hoping for the script to pick up elements at the next stage, but it did not worked as well, "find_element_by_xpath" is returning empty objects. I can't understand why is it happening that way when the x path is 100% accurate. I have used similar script for other site but never had any problem.
For clicking the search button i used java script, action chains as well, still failed.
#using selenium
#search=driver.find_element_by_xpath("//input[#class='DLGButton']").click()
#using js
#driver.execute_script("document.getElementsByClassName('DLGButton')[0].click()")
Here is my detailed code:
options = Options()
options.add_argument("--start-maximized")
driver = webdriver.Chrome(options=options, executable_path=r'D:\chromedriver.exe')
url='https://agta.org/directory/'
driver.get(url)
time.sleep(10)
search=driver.find_element_by_xpath("//input[#class='DLGButton']").click()
time.sleep(10)
listings=driver.find_elements_by_xpath("//div[#class='col-md-3']//a")
print(listings)
check=1
while(check==1):
index=1
for l in listings:
l=driver.find_element_by_xpath("(//div[#class='col-md-3']//a)[{}]".format(index)).get_attribute('href').strip()
print(l)
index=index+1
try:
clickNext=find_element_by_xpath("(//*[contains(text(), 'Next')])[1]").click()
time.sleep()
except Exception as e:
print(e)
check=0
Search button is in the frame. So you should switch the frame first.
I modify your code and click Search Button successfully.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.maximize_window()
url='https://agta.org/directory/'
driver.get(url)
time.sleep(5)
# find the frame
frame = driver.find_element_by_xpath("//iframe[#class='directory_iframe']")
driver.switch_to.frame(frame)
# wait element to be clickable
search = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//input[#class='DLGButton']")))
# check button can be found
print(search.get_attribute("value"))
# Scroll the page to show this button (or it will fail to click)
driver.execute_script("arguments[0].scrollIntoView();", search)
search.click()
And you can use Explicit Waits instead of time.sleep()
Ex: To confirm the presence of the element within the DOM Tree.
frame = WebDriverWait(driver, 10).until(EC.presence_of_element_located
((By.XPATH, "//iframe[#class='directory_iframe']")))
Reference: https://selenium-python.readthedocs.io/waits.html
I cannot seem to find a way to click on the right element in order to get the url I am looking for. In essence I am trying to click on the top video in a youtube search (the most highly ranked returned video).
How to resolve ElementNotInteractableException: Element is not visible in Selenium webdriver?
-> This is for Java but it let me in the right direction (knowing I needed to execute JavaScript)
http://www.teachmeselenium.com/2018/04/17/python-selenium-interacting-with-the-browser-executing-javascript-through-javascriptexecutor/
-> This shows me how I should try to execute the javascript in python.
I have also seen countless articles about waits but they do not solve my problem.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
wrds = ["Vivaldi four seasons", "The Beatles twist and shout", "50
cent heat"] #Random list of songs
driver = webdriver.Chrome()
for i in wrds:
driver.get("http://www.youtube.com")
elem = driver.find_element_by_id("search")
elem.send_keys(i)
elem.send_keys(Keys.RETURN)
time.sleep(5)
driver.execute_script("arguments[0].click()",driver.find_element_by_id('video-title')) #THIS CLICKS ON WRONG VIDEO
#elem = driver.find_element_by_id("video-title").click() #THIS FAILS
time.sleep(5)
url = driver.current_url
driver.close()
I get a ElementNotVisibleException: Message: element not interactable error when I do not execute any javascript (even though it has actually worked before it is just no way near robust). When I do execute the javascript it clicks on the wrong videos.
I have tried all types of waits "Explicit" and "Implicit" this did now work.
I am quite sure I need to execute some JavaScript but I don't know how.
You were almost there. You need to induce WebDriverWait for the element to be clickable and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wrds = ["Vivaldi four seasons", "The Beatles twist and shout", "50 cent heat"]
kwrd = ["Vivaldi", "Beatles", "50"]
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\\chromedriver.exe')
for i, j in zip(wrds, kwrd):
driver.get("https://www.youtube.com/")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#search"))).send_keys(i)
driver.find_element_by_css_selector("button.style-scope.ytd-searchbox#search-icon-legacy").click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "h3.title-and-badge.style-scope.ytd-video-renderer a"))).click()
WebDriverWait(driver, 10).until(EC.title_contains(j))
print(driver.current_url)
driver.quit()
That's one of the reasons you should never use JavaScript click, Selenium Webdriver is designed to stimulate as if a real user is able to click. Real user can't click an invisible element in the page but you can click through Javascript. If you search the element by that id video-title, it matches totally 53 videos. But I don't know which one you want to click. You may match that element by some other way(not by id).
I will give you an idea how to click that element but you need to find out the index first before you click.
driver.find_element_by_xpath("(//*[#id='video-title'])[1]").click
If the first one is invisible, then pass 2, [2] then three, figure out which one it's clicking the desired element. Or you may specify the exact element, we may try to locate that element by some other way.
I wrote a program to get some odds from the "Eurobet.it" site using selenium to open the page containing the "ChanceMix". It seems that it works, but when the games take place in more than one date the list I get is incomplete. It contains only those of the first/second date, but not the next ones. I tried various solutions but without results. Can someone help me?
This is my code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import ui
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
link = 'https://www.eurobet.it/it/scommesse/#!/calcio/eu-champions-league/'
driver = webdriver.Chrome()
driver.get(link)
# Find the "ChanceMix" button by xpath and store it as a webdriver object
element = driver.find_element_by_xpath("//a[contains(text(), 'ChanceMix')]")
# Click on the "ChanceMix" button to open the relevant page
element.click()
# Find the list of games
Games = ui.WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.box-sport")))
# Print the list of games
for x in range(0,len(Games)): print(Games[x].text)
I am new to Python 3.X and need to write a script to automate the process downloading US Census data from American Fact Finder. I am using selenium webdriver and the code I have so far is:
driver = webdriver.Chrome(chromePath)
#make driver navigate to American Fact Finder Download Center
driver.get('https://factfinder.census.gov/faces/nav/jsf/pages/download_center.xhtml')
#Make driver click 'Next' to go to Dataset page
driver.find_element_by_xpath('''//*[#id="nextButton"]''').click()
#this is where I need to locate the drop down and select American Community Survey'
On the 'Dataset' page I need to select 'American Community Survey' from the drop down list, but no matter how I try to locate the drop down(xpath, id, value, etc.) running the script returns NoSuchElementException: no such element: Unable to locate element:
I need help with locating the correct element and selecting 'American Community Survey' from the drop down menu.
Thank you!
I was actually able to solve it myself.
For anyone wondering, the error was caused because selenium was trying to find the element before the page had finished loading. The working code is:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
chromePath = r'chromedriver.exe'
driver = webdriver.Chrome(chromePath)
#make driver navigate to American Fact Finder Download Center
driver.get('https://factfinder.census.gov/faces/nav/jsf/pages/download_center.xhtml')
#Make driver click 'Next' to go to Dataset
driver.find_element_by_xpath('''//*[#id="nextButton"]''').click() #needs to be triple quoted
#make driver wait while page loads
timeout = 5
try:
element_present = EC.presence_of_element_located((By.XPATH, '//*[#id="filterDimensionListId'))
WebDriverWait(driver, timeout).until(element_present)
except TimeoutException:
print ("Timed out waiting for page to load")
#Choose ACS 5-year from drop down
driver.find_element_by_xpath('''//*[#id="filterDimensionListId"]/option[2]''').click()
You can try below code to get.
select = Select(driver.find_element_by_id('filterDimensionListId'))
for index in range(len(select.options)):
select = Select(driver.find_element_by_id('filterDimensionListId'))
select.select_by_index(1)
// you can use other selection method.