I am trying to click on the "Next" button at the bottom of a BusinessWire web page. I have some code that goes from the BusinessWire homepage to my desired search results page. I want to be able to click on that page's "Next" button, but I get the error message telling me that the "Next" element is not clickable at point(X,Y). The next button is at the bottom of the window. For some reason the
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
command is not scrolling as I would expect. Because the window is not scrolling as expected the element is not visible to be clicked (at least I believe that is the problem). I use this same command twice earlier in the code and it works fine in those two instances. Any help on getting the window to scroll would be greatly appreciated. Here is my code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
def BusinessWire():
browser = webdriver.Chrome()
browser.get('http://www.businesswire.com/portal/site/home/')
search_box_element = browser.find_element_by_id('bw-search-
input')
search_box_element.clear()
search_box_element.send_keys('biotechnology')
search_box_element.send_keys(Keys.ENTER)
browser.execute_script("window.scrollTo(0,
document.body.scrollHeight);")
search_box_element_two = browser.find_element_by_id('more-news-
results')
search_box_element_two.click()
browser.execute_script("window.scrollTo(0,
document.body.scrollHeight);")
time.sleep(5)
next_page_click_element = WebDriverWait(browser, 10).until(
EC.presence_of_element_located((By.XPATH, '//*[#id="more-news-pagination"]/div/div[1]/div/a'))
)
next_page_click_element.click()
Try using scrollIntoView of the element you want to scroll to, sorry this is in Java, but should work the same in python:
driver.manage().window().maximize();
driver.get("http://www.businesswire.com/portal/site/home/");
wait = new WebDriverWait(driver, 10);
driver.findElement(By.id("bw-search-input")).clear();
driver.findElement(By.id("bw-search-input")).sendKeys("biotechnology");
driver.findElement(By.id("bw-search-input")).sendKeys(Keys.ENTER);
WebElement clicklink = driver.findElement(By.id("more-news-results"));
((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", clicklink);
clicklink.click();
Thread.sleep(1000);
WebElement clicknext = wait.until(ExpectedConditions.presenceOfElementLocated(By.xpath("//*[#id=\"more-news-pagination\"]/div/div[1]/div/a")));
((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", clicknext);
clicknext.click();
StackOverFlow won't let me post any more comments? The url is in the code. But the homepage url is not the problem page. The problem page is the search page after the home page. The search page can only be gotten to through a search on the homepage. My code does all this.
Related
I wrote a python script that goes to a site, and interacts with some dropdowns. It works perfectly fine if after I run the script, quickly make the browser instance full screen so that the elements are in view. If I don't do that, I get the error "Element could not be scrolled into view".
Here is my script:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://example.com")
driver.implicitly_wait(5)
yearbtn = driver.find_element("id", "dropdown_year")
yearbtn.click()
year = driver.find_element("css selector", '#dropdown_ul_year li:nth-child(5)')
year.click()
makebtn = driver.find_element("id", "dropdown_make")
makebtn.click()
make = driver.find_element("css selector", '#dropdown_ul_make li:nth-child(2)')
make.click()
modelbtn = driver.find_element("id", "dropdown_model")
modelbtn.click()
model = driver.find_element("css selector", '#dropdown_ul_model li:nth-child(2)')
model.click()
trimbtn = driver.find_element("id", "dropdown_trim")
trimbtn.click()
trim = driver.find_element("css selector", '#dropdown_ul_trim li:nth-child(2)')
trim.click()
vehicle = driver.find_element("css selector", '#vehiclecontainer > div > p')
vdata = driver.find_element("css selector", '.top-sect .tow-row:nth-child(2)')
print("--------------")
print("Your Vehicle: " + vehicle.text)
print("Vehicle Data: " + vdata.text)
print("--------------")
print("")
driver.close()
Like I said, it works fine if I make the browser full-screen (or manually scroll) so that the elements in question are in view. It finds the element, so what's the issue here? I've tried both Firefox and Chrome.
Generally firefox opens in maximized mode. Incase for specific version of os it doesn't, the best practice is to open it in maximized mode as follows:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("start-maximized")
driver = webdriver.Firefox(options=options)
However in somecases the desired element may not get rendered within the due coarse, so you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:
vehicle = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#vehiclecontainer > div > p")))
vdata = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".top-sect .tow-row:nth-child(2)")))
print("--------------")
print("Your Vehicle: " + vehicle.text)
print("Vehicle Data: " + vdata.text)
print("--------------")
print("")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python
I'm not sure what all is happening on the page in question. I use a generic method to take care of common actions like click(), find(), etc. that contain a wait, scroll, and whatever else is useful for that action. I've put an example below. It waits for the element to be clickable (always a best practice), scrolls to the element, and then clicks it. You may want to tweak it to your specific use but it will get you pointed in the right direction.
def click(self, locator):
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable(locator))
driver.execute_script("arguments[0].scrollIntoView();", element)
element.click()
I would suggest you try this and then convert all of your clicks to use this method and see if that helps, e.g.
click((By.ID, "dropdown_year"))
or
locator = (By.ID, "dropdown_year")
click(locator)
Actually, i have a script with python 3.10, selenium, undetected-chromedriver and i want to click on two buttons on this webpage:
https://keepa.com/#!
Buttons are french flag, and ".fr" here:
And html code for these two buttons are here:
-first highlighted is flag fr
-second is country ".fr"
I've tested somes part of script, but not work actually:
For language flag fr:
driver.find_element(By.XPATH, "(//span[#id='lang_fr'])").click()
driver.find_element(By.ID,"lang_fr").click()
For country .fr:
driver.find_element(By.XPATH, "(//span[#setting='4'])").click()
driver.find_element(By.XPATH, "(.//span[contains(text(), '.fr')])").click()
I have somes errors on results, "element not found", or this:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
Anyone can help me or have an idea where is the problem please?
Thanks for help, bye!
You have to click on default flag first and then there will be 2 button which you can click like below:
Code:
driver_path = r'C:\\Users\\****\\***\\Desktop\\Automation\\chromedriver.exe'
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.get("https://keepa.com/#!")
wait = WebDriverWait(driver, 30)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#currentLanguage .languageMenuText"))).click()
flag = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "img#lang_fr")))
flag.click()
lang = wait.until(EC.visibility_of_element_located((By.XPATH, "//span[text()='.fr']")))
lang.click()
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can try the below code it will help you (this is using java though, ).
Actions actions = new Actions(driver);
WebElement ctr;
ctr= driver.findElement(By.xpath("//*[contains(text(),'YOUR DESIRED TEXT')]"));
Thread.sleep(1000);
actions.click(spatial).perform();
using imports:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
using this def to navigate to the exact iFrame that the button is under :
def changeIframetoManagement():
wait2 = WebDriverWait(driver, 10)
wait2.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "moduleManagement")))
my attempt to achieve that task :
changeIframetoManagement()
driver.find_element_by_css_selector("body > div.bootbox.modal.fade.show-order-info-bootbox-
container.in > div > div > div.modal-body > button").click()
im also tried to reach by xPath like this
driver.find_element_by_xpath("//html/body/div[5]/div/div/div[1]/button").click()
that is the last i frame i was in
def changeIframetoOrders():
wait3 = WebDriverWait(driver, 10)
wait3.until(EC.frame_to_be_available_and_switch_to_it((By.NAME,
"orderInfo")))
After switching into the iframe try this:
driver.find_element_by_css_selector("div.modal-dialog button.bootbox-close-button").click()
UPD
According to your last comments, since you previously switched into iframe located by orderInfo name, before switching into iframe located by moduleManagement id you have yo switch to the default content, like this:
changeIframetoOrders()
driver.switch_to.default_content()
changeIframetoManagement()
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.modal-dialog button.bootbox-close-button"))).click()
See if this xpath works:-
driver.find_element_by_xpath(".//button[contains(#class,'close-button')]").click()
The x button is in moduleManagement iframe.
and the goal is to click on it using Selenium.
as per OP
wait3.until(EC.frame_to_be_available_and_switch_to_it((By.NAME,
"orderInfo")))
looks like he was in a different iframe orderInfo.
So I would suggest whenever you are done with any iframe stuff, always switch the control back to default content like below :
driver.switch_to.default_content()
Now again you will have to switch to moduleManagement iframe like below :
wait = WebDriverWait(driver, 10)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.ID, "moduleManagement")))
and click on x button like this :
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[contains(#class, ''bootbox-close-button) and text()='x']"))).click()
I want to select an option that displays only after you've clicked on the dropdown (see attached image). I have been able to click on the dropdown to get the list, but have not been able to figure out how to click on an option, say option 1, 'Last Day' after the list comes into picture.
Here's what I have so far:
from selenium import webdriver
binary = FirefoxBinary('C:\\Program Files\\Firefox Developer Edition\\firefox.exe')
cap = DesiredCapabilities().FIREFOX
cap["marionette"] = True
url='https://www.glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=false&clickSource=searchBtn&typedKeyword=data+sc&sc.keyword=data+scientist&locT=C&locId=1154532&jobType='
driver = webdriver.Firefox(firefox_binary=binary, capabilities=cap, executable_path=GeckoDriverManager().install())
driver.get(url=url)
driver.implicitly_wait(10)
driver.maximize_window()
# clicking on dropdown
d = driver.find_element_by_id('filter_fromAge')
d.click()
I also tried using the following code (found on another SO answer) but it did not work either:
element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"ul#css-1dv4b0s ew8xong0")))
I'm new to web scraping and not really familiar with XPATHs and how to deal with actions. Help appreciated!
You can use Java script to click on your element. As element is present in DOM but its only appears when click on drop down, so normal click method mau work or may not. But with JS it will always click. Can use below code:
day = driver.find_element_by_xpath("//span[contains(text(),'Last Day')]") #Identify your element
driver.execute_script("arguments[0].click();", day) # CLick it with help of JS
out Put:
You can try the following approach:
driver.get('https://www.glassdoor.com/Job/boston-data-scientist-jobs-SRCH_IL.0,6_IC1154532_KO7,21.htm')
driver.maximize_window()
expand_element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.ID, 'filter_fromAge')))
expand_element.click()
target_text = 'Last 3 Days'
target_element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//ul[#class="css-1dv4b0s ew8xong0"]/li/span[text()="{}"]'.format(target_text))))
target_element.click()
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//div[#id="filter_fromAge"]/span[text()="{}"]'.format(target_text))))
To select the option with text as Last Day that displays only after clicking on the dropdown you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategy:
Using XPATH:
driver.get('https://www.glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=false&clickSource=searchBtn&typedKeyword=data+sc&sc.keyword=data+scientist&locT=C&locId=1154532&jobType=')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#filter_fromAge>span"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#id='PrimaryDropdown']/ul//li//span[#class='label' and contains(., 'Last Day')]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
I want to enter text into a textarea. The problem is the textarea is only created and shows up in the html after clicking twice on a visual element in the browser.
For this visual element Dubbelklik om je tekst te typen, I cannot find an element in the html that refers to it and perform a double_click(). There are only containers, that either are unclickable elements, or allow clicks, but don't perform an action.
Up until here it works and the visual cue to double click appears slightly below the center:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
driver = webdriver.Chrome()
driver.get("https://www.hallmark.nl/kaarten/verjaardag-man/")
#wait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href*='https://www.hallmark.nl:443/kaarten/verjaardag-man/grappig-m/make-that-the-cat-wise/happy-bursdeej-to-jou-3415094.aspx']"))).click()
wait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href*='https://www.hallmark.nl:443/kaarten/verjaardag-man/grappig-m/hallmark/een-jaguar-voor-je-verjaardag-3346861.aspx']"))).click()
wait(driver, 10).until(EC.element_to_be_clickable((By.ID, "btnShowSizepicker"))).click()
wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Standaard']"))).click()
wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//div[contains(#class, 'showDesktop')]//button[contains(text(),'Binnenkant')]"))).click()
But from then on I can't seem to find anything to make selenium click on the element.
I have tried clicking the things I can locate in the html:
# elem = driver.find_element_by_xpath("//div[contains(#class, 'canvasAnchor')]").click()
This gives an element not interactable error. I have also tried using an offset for a findable element:
elem = driver.find_element_by_class_name("canvas-container")
print(elem.location)
print(elem.size)
action = webdriver.common.action_chains.ActionChains(driver)
action.move_to_element_with_offset(elem, 0.5*elem.location.get('x'), (0.5*elem.location.get('y'))) #should click in the middle of this container
action.double_click()
action.perform()
But for different elements and values of x and y, I can only get it to do either nothing, or click on the contact sidebar (which highlights the word 'Neem').
If I click twice manually, a textarea appears in the html and I can execute send_keys('hello world')
The element where you have to click is actually canvas, so what you have to do is that move cursor to the position where double click is needed and then click there.
I tried this over your code and worked for me:
from selenium.webdriver.common.action_chains import ActionChains
elem = driver.find_element_by_xpath('//*[#class="canvasWrapper active"]//*[#class="canvas-container"]')
ActionChains(driver).move_to_element_with_offset(elem, 70, 60).double_click().perform()
Note:You may have to wait for the canvas element to load properly. I just tried from prompt.
You can read about this mouse action here.
UPDATE:
I noticed that somehow the double click was behaving as single click for me, so changed to this.
from selenium.webdriver.common.action_chains import ActionChains
import time
elem = driver.find_element_by_xpath('//*[#class="canvasWrapper active"]//*[#class="canvas-container"]')
ActionChains(driver).move_to_element_with_offset(elem, 70, 60).click().perform()
time.sleep(1)
ActionChains(driver).move_to_element_with_offset(elem, 70, 60).click().perform()