This script with Selenium in Python 3 helps me to scrape data from a page by clicking the see all button, but it stopped working and I do not know the cause. Could you help me with the code?
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
nav= Service(ChromeDriverManager().install())
driver= webdriver.Chrome(service= nav)
driver.get('https://getdaytrends.com/es/venezuela/#moreTrends')
driver.find_element(By.CLASS_NAME,'btn btn-outline-primary px-5')
driver.execute_script("arguments[0].click();", search)
datos= driver.find('table')
The Traceback is:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".btn btn-outline-primary px-5"}
To click on Ver todo 34 you need to scrollIntoView() the desired element and induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
driver.get("https://getdaytrends.com/es/venezuela/#moreTrends")
element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//span[#class='icon icon-longest-lasting']")))
driver.execute_script("arguments[0].scrollIntoView(true);", element)
WebDriverWait(driver, 7).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='btn btn-outline-primary px-5']"))).click()
Browser Snapshot:
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
References
You can find a couple of relevant discussions on NoSuchElementException in:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
Related
I am trying to download the data from https://projects.propublica.org/nonprofits for my research. When the page is open, a notification window pops up. I tried to use python selenium to close it. My code is as follows,
from selenium.webdriver import Chrome
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = Chrome()
driver.get('https://projects.propublica.org/nonprofits')
driver.find_element(By.XPATH, "/html/body/div[1]/div/div[2]/p[2]/a").click()
I got the error message: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[1]/div/div[2]/p[2]/a"}
(Session info: chrome=99.0.4844.51)
I revised my code as
driver.get('https://projects.propublica.org/nonprofits')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "/html/body/div[1]/div/div[2]/button"))).click()
The error message is TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Stacktrace:
Backtrace:
Ordinal0 [0x005E9943+2595139]
...
Any suggestion to overpass the notification windows is highly appreciated. Thank you.
The element you are trying to click is inside an iframe - you have to switch to that iframe first in order to access this element
You have to add waits to let the elements be loaded before accessing them.
This will work better:
from selenium.webdriver import Chrome
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver = Chrome()
driver.get('https://projects.propublica.org/nonprofits')
wait = WebDriverWait(driver, 20)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe.syndicated-modal")))
wait.until(EC.visibility_of_element_located((By.XPATH, "//a[contains(#href,'newsletter-roadblock')]"))).click()
When finished working inside the iframe you will need to switch to the default content with
driver.switch_to.default_content()
I want to Extract a Specific part in Python Selenium. I have done it with Pyautogui but I want to do it without that is it possible?
https://yopmail.com/
put in inbox tab
jenniferwilks09182
I want to extract exactly this code it is in a separate div
The element with the code is within an <iframe> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be visible.
You can use either of the following Locator Strategies:
Using XPATH and following:
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='ifmail']")))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[.='Your code is:']//following::div[1]"))).text)
Using XPATH and following-sibling:
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='ifmail']")))
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[.='Your code is:']//following-sibling::div[1]"))).text)
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Reference
You can find a couple of relevant discussions in:
Switch to an iframe through Selenium and python
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
I'm trying to access some elements inside of iframe but withour success. Basically, I found the iframe, switch to it, but I can get the element inside of it. All elements that I try, I always get a message saying that the element was not found.
Could you please help?
HTML:
You can use the below code to switch to frame and then interact with the desired element :-
wait = WebDriverWait(driver, 20)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[contains(#sandbox,'allow-popups-to-escape-sandbox')]")))
ele = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.analytics-ui-application")))
#ele is a web element so you can trigger .click, .text or any other web element methods on it.
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
As the elements are within an <iframe> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"frame-router[activeclient='analyticsUi'] > iframe[sandbox]")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "css_clickable_element"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//frame-router[#activeclient='analyticsUi']/iframe[#sandbox]")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "xpath_clickable_element"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Reference
You can find a couple of relevant discussions in:
Ways to deal with #document under iframe
Switch to an iframe through Selenium and python
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element while trying to click Next button with selenium
selenium in python : NoSuchElementException: Message: no such element: Unable to locate element
Here my code:
from selenium import webdriver
from time import sleep
browser=webdriver.Chrome(r"C:\Users\Desktop\chromedriver.exe")
browser.maximize_window()
browser.get("https://www.youtube.com/watch?v=3yjO6yfHLcU&ab_channel=TRT%C4%B0zleTRT%C4%B0zleDo%C4%9Fruland%C4%B1")
browser.find_element_by_class_name("ytp-play-button ytp-button").click()
sleep(2)
I can not able to play videos on YouTube with selenium and python. How can I do that?
This is the Error :
NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":".ytp-play-button ytp-button"}
You are missing a wait / delay there.
Immediately after browser.get("the_url) the page is still not loaded, it takes some time, so the element you are trying to click is still not there.
You have to add some delay.
The correct way to do this is to add explicit wait, like this:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(browser, 20).until(
EC.element_to_be_clickable((By.CSS_SELECTOR, ".ytp-play-button ytp-button")))
element.click();
I am trying to access the sign in button on the url as shown in the code below. I have verified the content of the url as well as the href. They are both consistent with what appears using inspect element dev tool.
But on clicking the extracted element I get the error:
Message: element not interactable
I have no idea why is this occurring.
Kindly help me solve this issue
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
# setup the browser
browser = webdriver.Chrome('./chromedriver')
browser.get('https://libraries.usc.edu/')
browser.maximize_window()
# access the relevant a tag after inspecting it in dev tool inspect element
a_tag_elt = WebDriverWait(browser, 10).until(lambda browser :
browser.find_element_by_css_selector('div.site-header__signin a'))
# sanity check by printing out the details
print(type(a_tag_elt))
print(a_tag_elt.get_attribute('href'), a_tag_elt.get_attribute('innerHTML'))
# produces Message: element not interactable error
a_tag_elt.click()
# quit the browser
browser.quit()
To click on Sign In link Induce WebDriverWait() and element_to_be_clickable() and following css selector.
WebDriverWait(browser,10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"a.main-navigation__navbar>.main-navigation__navbar-text"))).click()
Or following xpath.
WebDriverWait(browser,5).until(EC.element_to_be_clickable((By.XPATH,"//span[text()='Sign In']"))).click()
You need to import below libraries.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
To click on the element with text as Sign In you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.main-navigation__navbar span"))).click()
Using XPATH:
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='main-navigation__navbar ']//span"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC