I am trying do you Youtube data scraping, but stuck at the first step... accepting the cookie.
I have tried searching the "I agree" text and click but seems another element is obscuring it.
buttons = driver.find_elements_by_xpath("//*[contains(text(), 'I agree')]")
for btn in buttons:
btn.click()
Tried accessing div, seems to do the action but does not really click
buttons = driver.find_element_by_xpath('//div[#class="VfPpkd-dgl2Hf-ppHlrf-sM5MNb"]//button')
buttons.click
any pointer on how to click the button using python + selenium?
button html
https://consent.youtube.com/m?continue=https%3A%2F%2Fwww.youtube.com%2Fresults%3Fsearch_query%3Dpost%2Bdriver&gl=FR&m=0&pc=yt&uxe=23983172&hl=en&src=1
To click on I agree button to accept cookies on YouTube. The button is present inside an iframe and you need to switch the iframe. Use WebDriverWait() and wait for frame available and switch it.
WebDriverWait(driver,20).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"iframe")))
driver.find_element_by_xpath("//*[contains(text(), 'I agree')]").click()
To jump out from frame you need to use
driver.switch_to.default_content()
Please import below libraries
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
This function is working for me..Most of the time it's showing in a form and sometimes you need to click on this particular button. Sometimes it's not showing at all.
bypass_consent(driver):
try:
consent = WebDriverWait(driver, 30).until(EC.element_to_be_clickable(
(By.XPATH, "//input[#type='submit' and #value='I agree']")))
consent.submit()
except:
try:
consent = driver.find_element_by_css_selector(
'button.VfPpkd-LgbsSe.VfPpkd-LgbsSe-OWXEXe-k8QpJ.VfPpkd-LgbsSe-OWXEXe-dgl2Hf.nCP5yc.AjY5Oe.DuMIQc.IIdkle')
consent.click()
except:
pass
You need to import the followings:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
As YouTube changes so frequently and behaves individually region to region there are a variety of different solutions. From your feedback I'd suggest along the lines of below
bypass_popup(driver):
driver.get("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
consent_button_xpath = "//button[#aria-label='Agree to the use of cookies and other data for the purposes described']"
consent = WebDriverWait(driver, 30).until(EC.element_to_be_clickable(
(By.XPATH, consent_button_xpath)))
consent = driver.find_element_by_xpath(consent_button_xpath)
consent.click()
What you want to do is wait for the pop-up to appear first, this is WebDriverWait & then as you are already to find it with find_element_by_xpath and then a simple click()
Due to the obfuscated nature of the web today the above takes advantage of accessibility properties which should mean it remains a tweak away whenever YouTube next changes their UI again, as accessibility should always be present.
Related
I'm trying to use selenium 4 to click on a button in an iframe to expose the hidden input fields. Taking hints from the answers here and here, I'm still not quite able to click the desired Vote button on the source site.
Source Site
Python 3.10.5
Imports:
import selenium.webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Current code:
driver = selenium.webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))
driver.get('https://pnccontests.secondstreetapp.com/ChasChoice_2022/gallery/336759126?group=417500')
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element(By.XPATH, """//*[#id="tncms-block-544689"]/p/iframe""")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="ember2408"]/div[1]/div[2]/div[1]/button'))).click()
Line four is where I'm stumbling to find the button and click it. I've tried a variety of selectors by XPath and CSS identifiers, but am not quite able to get the Vote button to click. Continue to either receive error that the locator cannot be found or stack traces from Selenium timeouts.
Hoping someone can help identify the proper way of selecting the Vote button on that source page.
You can try the following:
driver = selenium.webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))
driver.get('https://pnccontests.secondstreetapp.com/ChasChoice_2022/gallery/336759126?group=417500')
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element(By.XPATH, "//*[#id='tncms-block-544689']/p/iframe")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ember2286 button"))).click()
Also, I see you used absolute xpaths. It is not a good practice. Try to use relative xpaths instead of absolute ones. They are easy on maintenance, readability, and execution per se.
I'm currently trying to select a radio button on the site linked in the code. I've tried finding it by the xpath, and by the ID, but both routes fail, giving me the 'Unable to locate element error'. So I was wondering if someone where could tell me what I'm doing wrong.
driver = webdriver.Chrome(executable_path="/Users/MrPete/Downloads/chromedriver_win32/chromedriver")
driver.get('https://www.pals.pa.gov/#/page/search')
radio = driver.find_element_by_id('optionsRadios1')
radio.click()
This is the radio button I'm trying to click
Induce WebDriverWait() and visibility_of_element_located() and following xpath.
Use Java scripts executor to click on the element.
code:
driver.get('https://www.pals.pa.gov/#/page/search')
radio=WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH,"//input[#id='optionsRadios1']/following::span[1]")))
driver.execute_script("arguments[0].click();", radio)
Import following libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser snapshot.
I'm having some difficulty interacting with a drag-able window that pops up.
After running the following code, a window shows up were I can do some customization. I am trying to click on the unselect all button that appears.
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
driver.get("https://stats.oecd.org/Index.aspx?DataSetCode=HOUSE_PRICES#")
try:
customize = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, "customize-icon")))
print("Customizing")
except TimeoutException:
print("Loading took too much time!")
selection = driver.find_element_by_id("customize-menu-0")
menu = driver.find_element_by_id("customize-menu-3")
Hover = ActionChains(driver).move_to_element(customize).move_to_element(selection).move_to_element(menu)
driver.execute_script("arguments[0].click()", menu)
I have tried the switch to alert and switch to popup functions, but I feel like the thing that shows up on this website isn't a pop up or an alert.
Any help to access and interact with the "popup" customization menu would greatly appreciated.
Thanks!
EDIT:
It seems that in the HTML, the popup table/window is called an iframe, and it has another html tag enclosing all of the content. The iframe is contained in a ui widget thing, and as a result I am unable to switch to the iframe directly.
Yesss, It works :)
Adding to the code I have above and referencing the iframe id directly:
driver.switch_to.frame("DialogFrame")
driver.find_element_by_id("lbtnClear_all").click()
I am able to access the "unselect all" item and click it.
I have difficulty in finding the social media links on this particular website:
https://www.digitalrealty.com/
The code I used:
url = 'https://www.digitalrealty.com/'
driver.get('https://www.digitalrealty.com/')
fb_link = driver.find_element_by_css_selector("a[href*='facebook.com']").get_attribute("href")
After I run the code, however, python returns "NoSuchElementException". Actually, when I look into the website and scroll down, the social media accounts are just at the bottom. I am not sure whether it is because the page is not loaded completely? But isn't .get() by default waiting for the page to load?
I am still new to handling these errors. Any help would be appreciated. Thanks in advance!
The website loads content on scroll, that's why to get to the social media accounts you need to scroll to the bottom of the page.
To wait for some conditions you have to use WebDriverWait and expected_conditions, more details about waits you can find here.
Here your code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 5)
url = "https://www.digitalrealty.com/"
driver.get(url)
#Scroll to the bottom
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
#Wait until presence of the facebook element. Presence of element=exist in the DOM, but can be not visible like here.
facebook = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "a[href*=facebook]")))
#Scroll to element will make element visible and we'll able to click or get text, attribute from it
driver.execute_script("arguments[0].scrollIntoView(true);", facebook)
#Wait for facebook element to be visible and click on it.
fb = wait.until(EC.visibility_of(facebook)).get_attribute("href")
print(fb)
Currently I am trying to click the "entry-level" link in an iframe, for this website:https://a127-jobs.nyc.gov/index_new.html?category=CAS
Unfortunately, the url does not change if I click entry level, so therefore I am forced to automate.
I am not sure what to do, if the problem is that I have entered the wrong iframe? there seems to be only one, but I could be wrong.
from selenium import webdriver
import time
#webdriver
driver = webdriver.Firefox(executable_path="/Users/alexandrubordei/Desktop/geckodriver")
#get the website
driver.get ("https://a127-jobs.nyc.gov/index_new.html?category=CAS")
time.sleep(10)
#switch to iframe
iframes = driver.find_elements_by_tag_name("iframe")
driver.switch_to.frame(iframes[0])
time.sleep(10)
#click element "entry level"
driver.find_element_by_xpath('//*[#id="ti_S13"]').click()
When I run my code everything seems to function, except the link is not clicked. My search results are not narrowed.
However I do not get any errors. I get an error as:
Process finished with exit code 0
you have to click the a element inside that ID
driver.find_element_by_xpath('//*[#id="ti_S13"]/a').click()
To click() on the element with text as entry-level as the the desired elements are within an <iframe> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
driver.get("https://a127-jobs.nyc.gov/index_new.html?category=CAS")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='ptifrmtgtframe' and #name='TargetContent']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Entry-Level"))).click()
Browser Snapshot:
Here you can find a relevant discussion on Ways to deal with #document under iframe