I have difficulty in finding the social media links on this particular website:
https://www.digitalrealty.com/
The code I used:
url = 'https://www.digitalrealty.com/'
driver.get('https://www.digitalrealty.com/')
fb_link = driver.find_element_by_css_selector("a[href*='facebook.com']").get_attribute("href")
After I run the code, however, python returns "NoSuchElementException". Actually, when I look into the website and scroll down, the social media accounts are just at the bottom. I am not sure whether it is because the page is not loaded completely? But isn't .get() by default waiting for the page to load?
I am still new to handling these errors. Any help would be appreciated. Thanks in advance!
The website loads content on scroll, that's why to get to the social media accounts you need to scroll to the bottom of the page.
To wait for some conditions you have to use WebDriverWait and expected_conditions, more details about waits you can find here.
Here your code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 5)
url = "https://www.digitalrealty.com/"
driver.get(url)
#Scroll to the bottom
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
#Wait until presence of the facebook element. Presence of element=exist in the DOM, but can be not visible like here.
facebook = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "a[href*=facebook]")))
#Scroll to element will make element visible and we'll able to click or get text, attribute from it
driver.execute_script("arguments[0].scrollIntoView(true);", facebook)
#Wait for facebook element to be visible and click on it.
fb = wait.until(EC.visibility_of(facebook)).get_attribute("href")
print(fb)
Related
I'm trying to use selenium 4 to click on a button in an iframe to expose the hidden input fields. Taking hints from the answers here and here, I'm still not quite able to click the desired Vote button on the source site.
Source Site
Python 3.10.5
Imports:
import selenium.webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Current code:
driver = selenium.webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))
driver.get('https://pnccontests.secondstreetapp.com/ChasChoice_2022/gallery/336759126?group=417500')
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element(By.XPATH, """//*[#id="tncms-block-544689"]/p/iframe""")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="ember2408"]/div[1]/div[2]/div[1]/button'))).click()
Line four is where I'm stumbling to find the button and click it. I've tried a variety of selectors by XPath and CSS identifiers, but am not quite able to get the Vote button to click. Continue to either receive error that the locator cannot be found or stack traces from Selenium timeouts.
Hoping someone can help identify the proper way of selecting the Vote button on that source page.
You can try the following:
driver = selenium.webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))
driver.get('https://pnccontests.secondstreetapp.com/ChasChoice_2022/gallery/336759126?group=417500')
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element(By.XPATH, "//*[#id='tncms-block-544689']/p/iframe")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ember2286 button"))).click()
Also, I see you used absolute xpaths. It is not a good practice. Try to use relative xpaths instead of absolute ones. They are easy on maintenance, readability, and execution per se.
I have a simple python program that opens this page with chromedriver.exe, then it clicks on the wallet icon located at the top right corner of that page, for then clicking on the MetaMask wallet button, here:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
opt.add_argument('--user-data-dir='+r'C:\Users\ResetStoreX\AppData\Local\Google\Chrome\User Data') #Add the user data path as an argument in selenium Options
opt.add_argument("--profile-directory=Default") #Add the profile directory as an argument in selenium Options
s = Service(r'C:\Users\ResetStoreX\AppData\Local\Programs\Python\Python39\Scripts\chromedriver.exe') #Use the chrome driver located at the corresponding path
driver = webdriver.Chrome(service=s, options=opt) #execute the chromedriver.exe with the previous conditions
driver.implicitly_wait(10)
driver.get('https://opensea.io/') #go to the opensea main page.
initial_page = driver.current_window_handle #make the previous page the current one
WebDriverWait(driver, 3).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="__next"]/div/div[1]/nav/ul/div[2]/li/button'))) #wait for the wallet button to be enabled for clicking
wallet_button = driver.find_element(By.XPATH, '//*[#id="__next"]/div/div[1]/nav/ul/div[2]/li/button')
wallet_button.click() #click that wallet button
#time.sleep(1)
wallet_options = driver.find_elements(By.XPATH, "//*[#data-testid='WalletSidebar--body']//li")
for i in wallet_options:
if "MetaMask" in i.get_attribute('innerText'): #find the MetaMask wallet button
i.click() #click the MetaMask wallet button
The problem I noticed when testing this program was that after clicking the MetaMask wallet button via automation, the expected Metamask Notification window was never displayed, and the current window kept loading that button eternally as shown below:
However, if the user manually clicked the same MetaMask wallet button in the same chrome instance, the Metamask Notification window would immediately appear as expected, asking the user to log in for then connecting the wallet to that site, as shown below:
I even checked if there was another hidden window handle after pressing the MetaMask wallet button (via automation) with this code, but there was nothing else:
for handle in driver.window_handles:
if handle != initial_page:
login_page = handle
driver.switch_to.window(login_page)
So, I'm confused, why does that button apparently only work when a user presses it but not when a program does? As always, I'd appreciate a lot if someone could explain what's happening and how could I fix this?
I figured it out already, and it was a foolish thing, I simply had to add the extension path (which must be compressed as a .crx file) as an argument to Selenium's Options(), here's the improved code:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
opt.add_argument('--user-data-dir='+r'C:\Users\ResetStoreX\AppData\Local\Google\Chrome\User Data') #Add the user data path as an argument in selenium Options
opt.add_argument("--profile-directory=Default") #Add the profile directory as an argument in selenium Options
extension_path = r'C:\Users\ResetStoreX\AppData\Local\Google\Chrome\User Data\Default\Extensions\nkbihfbeogaeaoehlefnkodbefgpgknn\10.8.1_0.crx' #assign the extension path of the compressed Metamask extension to this variable
opt.add_extension(extension_path) #add the extension_path as argument
s = Service(r'C:\Users\ResetStoreX\AppData\Local\Programs\Python\Python39\Scripts\chromedriver.exe') #Use the chrome driver located at the corresponding path
driver = webdriver.Chrome(service=s, options=opt) #execute the chromedriver.exe with the previous conditions
driver.implicitly_wait(10)
driver.get('https://opensea.io/') #go to the opensea main page.
initial_page = driver.current_window_handle #make the previous page the current one
WebDriverWait(driver, 3).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="__next"]/div/div[1]/nav/ul/div[2]/li/button'))) #wait for the wallet button to be enabled for clicking
wallet_button = driver.find_element(By.XPATH, '//*[#id="__next"]/div/div[1]/nav/ul/div[2]/li/button')
wallet_button.click() #click that wallet button
#time.sleep(1)
wallet_options = driver.find_elements(By.XPATH, "//*[#data-testid='WalletSidebar--body']//li")
for i in wallet_options:
if "MetaMask" in i.get_attribute('innerText'): #find the MetaMask wallet button
i.click() #click the MetaMask wallet button
So when you compile the enhanced code, it generates the following output:
Something I noticed was that this program ended up opening a chromedriver tab and a new window to access the Metamask wallet (ie both things are used for the same thing).
If anyone figures out how to prevent the MetaMask tab from opening as shown in the image, please let me know in the comments.
I am trying do you Youtube data scraping, but stuck at the first step... accepting the cookie.
I have tried searching the "I agree" text and click but seems another element is obscuring it.
buttons = driver.find_elements_by_xpath("//*[contains(text(), 'I agree')]")
for btn in buttons:
btn.click()
Tried accessing div, seems to do the action but does not really click
buttons = driver.find_element_by_xpath('//div[#class="VfPpkd-dgl2Hf-ppHlrf-sM5MNb"]//button')
buttons.click
any pointer on how to click the button using python + selenium?
button html
https://consent.youtube.com/m?continue=https%3A%2F%2Fwww.youtube.com%2Fresults%3Fsearch_query%3Dpost%2Bdriver&gl=FR&m=0&pc=yt&uxe=23983172&hl=en&src=1
To click on I agree button to accept cookies on YouTube. The button is present inside an iframe and you need to switch the iframe. Use WebDriverWait() and wait for frame available and switch it.
WebDriverWait(driver,20).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"iframe")))
driver.find_element_by_xpath("//*[contains(text(), 'I agree')]").click()
To jump out from frame you need to use
driver.switch_to.default_content()
Please import below libraries
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
This function is working for me..Most of the time it's showing in a form and sometimes you need to click on this particular button. Sometimes it's not showing at all.
bypass_consent(driver):
try:
consent = WebDriverWait(driver, 30).until(EC.element_to_be_clickable(
(By.XPATH, "//input[#type='submit' and #value='I agree']")))
consent.submit()
except:
try:
consent = driver.find_element_by_css_selector(
'button.VfPpkd-LgbsSe.VfPpkd-LgbsSe-OWXEXe-k8QpJ.VfPpkd-LgbsSe-OWXEXe-dgl2Hf.nCP5yc.AjY5Oe.DuMIQc.IIdkle')
consent.click()
except:
pass
You need to import the followings:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
As YouTube changes so frequently and behaves individually region to region there are a variety of different solutions. From your feedback I'd suggest along the lines of below
bypass_popup(driver):
driver.get("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
consent_button_xpath = "//button[#aria-label='Agree to the use of cookies and other data for the purposes described']"
consent = WebDriverWait(driver, 30).until(EC.element_to_be_clickable(
(By.XPATH, consent_button_xpath)))
consent = driver.find_element_by_xpath(consent_button_xpath)
consent.click()
What you want to do is wait for the pop-up to appear first, this is WebDriverWait & then as you are already to find it with find_element_by_xpath and then a simple click()
Due to the obfuscated nature of the web today the above takes advantage of accessibility properties which should mean it remains a tweak away whenever YouTube next changes their UI again, as accessibility should always be present.
I'm having some difficulty interacting with a drag-able window that pops up.
After running the following code, a window shows up were I can do some customization. I am trying to click on the unselect all button that appears.
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
driver.get("https://stats.oecd.org/Index.aspx?DataSetCode=HOUSE_PRICES#")
try:
customize = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.ID, "customize-icon")))
print("Customizing")
except TimeoutException:
print("Loading took too much time!")
selection = driver.find_element_by_id("customize-menu-0")
menu = driver.find_element_by_id("customize-menu-3")
Hover = ActionChains(driver).move_to_element(customize).move_to_element(selection).move_to_element(menu)
driver.execute_script("arguments[0].click()", menu)
I have tried the switch to alert and switch to popup functions, but I feel like the thing that shows up on this website isn't a pop up or an alert.
Any help to access and interact with the "popup" customization menu would greatly appreciated.
Thanks!
EDIT:
It seems that in the HTML, the popup table/window is called an iframe, and it has another html tag enclosing all of the content. The iframe is contained in a ui widget thing, and as a result I am unable to switch to the iframe directly.
Yesss, It works :)
Adding to the code I have above and referencing the iframe id directly:
driver.switch_to.frame("DialogFrame")
driver.find_element_by_id("lbtnClear_all").click()
I am able to access the "unselect all" item and click it.
Currently I am trying to click the "entry-level" link in an iframe, for this website:https://a127-jobs.nyc.gov/index_new.html?category=CAS
Unfortunately, the url does not change if I click entry level, so therefore I am forced to automate.
I am not sure what to do, if the problem is that I have entered the wrong iframe? there seems to be only one, but I could be wrong.
from selenium import webdriver
import time
#webdriver
driver = webdriver.Firefox(executable_path="/Users/alexandrubordei/Desktop/geckodriver")
#get the website
driver.get ("https://a127-jobs.nyc.gov/index_new.html?category=CAS")
time.sleep(10)
#switch to iframe
iframes = driver.find_elements_by_tag_name("iframe")
driver.switch_to.frame(iframes[0])
time.sleep(10)
#click element "entry level"
driver.find_element_by_xpath('//*[#id="ti_S13"]').click()
When I run my code everything seems to function, except the link is not clicked. My search results are not narrowed.
However I do not get any errors. I get an error as:
Process finished with exit code 0
you have to click the a element inside that ID
driver.find_element_by_xpath('//*[#id="ti_S13"]/a').click()
To click() on the element with text as entry-level as the the desired elements are within an <iframe> so you have to:
Induce WebDriverWait for the desired frame to be available and switch to it.
Induce WebDriverWait for the desired element to be clickable.
You can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe')
driver.get("https://a127-jobs.nyc.gov/index_new.html?category=CAS")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='ptifrmtgtframe' and #name='TargetContent']")))
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Entry-Level"))).click()
Browser Snapshot:
Here you can find a relevant discussion on Ways to deal with #document under iframe