I am new to Python 3.X and need to write a script to automate the process downloading US Census data from American Fact Finder. I am using selenium webdriver and the code I have so far is:
driver = webdriver.Chrome(chromePath)
#make driver navigate to American Fact Finder Download Center
driver.get('https://factfinder.census.gov/faces/nav/jsf/pages/download_center.xhtml')
#Make driver click 'Next' to go to Dataset page
driver.find_element_by_xpath('''//*[#id="nextButton"]''').click()
#this is where I need to locate the drop down and select American Community Survey'
On the 'Dataset' page I need to select 'American Community Survey' from the drop down list, but no matter how I try to locate the drop down(xpath, id, value, etc.) running the script returns NoSuchElementException: no such element: Unable to locate element:
I need help with locating the correct element and selecting 'American Community Survey' from the drop down menu.
Thank you!
I was actually able to solve it myself.
For anyone wondering, the error was caused because selenium was trying to find the element before the page had finished loading. The working code is:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
chromePath = r'chromedriver.exe'
driver = webdriver.Chrome(chromePath)
#make driver navigate to American Fact Finder Download Center
driver.get('https://factfinder.census.gov/faces/nav/jsf/pages/download_center.xhtml')
#Make driver click 'Next' to go to Dataset
driver.find_element_by_xpath('''//*[#id="nextButton"]''').click() #needs to be triple quoted
#make driver wait while page loads
timeout = 5
try:
element_present = EC.presence_of_element_located((By.XPATH, '//*[#id="filterDimensionListId'))
WebDriverWait(driver, timeout).until(element_present)
except TimeoutException:
print ("Timed out waiting for page to load")
#Choose ACS 5-year from drop down
driver.find_element_by_xpath('''//*[#id="filterDimensionListId"]/option[2]''').click()
You can try below code to get.
select = Select(driver.find_element_by_id('filterDimensionListId'))
for index in range(len(select.options)):
select = Select(driver.find_element_by_id('filterDimensionListId'))
select.select_by_index(1)
// you can use other selection method.
Related
ENV
Python 3.10
Selenium 4
Hi,
I use :
def findAllByXPath(p_driver, xpath, time=5):
try:
#return WebDriverWait(p_driver, time).until(EC.presence_of_all_elements_located((By.XPATH, (xpath))))
return WebDriverWait(p_driver, time).until(EC.presence_of_all_elements_located((By.XPATH, xpath)))
except Exception as ex:
print(f"ERROR findAllByXPath : {ex}")
posts = findAllByXPath(p_driver,"//article//a",2)
for index in range(0,len(posts)):
p_driver.execute_script("arguments[0].scrollIntoView()", posts[index])
time.sleep(random.uniform(0.5, 0.8))
action.move_to_element(posts[index]).perform()
To make a mouse over and try to get elements.
I am trying to get the number of comments on an Instagram post from a page profile.
If you pass manually the mouse over a post, the number of comments or likes is displayed.
But when I try to do it in python, it doesn't find the element.
I tried the XPath from console
$x(//article//li//span[string-length(text())>0]")
It gives results when I freeze the browser with F8.
action.move_to_element(post).perform() doesn't work? I suspect this version 4.3.0 removed many functions like ActionChains from the Selenium version 3, didn't it?
How can I extract this element?
Or Am I doing something wrong?
ActionChains works nicely in Selenium v4!
I don't have an Instagram account so I cannot try there but you can see by running this code that move_to_element() works and how it works.
The About button (top left corner of stackoverflow homepage) get highlighted.
Remember to don't move your mouse while running of course!
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Firefox()
actions = ActionChains(driver)
driver.get("https://stackoverflow.com/")
about_button = WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.XPATH, "//a[#href='https://stackoverflow.co/']")))
actions.move_to_element(about_button)
actions.perform()
EDIT: I am using Selenium 4.4.3 and Python 3.10 but it should work with Selenium 4.3.0 as well.
Website: https://vahan.parivahan.gov.in/vahan4dashboard/vahan/view/reportview.xhtml'
I'm trying to use selenium to download data from this website but it's setup in a confusing way. I need to figure out how to use the dropdown in the list called 'Y-Axis' and select 'Maker' from that list. Then I need to hit the 'refresh' button and the 'download excel' button. This is the html of the dropdown menu:
<select id="yaxisVar_input" name="yaxisVar_input" tabindex="-1" aria-hidden="true" onchange="PrimeFaces.ab({s:"yaxisVar",e:"change",f:"masterLayout_formlogin",p:"yaxisVar",u:"xaxisVar"});"><option value="Vehicle Category" data-escape="true">Vehicle Category</option><option value="Vehicle Class" selected="selected" data-escape="true">Vehicle Class</option><option value="Norms" data-escape="true">Norms</option><option value="Fuel" data-escape="true">Fuel</option><option value="Maker" data-escape="true">Maker</option></select>
This is the code I'm playing around with:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
driver = webdriver.Chrome('C:/Users/abhay.singh/chromedriver')
driver.get('https://vahan.parivahan.gov.in/vahan4dashboard/vahan/view/reportview.xhtml')
# Get the y-axis selector
# select = Select(driver.find_element_by_id('yaxisVar_input'))
# select.select_by_visible_text('Maker').click()
# print(select.options)
# print([o.text for o in select.options])
driver.find_element_by_xpath("//select[#name='yaxisVar_input']/option[text()='Maker']").click()
I'd appreciate your help in figuring this out!
This is the best I could do. One thing to note is that selenium is not really meant for downloading, so I added a sleep at the end to ensure the download completes. It can also be done with scripts to monitor the download status, but I don't really know how to do it. I also had to add a sleep in the middle to make sure the "Maker" click was being captured correctly. I'm sure there is a better way to do that as well.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
with webdriver.Chrome() as driver:
driver.get("https://vahan.parivahan.gov.in/vahan4dashboard/vahan/view/reportview.xhtml")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "label[id='yaxisVar_label']"))).click()
time.sleep(2)
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "li[data-label='Maker']"))).click()
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "j_idt61"))).click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "vchgroupTable:xls"))).click()
time.sleep(10)
You might be able to do this with a requests.post(). I didn't look into the headers or form data, but it is there.
I'm trying to automate some tedious copy / paste I do monthly from my bank's online service via Selenium and Python 3. Unfortunately, I can't get Selenium to click the log-in link.
It's the blue continue button at https://www1.bmo.com/onlinebanking/cgi-bin/netbnx/NBmain?product=5.
Strangely, when I try to click that link manually in the browser launched by Selenium, it doesn't work either - whereas it does work in a browser I launch manually.
I suspect the issue is that the bank's website is smart enough to detect that I'm automating the browser activity. Is there any way to get around that?
If not, could it be something else?
I've tried using Chrome and Firefox - to no avail. I'm using a 64 bit Windows 10 machine with Chrome 73.0.3683.103 and Firefox 66.0.
Relevant code is below.
#websites and log in information
bmo_login_path = 'https://www1.bmo.com/onlinebanking/cgi-bin/netbnx/NBmain?product=5'
bmo_un = 'fake_user_name'
bmo_pw = 'fake_password'
#Selenium setup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
chrome_driver_path = 'C:\\Path\\To\\Driver\\chromedriver.exe'
gecko_driver_path = 'C:\\Path\\To\\Driver\\geckodriver.exe'
browswer_bmo = webdriver.Firefox(executable_path = gecko_driver_path)
#browswer_bmo = webdriver.Chrome(executable_path = chrome_driver_path)
#log into BMO
browswer_bmo.get(bmo_login_path)
time.sleep(5)
browswer_bmo.find_element_by_id('siBankCard').send_keys(bmo_un)
browswer_bmo.find_element_by_id('regSignInPassword').send_keys(bmo_pw)
browswer_bmo.find_element_by_id('btnBankCardContinueNoCache1').click()
Sending the keys works perfectly. I may actually have the wrong element ID (I was trying to test that in Chrome when I realized I couldn't click the link manually) - but I think the bigger issue is that I can't manually click the link in the browser launched by Selenium. Thank you for any ideas.
EDIT
This is a screenshot that I get of all I get when I try to click the continue button.
Ultimately the error message I get in my IDE (Jupyter Notebook) is:
TimeoutException: Message: timeout
(Session info: chrome=74.0.3729.108)
(Driver info: chromedriver=74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729#{#29}),platform=Windows NT 10.0.17134 x86_64)
To click on the button with text as Continue you can fill up the Card Number and Password field inducing WebDriverWait for the element_to_be_clickable() and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument('start-maximized')
options.add_argument('disable-infobars')
options.add_argument('--disable-extensions')
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www1.bmo.com/onlinebanking/cgi-bin/netbnx/NBmain?product=5')
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.dijitReset.dijitInputInner#siBankCard[name='FBC_Number']"))).send_keys("1234567890112233")
driver.find_element_by_css_selector("input.dijitReset.dijitInputInner#regSignInPassword[name='FBC_Password']").send_keys("fake_password")
driver.find_element_by_css_selector("span.dijitReset.dijitInline.dijitIcon.dijitNoIcon").click()
# driver.quit()
Browser Snapshot:
I was able to fix this issue and solve the problem by adding the following line below the options variables. This disables the chrome check for automation. I used the whole sale code and then added the following line in the correct location before starting the driver.
options.add_experimental_option("excludeSwitches", ['enable-automation'])
ref: https://help.applitools.com/hc/en-us/articles/360007189411--Chrome-is-being-controlled-by-automated-test-software-notification
I cannot seem to find a way to click on the right element in order to get the url I am looking for. In essence I am trying to click on the top video in a youtube search (the most highly ranked returned video).
How to resolve ElementNotInteractableException: Element is not visible in Selenium webdriver?
-> This is for Java but it let me in the right direction (knowing I needed to execute JavaScript)
http://www.teachmeselenium.com/2018/04/17/python-selenium-interacting-with-the-browser-executing-javascript-through-javascriptexecutor/
-> This shows me how I should try to execute the javascript in python.
I have also seen countless articles about waits but they do not solve my problem.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
wrds = ["Vivaldi four seasons", "The Beatles twist and shout", "50
cent heat"] #Random list of songs
driver = webdriver.Chrome()
for i in wrds:
driver.get("http://www.youtube.com")
elem = driver.find_element_by_id("search")
elem.send_keys(i)
elem.send_keys(Keys.RETURN)
time.sleep(5)
driver.execute_script("arguments[0].click()",driver.find_element_by_id('video-title')) #THIS CLICKS ON WRONG VIDEO
#elem = driver.find_element_by_id("video-title").click() #THIS FAILS
time.sleep(5)
url = driver.current_url
driver.close()
I get a ElementNotVisibleException: Message: element not interactable error when I do not execute any javascript (even though it has actually worked before it is just no way near robust). When I do execute the javascript it clicks on the wrong videos.
I have tried all types of waits "Explicit" and "Implicit" this did now work.
I am quite sure I need to execute some JavaScript but I don't know how.
You were almost there. You need to induce WebDriverWait for the element to be clickable and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wrds = ["Vivaldi four seasons", "The Beatles twist and shout", "50 cent heat"]
kwrd = ["Vivaldi", "Beatles", "50"]
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver=webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\\chromedriver.exe')
for i, j in zip(wrds, kwrd):
driver.get("https://www.youtube.com/")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#search"))).send_keys(i)
driver.find_element_by_css_selector("button.style-scope.ytd-searchbox#search-icon-legacy").click()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "h3.title-and-badge.style-scope.ytd-video-renderer a"))).click()
WebDriverWait(driver, 10).until(EC.title_contains(j))
print(driver.current_url)
driver.quit()
That's one of the reasons you should never use JavaScript click, Selenium Webdriver is designed to stimulate as if a real user is able to click. Real user can't click an invisible element in the page but you can click through Javascript. If you search the element by that id video-title, it matches totally 53 videos. But I don't know which one you want to click. You may match that element by some other way(not by id).
I will give you an idea how to click that element but you need to find out the index first before you click.
driver.find_element_by_xpath("(//*[#id='video-title'])[1]").click
If the first one is invisible, then pass 2, [2] then three, figure out which one it's clicking the desired element. Or you may specify the exact element, we may try to locate that element by some other way.
I am trying to figure out how to use selenium to perform a next page click on a news release page. Here is my code that will go to the correct website and perform a search to obtain the correct news release articles topic page. This site is configured so that to see every news release on page 1 after the search has executed you must also select the More News Results button at the bottom of the page. I am able to get to the full page 1 news section without problem. Here is my code that performs the search and page clicks.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
browser = webdriver.Chrome()
browser.get('http://www.businesswire.com/portal/site/home/')
search_box_element = browser.find_element_by_id('bw-search-input')
search_box_element.clear()
search_box_element.send_keys('biotechnology')
search_box_element.send_keys(Keys.ENTER)
search_box_element_two = browser.find_element_by_id('more-news-results')
search_box_element_two.click()
That part of the code works fine but I want to be able to click on the next button to move to page 2 and then page 3 and so on. Here is the code that I thought would work but it doesn't:
next_page_click_element = browser.find_element_by_class_name("bw-paging-next")
next_page_click_element.click()
This portion of the code throws an error:
selenium.common.exceptions.ElementNotVisibleException: Message:
element not visible
I have also tried using
next_page_click_element = browser.find_element_by_xpath('//*[#id="more-news-pagination"]/div/div[1]/div/a')
but got the same error message. I have also tried using a wait by adding these lines of codes just before the next_page_click_element section.
element_present = EC.presence_of_element_located((By.ID, "bw-paging-next"))
WebDriverWait(browser, 10).until(element_present)
While this does cause the program to wait it returns this error message:
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Any suggestions on how to resolve this issue would be greatly appreciated.
Here is the Answer to your Question:
A few words about the solution-
As the Page 2,Page 3,Next etc elements are at the bottom of the page, you have to scroll down to bring those elements within the Viewport.
Once you scroll down you may consider to induce some ExplicitWait for the elements to be located properly on the HTML DOM.
Here is your own code with some simple tweaks which will finally scroll to the bottom of the page and click on Next link:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
browser=webdriver.Chrome("C:\\Utility\\BrowserDrivers\\chromedriver.exe")
browser.get('http://www.businesswire.com/portal/site/home/')
search_box_element = browser.find_element_by_id('bw-search-input')
search_box_element.clear()
search_box_element.send_keys('biotechnology')
search_box_element.send_keys(Keys.ENTER)
search_box_element_two = browser.find_element_by_id('more-news-results')
search_box_element_two.click()
last_height = browser.execute_script("return document.body.scrollHeight")
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")
next_page_click_element = WebDriverWait(browser, 10).until(
EC.presence_of_element_located((By.XPATH, "//div[#id='more-news-pagination']/div/div/div/a[text()='Next']"))
)
next_page_click_element.click()
Let me know if this Answers your Question.