I am trying to scrape data from this website using Selenium.
There are three features in data, "Value", "Net change" and "percent change", including values for net and percentage changes for 1, 3, 6, and 12 months. I want to fetch 1 month's net change and percent change. For that, I need to click on the check boxes and click on the update button.
Now, I performed these actions using selenium's find element by XPath method but for percent change, I needed to use the ActionChains command, as I was getting "Element not clickable error".
When I execute the code, all three features should occur in the downloaded csv. But that's not happening. I am just able to fetch "Value" and "1 Month Net change". If anyone knows, may I know, why the is not getting updated and or how to fix it? Thanks
My code:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
driver.get("https://beta.bls.gov/dataViewer/view/timeseries/CUUR0000SA0")
soup = BeautifulSoup(driver.page_source, "html.parser", from_encoding='utf-8')
driver.find_element(By.XPATH, '/html/body/div[2]/div/div/div[4]/div/div[1]/form/div[2]/fieldset/div[1]/table/tbody/tr[1]/td[1]/label/input').click() //1 month net change
element = WebDriverWait(driver, 60).until(EC.element_to_be_clickable((By.XPATH, '// [#id="percent_monthly_changes_div"]/table/tbody/tr[1]/td[1]/label/input')))
ActionChains(driver).move_to_element(element).click().perform() //1 month percent change
driver.find_element(By.XPATH, '/html/body/div[2]/div/div/div[4]/div/div[1]/form/div[4]/input').click() //update button
driver.find_element(By.XPATH, '//*[#id="csvclickCU"]').click() //download csv button
The website is showing N/A in the column of 1 Month Net change.
if you still not getting 1 month % change value you can do
driver.execute_script('document.querySelector("#percent_monthly_changes_div > table > tbody > tr:nth-child(1) > td:nth-child(1) > label > input").click()')
instead of:
element = WebDriverWait(driver, 60).until(EC.element_to_be_clickable((By.XPATH, '// [#id="percent_monthly_changes_div"]/table/tbody/tr[1]/td[1]/label/input')))
ActionChains(driver).move_to_element(element).click().perform()
this might not be the optimal solution, but it works fine.
and 1 month net change value is not given from the website itself.
To click on the elements with text as 1-Month Net Change and 1-Month % Change using ActionChains will be an overhead and you can avoid it easily.
Ideally, you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
driver.get("https://beta.bls.gov/dataViewer/view/timeseries/CUUR0000SA0")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[value='1N']"))).click()
driver.find_element(By.CSS_SELECTOR, "input[value='1P']").click()
Using XPATH:
driver.get("https://beta.bls.gov/dataViewer/view/timeseries/CUUR0000SA0")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#value='1N']"))).click()
driver.find_element(By.XPATH, "//input[#value='1P']").click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
Related
Actually, i have a script with python 3.10, selenium, undetected-chromedriver and i want to click on two buttons on this webpage:
https://keepa.com/#!
Buttons are french flag, and ".fr" here:
And html code for these two buttons are here:
-first highlighted is flag fr
-second is country ".fr"
I've tested somes part of script, but not work actually:
For language flag fr:
driver.find_element(By.XPATH, "(//span[#id='lang_fr'])").click()
driver.find_element(By.ID,"lang_fr").click()
For country .fr:
driver.find_element(By.XPATH, "(//span[#setting='4'])").click()
driver.find_element(By.XPATH, "(.//span[contains(text(), '.fr')])").click()
I have somes errors on results, "element not found", or this:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
Anyone can help me or have an idea where is the problem please?
Thanks for help, bye!
You have to click on default flag first and then there will be 2 button which you can click like below:
Code:
driver_path = r'C:\\Users\\****\\***\\Desktop\\Automation\\chromedriver.exe'
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
driver.get("https://keepa.com/#!")
wait = WebDriverWait(driver, 30)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#currentLanguage .languageMenuText"))).click()
flag = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "img#lang_fr")))
flag.click()
lang = wait.until(EC.visibility_of_element_located((By.XPATH, "//span[text()='.fr']")))
lang.click()
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can try the below code it will help you (this is using java though, ).
Actions actions = new Actions(driver);
WebElement ctr;
ctr= driver.findElement(By.xpath("//*[contains(text(),'YOUR DESIRED TEXT')]"));
Thread.sleep(1000);
actions.click(spatial).perform();
I am trying to create an Account on Walmart using selenium python. I successfully opened https://www.walmart.com/ and successfully go to create an account button under Sign In tab. Moreover, I also successfully entered the details of First name, Last name, Email Address and Password. However, once I clicked on Create account button, I got TimeoutException error despite using EC.visibility_of_element_located().click () method.
Can anyone kindly guide me what is wrong with my approach. Thanks in advance.
The source code of the website for Create Account button is as follows:
<button class="button m-margin-top text-inherit" type="submit" data-automation-id="signup-submit-btn" data-tl-id="signup-submit-btn" aria-label="Create Account, By clicking Create Account, the user is acknowledging that they have read and agreed to the Terms of Use and Privacy Policy">Create account</button>
My Python code is as follows:
import time
import requests
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
url = "https://www.walmart.com/"
first_name = "chuza"
last_name = "123"
email_id = "chuza123#gmail.com"
password = "Eureka1#"
options = Options()
s=Service('C:/Users/Samiullah/.wdm/drivers/chromedriver/win32/96.0.4664.45/chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
"source":
"const newProto = navigator.__proto__;"
"delete newProto.webdriver;"
"navigator.__proto__ = newProto;"
})
wait = WebDriverWait(driver, 20)
actions = ActionChains(driver)
driver.get(url)
sign_in_btn = wait.until(EC.visibility_of_element_located((By.XPATH, "//div[text()='Sign In']")))
actions.move_to_element(sign_in_btn).perform()
time.sleep(0.5)
wait.until(EC.visibility_of_element_located((By.XPATH, '//button[normalize-space()="Create an account"]'))).click()
f_name = driver.find_element(By.ID, 'first-name-su')
l_name = driver.find_element(By.ID, 'last-name-su')
email = driver.find_element(By.ID, 'email-su')
pswd = driver.find_element(By.ID, 'password-su')
f_name.send_keys(first_name)
driver.implicitly_wait(2)
l_name.send_keys(last_name)
driver.implicitly_wait(1.5)
email.send_keys(email_id)
driver.implicitly_wait(2)
pswd.send_keys(password)
driver.implicitly_wait(0.5)
###
wait.until(EC.visibility_of_element_located((By.XPATH, '//button[normalize-space()="Create account"]'))).click()
I see this css selector that represent the desired webelement:
button[data-automation-id='signup-submit-btn']
and xpath would be:
//button[#data-automation-id='signup-submit-btn']
there are 3 matching nodes for each CSS and XPath and Selenium will look for the first match, the CSS and XPath basically are first matching node.
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-automation-id='signup-submit-btn']"))).click()
or
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[#data-automation-id='signup-submit-btn']"))).click()
It makes more sense to use element_to_be_clickable when trying to click on a web element instead of visibility_of_element_located. Also, CSS are much better locator as compared to XPath.
//button[normalize-space()="Create account"] locator matches 3 elements on that page, you need to use more precise locator.
This locator is unique: //form[#id='sign-up-form']//button[#data-tl-id='signup-submit-btn']
So, this should work:
wait.until(EC.visibility_of_element_located((By.XPATH, "//form[#id='sign-up-form']//button[#data-tl-id='signup-submit-btn']"))).click()
This xpath based Locator Strategy...
//button[normalize-space()="Create account"]
...identifies three(3) elements within the DOM Tree and your desired element is the second in the list.
Solution
The desired element is a dynamic element so to click on the clickable element instead of visibility_of_element_located() you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategy:
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//form[#id='sign-up-form']//button[normalize-space()='Create account']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
This issue is because of selenium, you can fix this by creating it manually, follow the steps:
Create an account on Walmart IO platform here by clicking on the man icon just before the search box.
Login to the account and accept "Terms of Use"
Click on "Create Your Application" to create a new application and fill in appropriate details.
You can follow this tutorial to generate two sets of public/private keys -one set will be used for production and the other set will be used for stage.
Upload both public keys using this: https://walmart.io/key-upload?app_name=<your app name>
Consumer ID will be generated for both sets for prod and stage which can be seen on the dashboard
Click on "Request Access" for OPD APIs at here and fill out the form
I am trying to get the top stocks for the day so I go to https://finance.yahoo.com/gainers but I the want to edit the filters by pressing Edit.
driver = webdriver.Chrome()
driver.get("https://finance.yahoo.com/gainers")
element = driver.find_element_by_class_name("Bgc($linkColor).Bgc($linkActiveColor):h.C(white).Fw(500).Px(20px).Py(9px).Bdrs(3px).Bd(0).Fz(s).D(ib).Whs(nw).Miw(110px)")
element.click()
This doesn't work. How can I fix it?
To click on the element Edit you can use either of the following Locator Strategies:
Using xpath:
driver.get("https://finance.yahoo.com/gainers")
driver.find_element_by_xpath("//span[text()='Edit']").click()
Ideally, to click on the element you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using XPATH:
driver.get("https://finance.yahoo.com/gainers")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Edit']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
The java code below seems to work.
WebDriver driver = new ChromeDriver();
driver.get("https://finance.yahoo.com/gainers");
driver.manage().window().maximize();
WebDriverWait wait = new WebDriverWait(driver, 30);
WebElement editButton = wait
.until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("button[data-reactid=\"23\"]")));
editButton.click();
Cleaner locator
WebElement editButton = wait
.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//button/span[contains(text(),'Edit')]")));
I want to select an option that displays only after you've clicked on the dropdown (see attached image). I have been able to click on the dropdown to get the list, but have not been able to figure out how to click on an option, say option 1, 'Last Day' after the list comes into picture.
Here's what I have so far:
from selenium import webdriver
binary = FirefoxBinary('C:\\Program Files\\Firefox Developer Edition\\firefox.exe')
cap = DesiredCapabilities().FIREFOX
cap["marionette"] = True
url='https://www.glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=false&clickSource=searchBtn&typedKeyword=data+sc&sc.keyword=data+scientist&locT=C&locId=1154532&jobType='
driver = webdriver.Firefox(firefox_binary=binary, capabilities=cap, executable_path=GeckoDriverManager().install())
driver.get(url=url)
driver.implicitly_wait(10)
driver.maximize_window()
# clicking on dropdown
d = driver.find_element_by_id('filter_fromAge')
d.click()
I also tried using the following code (found on another SO answer) but it did not work either:
element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"ul#css-1dv4b0s ew8xong0")))
I'm new to web scraping and not really familiar with XPATHs and how to deal with actions. Help appreciated!
You can use Java script to click on your element. As element is present in DOM but its only appears when click on drop down, so normal click method mau work or may not. But with JS it will always click. Can use below code:
day = driver.find_element_by_xpath("//span[contains(text(),'Last Day')]") #Identify your element
driver.execute_script("arguments[0].click();", day) # CLick it with help of JS
out Put:
You can try the following approach:
driver.get('https://www.glassdoor.com/Job/boston-data-scientist-jobs-SRCH_IL.0,6_IC1154532_KO7,21.htm')
driver.maximize_window()
expand_element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.ID, 'filter_fromAge')))
expand_element.click()
target_text = 'Last 3 Days'
target_element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//ul[#class="css-1dv4b0s ew8xong0"]/li/span[text()="{}"]'.format(target_text))))
target_element.click()
WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '//div[#id="filter_fromAge"]/span[text()="{}"]'.format(target_text))))
To select the option with text as Last Day that displays only after clicking on the dropdown you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategy:
Using XPATH:
driver.get('https://www.glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=false&clickSource=searchBtn&typedKeyword=data+sc&sc.keyword=data+scientist&locT=C&locId=1154532&jobType=')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#filter_fromAge>span"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#id='PrimaryDropdown']/ul//li//span[#class='label' and contains(., 'Last Day')]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
I am trying to automate an extraction of stock prices in my broker website because yahoo and google finance have delays. But i need the code to wait for the 'home-broker' to be online so it can continue with scraping...
Here is my code:
expected = 'online'
while True:
try:
driver.find_element_by_xpath('//*[#id="spnStatusConexao"]').text == expected
except NoSuchElementException:
print('offline')
else:
print('online')
But, while testing it, it prints 'online' even when the homebroker displays 'offline' message.
I need to print 'offline' when the xpath text is equal to: offline . And to print 'online' when xpath text is equal to: online.
EDIT:
outter HTML:
<span id="spnStatusConexao" hover="DV_bgConexao" class="StatusConexao online">online</span>
XPath:
//*[#id="spnStatusConexao"]
Full XPath:
/html/body/form/div[9]/div/div/p[2]/span
expected_conditions in Python has a built in operation for this called text_to_be_present_in_element. The below code snippet will wait for the span element to contain the text online:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 10).until(EC.text_to_be_present_in_element((By.ID, "spnStatusConexao"), 'online'))
If this does not work, you can try invoking WebDriverWait on the presence_of_element_located and include the text in your XPath query:
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "//span[#id='spnStatusConexao' and contains(text(),'online')]")))