<div dojoattachpoint="titleNode" class="dijitTitlePaneTextNode" tabindex="0">Transfer ( XXXXX.YYYYYY#ABCD.COM 10-Aug-2021 02:24:39 PM GMT+05:30 ) Top Bottom</div>
The above snippet I got from inspect element on Google Chrome. I am want to get this "Transfer ( XXXXX.YYYYYY#ABCD.COM 10-Aug-2021 02:24:39 PM GMT+05:30 ) " value to be printed in output.
The problem is that XPATH get changed dynamically every time.
I have already tried the below, but it did not work.
get_div = driver.find_element_by_class_name("dijitTitlePaneTextNode")
get_div = driver.find_element_by_xpath('//a[contains(text(), "Transfer ( ")]')
get_div = driver.find_elements_by_xpath('//div[#class = "dijitTitlePaneTextNode"]')
Try this CSS selector and see if that helps:
div[dojoattachpoint='titleNode']
in code:
get_div = driver.find_element_by_css_selector("div[dojoattachpoint='titleNode']")
print(get_div.text)
if it needs explicit wait:
wait = WebDriverWait(driver, 10)
a = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[dojoattachpoint='titleNode']"))).text
print(a)
Imports:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Update 1 :
The below xpath should represent the first element,
(//div[#dojoattachpoint='titleNode'])[1]
and the below should represent the 2nd one..
(//div[#dojoattachpoint='titleNode'])[2]
so using xpath indexing you can get the unique match.
Related
I am trying to scrape data from this website using Selenium.
There are three features in data, "Value", "Net change" and "percent change", including values for net and percentage changes for 1, 3, 6, and 12 months. I want to fetch 1 month's net change and percent change. For that, I need to click on the check boxes and click on the update button.
Now, I performed these actions using selenium's find element by XPath method but for percent change, I needed to use the ActionChains command, as I was getting "Element not clickable error".
When I execute the code, all three features should occur in the downloaded csv. But that's not happening. I am just able to fetch "Value" and "1 Month Net change". If anyone knows, may I know, why the is not getting updated and or how to fix it? Thanks
My code:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
driver.get("https://beta.bls.gov/dataViewer/view/timeseries/CUUR0000SA0")
soup = BeautifulSoup(driver.page_source, "html.parser", from_encoding='utf-8')
driver.find_element(By.XPATH, '/html/body/div[2]/div/div/div[4]/div/div[1]/form/div[2]/fieldset/div[1]/table/tbody/tr[1]/td[1]/label/input').click() //1 month net change
element = WebDriverWait(driver, 60).until(EC.element_to_be_clickable((By.XPATH, '// [#id="percent_monthly_changes_div"]/table/tbody/tr[1]/td[1]/label/input')))
ActionChains(driver).move_to_element(element).click().perform() //1 month percent change
driver.find_element(By.XPATH, '/html/body/div[2]/div/div/div[4]/div/div[1]/form/div[4]/input').click() //update button
driver.find_element(By.XPATH, '//*[#id="csvclickCU"]').click() //download csv button
The website is showing N/A in the column of 1 Month Net change.
if you still not getting 1 month % change value you can do
driver.execute_script('document.querySelector("#percent_monthly_changes_div > table > tbody > tr:nth-child(1) > td:nth-child(1) > label > input").click()')
instead of:
element = WebDriverWait(driver, 60).until(EC.element_to_be_clickable((By.XPATH, '// [#id="percent_monthly_changes_div"]/table/tbody/tr[1]/td[1]/label/input')))
ActionChains(driver).move_to_element(element).click().perform()
this might not be the optimal solution, but it works fine.
and 1 month net change value is not given from the website itself.
To click on the elements with text as 1-Month Net Change and 1-Month % Change using ActionChains will be an overhead and you can avoid it easily.
Ideally, you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using CSS_SELECTOR:
driver.get("https://beta.bls.gov/dataViewer/view/timeseries/CUUR0000SA0")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[value='1N']"))).click()
driver.find_element(By.CSS_SELECTOR, "input[value='1P']").click()
Using XPATH:
driver.get("https://beta.bls.gov/dataViewer/view/timeseries/CUUR0000SA0")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#value='1N']"))).click()
driver.find_element(By.XPATH, "//input[#value='1P']").click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
Im using Selenium with Python3 on a Service Now Website.
So the process is as follows: selenium loads up the ServiceNow URL and then I use sendKeys to automate typing in of username and password, then the page is loaded which has a table of incidents I need to extract. Unfortunately I have to login in every single time because of the group policy I have.
This works up until I have to find the dynamically rendered Javascript table with data and I can't for the life of me seem to find it. I even tried to put a sleep in there for 15 seconds to allow it to load.
I also double checked the XPaths and Id / Class names and they match up. When I print query.page_source I don't see anything rendered by JS.
I've used beautiful soup too but that also doesn't work.
Any ideas?
from time import sleep
from collections import deque
from selenium import webdriver
from selenium.webdriver.support.ui import Select # for <SELECT> HTML form
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
query = webdriver.Firefox()
get_query = query.get("SERVICENOW URL")
query.implicitly_wait(10)
login_username = query.find_element_by_id('username')
login_password = query.find_element_by_id('password')
login_button = query.find_element_by_id('signOnButton')
username = "myUsername"
password = "myPassword"
login_username.send_keys(username)
login_password.send_keys(password)
login_button.click()
sleep(10)
incidentTableData = []
print(query.page_source)
// *** THESE ALL FAIL AND RETURN NONE ***
print(query.find_elements())
tableById = query.find_element_by_id('service-now-table-id')
tableByXPath = query.find_element_by_xpath('service-now-xpath')
tableByClass = query.find_element_by_id('service-now-table-class')
Since it's a dynamically rendered Javascript table, I would suggest you to implement explicit wait in your code.
so instead of this :
tableById = query.find_element_by_id('service-now-table-id')
tableByXPath = query.find_element_by_xpath('service-now-xpath')
tableByClass = query.find_element_by_id('service-now-table-class')
re-write these lines like this :
wait = WebDriverWait(query, 10)
service_now_with_id = wait.until(EC.element_to_be_clickable((By.ID, "service-now-table-id")))
service_now_with_xpath = wait.until(EC.element_to_be_clickable((By.XPATH, "service-now-xpath")))
service_now_with_class = wait.until(EC.element_to_be_clickable((By.ID, "service-now-table-class")))
You are gonna need to use the below imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as E
PS :- service_now_with_id, service_now_with_xpath, service_now_with_class, these are web elements returned by explicit waits. you may wanna have to interact with them as per your requirement meaning, clicking on it or sending keys or whatever.
I'm using selenium with Python 3 to retrieve the contents from an element on a webpage. There is an element with dynamic text.
<div id="accResp">Please wait...</div>
The "Please wait..." section changes to different strings like "Password Incorrect", "Loading".
How do I wait until the string changes to anything except "Please wait..."?
Based on your functionality you will see text in your web page so you can handle text using below xpath's:
wait = WebDriverWait(driver, 10)
password Incorrect
passwordIncorrect = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[contains(text(),'Password Incorrect')]")))
Loading
loading = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[contains(text(),'Loading')]")))
Please wait...
waitText= wait.until(EC.element_to_be_clickable((By.XPATH, "//div[contains(text(),'Please wait...')]")))
Note : please add below imports to your solution
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
I am having trouble filling the form which using java creates a pair of random numbers and upon entering the right answer one can proceed:
<div class="form-group has-feedback">
<label for="captcha" class="..." id="captchaOperation">6 + 20 =</label>
I want to extract the two numbers, add them up (it is mainly the sum operation, but generic solutions are also appreciated).
I have used different combinations with send_keys, e.g.:
mathcaptcha = action.send_keys("""document.querySelectorAll("span#captchaOperation").firstChild.nodeValue;""")
print(mathcaptcha)
which all return:
<selenium.webdriver.common.action_chains.ActionChains object at 0x11238cd68>
Then I used the execute_script with few different scripts, e.g.:
mathcaptcha = driver.execute_script("""document.getElementById("captchaOperation");""")
print(mathcaptcha)
and got None printed out.
How could I get the 6 + 20 out of the code or 26 as final result?
To make execute_script return a value to you, use return :
mathcaptha = driver.execute_script("return document.getElementById('captchaOperation').textContent;")
print(mathcaptha)
You can try the following code:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
mathcaptha = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "captchaOperation"))).text.replace(" =", "")
print(eval(mathcaptha))
As per the HTML you have shared to extract 6 + 20 you can use either of the following solution:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
#other lines of code
captcha_text = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//label[#id='captchaOperation' and #for='captcha']"))).get_attribute("innerHTML")
print((captcha_text.replace("=", "")), eval(captcha_text.replace("=", "")))
I'm trying to automatically run the currency converter in https://www.mastercard.us/en-us/consumers/get-support/convert-currency.html using Selenium in Python. Here is what I got so far:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link1 = 'https://www.mastercard.us/en-us/consumers/get-support/convert-currency.html'
driver1 = webdriver.PhantomJS()
driver1.get(link1)
script = """ var select = arguments[0];
for(var i = 0; i < select.options.length; i++) {
if(select.options[i].value == arguments[1]) {
select.options[i].selected = true;
}
}
"""
driver1.find_element_by_id('getDate').send_keys('05-Sep-2017')
select = driver1.find_element_by_id('firstID')
driver1.execute_script(script, select, 'USD');
driver1.find_element_by_name('txtTAmt').send_keys('1.00')
driver1.find_element_by_name('txtBankFee').send_keys('0.00')
select = driver1.find_element_by_id('newID')
driver1.execute_script(script, select, 'EUR');
driver1.find_element_by_id('btnSubmit').click()
wait = WebDriverWait(driver1, 100)
element = wait.until(EC.presence_of_element_located((By.XPATH,
'//*[#name="txtCardAmt" and text() != ""]')))
print(element.text)
The problem is that the field "txtCardAmt" never gets populated and I'm getting a timeout exception. My question is, how can I wait for the server to finish the computation?
PS: I know there is easier ways to select options using the Select class, however in this website they do not work for some reason.
Your problem is that you wait until the text of the element with name txtCardAmt is not empty. The problem is that this is always true.
If you take a look to the interested html:
<input type="text" name="txtCardAmt" ng-model="mcz.txtCardAmt"
class="mczreadonly ng-pristine ng-valid mczblue" placeholder="0"
readonly="readonly" disabled="">
you can see that the there isn't text.
The info that you are you looking for (not visible in the html) is in the attribute value:
That is 7.38 in my example.
So:
elem = driver1.find_element_by_name('txtCardAmt')
value = elem.get_attribute("value")
print(value)
Your code regarding the selection of the date and the currencies doesn't work. In my example I used the xpath in order to do that. I'm sure there are better way to do this tasks. I used the xpath returned by the tools of the inspector of my browser.
The entire example:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link1 = 'https://www.mastercard.us/en-us/consumers/get-support/convert-currency.html'
driver1 = webdriver.PhantomJS(executable_path=r'/pathTo/phantomjs')
driver1.get(link1)
driver1.find_element_by_id('getDate').click()
wait = WebDriverWait(driver1, 20)
wait.until(EC.presence_of_element_located((By.XPATH,"/html/body/div[1]/div/div/div/div[2]/div[3]/div/div/div[2]/div/div/div/a[1]/span")))
driver1.find_element_by_xpath("/html/body/div[1]/div/div/div/div[2]/div[3]/div/div/div[2]/div/div/div/a[1]/span").click()
driver1.find_element_by_xpath("//*[#id='transactiondatepicker']/div/table/tbody/tr[2]/td[3]/a").click()
#select = driver1.find_element_by_id('firstID')
#driver1.execute_script(script, select, 'USD');
driver1.find_element_by_xpath("//*[#id='mczRowC']/div[2]/button").click()
wait.until(EC.presence_of_element_located((By.XPATH,"//*[#id='mczRowC']/div[2]/div/ul/li[146]/a")))
driver1.find_element_by_xpath("//*[#id='mczRowC']/div[2]/div/ul/li[146]/a").click()
driver1.find_element_by_name('txtTAmt').send_keys('1.00')
driver1.find_element_by_name('txtBankFee').send_keys('2.00')
#select = driver1.find_element_by_id('newID')
#driver1.execute_script(script, select, 'EUR');
driver1.find_element_by_xpath("//*[#id='mczRowD']/div[2]/button").click()
wait.until(EC.presence_of_element_located((By.XPATH,"//*[#id='mczRowD']/div[2]/div/ul/li[49]/a")))
driver1.find_element_by_xpath("//*[#id='mczRowD']/div[2]/div/ul/li[49]/a").click()
driver1.find_element_by_id('btnSubmit').click()
time.sleep(3)
elem = driver1.find_element_by_name('txtCardAmt')
value = elem.get_attribute("value")
print(value)