I need help to select the option "Last 7 days" from the following dropdown using python and selenium:
here is the html for part of the dropdown:
<div class="date-range" id="yui_3_18_1_1_1614002255126_175" style="margin-right: 25px;">
<div class="option-select global default" id="yui_3_18_1_1_1614002255126_174">
<div class="select-open" id="yui_3_18_1_1_1614002255126_173">
<div class="select-title" id="yui_3_18_1_1_1614002255126_172">
<span class="option-title">DATE RANGE:</span>
<span class="option-selection" id="yui_3_18_1_1_1614002255126_170">Last 30 days</span>
</div>
<div class="dropdown-arrow"></div>
</div>
<div class="select-body" id="yui_3_18_1_1_1614002255126_190">
<div class="option">
Today
<span class="extra-info-wrapper">
<span> (</span>
<span class="extra-info">22 Feb</span>
<span>)</span>
</span></div>
<div class="option">
Yesterday
<span class="extra-info-wrapper">
<span> (</span>
<span class="extra-info">21 Feb</span>
<span>)</span>
</span></div>
<div class="option">
This week
<span class="extra-info-wrapper">
<span> (</span>
<span class="extra-info">Monday - Today</span>
<span>)</span>
</span></div>
<div class="option" id="yui_3_18_1_1_1614002255126_189">
Last 7 days
<span class="extra-info-wrapper">
<span> (</span>
<span class="extra-info">16 Feb - Today</span>
<span>)</span>
</span></div>
<div class="option">
This month
<span class="extra-info-wrapper">
<span> (</span>
<span class="extra-info">1 Feb - Today</span>
<span>)</span>
</span></div>
My code so far is:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
myUsername="Xxx”
myPassword="Xxx”
driver.get("https://uk.ixl.com/signin/sop")
driver.find_element_by_xpath('//*[#id="siusername"]').send_keys(myUsername)
driver.find_element_by_xpath('//*[#id="sipassword"]').send_keys(myPassword)
driver.find_element_by_xpath('//*[#id="custom-signin-button"]').click()
time.sleep(1)
#select report
driver.get("https://uk.ixl.com/analytics/students-quickview?teacherId=125756982")
time.sleep(5)
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'date-range'))).click()
driver.find_element_by_id('yui_3_18_1_1_1614002255126_189').click()
I am getting the error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate
element: {"method":"css selector","selector":"[id="yui_3_18_1_1_1614002255126_189"]"}
This is only my second python project so forgive any lack of understanding, this is just a hobby, but I've been stuck for 2 days trying all sorts but nothing works, any help would be appreciated (even links to videos that would help solve this so I can learn), thanks
You can hard code text in xpath like this:
wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'date-range'))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, '//div[text()="Last 7 days"]'))).click()
Please change the text if you want other options.
What I usually do is:
driver.find_element_by_xpath(f'//select[#name="nameOfYourButton"]/option[text()=""TextToBeSelected"]').click()
Where nameOfYourButton and TextToBeSelected must be replaced with your specific variable
In particular, I can see that TextToBeSelected should be Last 7 days while from the HTML snippet that you are showing I can't see the name of the button.
Use WebDriverWait() and following xpath to click on Last 7 days.
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='select-body']//div[#class='option' and contains(.,'Last 7 days')]"))).click()
Related
How to select text content from multiple DIV elements using selenium?
On the website I intend to collect information it contains div and span with the same class.
How can I collect this information separately?
I need the contents inside the panel-body div > span of each block
driver.find_element_by_xpath(".//div[#class='panel-body'][1]/span[1]").text
driver.find_element_by_xpath(".//div[#class='panel-body'][1]/span[2]").text
driver.find_element_by_xpath(".//div[#class='panel-body'][1]/span[3]").text
driver.find_element_by_xpath(".//div[#class='panel-body'][2]/span[1]").text
driver.find_element_by_xpath(".//div[#class='panel-body'][2]/span[2]").text
html
<div class="panel-heading">
<h3 class="panel-title">Identificação</h3>
</div>
<div class="panel-body">
<span class="spanValorVerde">TEXT</span><br>
<span style="font-size:small;color:gray">TEXT</span><br>
<br>
<span class="spanValorVerde">TEXT</span>
</div>
</div>
<div class="panel panel-success">
<div class="panel-heading">
<h3 class="panel-title">Situação Atual</h3>
</div>
<div class="panel-body">
<span class="spanValorVerde">TEXT</span> <br>
<span class="spanValorVerde">TEXT</span>
</div>
</div>
I am expecting that "select text" meant "get text".
first for loop:
count = driver.find_elements_by_xpath(".//div[#class='panel-body'][i]")
second for loop with count iteration:
driver.find_element_by_xpath(".//div[#class='panel-body'][i]/span[j]").text
If you search with findElements with ".//div[#class='panel-body'][i]" will give you the total element present, then add another loop for .//div[#class='panel-body'][i]/span[j] and then get text. Hope it helps!
To extract the texts e.g. TEXT, from each <span> using Selenium and python you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR and get_attribute("innerHTML"):
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.panel.panel-success div.panel-body span.spanValorVerde")))])
Using XPATH and text attribute:
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[#class='panel panel-success']//div[#class='panel-body']//span[#class='spanValorVerde']")))])
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Outro
Link to useful documentation:
get_attribute() method Gets the given attribute or property of the element.
text attribute returns The text of the element.
Difference between text and innerHTML using Selenium
Cant click on the link due to the fact, all of the value I keep trying to select has the same name as well within the tables.
Tried xpath,by link, by text, by css none of it works. My current code:
driver.find_element_by_xpath('//[#id="homeapp"]/div/div/div[3]/div[1]/table/tbody/tr[3]/td[2]/div[1]/div[2]/span[1]/a').click()
Here is the relevant html code:
<tr class="bui-table__row"><td class="peg-table__cell--no-label bui-table__cell" data-heading="ID" scope="row"><a href="../../extranet_ng/manage/index.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111" class="bui-link bui-link--secondary" target="_blank" data-track-ga="Groups: Home,Active Properties,Property ID">
1111(number i can click aswell)
</a></td> <td class="peg-table__cell--no-label bui-table__cell" data-heading="Name" scope="row"><div class="bui-avatar-block"><div class="peg-_-avatar-shrink bui-avatar"><img src="https://q-xx.bstatic.com/xdata/images/extra/square60/189090859.jpg?k=cc455d4d570f0e0c86a1b8329b16d53fdba96d78677cc87151d6181e45d38ec5&o=" alt="text i want to click" class="bui-avatar__image"></div> <div class="bui-avatar-block__text"><span class="bui-avatar-block__title"><a href="../../extranet_ng/manage/index.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111" class="peg-property-link bui-link bui-link--secondary" target="_blank" data-track-ga="Groups: Home,Active Properties,Property Name">
Text i want to click
<!----> <!----></a></span> <span class="bui-avatar-block__subtitle"><!----> </span></div></div> <div class="bui-spacer--smaller"></div> <div class="peg-score-bar__inline bui-score-bar"><div class="bui-score-bar__item"><div class="bui-score-bar__header"><h2 class="bui-score-bar__title"></h2> <span class="bui-score-bar__score">93%</span></div> <div class="bui-score-bar__bar"><span data-value="9.3" class="bui-score-bar__value" style="width: 93%;"></span></div></div></div></td> <td class="peg-table__cell--no-label bui-table__cell" data-heading="Location"><div class="peg-address-wrapper"><span class="peg-flag-wrapper"><div class="peg-flag bui-flag"><img src="https://q.bstatic.com/backend_static/common/flags/16/nl/314ce6500532e846e25d6e3a7c824ef17c968446.png" class="bui-flag__flag" style="height: auto;"> <!----></div></span>
TEXT2
</div></td> <td class="peg-table__cell--no-label bui-table__cell" data-heading="Status"><span>Open/Bookable</span></td> <td class="bui-table__cell bui-table__cell--center" data-heading="Arrivals/departures & tomorrow"><span class="peg-counter--arrivals"><a href="../../extranet_ng/manage/search_reservations.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111" data-track-ga="Groups: Home,Active Properties,Arrivals" target="_blank" class="peg-counter peg-counter--has"><span aria-label="80 unread" class="bui-bubble">
VALUE1
</span></a></span> <span class="peg-counter--departures"><a href="../../extranet_ng/manage/search_reservations.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111&type=departure" data-track-ga="Groups: Home,Active Properties,Departures" target="_blank" class="peg-counter peg-counter--has"><span aria-label="80 unread" class="bui-bubble">
VALUE2
</span></a></span></td> <td class="bui-table__cell bui-table__cell--center" data-heading="Guest Messages"><a href="../../extranet_ng/manage/messaging_inbox.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111" data-track-ga="Groups: Home,Active Properties,Guest Messages" target="_blank" class="peg-counter peg-counter--has"><span aria-label="102 unread" class="bui-bubble">
VALUE3
</span></a></td> <td class="bui-table__cell bui-table__cell--center" data-heading="extranet.com Messages"><a href="../../extranet_ng/manage/inbox.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111" data-track-ga="Groups: Home,Active Properties,extrabnet.com Messages" target="_blank" class="peg-counter"><span aria-label="0 unread" class="bui-bubble">
0
</span></a></td></tr>
If you want to click the link which contains avatar in its class attribute use the following XPath expression:
//div[contains(#class,'avatar')]/descendant::a[contains(text(), 'text i want to click')]
where:
contains() is XPath Function allowing partial match on the attributes values and/or text
descendant - XPath Axis which matches children of the current node and their respective children
last part limits the search to hyperlinks which contain text you're looking for
The desired element is a ReactJS enabled element, so you have to induce WebDriverWait for the element to be clickable() and you can use either of the following solutions:
Using PARTIAL_LINK_TEXT:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "text i want to click"))).click()
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.peg-property-link.bui-link.bui-link--secondary[href*='/extranet_ng/manage/index'][data-track-ga*='Active Properties']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='peg-property-link bui-link bui-link--secondary' and contains(#href, '/extranet_ng/manage/index')][starts-with(#data-track-ga, 'Groups')]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I am writing an automation for work and am stuck with a dropdown. The particular select box in question is as follows:
<span class="a-dropdown-container" id="select-size-bulk-asin">
<select name="display_type" class="a-native-dropdown">
<option value="SMALL-IMAGES">SMALL-IMAGES</option>
<option value="LARGE-IMAGES">LARGE-IMAGES</option>
<option value="TEXT">TEXT</option>
</select>
<span tabindex="-1" data-a-class="a-spacing-small" class="a-button a-button-dropdown a-spacing-small">
<span class="a-button-inner">
<span class="a-button-text a-declarative" data-action="a-dropdown-button" aria-haspopup="true" role="button" tabindex="0" aria-pressed="false" aria-owns="2_dropdown_combobox">
<span class="a-dropdown-prompt">SMALL-IMAGES</span>
</span>
<i class="a-icon a-icon-dropdown"></i>
</span>
</span>
</span>
It defaults to 'SMALL Images' and I would like to select the 'TEXT' option. I am receiving element not clickable error. The page is simple and the element is visible on the screen.
The list of methods I did try are:
Used WebDriverWait to wait for the element to be visible;
Used WebDriverWait to wait for the element to be clickable;
Used the select class to set the selected option;
I also read through a question.
I am thinking if I should just go to the next element and send Shift+Tabs until I reach this drop down and then down arrow keys. But would like to use that only as the last resort.
NOTE:
- I am using Python 3 and Chrome.
You can try this code to select value from drop down :
select = Select(driver.find_element_by_id('select-size-bulk-asin'))
select.select_by_visible_text('TEXT')
However,as you have mentioned you are receiving element not clickable exception. you can try this code :
WebDriverWait(browser, 30).until(EC.element_to_be_clickable((By.ID, "select-size-bulk-asin")))
As a last resort you can go ahead with :
drop_down= driver.find_element_by_id("select-size-bulk-asin")
drop_down.click()
actions = ActionChains(driver)
actions.send_keys(Keys.ARROW_DOWN)
actions.send_keys(Keys.ARROW_DOWN)
actions.send_keys(Keys.ENTER)
actions.perform()
I am currently trying to scrap information of a particular ecommerce site and i only want to get product information like product name, price, color and sizes of only products whose prices have been slashed.
i am currently using xpath
this is my python scraping code
from lxml import html
import requests
class CategoryCrawler(object):
def __init__(self, starting_url):
self.starting_url = starting_url
self.items = set()
def __str__(self):
return('All Items:', self.items)
def crawl(self):
self.get_item_from_link(self.starting_url)
return
def get_item_from_link(self, link):
start_page = requests.get(link)
tree = html.fromstring(start_page.text)
names = tree.xpath('//span[#class="name"][#dir="ltr"]/text()')
print(names)
Note this is not the original URL
crawler = CategoryCrawler('https://www.myfavoriteecommercesite.com/')
crawler.crawl()
When the program is Run ... These are the HTML Content Gotten from the E-commerce Site
Div of Products With Price Slash
div class="products-info">
<h2 class="title"><span class="brand ">Apple </span> <span class="name" dir="ltr">IPhone X 5.8-Inch HD (3GB,64GB ROM) IOS 11, 12MP + 7MP 4G Smartphone - Silver</span></h2>
<div class="price-container clearfix">
<span class="sale-flag-percent">-22%</span>
<span class="price-box ri">
<span class="price ">
<span data-currency-iso="NGN">₦</span>
<span dir="ltr" data-price="388990">388,990</span>
</span>
<span class="price -old ">
<span data-currency-iso="NGN">₦</span>
<span dir="ltr" data-price="500000">500,000</span>
</span>
</span>
</div>
div
Div of Products with No Price Slash
div class="products-info">
<h2 class="title"><span class="brand ">Apple </span> <span class="name" dir="ltr">IPhone X 5.8-Inch HD (3GB,64GB ROM) IOS 11, 12MP + 7MP 4G Smartphone - Silver</span></h2>
<div class="price-container clearfix">
<span class="price-box ri">
<span class="price ">
<span data-currency-iso="NGN">₦</span>
<span dir="ltr" data-price="388990">388,990</span>
</span>
</span>
</div>
div
Now this is my exact Question
i want to know how to select only the parent divs i.e
div class="price-container clearfix"> that also contains any of these children span classes
span class="price -old "> or
span class="sale-flag-percent">
Thank you all
One solution would be get all <div class="price-container clearfix"> and iterate, checking with the string of the whole element that your keywords exist.
But a better solution would be to use conditionals with xpath:
from lxml import html
htmlst = 'your html'
tree=html.fromstring(htmlst)
divs = tree.xpath('//div[#class="price-container clearfix" and .//span[#class = "price -old " or #class = "sale-flag-percent"] ]')
print(divs)
This get all divs where class="price-container clearfix" and then check if contains span with the searched classes.
I am trying to make a list of web elements, but it can not seem to find the elements on the web page, although did worked 3 days ago and i can not find any changes in the web page.
this is the html code :
<li id="wlg_41410" class="leagueWindow " dataid="41410">
<h5 style="cursor: pointer; cursor: hand;" onclick="TodaysEventsLeagueWindow.minimizeRestoreClick(41410)">Europa League</h5>
<div class="bet_type select" id="_bet_types"></div>
<div class="bet_type lastscore ">
<h6>1X2 FT </h6>
<div class="types_bg">
<!--[if IE]> <div id="IEroot"> <![endif]-->
<div class="first_buttons_line">
</div>
<!--[if IE]> </div> <![endif]-->
<div class="time"> 23/11 | 18:00 </div>
<div class="bets ml">
</div>
<div class="time"> 23/11 | 20:00 </div>
<div class="bets ml">
</div>
<div class="time"> 23/11 | 20:00 </div>
<div class="bets ml">
</div>
<div class="time"> 23/11 | 20:00 </div>
<div class="bets ml">
</div>
<div class="time"> 23/11 | 20:00 </div>
<div class="bets ml">
</div>
<div class="clr"></div>
</div>
</div> <span class="x" onclick="TodaysEventsLeagueWindow.closeLeagueWindow(41410)"></span>
</li>
i am trying to make a list from the <div class="bets ml"></div> elements
but keep getting the selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document exception , as if selenium can't find the web element.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import StaleElementReferenceException
import time
driver.get("https://www.luckia.es/apuestas")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it("sbtechBC"))
eventos_de_hoy = driver.find_element_by_id("today_event_btn")
eventos_de_hoy.click()
ligi_len = len(WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "leagueWindow "))))
print(ligi_len)
for index in range(ligi_len):
item = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "leagueWindow ")))[index]
driver.execute_script("arguments[0].scrollIntoView(true);", item)
nume_liga = item.find_element_by_tag_name("h5").text
time.sleep(3)
print('try', nume_liga)
meci = item.find_elements_by_xpath("//*[#class='bets ml']")
print("there are", len(meci), "in one liga")
the reason for the index is that the iframe refreshes every 25 sec.
i also tried meci = item.find_elements_by_css_selector('.bets.ml') and meci = item.find_elements_by_class_name('ml')
Why should i be able to extract the <h5></h5> element and not the other elements?
From your code block, its pretty clear you have just managed to cover up the real issue through time.sleep(3) as follows :
nume_liga = item.find_element_by_tag_name("h5").text
time.sleep(3)
print('try', nume_liga)
While invoking print() for a text, I am not sure why time.sleep(3) was induced. So our main issue got covered up there. But as the List was already created, you are able to print('try', nume_liga)
But next, when you do meci = item.find_elements_by_xpath("//*[#class='bets ml']") you face a StaleElementReferenceException because the HTML DOM have changed.
A closer look into the <h5> tag reveals it have a onclick() event as :
<h5 style="cursor: pointer; cursor: hand;" onclick="TodaysEventsLeagueWindow.minimizeRestoreClick(41410)">Europa League</h5>
A wild guess, while invoking .text on <h5> tag, the HTML DOM changes.
Solution :
A possible solution with your current code block may be to use getAttribute("innerHTML") instead of .text. So your line of code will be :
nume_liga = item.find_element_by_tag_name("h5").get_attribute("innerHTML")