python selenium selection option from dropdown - python-3.x

I need help to select the option "Last 7 days" from the following dropdown using python and selenium:
here is the html for part of the dropdown:
<div class="date-range" id="yui_3_18_1_1_1614002255126_175" style="margin-right: 25px;">
<div class="option-select global default" id="yui_3_18_1_1_1614002255126_174">
<div class="select-open" id="yui_3_18_1_1_1614002255126_173">
<div class="select-title" id="yui_3_18_1_1_1614002255126_172">
<span class="option-title">DATE RANGE:</span>
<span class="option-selection" id="yui_3_18_1_1_1614002255126_170">Last 30 days</span>
</div>
<div class="dropdown-arrow"></div>
</div>
<div class="select-body" id="yui_3_18_1_1_1614002255126_190">
<div class="option">
Today
<span class="extra-info-wrapper">
<span> (</span>
<span class="extra-info">22 Feb</span>
<span>)</span>
</span></div>
<div class="option">
Yesterday
<span class="extra-info-wrapper">
<span> (</span>
<span class="extra-info">21 Feb</span>
<span>)</span>
</span></div>
<div class="option">
This week
<span class="extra-info-wrapper">
<span> (</span>
<span class="extra-info">Monday - Today</span>
<span>)</span>
</span></div>
<div class="option" id="yui_3_18_1_1_1614002255126_189">
Last 7 days
<span class="extra-info-wrapper">
<span> (</span>
<span class="extra-info">16 Feb - Today</span>
<span>)</span>
</span></div>
<div class="option">
This month
<span class="extra-info-wrapper">
<span> (</span>
<span class="extra-info">1 Feb - Today</span>
<span>)</span>
</span></div>
My code so far is:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
myUsername="Xxx”
myPassword="Xxx”
driver.get("https://uk.ixl.com/signin/sop")
driver.find_element_by_xpath('//*[#id="siusername"]').send_keys(myUsername)
driver.find_element_by_xpath('//*[#id="sipassword"]').send_keys(myPassword)
driver.find_element_by_xpath('//*[#id="custom-signin-button"]').click()
time.sleep(1)
#select report
driver.get("https://uk.ixl.com/analytics/students-quickview?teacherId=125756982")
time.sleep(5)
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'date-range'))).click()
driver.find_element_by_id('yui_3_18_1_1_1614002255126_189').click()
I am getting the error:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate
element: {"method":"css selector","selector":"[id="yui_3_18_1_1_1614002255126_189"]"}
This is only my second python project so forgive any lack of understanding, this is just a hobby, but I've been stuck for 2 days trying all sorts but nothing works, any help would be appreciated (even links to videos that would help solve this so I can learn), thanks

You can hard code text in xpath like this:
wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'date-range'))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, '//div[text()="Last 7 days"]'))).click()
Please change the text if you want other options.

What I usually do is:
driver.find_element_by_xpath(f'//select[#name="nameOfYourButton"]/option[text()=""TextToBeSelected"]').click()
Where nameOfYourButton and TextToBeSelected must be replaced with your specific variable
In particular, I can see that TextToBeSelected should be Last 7 days while from the HTML snippet that you are showing I can't see the name of the button.

Use WebDriverWait() and following xpath to click on Last 7 days.
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='select-body']//div[#class='option' and contains(.,'Last 7 days')]"))).click()

Related

How to extract the text content from multiple span elements using Selenium and Python

How to select text content from multiple DIV elements using selenium?
On the website I intend to collect information it contains div and span with the same class.
How can I collect this information separately?
I need the contents inside the panel-body div > span of each block
driver.find_element_by_xpath(".//div[#class='panel-body'][1]/span[1]").text
driver.find_element_by_xpath(".//div[#class='panel-body'][1]/span[2]").text
driver.find_element_by_xpath(".//div[#class='panel-body'][1]/span[3]").text
driver.find_element_by_xpath(".//div[#class='panel-body'][2]/span[1]").text
driver.find_element_by_xpath(".//div[#class='panel-body'][2]/span[2]").text
html
<div class="panel-heading">
<h3 class="panel-title">Identificação</h3>
</div>
<div class="panel-body">
<span class="spanValorVerde">TEXT</span><br>
<span style="font-size:small;color:gray">TEXT</span><br>
<br>
<span class="spanValorVerde">TEXT</span>
</div>
</div>
<div class="panel panel-success">
<div class="panel-heading">
<h3 class="panel-title">Situação Atual</h3>
</div>
<div class="panel-body">
<span class="spanValorVerde">TEXT</span> <br>
<span class="spanValorVerde">TEXT</span>
</div>
</div>
I am expecting that "select text" meant "get text".
first for loop:
count = driver.find_elements_by_xpath(".//div[#class='panel-body'][i]")
second for loop with count iteration:
driver.find_element_by_xpath(".//div[#class='panel-body'][i]/span[j]").text
If you search with findElements with ".//div[#class='panel-body'][i]" will give you the total element present, then add another loop for .//div[#class='panel-body'][i]/span[j] and then get text. Hope it helps!
To extract the texts e.g. TEXT, from each <span> using Selenium and python you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR and get_attribute("innerHTML"):
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.panel.panel-success div.panel-body span.spanValorVerde")))])
Using XPATH and text attribute:
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[#class='panel panel-success']//div[#class='panel-body']//span[#class='spanValorVerde']")))])
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Outro
Link to useful documentation:
get_attribute() method Gets the given attribute or property of the element.
text attribute returns The text of the element.
Difference between text and innerHTML using Selenium

Selenium selecting link within table - with all the same name

Cant click on the link due to the fact, all of the value I keep trying to select has the same name as well within the tables.
Tried xpath,by link, by text, by css none of it works. My current code:
driver.find_element_by_xpath('//[#id="homeapp"]/div/div/div[3]/div[1]/table/tbody/tr[3]/td[2]/div[1]/div[2]/span[1]/a').click()
Here is the relevant html code:
<tr class="bui-table__row"><td class="peg-table__cell--no-label bui-table__cell" data-heading="ID" scope="row"><a href="../../extranet_ng/manage/index.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111" class="bui-link bui-link--secondary" target="_blank" data-track-ga="Groups: Home,Active Properties,Property ID">
1111(number i can click aswell)
</a></td> <td class="peg-table__cell--no-label bui-table__cell" data-heading="Name" scope="row"><div class="bui-avatar-block"><div class="peg-_-avatar-shrink bui-avatar"><img src="https://q-xx.bstatic.com/xdata/images/extra/square60/189090859.jpg?k=cc455d4d570f0e0c86a1b8329b16d53fdba96d78677cc87151d6181e45d38ec5&o=" alt="text i want to click" class="bui-avatar__image"></div> <div class="bui-avatar-block__text"><span class="bui-avatar-block__title"><a href="../../extranet_ng/manage/index.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111" class="peg-property-link bui-link bui-link--secondary" target="_blank" data-track-ga="Groups: Home,Active Properties,Property Name">
Text i want to click
<!----> <!----></a></span> <span class="bui-avatar-block__subtitle"><!----> </span></div></div> <div class="bui-spacer--smaller"></div> <div class="peg-score-bar__inline bui-score-bar"><div class="bui-score-bar__item"><div class="bui-score-bar__header"><h2 class="bui-score-bar__title"></h2> <span class="bui-score-bar__score">93%</span></div> <div class="bui-score-bar__bar"><span data-value="9.3" class="bui-score-bar__value" style="width: 93%;"></span></div></div></div></td> <td class="peg-table__cell--no-label bui-table__cell" data-heading="Location"><div class="peg-address-wrapper"><span class="peg-flag-wrapper"><div class="peg-flag bui-flag"><img src="https://q.bstatic.com/backend_static/common/flags/16/nl/314ce6500532e846e25d6e3a7c824ef17c968446.png" class="bui-flag__flag" style="height: auto;"> <!----></div></span>
TEXT2
</div></td> <td class="peg-table__cell--no-label bui-table__cell" data-heading="Status"><span>Open/Bookable</span></td> <td class="bui-table__cell bui-table__cell--center" data-heading="Arrivals/departures & tomorrow"><span class="peg-counter--arrivals"><a href="../../extranet_ng/manage/search_reservations.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111" data-track-ga="Groups: Home,Active Properties,Arrivals" target="_blank" class="peg-counter peg-counter--has"><span aria-label="80 unread" class="bui-bubble">
VALUE1
</span></a></span> <span class="peg-counter--departures"><a href="../../extranet_ng/manage/search_reservations.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111&type=departure" data-track-ga="Groups: Home,Active Properties,Departures" target="_blank" class="peg-counter peg-counter--has"><span aria-label="80 unread" class="bui-bubble">
VALUE2
</span></a></span></td> <td class="bui-table__cell bui-table__cell--center" data-heading="Guest Messages"><a href="../../extranet_ng/manage/messaging_inbox.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111" data-track-ga="Groups: Home,Active Properties,Guest Messages" target="_blank" class="peg-counter peg-counter--has"><span aria-label="102 unread" class="bui-bubble">
VALUE3
</span></a></td> <td class="bui-table__cell bui-table__cell--center" data-heading="extranet.com Messages"><a href="../../extranet_ng/manage/inbox.html?ses=8594ddf0718dec240a0c9e7991e108a8&name_id=11111" data-track-ga="Groups: Home,Active Properties,extrabnet.com Messages" target="_blank" class="peg-counter"><span aria-label="0 unread" class="bui-bubble">
0
</span></a></td></tr>
If you want to click the link which contains avatar in its class attribute use the following XPath expression:
//div[contains(#class,'avatar')]/descendant::a[contains(text(), 'text i want to click')]
where:
contains() is XPath Function allowing partial match on the attributes values and/or text
descendant - XPath Axis which matches children of the current node and their respective children
last part limits the search to hyperlinks which contain text you're looking for
The desired element is a ReactJS enabled element, so you have to induce WebDriverWait for the element to be clickable() and you can use either of the following solutions:
Using PARTIAL_LINK_TEXT:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "text i want to click"))).click()
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.peg-property-link.bui-link.bui-link--secondary[href*='/extranet_ng/manage/index'][data-track-ga*='Active Properties']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='peg-property-link bui-link bui-link--secondary' and contains(#href, '/extranet_ng/manage/index')][starts-with(#data-track-ga, 'Groups')]"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Select a dropdown using Python + Selenium

I am writing an automation for work and am stuck with a dropdown. The particular select box in question is as follows:
<span class="a-dropdown-container" id="select-size-bulk-asin">
<select name="display_type" class="a-native-dropdown">
<option value="SMALL-IMAGES">SMALL-IMAGES</option>
<option value="LARGE-IMAGES">LARGE-IMAGES</option>
<option value="TEXT">TEXT</option>
</select>
<span tabindex="-1" data-a-class="a-spacing-small" class="a-button a-button-dropdown a-spacing-small">
<span class="a-button-inner">
<span class="a-button-text a-declarative" data-action="a-dropdown-button" aria-haspopup="true" role="button" tabindex="0" aria-pressed="false" aria-owns="2_dropdown_combobox">
<span class="a-dropdown-prompt">SMALL-IMAGES</span>
</span>
<i class="a-icon a-icon-dropdown"></i>
</span>
</span>
</span>
It defaults to 'SMALL Images' and I would like to select the 'TEXT' option. I am receiving element not clickable error. The page is simple and the element is visible on the screen.
The list of methods I did try are:
Used WebDriverWait to wait for the element to be visible;
Used WebDriverWait to wait for the element to be clickable;
Used the select class to set the selected option;
I also read through a question.
I am thinking if I should just go to the next element and send Shift+Tabs until I reach this drop down and then down arrow keys. But would like to use that only as the last resort.
NOTE:
- I am using Python 3 and Chrome.
You can try this code to select value from drop down :
select = Select(driver.find_element_by_id('select-size-bulk-asin'))
select.select_by_visible_text('TEXT')
However,as you have mentioned you are receiving element not clickable exception. you can try this code :
WebDriverWait(browser, 30).until(EC.element_to_be_clickable((By.ID, "select-size-bulk-asin")))
As a last resort you can go ahead with :
drop_down= driver.find_element_by_id("select-size-bulk-asin")
drop_down.click()
actions = ActionChains(driver)
actions.send_keys(Keys.ARROW_DOWN)
actions.send_keys(Keys.ARROW_DOWN)
actions.send_keys(Keys.ENTER)
actions.perform()

How to select only divs with specific children span with xpath python

I am currently trying to scrap information of a particular ecommerce site and i only want to get product information like product name, price, color and sizes of only products whose prices have been slashed.
i am currently using xpath
this is my python scraping code
from lxml import html
import requests
class CategoryCrawler(object):
def __init__(self, starting_url):
self.starting_url = starting_url
self.items = set()
def __str__(self):
return('All Items:', self.items)
def crawl(self):
self.get_item_from_link(self.starting_url)
return
def get_item_from_link(self, link):
start_page = requests.get(link)
tree = html.fromstring(start_page.text)
names = tree.xpath('//span[#class="name"][#dir="ltr"]/text()')
print(names)
Note this is not the original URL
crawler = CategoryCrawler('https://www.myfavoriteecommercesite.com/')
crawler.crawl()
When the program is Run ... These are the HTML Content Gotten from the E-commerce Site
Div of Products With Price Slash
div class="products-info">
<h2 class="title"><span class="brand ">Apple </span> <span class="name" dir="ltr">IPhone X 5.8-Inch HD (3GB,64GB ROM) IOS 11, 12MP + 7MP 4G Smartphone - Silver</span></h2>
<div class="price-container clearfix">
<span class="sale-flag-percent">-22%</span>
<span class="price-box ri">
<span class="price ">
<span data-currency-iso="NGN">₦</span>
<span dir="ltr" data-price="388990">388,990</span>
</span>
<span class="price -old ">
<span data-currency-iso="NGN">₦</span>
<span dir="ltr" data-price="500000">500,000</span>
</span>
</span>
</div>
div
Div of Products with No Price Slash
div class="products-info">
<h2 class="title"><span class="brand ">Apple </span> <span class="name" dir="ltr">IPhone X 5.8-Inch HD (3GB,64GB ROM) IOS 11, 12MP + 7MP 4G Smartphone - Silver</span></h2>
<div class="price-container clearfix">
<span class="price-box ri">
<span class="price ">
<span data-currency-iso="NGN">₦</span>
<span dir="ltr" data-price="388990">388,990</span>
</span>
</span>
</div>
div
Now this is my exact Question
i want to know how to select only the parent divs i.e
div class="price-container clearfix"> that also contains any of these children span classes
span class="price -old "> or
span class="sale-flag-percent">
Thank you all
One solution would be get all <div class="price-container clearfix"> and iterate, checking with the string of the whole element that your keywords exist.
But a better solution would be to use conditionals with xpath:
from lxml import html
htmlst = 'your html'
tree=html.fromstring(htmlst)
divs = tree.xpath('//div[#class="price-container clearfix" and .//span[#class = "price -old " or #class = "sale-flag-percent"] ]')
print(divs)
This get all divs where class="price-container clearfix" and then check if contains span with the searched classes.

How to create list of web elements?

I am trying to make a list of web elements, but it can not seem to find the elements on the web page, although did worked 3 days ago and i can not find any changes in the web page.
this is the html code :
<li id="wlg_41410" class="leagueWindow " dataid="41410">
<h5 style="cursor: pointer; cursor: hand;" onclick="TodaysEventsLeagueWindow.minimizeRestoreClick(41410)">Europa League</h5>
<div class="bet_type select" id="_bet_types"></div>
<div class="bet_type lastscore ">
<h6>1X2 FT </h6>
<div class="types_bg">
<!--[if IE]> <div id="IEroot"> <![endif]-->
<div class="first_buttons_line">
</div>
<!--[if IE]> </div> <![endif]-->
<div class="time"> 23/11 | 18:00 </div>
<div class="bets ml">
</div>
<div class="time"> 23/11 | 20:00 </div>
<div class="bets ml">
</div>
<div class="time"> 23/11 | 20:00 </div>
<div class="bets ml">
</div>
<div class="time"> 23/11 | 20:00 </div>
<div class="bets ml">
</div>
<div class="time"> 23/11 | 20:00 </div>
<div class="bets ml">
</div>
<div class="clr"></div>
</div>
</div> <span class="x" onclick="TodaysEventsLeagueWindow.closeLeagueWindow(41410)"></span>
</li>
i am trying to make a list from the <div class="bets ml"></div> elements
but keep getting the selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document exception , as if selenium can't find the web element.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import StaleElementReferenceException
import time
driver.get("https://www.luckia.es/apuestas")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it("sbtechBC"))
eventos_de_hoy = driver.find_element_by_id("today_event_btn")
eventos_de_hoy.click()
ligi_len = len(WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "leagueWindow "))))
print(ligi_len)
for index in range(ligi_len):
item = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, "leagueWindow ")))[index]
driver.execute_script("arguments[0].scrollIntoView(true);", item)
nume_liga = item.find_element_by_tag_name("h5").text
time.sleep(3)
print('try', nume_liga)
meci = item.find_elements_by_xpath("//*[#class='bets ml']")
print("there are", len(meci), "in one liga")
the reason for the index is that the iframe refreshes every 25 sec.
i also tried meci = item.find_elements_by_css_selector('.bets.ml') and meci = item.find_elements_by_class_name('ml')
Why should i be able to extract the <h5></h5> element and not the other elements?
From your code block, its pretty clear you have just managed to cover up the real issue through time.sleep(3) as follows :
nume_liga = item.find_element_by_tag_name("h5").text
time.sleep(3)
print('try', nume_liga)
While invoking print() for a text, I am not sure why time.sleep(3) was induced. So our main issue got covered up there. But as the List was already created, you are able to print('try', nume_liga)
But next, when you do meci = item.find_elements_by_xpath("//*[#class='bets ml']") you face a StaleElementReferenceException because the HTML DOM have changed.
A closer look into the <h5> tag reveals it have a onclick() event as :
<h5 style="cursor: pointer; cursor: hand;" onclick="TodaysEventsLeagueWindow.minimizeRestoreClick(41410)">Europa League</h5>
A wild guess, while invoking .text on <h5> tag, the HTML DOM changes.
Solution :
A possible solution with your current code block may be to use getAttribute("innerHTML") instead of .text. So your line of code will be :
nume_liga = item.find_element_by_tag_name("h5").get_attribute("innerHTML")

Resources