I'm trying to generalize much as possible a code in order to scrape eBay website.
I am now stuck in this situation:
this is the following URL where I start:
https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw=iphone+12&_oac=1
I want to click on the "More filters" button and then select specifically the 'Brand' option and get the list of the available brands.
This is what I was able to do till now:
url = 'https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw=iphone+12&_oac=1'
driver = webdriver.Chrome()
driver.get(url)
driver.find_element_by_id('s0-14-11-0-1-2-6-2').click()
after click() the sub panel is open and I get what I need to change:
<div role="tab" class="x-overlay-aspect " data-aspecttitle="aspect-Brand" aria-selected="false" aria-controls="refineOverlay-subPanel" id="c3-mainPanel-Brand"><span class="x-overlay-aspect__label">Brand</span><svg focusable="false" aria-hidden="true" class="x-overlay-aspect__check-icon svg-icon icon-check" role="img" aria-label="Filter applied"><use xlink:href="#svg-icon-check"></use></svg></div>
if I select manually (i.e. clicking with the cursor) on 'Brand' here is what I get back:
<div role="tab" class="x-overlay-aspect active" tabindex="0" data-aspecttitle="aspect-Brand" aria-selected="true" aria-controls="refineOverlay-subPanel" id="c3-mainPanel-Brand"><span class="x-overlay-aspect__label">Brand</span><svg focusable="false" aria-hidden="true" class="x-overlay-aspect__check-icon svg-icon icon-check" role="img" aria-label="Filter applied"><use xlink:href="#svg-icon-check"></use></svg></div>
Unfortunately, I barely know Javascript so, althought there are other similar posts, I don't know how to keep going from here in order to get finally the information I need.
I hope you can help and I thank you in advance!
To click More filters and then to click on Brand you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Code Block:
driver.get("https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw=iphone+12&_oac=1")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[contains(., 'More filters')]"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Brand']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
Related
I've been trying to scrape this section of the website with Selenium on Python 3.7, there is nothing unique about it, there are many sections of the website with the exact same style.
<tr><td style="black 2px solid" colspan="1"></td></tr>
<tr><td>Total</td><td style="color: blue" valign="top">1,233</td><td style="color: green" valign="top"> (86.35%) </td></tr>
<tr><td colspan="1"> </td></tr>
The only thing unique is the value being 'Total', I need to scrape the td after that which is '1,233'.
When using: find elements by xpath I use this xpath:
('.//td[text()="Total"]')
I have been playing around with paths and with no luck.
Would it be possible to get xpath location from value?
Thank you :-)
To get the value 1,233 use either of the following xpath.
//td[text()='Total']/following-sibling::td[1]
OR
//tr[.//td[text()='Total']]/td[2]
In selenium using python
print(driver.find_element_by_xpath("//tr[.//td[text()='Total']]/td[2]").text)
Or
print(driver.find_element_by_xpath("//td[text()='Total']/following-sibling::td[1]").text)
For best practice induce WebDriverWait() and wait for visibility_of_element_located()
print(WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH,"//td[text()='Total']/following-sibling::td[1]"))).text)
You need to import below libraries.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
This should work:
//td[text()="Total"]/following-sibling::td[1]
You can test it here: https://www.freeformatter.com/xpath-tester.html
How do I write an xpath so that I can click on the settings button with selenium in python for the following html. Clicking the element is made harder by the fact that the settings element doesn't have an own id.
<ul class="nav nav-tabs">
<li class="active">
<a data-toggle="tab" href="#editor">Structure</a>
</li>
<li>
<a data-toggle="tab" href="#feedback">Feedback</a>
</li>
<li>
<a data-toggle="tab" href="#settings">General settings</a>
</li>
I have tried multiple commands, but I don't seem to get it right.
I am using elements (plural) so that I can see if it matches any xpaths. Figured it would be the cleanest way to test out different xpaths. If I get the xpath correctly I will use element (singular) and add .click() to the end. I have tried at least these following codes.
driver.find_elements_by_xpath("//*[#class='nav nav-tabs']//*[href='#settings']")
--> return [] (xpath doesn't match?)
driver.find_elements_by_xpath("//ul[contains(#class, 'nav-tabs')]")
--> returns []
driver.find_elements_by_xpath(".//a[#href='#settings']")
--> return []
When I 'click' the element, it usually gives the following error message:
NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":".//a[#href='#settings']"}
#MosheSlavin was close. However, to click() on the element you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using LINK_TEXT:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "General settings"))).click()
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "ul.nav.nav-tabs li>a[href$='settings']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//ul[#class='nav nav-tabs']//li/a[text()='General settings']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Here you can find a detailed discussion on Selenium “selenium.common.exceptions.NoSuchElementException” when using Chrome
The element might not be loaded yet, so use WebDriverWait to wait for the element:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(self.driver,10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "a[href='#settings']")))
Or
WebDriverWait(self.driver,10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href='#settings']")))
The solution was that selenium was looking for the element in the wrong frame. I inspected the html code and found two iframe id's. I inserted one of the iframe id's in this function
driver.switch_to.frame("iframe_name")
and then
driver.find_element_by_xpath(".//a[#href='#settings']").click()
On the website I am trying to fill in some fields on, there is a checkbox that I need to click to add the check mark in it:
<div class="rc-anchor-content"><div class="rc-inline-block"><div class="rc-anchor-center-container"><div class="rc-anchor-center-item rc-anchor-checkbox-holder"><span class="recaptcha-checkbox goog-inline-block recaptcha-checkbox-unchecked rc-anchor-checkbox recaptcha-checkbox-expired" role="checkbox" aria-checked="false" id="recaptcha-anchor" dir="ltr" aria-labelledby="recaptcha-anchor-label" aria-disabled="false" tabindex="0"><div class="recaptcha-checkbox-border" role="presentation" style=""></div><div class="recaptcha-checkbox-borderAnimation" role="presentation" style=""></div><div class="recaptcha-checkbox-spinner" role="presentation" style="transform: rotate(180deg);"></div><div class="recaptcha-checkbox-spinnerAnimation" role="presentation" style=""></div><div class="recaptcha-checkbox-checkmark" role="presentation"></div></span></div></div></div><div class="rc-inline-block"><div class="rc-anchor-center-container"><label class="rc-anchor-center-item rc-anchor-checkbox-label" aria-hidden="true" role="presentation" id="recaptcha-anchor-label"><span aria-live="polite" aria-labelledby="recaptcha-accessible-status"></span>I'm not a robot</label></div></div></div>
Using Selenium in VBA, I tried the following
.FindElementByCss("div.recaptcha-checkbox-border").Click
And also I tried
.FindElementByCss("span.recaptcha-checkbox").Click
But I got an error at this line.
Here's the link of the website to see the whole HTML
https://www.moj.gov.kw/AR/E-Gov/Pages/eServices01.aspx
To click() on the element, as the desired element is within an <iframe> so you have to:
Induce a waiter and switch to the desired frame.
Induce a waiter for the desired element to be clickable.
You can use the following solution:
.SwitchToFrame.FindElementByXPath("//iframe[contains(#src, 'recaptcha') and not(#title='recaptcha challenge')]", timeout:=10000)
.FindElementByCss("div.recaptcha-checkbox-checkmark").Click
You can find similar discussions in:
How to click on the reCaptcha using Selenium and Java
Find the reCAPTCHA element and click on it — Python + Selenium
Here you can find a relevant discussion on Ways to deal with #document under iframe
I am having issues locating button and clicking on it. It is pop up in internet explorer. I am using by classname but don't work.
browser.find_element_by_css_selector("ui-button ui-corner-all ui-widget").click()
HTML:
To locate and click the element with text as Accept you need to induce WebDriverWait for the element to be clickable and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.ui-dialog-buttonset>button.ui-button.ui-corner-all.ui-widget"))).click()
Using XPATH:
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='ui-dialog-buttonset']//button[#class='ui-button ui-corner-all ui-widget' and text()='Accept']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I'm learning Selenium with Python, and attempting to write happy path flow for a website. Where I get stuck is a part of the flow where an iframe is automatically launched after clicking through a pop-up window. I've tried several different methods, but am unable to locate the frame, or wait for it appear. Either it's not found or times out.
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.ID, 'continue- reservation'))).click()
# cvv2 form
WebDriverWait(driver, 5).until(EC.frame_to_be_available_and_switch_to_it((By.ID, 'cvv_iframe')))
Error:
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.ID, 'cvv_iframe')))
File "/Users/anutter/venv/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
HTML is as follows:
<div id="cpstaging" class="emptystage">
<div class="fixed h-v-centered new-preload" style="display: none;"><span class="fa fa-spinner f-50 f-grn fa-pulse"></span></div>
</div>
<div id="cvv-box" class="pad-30-lr">
<div id="enter-cvv-title" class="blk mar-20-b f-18 border-b pad-10-b">Enter CVV Code</div>
<iframe src="https://qa-hotels.ecbsn.com/cvv?oauth_token=u7q99%2Fe8I%2BkUrkLMr4dGR2t4gmcDbVtr&type=visa&src=web-desktop" id="cvv_iframe" name="cvv_iframe" width="430" height="160" frameborder="0" seamless="seamless" scrolling="no" sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe>
</div>
</div>
</div>
</div>
As per the HTML you have shared to access the elements within the <iframe> you have to:
Induce WebDriverWait for the desired frame to be available and switch to it using either of the Locator Strategies:
Using ID:
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"cvv_iframe")))
Using NAME:
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,"cvv_iframe")))
Using CSS_SELECTOR:
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#cvv_iframe[name='cvv_iframe']")))
Using XPATH:
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='cvv_iframe' and #name='cvv_iframe']")))
Note: As per your code trials the ID of the <iframe> isn't cvv-iframe but cvv_iframe
Here you can find a relevant discussion on Ways to deal with #document under iframe