I've been trying to scrape this section of the website with Selenium on Python 3.7, there is nothing unique about it, there are many sections of the website with the exact same style.
<tr><td style="black 2px solid" colspan="1"></td></tr>
<tr><td>Total</td><td style="color: blue" valign="top">1,233</td><td style="color: green" valign="top"> (86.35%) </td></tr>
<tr><td colspan="1"> </td></tr>
The only thing unique is the value being 'Total', I need to scrape the td after that which is '1,233'.
When using: find elements by xpath I use this xpath:
('.//td[text()="Total"]')
I have been playing around with paths and with no luck.
Would it be possible to get xpath location from value?
Thank you :-)
To get the value 1,233 use either of the following xpath.
//td[text()='Total']/following-sibling::td[1]
OR
//tr[.//td[text()='Total']]/td[2]
In selenium using python
print(driver.find_element_by_xpath("//tr[.//td[text()='Total']]/td[2]").text)
Or
print(driver.find_element_by_xpath("//td[text()='Total']/following-sibling::td[1]").text)
For best practice induce WebDriverWait() and wait for visibility_of_element_located()
print(WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.XPATH,"//td[text()='Total']/following-sibling::td[1]"))).text)
You need to import below libraries.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
This should work:
//td[text()="Total"]/following-sibling::td[1]
You can test it here: https://www.freeformatter.com/xpath-tester.html
Related
I'm trying to generalize much as possible a code in order to scrape eBay website.
I am now stuck in this situation:
this is the following URL where I start:
https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw=iphone+12&_oac=1
I want to click on the "More filters" button and then select specifically the 'Brand' option and get the list of the available brands.
This is what I was able to do till now:
url = 'https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw=iphone+12&_oac=1'
driver = webdriver.Chrome()
driver.get(url)
driver.find_element_by_id('s0-14-11-0-1-2-6-2').click()
after click() the sub panel is open and I get what I need to change:
<div role="tab" class="x-overlay-aspect " data-aspecttitle="aspect-Brand" aria-selected="false" aria-controls="refineOverlay-subPanel" id="c3-mainPanel-Brand"><span class="x-overlay-aspect__label">Brand</span><svg focusable="false" aria-hidden="true" class="x-overlay-aspect__check-icon svg-icon icon-check" role="img" aria-label="Filter applied"><use xlink:href="#svg-icon-check"></use></svg></div>
if I select manually (i.e. clicking with the cursor) on 'Brand' here is what I get back:
<div role="tab" class="x-overlay-aspect active" tabindex="0" data-aspecttitle="aspect-Brand" aria-selected="true" aria-controls="refineOverlay-subPanel" id="c3-mainPanel-Brand"><span class="x-overlay-aspect__label">Brand</span><svg focusable="false" aria-hidden="true" class="x-overlay-aspect__check-icon svg-icon icon-check" role="img" aria-label="Filter applied"><use xlink:href="#svg-icon-check"></use></svg></div>
Unfortunately, I barely know Javascript so, althought there are other similar posts, I don't know how to keep going from here in order to get finally the information I need.
I hope you can help and I thank you in advance!
To click More filters and then to click on Brand you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Code Block:
driver.get("https://www.ebay.co.uk/sch/i.html?_from=R40&_nkw=iphone+12&_oac=1")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[contains(., 'More filters')]"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Brand']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Browser Snapshot:
How do I write an xpath so that I can click on the settings button with selenium in python for the following html. Clicking the element is made harder by the fact that the settings element doesn't have an own id.
<ul class="nav nav-tabs">
<li class="active">
<a data-toggle="tab" href="#editor">Structure</a>
</li>
<li>
<a data-toggle="tab" href="#feedback">Feedback</a>
</li>
<li>
<a data-toggle="tab" href="#settings">General settings</a>
</li>
I have tried multiple commands, but I don't seem to get it right.
I am using elements (plural) so that I can see if it matches any xpaths. Figured it would be the cleanest way to test out different xpaths. If I get the xpath correctly I will use element (singular) and add .click() to the end. I have tried at least these following codes.
driver.find_elements_by_xpath("//*[#class='nav nav-tabs']//*[href='#settings']")
--> return [] (xpath doesn't match?)
driver.find_elements_by_xpath("//ul[contains(#class, 'nav-tabs')]")
--> returns []
driver.find_elements_by_xpath(".//a[#href='#settings']")
--> return []
When I 'click' the element, it usually gives the following error message:
NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":".//a[#href='#settings']"}
#MosheSlavin was close. However, to click() on the element you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using LINK_TEXT:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.LINK_TEXT, "General settings"))).click()
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "ul.nav.nav-tabs li>a[href$='settings']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//ul[#class='nav nav-tabs']//li/a[text()='General settings']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Here you can find a detailed discussion on Selenium “selenium.common.exceptions.NoSuchElementException” when using Chrome
The element might not be loaded yet, so use WebDriverWait to wait for the element:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(self.driver,10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "a[href='#settings']")))
Or
WebDriverWait(self.driver,10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href='#settings']")))
The solution was that selenium was looking for the element in the wrong frame. I inspected the html code and found two iframe id's. I inserted one of the iframe id's in this function
driver.switch_to.frame("iframe_name")
and then
driver.find_element_by_xpath(".//a[#href='#settings']").click()
On the website I am trying to fill in some fields on, there is a checkbox that I need to click to add the check mark in it:
<div class="rc-anchor-content"><div class="rc-inline-block"><div class="rc-anchor-center-container"><div class="rc-anchor-center-item rc-anchor-checkbox-holder"><span class="recaptcha-checkbox goog-inline-block recaptcha-checkbox-unchecked rc-anchor-checkbox recaptcha-checkbox-expired" role="checkbox" aria-checked="false" id="recaptcha-anchor" dir="ltr" aria-labelledby="recaptcha-anchor-label" aria-disabled="false" tabindex="0"><div class="recaptcha-checkbox-border" role="presentation" style=""></div><div class="recaptcha-checkbox-borderAnimation" role="presentation" style=""></div><div class="recaptcha-checkbox-spinner" role="presentation" style="transform: rotate(180deg);"></div><div class="recaptcha-checkbox-spinnerAnimation" role="presentation" style=""></div><div class="recaptcha-checkbox-checkmark" role="presentation"></div></span></div></div></div><div class="rc-inline-block"><div class="rc-anchor-center-container"><label class="rc-anchor-center-item rc-anchor-checkbox-label" aria-hidden="true" role="presentation" id="recaptcha-anchor-label"><span aria-live="polite" aria-labelledby="recaptcha-accessible-status"></span>I'm not a robot</label></div></div></div>
Using Selenium in VBA, I tried the following
.FindElementByCss("div.recaptcha-checkbox-border").Click
And also I tried
.FindElementByCss("span.recaptcha-checkbox").Click
But I got an error at this line.
Here's the link of the website to see the whole HTML
https://www.moj.gov.kw/AR/E-Gov/Pages/eServices01.aspx
To click() on the element, as the desired element is within an <iframe> so you have to:
Induce a waiter and switch to the desired frame.
Induce a waiter for the desired element to be clickable.
You can use the following solution:
.SwitchToFrame.FindElementByXPath("//iframe[contains(#src, 'recaptcha') and not(#title='recaptcha challenge')]", timeout:=10000)
.FindElementByCss("div.recaptcha-checkbox-checkmark").Click
You can find similar discussions in:
How to click on the reCaptcha using Selenium and Java
Find the reCAPTCHA element and click on it — Python + Selenium
Here you can find a relevant discussion on Ways to deal with #document under iframe
I am having issues locating button and clicking on it. It is pop up in internet explorer. I am using by classname but don't work.
browser.find_element_by_css_selector("ui-button ui-corner-all ui-widget").click()
HTML:
To locate and click the element with text as Accept you need to induce WebDriverWait for the element to be clickable and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.ui-dialog-buttonset>button.ui-button.ui-corner-all.ui-widget"))).click()
Using XPATH:
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#class='ui-dialog-buttonset']//button[#class='ui-button ui-corner-all ui-widget' and text()='Accept']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I am trying to automate a process using Python and selenium webdriver.
I am able to login successfully and navigating to the page where I want to post something but for some reason the xpath is not recognized by the system.
It gives the below error:
NoSuchElementException: Unable to locate element: //*[#id="widecol"]/div/form/p[1]/input
Here is my code:
from selenium import webdriver
mydriver = webdriver.Firefox()
mydriver.get("https://www.phishtank.com/")
mydriver.maximize_window()
mydriver.find_element_by_xpath('//*[#id="username"]').send_keys("myusername")
mydriver.find_element_by_xpath('//*[#id="password"]').send_keys("mypassword")
mydriver.find_element_by_xpath('//*[#id="header"]/div[2]/form/input[3]').click()
mydriver.find_element_by_xpath('//*[#id="nav"]/ul/li[2]/a').click()
mydriver.find_element_by_xpath('//*[#id="widecol"]/div/form/p[1]/input').send_keys("sample text testing to fill form")
This is my HTML code
<div id="widecol">
<div class="padded">
<form method="POST">
<h2>Add A Phish</h2>
<ol>
<li>Visit our <b>What is phishing?</b> page to confirm that the suspected phish meets all of the criteria.</li>
<li>Add a phish using the form below, or even better, submit a phish directly via email.</li>
</ol>
<h3>Phish URL:</h3>
<p>
<input type="text" name="phish_url" style="width:90%;" value="" /><br />
<span class="small">Copy and paste the URL of the phishing website.</span>
</p>
<h3>What is the organization referenced in the email?</h3>
<p class="slim">
<select name="phish_target">
which gives me the following error:
NoSuchElementException: Unable to locate element: //*[#id="widecol"]/div/form/p[1]/input
Here is the HTML code:
<input type="text" name="phish_url" style="width:90%;" value=""> outer HTML code
And this is my XPath:
//*[#id="widecol"]/div/form/p[1]/input - Xpath
Please let me know where to look, thank you.
You need to induce WebDriverWait for the element to be clickable and you can use the following line of code :
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#name='phish_url' and #type='text']"))).send_keys("sample text testing to fill form")
Without the full HTML, we can't tell if the XPath is correct or no.
Can you please try and include more outer Html code?
You can try and wait after clicking on that last URL, maybe the element wasn't loaded yet.
You can also check if the input is in an iframe, that might be causing the problem.
Could you try the following instead?
//*[#id='widecol']//input[#name='phish_url']
Should work for that input. Just for trial/error sake, try a small wait before doing this.
The new Selenium version doesn't allow it.
Try the below code
from selenium.webdriver.common.by import By
And then Change the 'find_element_by_xpath statements' to
mydriver.find_element(By.XPATH,'//[#id="username"]').send_keys("myusername")
You can refer to the link to know more