I'm trying to use selenium to webscrape the webull screener. However, when I print the xpath to the webelement it just comes up with <selenium.webdriver.remote.webelement.WebElement (session="d9d2a95d8fde18ce1e914b8ca867a370", element="869bb784-0838-498e-8420-f56e67f42e68")>
and if I try executing it with execute_script it says it must be a str.
Here's a copy of my code
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(options=options)
driver.get('https://app.webull.com/screener')
driver.implicitly_wait(15)
peice =
driver.find_element_by_xpath('/html/body/div/div/div[3]/div/div[1]/div/div[2]/div/div[2]/div/div/div/div[3]/table/tbody/tr[2]/td[2]')
print(driver.execute_script(peice))
All I'm trying to do is print the actual symbol in the web screener.
Thank you
peice , contains a webelement object so if you print it you will get object representation printed
use peice.text or peice.get_attribute("textContent")
this will return the text from the element and prints it.
if you simply pass web element to execute_script you are not executing anything
you should execute something and return something eg:
print(driver.execute_script("return arguments[0].text()" , peice)
Related
I would like to scrape https://www.sportsbookreview.com/betting-odds/college-football/totals/1st-half/?date=20180825 where I would like to get only text from class adjust-1uDgI which is a child of class numbersContainer-29L5c with specific tag data-vertical-sbid="238". So I would like to get odds from Pinnacle and I should get 30.5 and 23 for this page.
I'm not sure how can I get it, should I use driver.find_elements_by_xpath() or driver.find_elements_by_class_name()
This is my code for now where I wanted to get data with that class but I get an empty list
driver = webdriver.Chrome(executable_path=binary_path)
driver.get('https://www.sportsbookreview.com/betting-odds/college-football/totals/1st-half/?date=20180825')
element = elements = driver.find_elements_by_class_name("adjust-1uDgI")
print(element)
driver.quit()
What might be the problem is that you didn't wait for the webpage to load. Try this:
from selenium.webdriver import Chrome
from selenium.webdriver.support.ui import WebDriverWait
driver = Chrome(executable_path=binary_path)
driver.get('https://www.sportsbookreview.com/betting-odds/college-football/totals/1st-half/?date=20180825')
element = elements = WebDriverWait(driver, 5).until(lambda d: d.find_elements_by_class_name("adjust-1uDgI"))
print(element)
driver.quit()
In the code below I am trying to print the tab title and it open the tab but after that python giving me a TypeError and the program crashes.
from selenium import webdriver
path = "C:\Program Files\chromedriver.exe"
driver = webdriver.Chrome(path)
driver.get() # in the parentheses is a link like https://link.com/
print(driver.title())
driver.quit()
What is the problem? and how can I fix it?
driver.title
title is an attritibute, but not a method. It returns the title of the current page.
Usage:
title = driver.title
Solution
You need to remove the parenthesis. So effectively, the line of code will be:
print(driver.title)
The output of the salary_estimate variable is '$39K-$72K (Glassdoor est.)'
The goal is to retrieve salary value such as '$39K-$72K' and not including '(Glassdoor est.)'
from selenium import webdriver
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(
executable_path="C:/webdrivers/chromedriver", options=options)
url = 'https://www.glassdoor.com/Job/data-scientist-jobs-SRCH_KO0,14_IP2.htm'
driver.get(url)
salary_estimate = driver.find_element_by_xpath('.//span[#class="css-1uyte9r css-hca4ks e1wijj242"]').text
You have to get rid of the child nodes you don't want. And you can do that executing a javascript with your driver.
Append this to your code:
salary = driver.execute_script('return arguments[0].firstChild.textContent;', salary_estimate).strip()
print(salary)
This will ensure you get only the first element text.
I am scraping a website using selenium and python, and the website has many h2 tags in it that I want to scrape.
<h2>john paplos<h2>
<h2>john smith</h2>
....
I want to scrape all these h2 tags and display them on a terminal.
Here is my code:
for i in temp:
temp = driver.find_element_by_tag_name('h2').text
print (temp)
This code works fine without a for loop, but when I add a for loop, it give me a name error
for i in temp:
NameError: name 'temp' is not defined
What am I doing wrong?
In order to locate multiple elements and not just one, selenium offers find_elements_by_*() set of methods. In your case find_elements_by_tag_name() would do the job:
for h2 in driver.find_elements_by_tag_name('h2'):
print(h2.text)
To break it down a bit further:
find_elements_by_tag_name() returns a list of WebElement instances
which means that h2 variable is of WebElement type
.text is a property of a WebElement which we print out
This error message...
for i in temp:
NameError: name 'temp' is not defined
...implies that the variable temp wasn't defined/initialized when you tried to use it within the line:
for i in temp:
However it seems you were close.To print all the text within the <h2> tags you can induce WebDriverWait for the visibility_of_all_elements_located() and you can use the following Locator Strategy
Using TAG_NAME and text attribute:
print([my_elem.text for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.TAG_NAME, "h2")))])
Using TAG_NAME and get_attribute("innerHTML"):
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 5).until(EC.visibility_of_all_elements_located((By.TAG_NAME, "h2")))])
I am made this simple whatsapp bot using python and selenium.
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://web.whatsapp.com/')
target = "Someone"
msg = "Something"
input('Press Enter after scanning QR Code...')
TargetXML = driver.find_element_by_xpath('//span[#title = "
{}"]'.format(target))
TargetXML.click()
MsgBox = driver.find_elements_by_class_name('_1Plpp')
MsgBox[0].send_keys(msg)
SendButton = driver.find_elements_by_class_name('_35EW6')
SendButton[0].click()
At first run, I had MsgBox.send_keys(msg) and SendButton.click() instead of what you see in the script which gave an error AttributeError: 'list' object has no attribute 'send_keys' and AttributeError: 'list' object has no attribute 'click'.
I changed them to index 0 which solved the error and the script worked perfectly fine but I couldn't really understand why it worked with the element in 0th index so I tried to print the element and got the output as <selenium.webdriver.remote.webelement.WebElement (session="bd86fe53729956ba1fc3b16cf841a1a8", element="0.5125252686493715-2")> I am still not convinced with it and have that question in mind. Any help would be appreciated! Thank you!
The method 'find_elements_by_class_name' returns a list of elements that satisfy the class name in the parameter. This list consists of WebElement when you select the 0th element, you get the object of WebElement, over which you can apply the methods send_keys() and click().
For more information on Selenium and WebElement object refer to this documentation.