i have this element : <span class="text-robux-lg wait-for-i18n-format-render">515</span>
and i wanna extract 515, i cant use requests and i can use selenium but that is very slow since i want it to be checked very 0,1 seconds. i tried requests but couldn't find anything on how to do it.
does anyone know how to do it? maybe beautifulsoup or other requests modules?
i tried this
from bs4 import BeautifulSoup
response = requests.get('https://www.roblox.com/catalog/20573078/Shaggy')
soup = BeautifulSoup(response.text, 'html.parser')
mydivs = soup.find_all("a", {"class": "text-robux-lg wait-for-i18n-format-render"})
print(mydivs)
but it doesn't work it just prints "[]"
It doesn't work, because the tag is <span>, not <a>:
import requests
from bs4 import BeautifulSoup
url = 'https://www.roblox.com/catalog/20573078/Shaggy'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')
t = soup.find("span", {"class": "text-robux-lg wait-for-i18n-format-render"}).text
print(t)
Prints:
515
Alternatively, you can use CSS selector:
print(soup.select_one('[data-expected-price]')['data-expected-price'])
Related
I'm trying to figure out how to scrape/extract a image url out of response.content.
This is the url I'm trying to extract <img src="/Content/images/asos-logo-2022-93x28.png"
The problem is that everything after the /Content/images/ part can change...
Any help appreciated !!!
You can use Beautiful Soup for this:
>>> import requests
>>> from bs4 import BeautifulSoup
>>> r = requests.get("https://stackoverflow.com/q/71636643/1416672")
>>> html = r.text
>>> soup = BeautifulSoup(html, 'html.parser')
>>> for item in soup.find_all('img'): print(item['src'])
...
https://cdn.sstatic.net/Img/teams/teams-illo-free-sidebar-promo.svg?v=47faa659a05e
https://www.gravatar.com/avatar/f96b33e2715bf57ba8e434140f0aeeba?s=64&d=identicon&r=PG&f=1
/posts/71636643/ivc/9bb6
https://sb.scorecardresearch.com/p?c1=2&c2=17440561&cv=3.6.0&cj=1
If you want to match a specific image then check the docs how to search by CSS class or any other CSS selectors.
I am just starting out with web scraping. I am having trouble with beautiful soup. I have tried changing the div class to other classes as well but it always returns []. Here is my code.
import time
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome(executable_path="C:/Users/MuhIsmail/Downloads/cd79/chromedriver.exe")
url = "https://www.cricbuzz.com/cricket-match/live-scores"
driver.get(url)
driver.maximize_window()
time.sleep(4)
content = driver.page_source
soup = BeautifulSoup(content, "html.parser")
scores = soup.find_all('div', class_='col-xs-9 col-lg-9 dis-inline')
print(scores)
import requests
from bs4 import BeautifulSoup
r = requests.get("https://www.cricbuzz.com/cricket-match/live-scores")
soup = BeautifulSoup(r.text, 'html.parser')
for item in soup.select("a.cb-mat-mnu-itm:nth-child(5)"):
print(item.text)
Output:
MLR vs SYS - SYS Won
It is returning [] because there are no elements on the page with that class.
If you open your browser console and do a simple
document.getElementsByClassName('col-xs-9 col-lg-9 dis-inline')
it will return no results.
I tried this as well:
import requests
from bs4 import BeautifulSoup
url = "https://www.cricbuzz.com/cricket-match/live-scores"
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
scores = soup.find_all('div', {'class':'col-xs-9 col-lg-9 dis-inline'})
print(scores)
I just want the written text out of this website: https://algorithms-tour.stitchfix.com/ so I can put it in Word doc and read it.
When I run the code, I get all the html and the tags, at the very end I get what I want, but I just want to separate the text.
import requests
from bs4 import BeautifulSoup
url = "https://algorithms-tour.stitchfix.com"
response = requests.get(url)
html = response.text
soup = BeautifulSoup(html, "html.parser")
item = soup.find_all("p")
print(item)
Is there a way to get just content so I can clean it up some more?
You have a few options for this. If you only want text found within p tags, you can do this:
import requests
from bs4 import BeautifulSoup
url = "https://algorithms-tour.stitchfix.com"
response = requests.get(url)
html = response.text
soup = BeautifulSoup(html, "html.parser")
items = soup.find_all("p")
result = []
for item in items:
result.append(item.string)
print(result)
Note that soup.find_all returns an iterable list, and not a single object.
An alternative, and easier method is to just use soup.get_text:
import requests
from bs4 import BeautifulSoup
url = "https://algorithms-tour.stitchfix.com"
response = requests.get(url)
html = response.text
soup = BeautifulSoup(html, "html.parser")
print(soup.get_text())
I am trying to get data from yellowpages, but i need only numbered plumbers. But i can't get text numbers in h2 class='n'. I can get a class="business-name" text but i need only numbered plumbers not with advertisement. What is my mistake? Thank you very much.
This html :
<div class="info">
<h2 class="n">1. <a class="business-name" href="/austin-tx/mip/johnny-rooter-11404675?lid=171372530" rel="" data-impressed="1"><span>Johnny Rooter</span></a></h2>
</div>
And this is my python code:
import requests
from bs4 import BeautifulSoup as bs
url = "https://www.yellowpages.com/austin-tx/plumbers"
req = requests.get(url)
data = req.content
soup = bs(data, "lxml")
links = soup.findAll("div", {"class": "info"})
for link in links:
for content in link.contents:
try:
print(content.find("h2", {"class": "n"}).text)
except:
pass
You need a different class selector to limit to that section
import requests
from bs4 import BeautifulSoup as bs
url = "https://www.yellowpages.com/austin-tx/plumbers"
req = requests.get(url)
data = req.content
soup = bs(data, "lxml")
links = [item.text.replace('\xa0','') for item in soup.select('.organic h2')]
print(links)
.organic is a single class selector, from a compound class, for a parent element which restricts to all the numbered plumbers. Observe how the highlighting starts after the ads:
Output:
I am trying to get the href of anchor tag of the very first video search on YouTube using Beautiful Soup. I am searching it by using the "a" and class_="yt-simple-endpoint style-scope ytd-video-renderer".
But I am getting None output:
from bs4 import BeautifulSoup
import requests
source = requests.get("https://www.youtube.com/results?search_query=MP+election+results+2018%3A+BJP+minister+blames+conspiracy+as+reason+while+losing").text
soup = BeautifulSoup(source,'lxml')
# print(soup2.prettify())
a =soup.findAll("a", class_="yt-simple-endpoint style-scope ytd-video-renderer")
a_fin = soup.find("a", class_="compact-media-item-image")
#
print(a)
from bs4 import BeautifulSoup
import requests
source = requests.get("https://www.youtube.com/results?search_query=MP+election+results+2018%3A+BJP+minister+blames+conspiracy+as+reason+while+losing").text
soup = BeautifulSoup(source,'lxml')
first_serach_result_link = soup.findAll('a',attrs={'class':'yt-uix-tile-link'})[0]['href']
heavily inspired by
this answer
Another option is to render the page first with Selenium.
import bs4
from selenium import webdriver
url = 'https://www.youtube.com/results?search_query=MP+election+results+2018%3A+BJP+minister+blames+conspiracy+as+reason+while+losing'
browser = webdriver.Chrome('C:\chromedriver_win32\chromedriver.exe')
browser.get(url)
source = browser.page_source
soup = bs4.BeautifulSoup(source,'html.parser')
hrefs = soup.find_all("a", class_="yt-simple-endpoint style-scope ytd-video-renderer")
for a in hrefs:
print (a['href'])
Output:
/watch?v=Jor09n2IF44
/watch?v=ym14AyqJDTg
/watch?v=g-2V1XJL0kg
/watch?v=eeVYaDLC5ik
/watch?v=StI92Bic3UI
/watch?v=2W_4LIAhbdQ
/watch?v=PH1WZPT5IKw
/watch?v=Au2EH3GsM7k
/watch?v=q-j1HEnDn7w
/watch?v=Usjg7IuUhvU
/watch?v=YizmwHibomQ
/watch?v=i2q6Fm0E3VE
/watch?v=OXNAMyEvcH4
/watch?v=vdcBtAeZsCk
/watch?v=E4v2StDdYqs
/watch?v=x7kCuRB0f7E
/watch?v=KERtHNoZrF0
/watch?v=TenbA4wWIJA
/watch?v=Ey9HfjUyUvY
/watch?v=hqsuOT0URJU
It dynamic html you can use Selenium or to get static html use GoogleBot user-agent
headers = {'User-Agent' : 'Googlebot/2.1 (+http://www.google.com/bot.html)'}
source = requests.get("https://.......", headers=headers).text
soup = BeautifulSoup(source, 'lxml')
links = soup.findAll("a", class_="yt-uix-tile-link")
for link in links:
print(link['href'])
Try looping over the matches:
import urllib2
data = urllib2.urlopen("some_url")
html_data = data.read()
soup = BeautifulSoup(html_data)
for a in soup.findAll('a',href=True):
print a['href']
The class which you're searching does not exist in the scraped html. You can identify it by printing the soup variable.
For example:
a =soup.findAll("a", class_="sign-in-link")
gives output as:
[<a class="sign-in-link" href="https://accounts.google.com/ServiceLogin?passive=true&continue=https%3A%2F%2Fwww.youtube.com%2Fsignin%3Faction_handle_signin%3Dtrue%26app%3Ddesktop%26feature%3Dplaylist%26hl%3Den%26next%3D%252Fresults%253Fsearch_query%253DMP%252Belection%252Bresults%252B2018%25253A%252BBJP%252Bminister%252Bblames%252Bconspiracy%252Bas%252Breason%252Bwhile%252Blosing&uilel=3&hl=en&service=youtube">Sign in</a>]