Not able to extract urls from HTML BeautifulSoup object - python-3.x

Question

I am looking to extract following url "https://mania.bg/p/pulover-alexander-mcqueen-p409648" from html (BeautifulSoup object) named urls that looks like:
[<a class="product sellout product-sellout float-left status-1" data-id="409648" data-producturl="https://mania.bg/p/pulover-alexander-mcqueen-p409648" data-status="1" href="https://mania.bg/p/pulover-alexander-mcqueen-p409648"> <div class="product-hover clearfix prevent-flicker"><div class="module-icons favourite tooltip" data-id="409648" data-title=" Любима находка на 24 клиент/и. "> <img alt="" class="favourite-product like-product unactivated" data-id="409648" src="dist/assets/icon_favourite_off.png"/></div> <div class="campaign" style="color: #FFF;background-color: #000000;"> NIGHT </div> <div class="profit-icons-wrapper clearfix"> </div> <div class="product-basic-info"> <div class="image-wrapper"> <img alt="Пуловер Alexander McQueen" class="front" data-url="https://media.mania.bg/product/048/409648/300/pulover-alexander-mcqueen-2.jpg" src="https://media.mania.bg/product/048/409648/300/pulover-alexander-mcqueen-1.jpg" title="Пуловер Alexander McQueen - Mania"> </img></div> <div class="clearfix brand-line"> <div class="brand float-left text-uppercase">Alexander McQueen</div> <div class="size float-right">S</div> </div> </div> <div class="prices-section"> <div class="prices-inner-section"> <div class="price-wrapper clearfix"> <div class="price-title text-uppercase float-left"> Начална цена </div> <div class="price old"> <span>98.00</span> <span class="currency">лв.</span> </div> </div> <div class="price-wrapper clearfix"> <div class="discount price-title text-uppercase float-left"> -40% </div> <div class="price old"> <span>58.80</span> <span class="currency">лв.</span> </div> </div> <div class="price-wrapper clearfix"> <div class="discount price-title text-uppercase float-left" style="color: #FFF;background-color: #000000"> -40% </div> <div class="price"> <span>35.28</span> <span class="currency">лв.</span> </div> </div> </div> </div> </div> <div class="button button-auction buy-now text-center float-left tooltip prevent-popup-close" data-id="409648" data-title="Може да добавите този продукт към количката.">ДОБАВЯМ<img alt="" class="bag-icon" src="dist/assets/icon_bag_button.svg"> </img></div> </a>]
With following code:
for num in range(len(urls)):
url = urls[num - 1].a['href']
I also tried to use:
url = urls[num - 1].a['data-producturl']
I get "TypeError: 'NoneType' object is not subscriptable" as url is None.

Answer 1

import requests
import bs4
url = 'https://mania.bg/p/pulover-alexander-mcqueen-p409648'
data = requests.get(url)
soup = bs4.BeautifulSoup(data.text,'html.parser')
urls = soup.find_all('a', attrs={'class': 'product sellout product-sellout float-left status-1'})
for num in range(len(urls)):
url = urls[num]['href']
print(url)

Answer 2

Try this. Here's an example:https://github.com/yiyedata/simplified-scrapy-demo/tree/master/doc_examples
from simplified_scrapy.simplified_doc import SimplifiedDoc
html ='''
[<a class="product sellout product-sellout float-left status-1" data-id="409648" data-producturl="https://mania.bg/p/pulover-alexander-mcqueen-p409648" data-status="1" href="https://mania.bg/p/pulover-alexander-mcqueen-p409648"> <div class="product-hover clearfix prevent-flicker"><div class="module-icons favourite tooltip" data-id="409648" data-title=" Любима находка на 24 клиент/и. "> <img alt="" class="favourite-product like-product unactivated" data-id="409648" src="dist/assets/icon_favourite_off.png"/></div> <div class="campaign" style="color: #FFF;background-color: #000000;"> NIGHT </div> <div class="profit-icons-wrapper clearfix"> </div> <div class="product-basic-info"> <div class="image-wrapper"> <img alt="Пуловер Alexander McQueen" class="front" data-url="https://media.mania.bg/product/048/409648/300/pulover-alexander-mcqueen-2.jpg" src="https://media.mania.bg/product/048/409648/300/pulover-alexander-mcqueen-1.jpg" title="Пуловер Alexander McQueen - Mania"> </img></div> <div class="clearfix brand-line"> <div class="brand float-left text-uppercase">Alexander McQueen</div> <div class="size float-right">S</div> </div> </div> <div class="prices-section"> <div class="prices-inner-section"> <div class="price-wrapper clearfix"> <div class="price-title text-uppercase float-left"> Начална цена </div> <div class="price old"> <span>98.00</span> <span class="currency">лв.</span> </div> </div> <div class="price-wrapper clearfix"> <div class="discount price-title text-uppercase float-left"> -40% </div> <div class="price old"> <span>58.80</span> <span class="currency">лв.</span> </div> </div> <div class="price-wrapper clearfix"> <div class="discount price-title text-uppercase float-left" style="color: #FFF;background-color: #000000"> -40% </div> <div class="price"> <span>35.28</span> <span class="currency">лв.</span> </div> </div> </div> </div> </div> <div class="button button-auction buy-now text-center float-left tooltip prevent-popup-close" data-id="409648" data-title="Може да добавите този продукт към количката.">ДОБАВЯМ<img alt="" class="bag-icon" src="dist/assets/icon_bag_button.svg"> </img></div> </a>]
'''
doc = SimplifiedDoc(html)
urls = doc.selects('a.product sellout product-sellout float-left status-1')
print ([(url.href,url['data-producturl']) for url in urls])
Result:
[('https://mania.bg/p/pulover-alexander-mcqueen-p409648', 'https://mania.bg/p/pulover-alexander-mcqueen-p409648')]

Answer 3

find_all already gives you the list of a elements; you just need to get the href from each.
from bs4 import BeautifulSoup
import requests
url = 'https://mania.bg/p/pulover-alexander-mcqueen-p409648'
r = requests.get(url)
soup = BeautifulSoup(r.content, 'html.parser')
for a in soup.find_all(
'a',
attrs={'class':
'product sellout product-sellout float-left status-1'}):
print(a['data-producturl'])