How do I combine url for my beautifulsoup project? - python-3.x

Question

This is the code that I have so far :
from grp import struct_group
from bs4 import BeautifulSoup
import requests
import pandas as pd
import urllib.parse
name = []
price = []
mileage = []
dealer_name =[]
source = []
for i in range (1,13):
#Allow to Crawl multiple pages:
website ='https://www.cars.com/shopping/results/?page=' + str(i) + '&page_size=20&dealer_id=&keyword=&list_price_max=&list_price_min=&makes[]=&maximum_distance=all&mileage_max=&sort=best_match_desc&stock_type=used&year_max=&year_min=&zip=95355'
#Requesting using requests lib
response = requests.get(website)
soup = BeautifulSoup(response.content, 'html.parser')
#Finding results
results = soup.find_all('div', {'class':'vehicle-card'})
url_combine = []
root_url = 'http://www.cars.com'
url_combine = root_url , source
for result in results:
# name
try:
name.append(result.find('h2').get_text())
except:
name.append('n/a')
#price
try:
price.append(result.find('span', {'class':'primary-price'}).get_text())
except:
price.append('n/a')
# mileage
try:
mileage.append(result.find('div', {'class':'mileage'}).get_text())
except:
mileage.append('n/a')
# dealer_name
try:
dealer_name.append(result.find('div', {'class':'dealer-name'}).get_text().strip())
except:
dealer_name.append('n/a')
#link
try:
source.append(result.find('a', {'class':'vehicle-card-visited-tracking-link'}).get('href'))
except:
source.append('n/a')
for link in source:
url_combine.append(urllib.parse.urljoin(root_url, link))
#Using Pandas to create a dictionary and import to Excel
car_listings = pd.DataFrame({'Name': name, 'Mileage':mileage, 'Price': price, 'Dealer Name':dealer_name,'Link': source})
car_listings.to_excel('car_listings_page4.xlsx')
However, I keep running into prolem where it says that AttributeError: 'tuple' object has no attribute 'append'.
I know I need to make everything a list instead of a tuple but I can't seem to find where my mistake is. I believe this is one way to get the full url from href. If so, is there any other way I can implement into my code?

Answer 1

Avoid all these lists and use dicts instead, simplify and store info in more structured way - You could also use if-statement.
There are various ways to perform string concatination:
+ operator
join() method
% operator
format() function
f-string literal string interpolation (check in example for assignment
to website)
However simplest one is using + operator:
root_url+link
Concerning my example using dicts it would look like:
'link': root_url+result.find('a', {'class':'vehicle-card-visited-tracking-link'}).get('href') if result.find('a', {'class':'vehicle-card-visited-tracking-link'}) else None
or a bit shorter with walrus operator (Python 3.8 and later):
'link': root_url+a.get('href') if (a:=result.find('a', {'class':'vehicle-card-visited-tracking-link'})) else None
Example
from bs4 import BeautifulSoup
import requests
import pandas as pd
root_url = 'http://www.cars.com'
data = []
for i in range (1,2):
website =f'https://www.cars.com/shopping/results/?page={i}&page_size=20&dealer_id=&keyword=&list_price_max=&list_price_min=&makes[]=&maximum_distance=all&mileage_max=&sort=best_match_desc&stock_type=used&year_max=&year_min=&zip=95355'
response = requests.get(website)
soup = BeautifulSoup(response.content, 'html.parser')
results = soup.find_all('div', {'class':'vehicle-card'})
for result in results:
data.append({
'name': result.find('h2').get_text() if result.find('h2') else None,
'price': result.find('span', {'class':'primary-price'}).get_text() if result.find('span', {'class':'primary-price'}) else None,
'link': root_url+a.get('href') if (a:=result.find('a', {'class':'vehicle-card-visited-tracking-link'})) else None
### all the other info
})
pd.DataFrame(data)
Output
name
price
link
0
2017 Lexus IS 200t Base
$28,900
http://www.cars.com/vehicledetail/6942c51b-c26c-4614-97f1-acb0b7517b82/
1
2021 Lincoln Corsair Reserve
$43,797
http://www.cars.com/vehicledetail/e575219a-90fa-4a95-ade5-d2740e746cd0/
2
2021 Hyundai IONIQ Hybrid SE
$26,997
http://www.cars.com/vehicledetail/716b65ec-3abd-42e4-b19b-9024d2ad58f1/
3
2021 GMC Yukon XL Denali
$74,888
http://www.cars.com/vehicledetail/475045f6-142a-440f-80e7-2c3ae289fee2/
4
2007 Chevrolet Silverado 1500
$12,688
http://www.cars.com/vehicledetail/56080319-0bb9-49e0-8758-24f58d0d5d76/
...