How to actually use loop in beautifulsoup? - python-3.x

Question

I'm trying to make a web crawler. I use some loop inside. The loop run well for the first loop, but not to the second. I always get this message: "During handling of the above exception, another exception occurred"
import requests
from bs4 import BeautifulSoup
result =
requests.get("http://desaku.bandungkab.go.id/desaonline/")
#This url is the main web, inside this web there are 270 links of
#other website. I get into that 270 webs and open every article in
#each
web
src = result.content
soup = BeautifulSoup(src, 'lxml')
links = soup.find_all('a')
urls = []
for link in links:
if "www" in link.text:
url = link.attrs['href']
urls.append(url)
num1=len(urls)
b=0
while b<num1:
result2 = requests.get(urls[b])
src2 = result2.content
soup = BeautifulSoup(src2, 'lxml')
links2 = soup.find_all('a')
urls2 = []
for link in links2:
if "selengkapnya" in link.text:
url2 = link.attrs['href']
urls2.append(url2)
b+=1
#the code run well until this part. If i print this, it will result
#url that take me directly to specific article
num=len(urls2)
i=0
while i<num:
result2 = requests.get(urls2[i])
src2 = result2.content
soup = BeautifulSoup(src2, 'lxml')
links2 = soup.find_all('a')
artikel=[]
isi = link.attrs['href']
artikel.append(isi)
print(artikel)
i+=1
I expect to get all the link of article from the website and put them into a list called artikel=[]

Answer 1

The problem is you are assigning to urls2 = [] and to artikel=[] each loop iteration, effectively replacing the list. At the end of iteration you have empty array. You could use this code as a start:
import requests
from bs4 import BeautifulSoup
result = requests.get("http://desaku.bandungkab.go.id/desaonline/")
#This url is the main web, inside this web there are 270 links of
#other website. I get into that 270 webs and open every article in
#each web
src = result.content
soup = BeautifulSoup(src, 'lxml')
urls = []
urls2 = []
for link in soup.select('a[href]:contains(www)'):
urls.append(link['href'])
print('Urls:')
for url in urls:
print('Downloading {}'.format(url))
result2 = requests.get(url)
src2 = result2.content
soup = BeautifulSoup(src2, 'lxml')
for link in soup.select('a[href]:contains(selengkapnya)'):
print('\tFound link {}'.format(link['href']))
urls2.append(link['href'])
print('Articles:')
articles = []
for url2 in urls2:
print('Downloading {}'.format(url2))
result2 = requests.get(url2)
src2 = result2.content
soup = BeautifulSoup(src2, 'lxml')
for link in soup.find_all('a[href]'):
articles.append(link['href'])
print(articles)
Prints:
Urls:
Downloading http://www.ancolmekar.desa.id
Found link http://www.ancolmekar.desa.id/first/artikel/423
Found link http://www.ancolmekar.desa.id/first/artikel/421
Found link http://www.ancolmekar.desa.id/first/artikel/420
Found link http://www.ancolmekar.desa.id/first/artikel/419
Found link http://www.ancolmekar.desa.id/first/artikel/414
Found link http://www.ancolmekar.desa.id/first/artikel/413
Found link http://www.ancolmekar.desa.id/first/artikel/412
Found link http://www.ancolmekar.desa.id/first/artikel/410
Found link http://www.ancolmekar.desa.id/first/artikel/410
Found link http://www.ancolmekar.desa.id/first/artikel/100
Downloading http://www.arjasari.desa.id
Found link http://www.arjasari.desa.id/first/artikel/180
Found link http://www.arjasari.desa.id/first/artikel/190
...and so on.