Scraping information from website using beautifullsoup wont work - python-3.x

Question

I have been using beautiful soup to extract info from the website http://slc.bioparadigms.org
But I am only interested in the diseases and OMIM number, so for each SLC transporter which i already have in a list i want to extract these 2 characteristics. The thing is that both are related to class prt_col2. So if i search for this class i get a lot of hits. How can I only get the diseases? Also sometimes there are no diseases related to the SLC transporter or sometimes there is no OMIM number. How can i extract the information? I put some screenshots below to show you how it looks like. Any help will be highly appreciated! This is my first post here so forgive me for any mistakes or missing information. Thank you!
http://imgur.com/aTiGi84 other one is /L65HSym
So ideally the output will be for example:
transporter: SLC1A1
Disease: Epilepsy
OMIM: 12345
Edit: the code i have so far:
import os
import re
from bs4 import BeautifulSoup as BS
import requests
import sys
import time
def hasNumbers(inputString): #get transporter names which contain numbers
return any(char.isdigit() for char in inputString)
def get_list(file): #get a list of transporters
transporter_list=[]
lines = [line.rstrip('\n') for line in open(file)]
for line in lines:
if 'SLC' in line and hasNumbers(line) == True:
get_SLC=line.split()
if 'SLC' in get_SLC[0]:
transporter_list.append(get_SLC[0])
return transporter_list
def get_transporter_webinfo(transporter_list):
output_Website=open("output_website.txt", "w") # get the website content of all transporters
for transporter in transporter_list:
text = requests.get('http://slc.bioparadigms.org/protein?GeneName=' + transporter).text
output_Website.write(text) #ouput from the SLC tables website
soup=BS(text, "lxml")
disease = soup(text=re.compile('Disease'))
characteristics=soup.find_all("span", class_="prt_col2")
memo=soup.find_all("span", class_='expandable prt_col2')
print(transporter,disease,characteristics[6],memo)
def convert(html_file):
file2= open(html_file, 'r')
clean_file= open('text_format_SLC','w')
soup=BS(file2,'lxml')
clean_file.write(soup.get_text())
clean_file.close()
def main():
start_time=time.time()
os.chdir('/home/Programming/Fun stuff')
sys.stdout= open("output_SLC.txt","w")
SLC_list=get_list("SLC.txt")
get_transporter_webinfo(SLC_list) #already have the website content so little redundant
print("this took",time.time() - start_time, "seconds to run")
convert("output_SLC.txt")
sys.stdout.close()
if __name__ == "__main__":
main()

Answer 1

No offence intended, I didn't feel like reading such a large piece of code as what you put in your question.
I would say it can be simplified.
You can get the complete list of links to the SLCs in the line at SLCs =. The next line shows how many there are, and the line beyond that exhibits the href attribute that the last link contains, as an example.
In each SLC's page I look for the string 'Disease' and then, if it's there, I navigate to the link nearby. I find the OMIM in a similar way.
Notice that I process only the first SLC.
>>> import requests
>>> import bs4
>>> main_url = 'http://slc.bioparadigms.org/'
>>> main_page = requests.get(main_url).content
>>> main_soup = bs4.BeautifulSoup(main_page, 'lxml')
>>> stem_url = 'http://slc.bioparadigms.org/protein?GeneName=SLC1A1'
>>> SLCs = main_soup.select('td.slct.tbl_cell.tbl_col1 a')
>>> len(SLCs)
418
>>> SLCs[-1].attrs['href']
'protein?GeneName=SLC52A3'
>>> stem_url = 'http://slc.bioparadigms.org/'
>>> for SLC in SLCs:
... SLC_page = requests.get(stem_url+SLC.attrs['href'], 'lxml').content
... SLC_soup = bs4.BeautifulSoup(SLC_page, 'lxml')
... disease = SLC_soup.find_all(string='Disease: ')
... if disease:
... disease = disease[0]
... diseases = disease.findParent().findNextSibling().text.strip()
... else:
... diseases = 'No diseases'
... OMIM = SLC_soup.find_all(string='OMIM:')
... if OMIM:
... OMIM = OMIM[0]
... number = OMIM.findParent().findNextSibling().text.strip()
... else:
... OMIM = 'No OMIM'
... number = -1
... SLC.text, number, diseases
... break
...
('SLC1A1', '133550', "Huntington's disease, epilepsy, ischemia, Alzheimer's disease, Niemann-Pick disease, obsessive-compulsive disorder")