Web scraping with Beautiful Soup (Not capturing all Information) - python-3.x

Question

I have used the beautiful soup package a few times, but this is the first time it doesn't have all the information I need. How do I get the full webpage? I need to extract all the publications and hyperlinks to the papers.
from bs4 import BeautifulSoup
import requests
url = 'https://openreview.net/group?id=ICLR.cc/2018/Conference'
source = requests.get(url).text
soup = BeautifulSoup(source, 'html.parser')

Answer 1

There are other HTTP requests that are filling in the webpage.
A good way of seeing these is using the inspector provided in a web browser.
In Chrome, you can see these requests under the 'Network' tab in the inspector.
The requests are as follows:
GET https://openreview.net/notes?invitation=ICLR.cc%2F2018%2FConference%2F-%2FBlind_Submission&details=replyCount&offset=0&limit=1000
GET https://openreview.net/notes?invitation=ICLR.cc%2F2018%2FConference%2F-%2FWithdrawn_Submission&noDetails=true&offset=0&limit=1000
GET https://openreview.net/notes?invitation=ICLR.cc%2F2018%2FConference%2F-%2FAcceptance_Decision&noDetails=true&offset=0&limit=1000
It appears that each one returns JSON text with the information you are looking for (the publications and hyperlinks to the papers),
so you can just create an individual request for each of these URL's and access the returned JSON in the following manner:
import json
source = requests.get(new_url).text
# json.loads returns a Python dictionary
data = json.loads(source)
for publication in data['notes']:
publication_info = publication['_bibtex']
url = publication_info.split('\nurl={')[1].split('}')[0]
The element containing the URL for each publication is rather difficult to parse since it has characters not allowed in dictionary names (i.e. '#'),
but this solution should work.
Note that I have not tested this solution, so there might be some errors, but the underlying logic behind the solution should be correct.
Alternatively:
You can use Splash, which is used to render Javascript-based pages. You can run Splash in Docker quite easily, and just make HTTP requests to the Splash container which will return HTML that looks just like the webpage as rendered in a web browser.
Although this sounds overly complicated, it is actually quite simple to set up since you don't need to modify the Docker image at all, so you need no previous knowledge of docker to work. It requires just a single line to start a local Splash server:
docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash
You then just modify any existing requests you have in your Python code to route to splash instead:
i.e. http://example.com/ becomes
http://localhost:8050/render.html?url=http://example.com/