Crawling data but the url doesn't change - python-3.x

Question

I want to crawl data using python from this webpage:
https://www.discountoptiondata.com/freedata/
by keep same value for expiration date and symbol but iterating over all values of the start date.
The problem is that the URL stays same for all combinations and hence I cannot get a list of the URLs I want to crawl.
anybody has ideas about how I can do that?

Answer 1

The website you are trying to parse is dynamic, which means it runs some code when you download it in your browser. In your case, the code is set to fetch the data when the "Get OptionData" button is clicked.
You can actually see the browser fetch the data in the Network tab of your browsers Developer Tools. F12 → Network → (Refresh the page) → Fill out the form and Click "Get OptionData". It will show up as a XHR request in the Network Tab list.
The response of the data fetch will look a bit like this
{
"AskPrice": "5.7",
"AskSize": "",
"BidPrice": "0.85",
"ExpirationDate": "2019-06-21",
"LastPrice": "4.4",
"StrikePrice": "1000",
"Symbol": "SPX"
}
The data returned from the data fetch is encoded as JSON, and lucky for us, its very easy to parse in Python. You can get the above JSON code by investigating the XHR request in the Network tab, this was the URL for me
https://www.discountoptiondata.com/freedata/getoptiondatajson?symbol=spx&datadate=2018-06-01&expirationDate=2018-06-15
I am unfamiliar with scrapy, but for JSON based parsing, I would recommend the 'requests' module. Here is an example program that will fetch the data shown on the webpage
import requests
ROOT_URL = "https://www.discountoptiondata.com/freedata/getoptiondatajson"
def fetch_option_data(symbol, datadate, expiration_date):
response = requests.get(ROOT_URL, params={"symbol": symbol, "datadate": datadate, "expirationDate": expiration_date})
return response.json()
data = fetch_option_data('spx', '2018-06-01', '2018-06-15')
for item in data:
print("AskPrice:", item['AskPrice'], "Last Price:", item["LastPrice"])

Answer 2

To view the request or response HTTP headers in Google Chrome, take the following steps :
In Chrome, visit a URL, right click, select Inspect to open the developer tools.
Select Network tab.
Reload the page, select any HTTP request on the left panel, and the HTTP headers will be displayed on the right panel.
Source
In your case,
Open https://www.discountoptiondata.com/freedata/ in Google Chrome
Right click, select Inspect and select Network Tab
Now if you Select start date, you will find the request url under "Headers" Tab.
Same way you can view the response under "Response" Tab.
Here are the screenshots:
start_date_request_url
option_data_request_url
Example:
Start Date Request URL:
https://www.discountoptiondata.com/freedata/getexpirationdates?symbol=spx&datadate=2018-06-01
Option Data Request URL:
https://www.discountoptiondata.com/freedata/getoptiondatajson?symbol=spx&datadate=2018-06-01&expirationDate=2018-06-15