How to scrape location names from a website using VBA - excel

Question

I am trying to scrape the pickup branch locations from a car rental website home page. The idea is to see exactly where pickup branches exist for a given company.
I have successfully created this before but this company revamped their website recently and now my code doesn't work. The branch locations seem to be hidden within a form of some sort, the locations only become visible in the html once you click on the pickup location space.
My current code looks as below:
Option Explicit
Private Sub pickuplocations()
Dim html As Object
Dim ws As Worksheet
Dim headers()
Dim i As Long
Dim r As Long
Dim c As Long
Dim numrows As Long
Set ws = ThisWorkbook.Worksheets("Europcar Branches(2)")
Set html = New HTMLDocument
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://www.europcar.co.za", False
.send
html.body.innerHTML = .responseText 'fetches all html from the website
Dim pickupbranches As Object
Dim pickupbranchresults()
Set pickupbranches = html.getElementById("_location-search-widget_15").getElementsByTagName("span")
headers = Array("Pickup Location", "Option value") 'for the ws
numrows = pickupbranches.Length - 1 'sets the row length
ReDim pickupbranchresults(1 To numrows, 1 To 2) 'sets array size for the results
For i = 1 To numrows
pickupbranchresults(i, 1) = pickupbranches.Item(i).innerText
pickupbranchresults(i, 2) = pickupbranches.Item(i).Value
Next
With ws
.Cells(1, 1).Resize(1, UBound(headers) + 1) = headers 'sets the column headers
.Cells(2, 1).Resize(UBound(pickupbranchresults, 1), UBound(pickupbranchresults, 2)) = pickupbranchresults
End With
End With
End Sub

Answer 1

Your current code requests the source HTML code and attempts to scrape it.
However, as explained in the comments, the list of locations is dynamically loaded when you click on the search bar and it is not a part of the page's source HTML. For this reason your code will yield no results.
It makes more sense to scrape the page dedicated to the locations:
https://www.europcar.co.za/rental-locations/
Now, if you navigate to this page and inspect the network traffic in your browser's developer tools (F12) when the page is loaded, you will see that an XHR request is being sent. It looks like so:
If you go through the Headers and the Params of the request you will see how the url, the body and the headers should look like. In this particular case, there are no parameters and the headers are not essential to the success of the request, so all you need is the url.
The response's payload is in json format. You can inspect its structure using a tool like this. Here's how it looks like:
Basically, the JSON consists of the different countries, each country consists of provinces and each province consists of the corresponding branches. Each branch consists of all the corresponding info.
To parse a response like that you need a JSON parser (look at the end of this post).
TL;DR
Here's how the code should look like:
Option Explicit
Sub getLocations()
Dim req As New WinHttpRequest
Dim url As String, results() As String
Dim sht As Worksheet
Dim responseJSON As Object, country As Object, province As Object, branch As Object
Dim i As Long
Dim rng As Range
Set sht = ThisWorkbook.Worksheets("Name of your Worksheet")
url = "https://www.europcar.co.za/api/rentalLocations/impressLocations"
With req
.Open "GET", url, False
.send
Set responseJSON = JsonConverter.ParseJson(.responseText)
End With
For Each country In responseJSON
For Each province In country("provinces")
i = 0
ReDim results(1 To province("branches").Count, 1 To 5)
For Each branch In province("branches")
i = i + 1
results(i, 1) = country("name")
results(i, 2) = province("name")
results(i, 3) = branch("name")
results(i, 4) = branch("emailAddress")
results(i, 5) = branch("contactNumber")
Next branch
With sht
Set rng = .Range("A" & .Rows.Count).End(xlUp).Offset(1, 0)
End With
rng.Resize(UBound(results, 1), UBound(results, 2)) = results
Next province
Next country
End Sub
For demonstration purposes the code above prints out the results in the following way:
Having in mind the JSON structure and the sample code I provided, you can easily modify it to fit your needs.
For the code to work you will need to add the following references to your project (VBE>Tools>References):
1. Microsoft WinHTTP Services version 5.1
2. Microsoft Scripting Runtime
You will also need to add this JSON parser to your project. Follow the installation instructions in the link and you should be set to go.