Fetch Data from HTML Website using VBA - FREEMAPTOOLS.COM - excel

I am trying to input a post code into this website and pull the results into Excel using VBA
http://www.freemaptools.com/find-uk-postcodes-inside-radius.htm
In short you input a post code and set a radius either in miles or KM and it gives you all the post codes within that area. As you can imagine this tool would be very useful!
This is what I have so far:
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = 0
url = "http://www.freemaptools.com/find-uk-postcodes-inside-radius.htm"
ie.Navigate url
state = 0
Do Until state = 4
DoEvents
state = ie.readyState
Loop
It would be good if say cell A1 had the post code and cell A2 had the distance in KM. This script would then look at this as the variable.
I am not 100% sure put I think I then need to Parse the result to put them each into there own cell.
Any help with this would be incredible!

Here you go
Download the file
Sub postcode()
Dim URL As String, str_output As String, arr_output() As String, row As Long
Dim obj_Radius As Object, obj_Miles As Object, post_code As Object
Dim btn As Object, btn_Radius As Object, tb_output As Object
URL = "http://www.freemaptools.com/find-uk-postcodes-inside-radius.htm"
Dim IE As Object
Set IE = CreateObject("internetexplorer.application")
IE.Visible = True
IE.navigate URL
Do While IE.readystate <> 4
DoEvents
Loop
delay 5
Set obj_Radius = IE.document.getelementbyid("tb_radius")
obj_Radius.Value = ThisWorkbook.Sheets(1).Range("B1")
Set obj_Miles = IE.document.getelementbyid("tb_radius_miles")
obj_Miles.Value = ThisWorkbook.Sheets(1).Range("B2")
Set post_code = IE.document.getelementbyid("goto")
post_code.Value = ThisWorkbook.Sheets(1).Range("B3")
Set btn_Radius = IE.document.getelementsbytagname("Input")
For Each btn In btn_Radius
If btn.Value = "Draw Radius" Then
btn.Click
End If
Next
Do While IE.readystate <> 4
DoEvents
Loop
delay 10
Set tb_output = IE.document.getelementbyid("tb_output")
str_output = tb_output.innerText
arr_output = Split(str_output, ",")
row = 1
For i = LBound(arr_output) To UBound(arr_output)
ThisWorkbook.Sheets(1).Range("C" & row) = arr_output(i)
row = row + 1
Next
End Sub
Private Sub delay(seconds As Long)
Dim endTime As Date
endTime = DateAdd("s", seconds, Now())
Do While Now() < endTime
DoEvents
Loop
End Sub

Related

VBA for Web Scraping works as Sub but Not as Function

I have written a VBA to scrape the status of a shipment from a cargo tracking site with the help of you guys here. I am trying to convert it to a function. The code works as a sub but does not work as a function. It returns a #Value error. Can someone please tell me what I am doing wrong.
Here is the code as a sub
Sub FlightStat_AFL()
Dim url As String
Dim ie As Object
Dim MAWBStatus As String
Dim MAWBNo As String
MAWBNo = Sheets("Sheet3").Range("H3").Value
'You can handle the parameters id and pfx in a loop to scrape dynamic numbers
url = "https://www.afklcargo.com/mycargo/shipment/detail/057-" & MAWBNo
'Initialize Internet Explorer, set visibility,
'call URL and wait until page is fully loaded
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = False
ie.navigate url
Do Until ie.readyState = 4: DoEvents: Loop
'Wait to load dynamic content after IE reports it's ready
'We do that with a fix manual break of a few seconds
'because the whole page will be "reload"
'The last three values are hours, minutes, seconds
Application.Wait (Now + TimeSerial(0, 0, 3))
'Get the status from the table
MAWBStatus = ie.document.getElementsByClassName("fs-12 body-font-bold")(1).innertext
Debug.Print MAWBStatus
'Clean up
ie.Quit
Set ie = Nothing
End Sub
Here is the code I am trying to make it work as a function.
Function FlightStat_AF(MAWBNo As Range) As String
Dim url As String
Dim ie As Object
Dim MAWBStatus As String
url = "https://www.afklcargo.com/mycargo/shipment/detail/057-" & MAWBNo
'Initialize Internet Explorer, set visibility,
'call URL and wait until page is fully loaded
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = False
ie.navigate url
Do Until ie.readyState = 4: DoEvents: Loop
'Wait to load dynamic content after IE reports it's ready
'We do that with a fix manual break of a few seconds
'because the whole page will be "reload"
'The last three values are hours, minutes, seconds
Application.Wait (Now + TimeSerial(0, 0, 3))
'Get the status from the table
MAWBStatus = ie.document.getElementsByClassName("fs-12 body-font-bold")(1).innertext
FlightStat_AF = MAWBStatus
'Clean up
ie.Quit
Set ie = Nothing
End Function
Try the next code, please:
Function FlightStat_AF(cargoNo As Variant) As String
Dim url As String, ie As Object, result As String
url = "https://www.afklcargo.com/mycargo/shipment/detail/" & cargoNo
Set ie = CreateObject("InternetExplorer.Application")
With ie
.Visible = False
.navigate url
Do Until .readyState = 4: DoEvents: Loop
End With
'wait a little for dynamic content to be loaded
Application.Wait (Now + TimeSerial(0, 0, 1))
'Get the status from the table
Do While result = ""
DoEvents
On Error Resume Next
result = Trim(ie.document.getElementsByClassName("fs-12 body-font-bold")(1).innerText)
On Error GoTo 0
Application.Wait (Now + TimeSerial(0, 0, 1))
Loop
ie.Quit: Set ie = Nothing
'Return value of the function
FlightStat_AF = result
End Function
IE Function
You can try this if you really want a range. Usually it should be a string which you can easily change.
You can test the function (2nd procedure) with the first procedure. Just adjust the values in the constants section.
The Code
Option Explicit
Sub getFlightStat()
' Constants
Const wsName As String = "Sheet3"
Const FirstRow As Long = 3
Const CritCol As Variant = "H"
Const ResCol As Variant = "I"
Dim wb As Workbook: Set wb = ThisWorkbook
' Define worksheet.
Dim ws As Worksheet
Set ws = wb.Worksheets(wsName)
' Calculate the row of the last non-blank cell in column 'CritCol'.
Dim LastRow As Long
LastRow = ws.Cells(ws.Rows.Count, CritCol).End(xlUp).Row
' Loop through rows and for each value in cell of column 'CritCol',
' write the value retrieved via 'FlightStat_AF' to the cell
' in the same row, but in column 'ResCol'.
Dim i As Long
For i = FirstRow To LastRow
ws.Cells(i, ResCol).Value = FlightStat_AF(ws.Cells(i, CritCol))
Next i
' Inform user.
MsgBox "Data transferred", vbInformation, "Success"
End Sub
Function FlightStat_AF(MAWBNo As Range) As String
Dim url As String
Dim ie As Object
Dim MAWBStatus As String
'You can handle the parameters id and pfx in a loop to scrape dynamic numbers
url = "https://www.afklcargo.com/mycargo/shipment/detail/057-" & MAWBNo.Value
'Initialize Internet Explorer, set visibility,
'call URL and wait until page is fully loaded
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = False
ie.navigate url
Do Until ie.readyState = 4: DoEvents: Loop
'Wait to load dynamic content after IE reports it's ready
'We do that with a fix manual break of a few seconds
'because the whole page will be "reload"
'The last three values are hours, minutes, seconds
Application.Wait (Now + TimeSerial(0, 0, 3))
'Get the status from the table
MAWBStatus = ie.document.getElementsByClassName("fs-12 body-font-bold")(1).innertext
FlightStat_AF = MAWBStatus
'Clean up
ie.Quit
Set ie = Nothing
End Function

How to scrape a class and if not found scrape another

I am using VBA to scrape a website. The scraper made by me works but I want to implement 2 more functions and don't really know how to do it. This is the code:
Sub pronutrition()
Set ie = CreateObject("InternetExplorer.Application")
my_url = "https://www.myprotein.ro/"
ie.Visible = True
i = 20
LastRow = ActiveSheet.Range("A" & ActiveSheet.Rows.Count).End(xlUp).Row
Set Rng = ActiveSheet.Range("A20:A" & LastRow)
For Each cell In Rng
ie.navigate my_url
Do While ie.Busy
DoEvents
Loop
Wait 1
ie.Document.getElementsByName("search")(0).Value = cell
ie.Document.getElementsByClassName("headerSearch_button")(0).Click
Do While ie.Busy
DoEvents
Loop
Wait 2
ActiveSheet.Range("B" & i) = ie.Document.getElementsByClassName("athenaProductBlock_productName")(0).innerText + ie.Document.getElementsByClassName("athenaProductBlock_fromValue")(0).innerText
Do While ie.Busy
DoEvents
Loop
Wait 2
ActiveSheet.Range("C" & i) = ie.Document.getElementsByClassName("athenaProductBlock_productName")(1).innerText + ie.Document.getElementsByClassName("athenaProductBlock_fromValue")(1).innerText
Do While ie.Busy
DoEvents
Loop
Wait 2
ActiveSheet.Range("D" & i) = ie.Document.getElementsByClassName("athenaProductBlock_productName")(2).innerText '+ ie.Document.getElementsByClassName("athenaProductBlock_priceValue")(2).innerText
Do While ie.Busy
DoEvents
Loop
Wait 2
ActiveSheet.Range("E" & i) = ie.Document.getElementsByClassName("athenaProductBlock_productName")(3).innerText '+ ie.Document.getElementsByClassName("athenaProductBlock_priceValue")(3).innerText
Do While ie.Busy
DoEvents
Loop
Wait 2
i = i + 1
Next cell
ie.Quit
MsgBox "Done"
End Sub
First I want to search for "athenaProductBlock_fromValue" class and if it doesn't find it to search for "athenaProductBlock_priceValue", and second, if it doesn't find more than 1 or 2 products (the range is set to 4) to stop the search (right now it returns and error if it doesn't find a 2nd or a 3rd product and won't go to search the next keyword).
Any advice would be appreciated.
Thank you!
Use a helper method to extract the HTMLCollection returned by the getElementsByClassName method. You can then check if the method returned any results.
Once you get back the collection filled, it's up to you how to handle it. You can loop and fill individual cells or join the results to fill a single cell. Also, if the Count is less then 2, ignore it etc.
Private Function TryExtractElementsByClassName(ByVal ie As Object,
ByVal className As String,
ByRef objCollection As VBA.Collection) As Boolean
'if ie is null, return false
If ie Is Nothing Then Exit Function
'if elements (HTMLCollection) is null, return false
Dim elements As Object
Set elements = ie.Document.getElementsByClassName(className)
If elements Is Nothing Then Exit Function
'fill collection
Dim element As Object, idx As Long
For idx = 0 To elements.Length
Set element = elements(idx)
If Not element Is Nothing Then objCollection.Add element
Next idx
'return
TryExtractElementsByClassName = objCollection.Count > 0
End Function
To call the helper method:
Sub Test()
Dim ie As Object
Set ie = CreateObject("InternetExplorer.Application")
Dim objColl As New VBA.Collection
'search single class name
If TryExtractElementsByClassName(ie, "athenaProductBlock_priceValue", objColl) Then
'handle results stored in objColl
End If
'search multiple class names separated by a space
If TryExtractElementsByClassName(ie, "athenaProductBlock_priceValue athenaProductBlock_fromValue", objColl) Then
'handle results stored in objColl
End If
End Sub

web scraping using excel and VBA

i wrote my VBA code in excel sheet as below but it is not scrape data for me and also i don't know why please any one help me. it gave me reullt as "click her to read more" onlyi want to scrape enitre data such as first name last name state zip code and so on
Sub extractTablesData()
Dim IE As Object, obj As Object
Dim myState As String
Dim r As Integer, c As Integer, t As Integer
Dim elemCollection As Object
Set IE = CreateObject("InternetExplorer.Application")
myState = InputBox("Enter the city where you wish to work")
With IE
.Visible = True
.navigate ("http://www.funeralhomes.com/go/listing/Search? name=&city=&state=&country=USA&zip=&radius=")
While IE.readyState <> 4
DoEvents
Wend
For Each obj In IE.document.all.item("state").Options
If obj.innerText = myState Then
obj.Selected = True
End If
Next obj
IE.document.getElementsByValue("Search").item.Click
Do While IE.Busy: DoEvents: Loop
ThisWorkbook.Sheets("Sheet1").Range("A1:K1500").ClearContents
Set elemCollection = IE.document.getElementsByTagName("TABLE")
For t = 0 To (elemCollection.Length - 1)
For r = 0 To (elemCollection(t).Rows.Length - 1)
For c = 0 To (elemCollection(t).Rows(r).Cells.Length - 1)
ThisWorkbook.Worksheets(1).Cells(r + 1, c + 1) = elemCollection(t).Rows(r).Cells(c).innerText
Next c
Next r
Next t
End With
Set IE = Nothing
End Sub
Using the same URL as the answer already given you could alternatively select with CSS selectors to get the elements of interest, and use split to get just the names and address parts from the text. We can also do away with the browser altogether to get faster results from first results page.
Business name:
You can get the name with the following selector (using paid listing example):
div.paid-listing .listing-title
This selects (sample view)
Try
Address info:
The associated descriptive information can be retrieved with the selector:
div.paid-listing .address-summary
And then using split we can parse this into just the address information.
Code:
Option Explicit
Public Sub GetTitleAndAddress()
Dim oHtml As HTMLDocument, nodeList1 As Object, nodeList2 As Object, i As Long
Const URL As String = "http://www.funeralhomes.com/go/listing/ShowListing/USA/New%20York/New%20York"
Set oHtml = New HTMLDocument
With CreateObject("WINHTTP.WinHTTPRequest.5.1")
.Open "GET", URL, False
.send
oHtml.body.innerHTML = .responseText
End With
Set nodeList1 = oHtml.querySelectorAll("div.paid-listing .listing-title")
Set nodeList2 = oHtml.querySelectorAll("div.paid-listing .address-summary")
With Worksheets("Sheet3")
.UsedRange.ClearContents
For i = 0 To nodeList1.Length - 1
.Range("A" & i + 1) = nodeList1.Item(i).innerText
.Range("B" & i + 1) = Split(nodeList2.Item(i).innerText, Chr$(10))(0)
Next i
End With
End Sub
Example output:
Yeah, without an API, this can be very tricky at best, and very inconsistent at worst. For now, you can try the script below.
Sub DumpData()
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
URL = "http://www.funeralhomes.com/go/listing/ShowListing/USA/New%20York/New%20York"
'Wait for site to fully load
IE.Navigate2 URL
Do While IE.Busy = True
DoEvents
Loop
RowCount = 1
With Sheets("Sheet1")
.Cells.ClearContents
RowCount = 1
For Each itm In IE.document.all
If itm.classname Like "*free-listing*" Or itm.classname Like "*paid-listing*" Then
.Range("A" & RowCount) = itm.classname
.Range("B" & RowCount) = Left(itm.innertext, 1024)
RowCount = RowCount + 1
End If
Next itm
End With
End Sub
You probably want some kind of input box to capture the city and state and radius from the user, or capture those variable in cells in your worksheet.
Notice, the '%20' is a space character.
I got this idea from a friend of mine, Joel, a long time ago. That guy is great!

Looping through a row and copying each cell In a specific procedure

What I have to do is use Excel VBA to:
login to Amazon Seller
open a workbook
loop through a column to get an order number
put it in the search box
hit the search button
go to the order page and extract the data
then have the extracted data go back into a specified column in
another Excel workbook
The loop and order number parts are what I'm currently stumped on. I've figured out this much code as of this moment:
Sub MyAmazonSeller()
Dim MyHTML_Element As IHTMLElement
Dim MyURL As String
Dim oSignInLink As HTMLLinkElement
Dim oInputEmail As HTMLInputElement
Dim oInputPassword As HTMLInputElement
Dim oInputSigninButton As HTMLInputButtonElement
'InputSearchOrder will be the destination for order numbers taken from the workbook
Dim InputSearchOrder As HTMLInputElement
Dim InputSearchButton As HTMLInputButtonElement
Dim IE As InternetExplorer
Dim AAOrder As Workbook
Dim AAws As Worksheet
MyURL = "https://sellercentral.amazon.com/gp/homepage.html"
Set IE = New InternetExplorer
' Open the browser and navigate.
With IE
.Silent = True
.Navigate MyURL
.Visible = True
Do
DoEvents
Loop Until .ReadyState = READYSTATE_COMPLETE
End With
' Get the html document.
Set HTMLDoc = IE.Document
' See if you have the sign in link is because you are in the main
' page
Set oSignInLink = HTMLDoc.getElementById("signin-button-container")
If Not oSignInLink Is Nothing Then
oSignInLink.Click
Do
DoEvents
Loop Until IE.ReadyState = READYSTATE_COMPLETE
End If
' Get the email field and the next button
Set oInputEmail = HTMLDoc.getElementById("username")
Set oInputPassword = HTMLDoc.getElementById("password")
' Click the button and wait
oInputEmail.Value = "xxxxxx#xxxxxx.net"
' Get the password field and the sign in button
Set oInputPassword = HTMLDoc.getElementById("password")
Set oInputSigninButton = HTMLDoc.getElementById("sign-in-button")
' Click the button and wait
oInputPassword.Value = "xxxxxxxx"
oInputSigninButton.Click
Do
DoEvents
Loop Until IE.ReadyState = READYSTATE_COMPLETE
Application.Wait (Now + TimeValue("0:00:05"))
Set AAOrder = Application.Workbooks.Open("Z:\Employee Folders\Employee\trackingnumber_sample_spreadsheet.xls")
Set AAws = AAws.Worksheets("PrimeOrdersWithNoFulfillmentRe")
Set InputSearchOrder = HTMLDoc.getElementById("sc-search-field")
'What I'm currently stuck on
InputSearchOrder.Value = "001-7163923-7572632"
Set InputSearchButton = HTMLDoc.getElementsByClassName("sc-search-button")(0)
InputSearchButton.Click
Do
DoEvents
Loop Until IE.ReadyState = READYSTATE_COMPLETE
'Was able to add this snippet, but I'm getting an error 13, most likely with
'my e variable. I'm basically trying to do a loop within a loop, extracting 5
'pieces of data and sticking them back into their respective columns in the
'original Excel sheet. The problem comes when scraping the HTML. I'm basically
'trying to get text in the tables which have a few levels and it's frustrating
'me to no end.
With HTMLDoc
Set elems = HTMLDoc.getElementsByTagName("td")
For Each e In elems
If e.innerText Like "*1Z*" Then
Range("D2").Value = e.innerText
End If
Next e
End With
Err_Clear:
If Err <> 0 Then
Err.Clear
Resume Next
End If
End Sub

How can I avoid empty values on IE.Navigate and GetElementById?

I code some vba scritp that open ie.navigate and get some values from webpage into excell values.
But sometimes that url does not load propetly on webpage and the values are not filled. So when the getelement try get any valume my loop camer fatal error and stoped.
another point is, imagine the all page is loadade..but for this criteria some geElementID is empyt...what I can do to avoid it.
my code here:
Sub Extract_One_Airport()
Dim IE As New InternetExplorer
Dim dtStart As Date
Datec = 0
CountRange = 1
For lSCtr = 0 To 5
Set P1 = Sheets("Inicio").Range("A9") 'Primeira p do link
Set P2 = Sheets("Inicio").Range("A10") 'Origem
link = P1 & P2
IE.navigate link
IE.Visible = True
Application.Wait (Now() + TimeValue("00:00:45")) ' For internal page refresh or loading
Dim doc As HTMLDocument
Set doc = IE.document
Dim dd As Variant
dd = doc.getElementsByClassName("valuefortoday")(0).innerText
Sheets("Resul").Range("C" & CountRange).Value = dd
Count = Count + 1
CountRange = CountRange + 1
Next
End Sub
As I said..sometimes the getElementsByClassName("valuefortoday")(0).innerText cames empty, cause the site there no information for this day...how can I avoid empty values and skip it for the next day?
This write the clipboard to StdOut. As you can only use IE's clipboard in the intranet zone without security dialogs, I navigate to a local file and wait for it to finish loading.
You can also do it with events.
Sub Clip
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = 0
ie.Navigate2 FilterPath & "Filter.html"
Do
wscript.sleep 100
Loop until ie.document.readystate = "complete"
txt=ie.document.parentwindow.clipboardData.GetData("TEXT")
ie.quit
If IsNull(txt) = true then
outp.writeline "No text on clipboard"
else
outp.writeline txt
End If
End Sub
You can check the lenght property of valueForToday. If there is no such element then lenght will be equal to zero.
IE.navigate link
' wait until the page is fully loaded
Do Until IE.readyState = 4: DoEvents: Loop
IE.Visible = True
Dim valueForToday
Set valueForToday = doc.getElementsByClassName("valuefortoday")
If valueForToday.Length > 0 Then
' element exists, get its inner text
dd = valueForToday(0).innerText
Sheets("Resul").Range("C" & CountRange).Value = dd
Count = Count + 1
CountRange = CountRange + 1
End If

Resources