VBA code - connect to webpage and retrieve value - excel

I have the following
Column A == FEdEX AWB #s
Column B == Delivery date (Empty)
I would like to write a function where it reads the tracking number on Column A and extracts the delivery date from the website - all AWB # are delivered - 100% sure
The code I have writes all the info found in the website into the sheet - not sure how to extract only the delivered date.
Sub Macro1()
With ActiveSheet.QueryTables.Add(Connection:= _
"URL;https://www.bing.com/packagetrackingv2?
packNum=727517426234&carrier=Fedex&FORM=PCKTR1" _
, Destination:=Range("$A$1"))
.Name = _
"https://www.bing.com/packagetrackingv2?
packNum=727517426234&carrier=Fedex&FORM=PCKTR1"
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = True
.WebConsecutiveDelimitersAsOne = True
.WebSingleBlockTextImport = False
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.Refresh BackgroundQuery:=False
End With
End Sub

A function, getting passing the airway bill number and returning the date would be quite enough:
Function GetDateFromAwb(awbNumber As String) As String
Dim objIE As New InternetExplorer 'Microsoft Internet Controls library added
objIE.Visible = False 'Or put True, if you want to see the IE
objIE.navigate "https://www.fedex.com/apps/fedextrack/?tracknumbers=" & awbNumber
Do While objIE.Busy = True Or objIE.readyState <> 4: DoEvents: Loop
Application.Wait (Now + TimeValue("0:00:05"))
GetDateFromAwb = objIE.Document.getElementsByClassName("redesignSnapshotTVC snapshotController_date dest").Item.InnerText
objIE.Quit
End Function
The idea of the function is to append the airbill string number to the URL and to open the corresponding site. Then, using the class "redesignSnapshotTVC snapshotController_date dest", the corresponding date is taken.
This is a possible way to call the function, displaying the date in a MsgBox:
Sub Main()
Dim awbNumber As String
awbNumber = 727517426234#
Dim awbDate As String
awbDate = GetDateFromAwb(awbNumber)
MsgBox awbDate
End Sub
Make sure that the library "Microsoft Internet Controls" is added from the VBE menu>Extras>References:

Rather than using a browser you could use xmlhttp request which is quicker.
The page does a form XHR POST request which returns json you can parse (lots of info returned including a delivery date field). You can use this as a function in the sheet. I also show a test call. The id (tracking number) is passed as an argument to the function GetDeliveryDate.
Here is the request made when you submit your tracking number on the site:
As you can see from the above, and further detailed in code, the tracking number is part of the body sent in the request (data param); it is also part of one of the request headers.
I use jsonconverter.bas to parse the json response. After adding the code from there to your project you need go VBE > Tools > References and add a reference to Microsoft Scripting Runtime.
View the json response here
As you say all requests will return a delivery date, if you don't want to load this external library you could use split to isolate the date.
Relevant json:
You can see relevant part of json here:
I use the field actDeliveryDt for version of code using split as I can separate an unambiguous date yyyy-mm-dd from the datetime string. I use displayActDeliveryDt for json parsing though you could use either (removing time part with split if usnig the former as shown in examples below)
Caveat: I have had only one delivery id to use for testing.
TODO:
You could add in a test for whether a valid request was made as the json response includes a field for this.
If performing this for multiple requests I would recommend, for efficiency, to re-write using a sub which loops an array of the tracking numbers, stores results in an array and writes that array out in go at end.
VBA:
JSON parsing:
Option Explicit 'example test call from VBE
Public Sub test()
Debug.Print GetDeliveryDate(727517426234#)
End Sub
Public Function GetDeliveryDate(ByVal id As Double) As Date
Dim json As Object, body As String '< VBE > Tools > References > Microsoft Scripting Runtime
body = "data={""TrackPackagesRequest"":{""appType"":""WTRK"",""appDeviceType"":""DESKTOP"",""supportHTML"":true,""supportCurrentLocation"":true,""uniqueKey"":"""",""processingParameters"":{},""trackingInfoList"":[{""trackNumberInfo"":{""trackingNumber"":" & Chr$(34) & CStr(id) & Chr$(34) & ",""trackingQualifier"":"""",""trackingCarrier"":""""}}]}}"
body = body & "&action=trackpackages&locale=en_US&version=1&format=json"
With CreateObject("MSXML2.XMLHTTP")
.Open "POST", "https://www.fedex.com/trackingCal/track", False
.setRequestHeader "Referer", "https://www.fedex.com/apps/fedextrack/?tracknumbers=" & CStr(id)
.setRequestHeader "User-Agent", "Mozilla/5.0"
.setRequestHeader "X-Requested-With", "XMLHttpRequest"
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded; charset=UTF-8"
.send body
Set json = JsonConverter.ParseJson(.responseText)
End With
GetDeliveryDate = json("TrackPackagesResponse")("packageList")(1)("displayActDeliveryDt")
End Function
Using split:
Option Explicit
Public Sub test()
Debug.Print GetDeliveryDate(727517426234#)
End Sub
Public Function GetDeliveryDate(ByVal id As Double) As Date
Dim s As String, body As String
body = "data={""TrackPackagesRequest"":{""appType"":""WTRK"",""appDeviceType"":""DESKTOP"",""supportHTML"":true,""supportCurrentLocation"":true,""uniqueKey"":"""",""processingParameters"":{},""trackingInfoList"":[{""trackNumberInfo"":{""trackingNumber"":" & Chr$(34) & CStr(id) & Chr$(34) & ",""trackingQualifier"":"""",""trackingCarrier"":""""}}]}}"
body = body & "&action=trackpackages&locale=en_US&version=1&format=json"
With CreateObject("MSXML2.XMLHTTP")
.Open "POST", "https://www.fedex.com/trackingCal/track", False
.setRequestHeader "Referer", "https://www.fedex.com/apps/fedextrack/?tracknumbers=" & CStr(id)
.setRequestHeader "User-Agent", "Mozilla/5.0"
.setRequestHeader "X-Requested-With", "XMLHttpRequest"
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded; charset=UTF-8"
.send body
s = .responseText
End With
GetDeliveryDate = Split(Replace$(Split(Split(s, """actDeliveryDt"":""")(1), Chr$(34))(0), "\u002d", "-"), "T")(0)
End Function
Example usage in sheet:
Note: I have UK format dd/mm/yyyy in sheet

Related

VBA error 438 object doesn't support this property or method while trying to Web Scraping from a site

I'm with a problem while trying to get elements from a website with VBA, I've searched this problem in StackOverflow but anyone of the answers that I tried have solved my problem.
I want to get the text that are in the element Strong, but this element are into an Div.
Sub StatusInvest()
Dim html As HTMLDocument
Set html = New HTMLDocument
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://statusinvest.com.br/acoes/eua/aapl", False
.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT" 'to deal with potential caching
.send
html.body.innerHTML = .responseText
End With
Application.ScreenUpdating = False
Set v = html.getElementsByClass("info special w-100 w-md-33 w-lg-20")(0).Value = "00000"
Set test = valor.getElementsByTagName("Strong").innerText
Range("B1").Value = teste
Application.ScreenUpdating = True
End Sub
This is the element that I'm trying to get:
Sample of the HTML
You got the method name wrong - It should be getElementsByClassName.
getElementsByTagName and getElementsByClassName both returns a collection of element, whether there is one or more so you have to refer to index 0 when you are trying to get strong element.
innertext property returns a string so you shouldn't use Set statement but simply assign test.
Sub StatusInvest()
Dim html As HTMLDocument
Dim valor As Object
Dim test As String
Set html = New HTMLDocument
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://statusinvest.com.br/acoes/eua/aapl", False
.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT" 'to deal with potential caching
.send
html.body.innerHTML = .responseText
End With
Application.ScreenUpdating = False
Set valor = html.getElementsByClassName("special")(0)
test = valor.getElementsByTagName("Strong")(0).innerText
Range("B1").Value = test
Application.ScreenUpdating = True
End Sub
In addition to what was mentioned in Raymond's answer, it would be faster, and more robust, to use a single class, via css class selector, to target the desired element. The target element itself has a single class you can use. By using querySelector you stop matching after first match retrieved, rather than returning an entire collection and then indexing.
Use the commented out line if you want to include the currency symbol in front of the value.
Option Explicit
Public Sub StatusInvest()
Dim html As MSHTML.HTMLDocument
Set html = New MSHTML.HTMLDocument
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://statusinvest.com.br/acoes/eua/aapl", False
.SetRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT" 'to deal with potential caching
.send
html.body.innerHTML = .responseText
End With
ActiveSheet.Range("B1").Value = html.querySelector(".value").innerText
'ActiveSheet.Range("B1").Value = html.querySelector(".icon").innerText & Chr$(32) & html.querySelector(".value").innerText
End Sub

Unable to create SharePoint item & update external columns content (business connectivity services)?

I am having trouble creating a new item within a SharePoint 2013 list. I'm able to actually create the item, though the external data (business connectivity services [bcs]) does not autopopulate as it would during the creation of the item via the SharePoint UI. Is there any way to actually make this work, or to send the function that needs to run via POST? I've pasted my current item creation sub below.
Dim oXMLHTTP As Object
Dim sListNameOrGuid As String
Dim sBatchXml As String
sWTF As String
Set oXMLHTTP = CreateObject("MSXML2.XMLHTTP.6.0")
sCurrentUser = fGetUserID
With oXMLHTTP
.Open "POST", "MYSPSITE.COM/_api/web/lists/GetByTitle('THELIST')/items", True
.setRequestHeader "Accept", "application/json;odata=verbose"
.setRequestHeader "Content-Type", "application/json;odata=verbose"
' .setRequestHeader "X-HTTP-Method", "PATCH"
' .setRequestHeader "If-Match", "*"
.setRequestHeader "X-RequestDigest", fGetLoginToken
sWTF = "{ 'User': " & Int(sCurrentUser) & ", 'newComment': 'TESTING TESTING'}"
.send (sWTF)
Do While .readyState <> 4:
DoEvents
Loop
Debug.Print (.responseText)
.abort
End With
Set oXMLHTTP = Nothing
End Sub
Lower the timer job frequency.

Attempting to set HTML element object by using getElementsByID returns nothing; ID exists in HTML page [duplicate]

Thanks to the help and code from #QHarr I have got the tracking info from Fedex, DHL and Startrack working. I have been trying to use his code and the UPS tracking Web Service Developer Guide and Tracking JSON Developer Guides to get UPS to work as well within Excel. The JSON converter code is from here https://github.com/VBA-tools/VBA-JSON/blob/master/JsonConverter.bas
The code I have tried is as follows
Public Function GetUPSDeliveryDate(ByVal id As String) As String
Dim body As String, json As Object
body = "data={""TrackPackagesRequest"":{""appType"":""WTRK"",""appDeviceType"":""DESKTOP"",""supportHTML"":true,""supportCurrentLocation"":true,""uniqueKey"":"""",""processingParameters"":{},""trackingInfoList"":[{""trackNumberInfo"":{""trackingNumber"":" & Chr$(34) & id & Chr$(34) & ",""trackingQualifier"":"""",""trackingCarrier"":""""}}]}}"
body = body & "&action=trackpackages&locale=en_AU&version=1&format=json"
With CreateObject("MSXML2.XMLHTTP")
.Open "POST", "http://wwwapps.ups.com/WebTracking", False
.setRequestHeader "Referer", "https://www.ups.com/track?loc=en_AU&tracknum=" & id
.setRequestHeader "User-Agent", "Mozilla/5.0"
.setRequestHeader "X-Requested-With", "XMLHttpRequest"
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded; charset=UTF-8"
.send body
Set json = JSONConverter.ParseJson(.responseText)
End With
GetUPSDeliveryDate = Format$(json("ResponseStatus")("ShipmentType")(1)("DeliveryDate"), "dddd, mmm dd, yyyy")
End Function
I am not getting any errors in the code per-say, but when I use the =GetUPSDeliveryDate() function I am getting a #VALUE! response instead of the delivered date of 7th May 2019, so I am guessing I have got the following bit wrong
GetUPSDeliveryDate = Format$(json("ResponseStatus")("ShipmentType")(1)("DeliveryDate"), "dddd, mmm dd, yyyy")
I have also tried the following, but no luck.
If json("results")(1)("delivery")("status") = "delivered" Then
GetUPSDeliveryDate = json("results")(1)("checkpoints")(1)("date")
Else
GetUPSDeliveryDate = vbNullString
End If
A sample UPS tracking number is 1Z740YX80140148107
Any help would be greatly appreciated.
Thanks
The following is by mimicking of this UPS tracking site. The json parser used is jsonconverter.bas: Download raw code from here and add to standard module called jsonConverter . You then need to go VBE > Tools > References > Add reference to Microsoft Scripting Runtime.
Option Explicit
Public Sub test()
Debug.Print GetUPSDeliveryDate("1Z740YX80140148107")
End Sub
Public Function GetUPSDeliveryDate(ByVal id As String) As String
Dim body As String, json As Object
body = "{""Locale"":""en_US"",""TrackingNumber"":[""" & id & """]}"
With CreateObject("MSXML2.XMLHTTP")
.Open "POST", "https://www.ups.com/track/api/Track/GetStatus?loc=en_US", False
.setRequestHeader "Referer", "https://www.ups.com/track?loc=en_US&requester=ST/"
.setRequestHeader "User-Agent", "Mozilla/5.0"
.setRequestHeader "DNT", "1"
.setRequestHeader "Content-Type", "application/json"
.setRequestHeader "Accept", "application/json, text/plain, */*"
.send body
Set json = JsonConverter.ParseJson(.responseText)
End With
If json("trackDetails")(1)("packageStatus") = "Delivered" Then
GetUPSDeliveryDate = json("trackDetails")(1)("deliveredDate")
Else
GetUPSDeliveryDate = "Not yet delivered"
End If
End Function
The Tracking Web Service Developer Guide.pdf contains all you need to know to set up using the official tracking API.

Scrape data that is not in the source code, using VBA

I'm trying to scrape whole div from one website. The data is not visible in the source code, it changes based on the variable in the URL (link).
I was looking for any solution to copy to the excel sheet everything from
<div id="div_measures_for_2103909010" class="measures_detail">
Unfortunately since there is no data in direct source code I have found a way to display only data from the div provided above Link
However to get this data I would need at first get the link to the direct data (the link is in the source code).
Do you have any idea how to deal with it the best possible way?
I've tried to download the source code, search for the link, open the link and copy all the data, but I have troubles downloading the source code (excel downloads only part of it due to cell data limitations). Here is my current code:
Sub Open_Webpage()
Set objHTTP = CreateObject("MSXML2.ServerXMLHTTP")
URL = "https://ec.europa.eu/taxation_customs/dds2/taric/measures.jsp?Lang=en&SimDate=20190329&Area=&MeasType=&StartPub=&EndPub=&MeasText=&GoodsText=&op=&Taric=2103909010&search_text=goods&textSearch=&LangDescr=pl&OrderNum=&Regulation=&measStartDat=&measEndDat="
objHTTP.Open "GET", URL, False
objHTTP.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
objHTTP.send ("")
html = objHTTP.responseText
Range("A1").Value = html
End Sub
If I am able to have full code in one cell I can then look for the link in the source code and use it:
=MID(LEFT(A1,FIND("' width='100%'",A1)-1),FIND("' src='",A1)+7,LEN(A1))
I know that there must be some better solution, but I'm not so proficient in VBA to figure it out...
You can regex out the required url, do a little cleaning then pass on to xhr. For some reason I was unable to simply use getAttribute("onclick") so had to use outerHTML (innerHTML also fine) on the element
Option Explicit
Public Sub GetInfo()
Dim html As HTMLDocument, s As String, re As Object, url As String
Set re = CreateObject("vbscript.regexp")
Set html = New HTMLDocument '< VBE > Tools > References > Microsoft Scripting Runtime
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://ec.europa.eu/taxation_customs/dds2/taric/measures.jsp?Lang=en&SimDate=20190329&Area=&MeasType=&StartPub=&EndPub=&MeasText=&GoodsText=&op=&Taric=2103909010&search_text=goods&textSearch=&LangDescr=pl&OrderNum=&Regulation=&measStartDat=&measEndDat=", False
.send
html.body.innerHTML = .responseText
s = html.querySelector("[id$='_end_goods']").outerHTML
With re
.Global = True
.MultiLine = True
.IgnoreCase = True
.Pattern = "measures_details\.jsp(.*)'\);"
If .Test(s) Then
url = "https://ec.europa.eu/taxation_customs/dds2/taric/measures_details.jsp" & .Execute(s)(0).SubMatches(0)
url = Replace$(url, "&", "&")
End If
End With
If Len(url) > 0 Then
.Open "GET", url, False
.send
html.body.innerHTML = .responseText
ActiveSheet.Cells(1, 1) = html.querySelector(".measures_detail").innerText
End If
End With
End Sub
Try the regex here
References:
VBE > Tools > References > Microsoft HTML Object Library

Text retrieved on multiple rows on Excel VBA

I'm a kind of new to Excel VBA. Here's the problem:
Given a identifier, I want to retireve some text from a web page. Ideally I want to store the text of the page in a single cell. I created a function that creates a QueryTable but, sometimes, the retrieved text is copied on multiple rows.
Is there a way to place all the text on a single cell?
Here's the code of my function:
Function Articolo(myRange As Range, code As String)
Dim myURL As String
Dim myName As String
myURL = "URL;http://techstore.runner.it/feed/dettagli_csv.php?codcli=111367&pwd=03142110786&sku=" & code
myName = "dettagli_csv.php?codcli=111367&pwd=03142110786&sku=" & code
With ActiveSheet.QueryTables.Add(Connection:= _
myURL _
, Destination:=myRange)
.Name = myName
.FieldNames = True
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = False
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlOverwriteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = False
.RefreshPeriod = 0
.WebSelectionType = xlEntirePage
.WebFormatting = xlWebFormattingNone
.WebPreFormattedTextToColumns = False
.WebConsecutiveDelimitersAsOne = False
.WebSingleBlockTextImport = True
.WebDisableDateRecognition = False
.WebDisableRedirections = False
.WebConsecutiveDelimitersAsOne = True
.Refresh BackgroundQuery:=False
End With
End Function
As test you can use 8E4374 as code
Thanks for your help!
Querytables are often slow and cumbersome. If you use one of the httprequest objects instead it's much quicker and you have more control over how to parse the response. Below is basic example that doesn't manage sessions or check if the page is cached.
Option Explicit
Sub test()
Dim rng As Range
Dim code As String
Set rng = Sheet1.Range("A1")
code = "8E4374"
Articolo rng, code
End Sub
Sub Articolo(myRange As Range, code As String)
Dim myURL As String
Dim myName As String
myURL = "http://techstore.runner.it/feed/dettagli_csv.php?codcli=111367&pwd=03142110786&sku=" & code
myRange.Value = ExecuteWebRequest(myURL)
End Sub
Function ExecuteWebRequest(ByVal url As String) As String
Dim oXHTTP As Object
Set oXHTTP = CreateObject("MSXML2.XMLHTTP")
oXHTTP.Open "GET", url, False
oXHTTP.send
ExecuteWebRequest = oXHTTP.responseText
Set oXHTTP = Nothing
End Function
EDIT: the above code is designed to run as a Sub rather than a UDF. Since a UDF cannot affect other cells the only option is to return the string to the calling cell or call set up the code to run as either an event or from a control (eg a button)
Below is an example UDF, it's called from Excel using =Articolo(C1) where C1 is any cell containing the required code eg 8E4374
Option Explicit
Function Articolo(ByVal code As String) As String
Dim myURL As String
myURL = "http://techstore.runner.it/feed/dettagli_csv.php?codcli=111367&pwd=03142110786&sku=" & code
Articolo = ExecuteWebRequest(myURL)
End Function
Function ExecuteWebRequest(ByVal url As String) As String
Dim oXHTTP As Object
Set oXHTTP = CreateObject("MSXML2.XMLHTTP")
oXHTTP.Open "GET", url, False
oXHTTP.send
ExecuteWebRequest = oXHTTP.responseText
Set oXHTTP = Nothing
End Function
Also, as this is making http requests, everytime you force a full re-calc your UDF's will also re-calc which is probably not what you want as you could be making hundreds or thousands of requests. I would suggest running it once only from a defined loop such as
For Each code in Listofcodes : <Download Page> : Next code
It looks like the data coming from that URL has carriage return and line feeds embedded in it. That's why it's separating into different cells in excel.
One solution would be to run VBA code to do the query, strip out the carriage return/line feed characters and then put the results into a cell. The problem would be that you'd have to run the code to update, rather than excel taking care of the refreshes.
A simpler answer might be to add another cell with a formula like this:
=A1&" "&A2&" "&A3&" "&A4

Resources