Help gratefully received on this one. I have some VBA running in Excel that inspects a series of webpages displaying betting odds for football matches and puts the odds into my spreadsheet. It has been working perfectly for months and has stopped working in the last few weeks. Here is a simplified version of the code I'm using:
Sub TestImport()
Dim http As New MSXML2.XMLHTTP60
Dim html As New MSHTML.HTMLDocument
Dim htmlEle1 As MSHTML.IHTMLElement
Dim columncounter As Integer
Dim rowccounter As Integer
Dim targetwebpage As String
Dim ColumnHeader As Variant
On Error GoTo ErrorStop
trowNum = 1
targetwebpage = "https://www.oddschecker.com/football/english/premier-league"
With http
.Open "get", targetwebpage, False
.send
End With
Set table_data = html.getElementsByTagName("tr")
If table_data.Length = 0 Then GoTo SkipLeague
For Each trow In table_data
For Each tcell In trow.Children
If tcell.innerText <> "TIP" Then 'Ignore this
tcellNum = tcellNum + 1
Cells(trowNum, tcellNum) = tcell.innerText
End If
Next tcell
Cells(trowNum, 1) = Worksheets("Leagues").Cells(j, 1)
trowNum = trowNum + 1
tcellNum = 1
Next trow
SkipLeague:
ErrorStop:
End Sub
No data gets returned because [table_data] is always null. It's always null because there are no tags in my variable, [html]. Instead, [html] seems to be simply this:
"<HEAD></HEAD>
<BODY>
<P> </P></BODY>"
Why would [html] return this value when the actual webpage (https://www.oddschecker.com/football/english/premier-league) is much more complex when displayed in my browser? And why has this problem only started in the last few weeks?
I'd be grateful for any help on this.
I did a quick test and had no issue. Some page, like Google require the User-Agent to be sent, but not the oddschecker page.
Sub TestURL()
Debug.Print GetResult("https://www.oddschecker.com/football/english/premier-league")
End Sub
Function GetResult(url As String) As String
Dim XMLHTTP As Object, ret As String
Set XMLHTTP = CreateObject("MSXML2.ServerXMLHTTP")
XMLHTTP.Open "GET", url, False
XMLHTTP.setRequestHeader "Content-Type", "text/xml"
XMLHTTP.setRequestHeader "Cache-Control", "no-cache"
XMLHTTP.setRequestHeader "Pragma", "no-cache"
XMLHTTP.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1; rv:25.0) Gecko/20100101 Firefox/25.0"
XMLHTTP.send
ret = XMLHTTP.responseText
GetResult = ret
End Function
Related
I am beginner at VBA. I have done the below code by referring to lot of articles found online.
I am trying to fetch API data from a website. It is taking the first fetch and I need the data to be fetched every 5 mins. But it is not refreshing at all. What can I do? Can anyone have a look at the code and advise?
I am using the below code to get the JSON data and later I am extracting using a JSON parser.
Sub FetchOptionChain()
Dim Json As Object
Dim webURL, webURL2 As String, mainString, subString
Dim i As Integer
Dim j As Integer
Dim k As Integer
Dim l As Integer
Dim dtArr() As String
Dim request, request2 As Object
Dim HTML_Content As Object
Dim requestString As String
webURL2 = "https://www.nseindia.com/"
webURL = "https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY"
subString = "Resource not found"
Set HTML_Content = CreateObject("htmlfile")
'Get the WebPage Content to HTMLFile Object
With CreateObject("msxml2.xmlhttp")
.Open "GET", webURL2, False
.send
End With
FetchAgain:
With CreateObject("msxml2.xmlhttp")
.Open "GET", webURL, False
'Found online that I have to add the below to remove the cached results. Adding this is hanging the excel and it never comes out of it. Excel is hanging here
.setRequestHeader "Content-Type", "application/json"
.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT"
.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36"
.send
mainString = .ResponseText
If InStr(mainString, subString) <> 0 Then
' Data has not been fetched properly. Will wait two seconds and try again.
Application.Wait (Now + TimeValue("0:00:2"))
GoTo FetchAgain
Added end with, end if and end sub. And fixed indenting to make code easier to read.
Sub FetchOptionChain()
Dim Json As Object
Dim webURL, webURL2 As String, mainString, subString
Dim i As Integer
Dim j As Integer
Dim k As Integer
Dim l As Integer
Dim dtArr() As String
Dim request, request2 As Object
Dim HTML_Content As Object
Dim requestString As String
webURL2 = "https://www.nseindia.com/"
webURL = "https://www.nseindia.com/api/option-chain-indices?symbol=BANKNIFTY"
subString = "Resource not found"
'''''''''''''''''''''''''''''''''''''''''''''
''' I don't understand this part though '''
'''''''''''''''''''''''''''''''''''''''''''''
Set HTML_Content = CreateObject("htmlfile")
'Get the WebPage Content to HTMLFile Object
With CreateObject("msxml2.xmlhttp")
.Open "GET", webURL2, False
.send
End With
'''''''''''''''''''''''''''''''''''''''''''''
'''''''''''''''' To here ''''''''''''''''''
'''''''''''''''''''''''''''''''''''''''''''''
FetchAgain:
With CreateObject("msxml2.xmlhttp")
.Open "GET", webURL, False
'Found online that I have to add the below to remove the cached results. Adding this is hanging the excel and it never comes out of it. Excel is hanging here
.setRequestHeader "Content-Type", "application/json"
.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT"
.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36"
.send
mainString = .responseText
End With
If InStr(mainString, subString) <> 0 Then
' Data has not been fetched properly. Will wait two seconds and try again.
Application.Wait (Now + TimeValue("00:00:02"))
GoTo FetchAgain
End If
End Sub
But it runs and works as expected for me.
The following snippet of code sends a XML request to the following site
Sub GetContents()
Dim XMLReq As New MSXML2.XMLHTTP60
XMLReq.Open "Get", "https://echa.europa.eu/brief-profile/-/briefprofile/100.028.723", False
XMLReq.send
End Sub
I have another Sub routine GetURL() which prints out the desired URL in this case: https://echa.europa.eu/brief-profile/-/briefprofile/100.028.723
How can I essentially concatenate the output of GetURL() into the BstrUrl? i.e.
XMLReq.Open "Get", "x", False where x is the output of GetURL()
Despite various attempts the syntax is not accepted as a URL.
Assuming you are combining from your earlier question then you need to ensure you write a function which returns the url (as Tim Williams has pointed out). I would expand upon this, in that I think you would need to consider adding a test to ensure both the request succeeded, there were results, and to pass the searchKeyWord as an argument to make your function more reusable. Along the same lines, you could pass the xmlhttp object into the function, so as to avoid continually creating and destroying them.
Avoid auto-instantiation, as you can get unexpected results, and Hungarian style notation. Personally, I also avoid those type characters, as they are harder to read.
vbNullString will offer faster assignment than = "".
I would also use a shorter, faster, and more reliable css pattern to retrieve the url, based on classes and a parent child relationship of two elements.
Public Sub GetContents()
Dim searchKeyWord As String, xmlReq As MSXML2.XMLHTTP60, html As MSHTML.HTMLDocument, url As String
searchKeyWord = "Acetone"
Set xmlReq = New MSXML2.XMLHTTP60
url = GetUrl(searchKeyWord, xmlReq)
Set html = New MSHTML.HTMLDocument
If url <> "N/A" Then
With xmlReq
.Open "GET", url, False
.send
If .Status = 200 Then
html.body.innerHTML = .responseText
Debug.Print html.querySelector("title").innerText
End If
End With
End If
End Sub
Public Function GetUrl(ByVal searchKeyWord As String, ByVal http As MSXML2.XMLHTTP60) As String
Const url = "https://echa.europa.eu/search-for-chemicals?p_auth=5ayUnMyz&p_p_id=disssimplesearch_WAR_disssearchportlet&p_p_lifecycle=1&p_p_state=normal&p_p_col_id=_118_INSTANCE_UFgbrDo05Elj__column-1&p_p_col_count=1&_disssimplesearch_WAR_disssearchportlet_javax.portlet.action=doSearchAction&_disssimplesearch_WAR_disssearchportlet_backURL=https%3A%2F%2Fecha.europa.eu%2Finformation-on-chemicals%3Fp_p_id%3Ddisssimplesearchhomepage_WAR_disssearchportlet%26p_p_lifecycle%3D0%26p_p_state%3Dnormal%26p_p_mode%3Dview%26p_p_col_id%3D_118_INSTANCE_UFgbrDo05Elj__column-1%26p_p_col_count%3D1%26_disssimplesearchhomepage_WAR_disssearchportlet_sessionCriteriaId%3D"
Dim html As MSHTML.HTMLDocument, dict As Object, i As Long, r As Long
Dim dictKey As Variant, payload$, ws As Worksheet
Set html = New MSHTML.HTMLDocument
Set dict = CreateObject("Scripting.Dictionary")
Set ws = ThisWorkbook.Worksheets("Sheet1")
dict("_disssimplesearchhomepage_WAR_disssearchportlet_formDate") = "1621017052777" 'timestamp
dict("_disssimplesearch_WAR_disssearchportlet_searchOccurred") = "true"
dict("_disssimplesearch_WAR_disssearchportlet_sskeywordKey") = searchKeyWord
dict("_disssimplesearchhomepage_WAR_disssearchportlet_disclaimer") = "true"
dict("_disssimplesearchhomepage_WAR_disssearchportlet_disclaimerCheckbox") = "on"
payload = vbNullString
For Each dictKey In dict
payload = IIf(Len(dictKey) = 0, WorksheetFunction.EncodeURL(dictKey) & "=" & WorksheetFunction.EncodeURL(dict(dictKey)), _
payload & "&" & WorksheetFunction.EncodeURL(dictKey) & "=" & WorksheetFunction.EncodeURL(dict(dictKey)))
Next dictKey
With http
.Open "POST", url, False
.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36"
.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
.send (payload)
If .Status = 200 Then
html.body.innerHTML = .responseText
Else
GetUrl = "N/A"
Exit Function
End If
End With
Dim result As Boolean
result = html.querySelectorAll(".lfr-search-container .substanceNameLink").Length > 0
GetUrl = IIf(result, html.querySelector(".lfr-search-container .substanceNameLink").href, "N/A")
End Function
If GetURL is a function returning a string then this should work:
Sub GetContents()
Dim XMLReq As New MSXML2.XMLHTTP60
Dim url
url = GetURL()
XMLReq.Open "Get", url, False
XMLReq.send
End Sub
I am trying to download the hi-res pictures from Amazon.co.uk. I tried the code given in the
[thread][1] and I am getting some issues.
the code by #QHarr works well for given Amazon.in website but when I try for Amazon.co.uk the
.querySelector("#landingImage").getAttribute("data-old-hires") returns nothing. here is the code I am testing.
Public Sub GetInfo()
Dim Html As HTMLDocument, results()
Set Html = New HTMLDocument
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://www.amazon.co.uk/dp/product/B01GFJWHZ0", True
.send
Html.body.innerHTML = .responseText
With ThisWorkbook.Worksheets("Sheet1")
.Cells(1, 2) = Html.querySelector("#productTitle").innerText
.Cells(1, 2).Offset(0, 1) = Html.querySelector("#landingImage").getAttribute("data-old-hires")
End With
End With
End Sub
any idea what exactly I am doing wrong?
My guess is the response html is mangled. You can easily regex out that bit from the response string however.
Note that the async argument is also set to False to allow time for data to load. I'm somewhat surprised your code got as far as you said it did. It should also have tripped over on the fact there is no match for that selector in the HTMLDocument.
Public Sub GetInfo()
'tools > references > Microsoft HTML Object Library
Dim re As Object, html As MSHTML.HTMLDocument, xhr As Object, s As String
Set re = CreateObject("VBScript.RegExp")
Set xhr = CreateObject("MSXML2.XMLHTTP")
Set html = New MSHTML.HTMLDocument
re.Pattern = """hiRes"":""(.*?)"""
With xhr
.Open "GET", "https://www.amazon.co.uk/dp/product/B01GFJWHZ0", False
.setRequestHeader "User-Agent", "Mozilla/5.0"
.send
s = .responseText
html.body.innerHTML = s
End With
With ThisWorkbook.Worksheets("Sheet1")
.Cells(1, 2) = html.querySelector("#productTitle").innerText
.Cells(1, 3) = re.Execute(s)(0).SubMatches(0)
End With
End Sub
Regex:
I'm trying to use the google translate API via VBA (works in python so far), and I found that the only difference between the python request and the VBA one is on the header "accept-encoding", the python one uses the "application/gzip" and works, but the VBA one is automatically changed to "gzip,deflate" even if I change it through code. Here is the code:
Function Test_GoogleTranslate()
Dim strTranslate As String
Dim objRequest As Object
Dim strUrl As String
Dim blnAsync As Boolean
Dim strResponse As String
Dim strWeather As String
Dim payload As String
Set objRequest = CreateObject("MSXML2.XMLHTTP")
payload = "target=es&q=something&source=en"
strTranslate = "https://google-translate1.p.rapidapi.com/language/translate/v2"
strTranslate = strTranslate & "?" & payload
With objRequest
.Open "POST", strTranslate, True
.setRequestHeader "host", "google-translate1.p.rapidapi.com"
.setRequestHeader "x-forwarded-port", "443"
.setRequestHeader "x-forwarded-proto", "https"
.setRequestHeader "connection", "keep-alive"
.setRequestHeader "content-type", "application/x-www-form-urlencoded"
.setRequestHeader "accept-encoding", "application/gzip"
.setRequestHeader "x-rapidapi-host", "google-translate1.p.rapidapi.com"
.setRequestHeader "x-rapidapi-key", "856e8ba78dmsh443766612c5a923p14f661jsn72323e803261"
.Send
While objRequest.readyState <> 4
DoEvents
Wend
strResponse = .ResponseText
End With
MsgBox (strResponse)
End Function
When I changed the accept-encoding to "gzip, deflate" in python it crashed, so I figured that might be the problem.
Any help is greatly appreciated
I had some success with Google Translate and Excel VBA using the MSXML2.ServerXMLHTTP object. I note you are using MSXML2.XMLHTTP. The solution appears to work well only setting a User-Agent request header so I did not delve into accept-encoding etc.
The differences between MSXML2.ServerXMLHTTP and MSXML2.XMLHTTP are touched upon in this question which might be useful for you.
Working code using MSXML2.ServerXMLHTTP:
Option Explicit
Sub Test()
Debug.Print Translate("Hello", "en", "fr", True) ' french
Debug.Print Translate("Hello", "en", "de", True) ' german
Debug.Print Translate("Hello", "en", "pt", True) ' portuguese
Debug.Print Translate("Hello", "en", "ru", False) ' russian - use romanised alphabet
Debug.Print Translate("Hello", "en", "ru", True) ' russian - use cyrillic
' ThisWorkbook.Sheets(1).Range("A1").Value = Translate("Hello", "en", "ru", True)
Debug.Print Translate("Hello", "en", "zh-CN", False) ' chinese simplified - use romanised alphabet
Debug.Print Translate("Hello", "en", "zh-CN", True) ' chinese simplified - use chinese script
' ThisWorkbook.Sheets(1).Range("B1").Value = Translate("Hello", "en", "zh-CN", True)
End Sub
Public Function Translate(strInput As String, strFromLanguageCode As String, strToLanguageCode As String, blnTargetAlphabet As Boolean) As String
Dim strURL As String
Dim objHTTP As Object
Dim objHTML As Object
Dim objDivs As Object, objDiv
Dim strTranslatedT0 As String
Dim strTranslatedO1 As String
' send query to web page
strURL = "https://translate.google.com/m?hl=" & strFromLanguageCode & _
"&sl=" & strFromLanguageCode & _
"&tl=" & strToLanguageCode & _
"&ie=UTF-8&prev=_m&q=" & strInput
Set objHTTP = CreateObject("MSXML2.ServerXMLHTTP")
objHTTP.Open "GET", strURL, False
objHTTP.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
objHTTP.send ""
' create a html document
Set objHTML = CreateObject("htmlfile")
With objHTML
.Open
.Write objHTTP.responseText
.Close
End With
' o1 has Anglicised translation, t0 as tranlsation in target language
Set objDivs = objHTML.getElementsByTagName("div")
For Each objDiv In objDivs
If objDiv.className = "o1" Then
strTranslatedO1 = objDiv.innerText
End If
If objDiv.className = "t0" Then
strTranslatedT0 = objDiv.innerText
End If
Next objDiv
' choose which to return
If blnTargetAlphabet Then
Translate = strTranslatedT0
Else
Translate = strTranslatedO1
End If
CleanUp:
Set objHTML = Nothing
Set objHTTP = Nothing
End Function
Result:
Bonjour
Hallo
Olá
Privet
??????
Ni hao
??
The VBA immediate window doesn't print Cyrillic or Chinese characters but you can see this feature working by outputing to a cell:
December 2020 update
Looks like this method will no longer work going back maybe to mid November.
Looking at the response
the div class names have changed to something more obscure
there's some esoteric c-wiz elements doing something wonderful...
also, I suspect that some client side script is calling for the actual translation after the document is retrieved
Options: Selenium, Microsoft Translate, free and paid tiers for Google translation APIs ;)
I am attempting to update a SharePoint item via REST API, though I am receiving the below error:
{"odata.error":{"code":"-1, System.InvalidOperationException","message":{"lang":"en-US","value":"The type of data at position 0 is different than the one expected."}}}
I'm not quite sure what is causing this- I have the request type set to JSON, but it does not seem to accept my input. Any help is appreciated :-)
My code:
Sub Work_Damn_You()
Dim oXMLHTTP As Object
Dim sListNameOrGuid As String
Dim sBatchXml As String
Dim sSoapBody As String
Dim sWTF as string
Set oXMLHTTP = CreateObject("MSXML2.XMLHTTP.6.0")
sListNameOrGuid = sListName
With oXMLHTTP
.Open "POST", "http://SPSITE.COM/_api/web/lists/GetByTitle('PAGE')/items(22)", True
.setRequestHeader "X-RequestDigest", testerino
.setRequestHeader "Accept", "application/json;odata=nometadata"
.setRequestHeader "Content-Type", "application/json;odata=verbose"
.setRequestHeader "__metadata", "(""type"":""SP.Data.QATrackerListItem"""
sWTF = """preTestComment""=""Hello"""
.send (sWTF)
Debug.Print (.responseText)
' Check response
If .Status = 200 Then
Debug.Print .Status & " [Happy Days!]"
Else
Debug.Print .Status & " [Sad Days :-(]"
End If
End With
Set oXMLHTTP = Nothing
End Sub
'''