I've been doing quite a bit of web scraping over the past year and at some point, for reasons I don't remember anymore, I decided to use the Microsoft WinHTTP Services version 5.1 library as my default solution when sending HTTP requests.
I've never had any problems with it and I have achieved anything I ever attempted to do as far as web scraping is concerned.
That is, until i tried the following:
Sub nse()
Dim req As New WinHttpRequest
Dim url As String, requestPayload As String
url = "https://www.niftyindices.com/Backpage.aspx/getHistoricaldatatabletoString"
requestPayload = "{'name':'NIFTY 50','startDate':'01-Feb-2020','endDate':'01-Feb-2020'}"
With req
.Open "POST", url, False
.setRequestHeader "Content-Type", "application/json; charset=UTF-8"
.send requestPayload
Debug.Print .responseText
End With
End Sub
The .send method fails with a
Run-time error -2147012894 (80072ee2) Automation error
Changing to Dim req As New MSXML2.XMLHTTP60 solves the issue completely.
What am I missing here? Could it be website specific somehow? Is there something in the inner workings of these 2 libraries I should know?
Any input would be appreciated.
Related
I have been using the following Excel VBA macro to bring back data from a website. It worked fine until a few days ago when the website stopped supporting IE. Of course the macro just fails now as there is no data on the webpage to bring back to Excel, just a message saying, "Your browser, Internet Explorer, is no longer supported." Is there a way to have the "Get method" (MSXML2.XMLHTTP) use Chrome instead of IE to interact with the website? BTW, my default browser is already set to "Chrome".
Dim html_doc As HTMLDocument ' note: reference to Microsoft HTML Object Library must be set
Sub KS()
' Define product url
KS_url = "https://www.kingsoopers.com/p/r-w-knudsen-just-blueberry-juice/0007468210784"
' Collect data
Set html_doc = New HTMLDocument
Set xml_obj = CreateObject("MSXML2.XMLHTTP")
xml_obj.Open "GET", KS_url, False
xml_obj.send
html_doc.body.innerHTML = xml_obj.responseText
Set xml_obj = Nothing
KS_product = html_doc.getElementsByClassName("ProductDetails-header")(0).innerText
KS_price = "$" & html_doc.getElementsByClassName("kds-Price kds-Price--alternate mb-8")(1).Value
do Stuff
End Sub
The check for this is a basic server check on user agent. Tell it what it wants to "hear" by passing a supported browser in the UA header...(or technically, in this case, just saying the equivalent of: "Hi, I am not Internet Explorer".)
It can be as simple as xml.setRequestHeader "User-Agent", "Chrome". I said basic because you could even pass xml.setRequestHeader "User-Agent", "I am a unicorn", so it is likely an exclusion based list on the server for Internet Explorer.
Option Explicit
Public Sub KS()
Dim url As String
url = "https://www.kingsoopers.com/p/r-w-knudsen-just-blueberry-juice/0007468210784"
Dim html As MSHTML.HTMLDocument, xml As Object
Set html = New MSHTML.HTMLDocument
Set xml = CreateObject("MSXML2.XMLHTTP")
xml.Open "GET", url, False
xml.setRequestHeader "User-Agent", "Mozilla/5.0"
xml.send
html.body.innerHTML = xml.responseText
Debug.Print html.getElementsByClassName("ProductDetails-header")(0).innerText
Debug.Print "$" & html.getElementsByClassName("kds-Price kds-Price--alternate mb-8")(1).Value
Stop
End Sub
Compare that with adding no UA or adding xml.setRequestHeader "User-Agent", "MSIE".
Study the article here by Daniel Pineault and this paragraph:
Feature Browser Emulation
Also note my comment dated 2020-09-13.
I have a code in which I declared a variable like that
Dim http As New XMLHTTP60
then in the code, I used this line
http.Open "POST", urlexample, False
http.Send strArg
How can I use the waitForResponse to wait for the server to respond to the request. I tried looping like that after the .Send statement
While http.ReadyState <> 4: DoEvents: Wend
But this works sometimes well and sometimes doesn't return anything.
Looking through object model seems that this method is not available through MSXML2.XMLHTTP60 but is with WinHttp.WinHttpRequest. The latter does not have a ReadyState property however.
You'll need to investigate the default timeout and see if you need SetTimeOuts on WinHttp object to obtain longer times.
Option Explicit
Public Sub testing()
Dim http As winHttp.WinHttpRequest
Set http = New winHttp.WinHttpRequest
With http
'.SetTimeouts ........
.Open "GET", "http://books.toscrape.com/", True
.setRequestHeader "User-Agent", "Mozilla/5.0"
.send
.waitForResponse 1000
End With
Stop
End Sub
N.B. Your request will not be async if the async arg is set to FALSE in the .Open
Interesting reading: https://github.com/VBA-tools/VBA-Web/issues/337
Seems Tim Hall's VBA-Web would be a nice pre-packaged way to go about this.
WinHTTP
I am trying to connect excel (through vba) to the "teamleader" api v1, via post calls.
api docs here
I tried some solutions which I have found, regarding post calls in vba, but seem to be getting nowhere.
This is what I have so far:
Sub test2()
Dim url As String
Dim objhttp As Object
Set objhttp = CreateObject("MSXML2.ServerXMLHTTP")
url = "https://app.teamleader.eu/api/helloWorld.php"
With objhttp
.Open "POST", url, False
.setRequestHeader "api_group", "xxx"
.setRequestHeader "api_secret", "xxx"
.send
Debug.Print .responseText
End With
End Sub
This however gives me this response:
{"status":"failed","reason":"Please set api_group."}
Does anyone have any idea how to get this to work?
Turns out I needed the put all the authentication in the url itself, this question can be closed.
I use XMLHTTP to connect to a API of a webserver and retrieve some data.
I connect to a http url so I can access from every position, but I want to do that when I am in office, if there is no internet connection, the macro is continuing to working connecting to a second url (internal IP).
I was thinking something like that:
Dim basic_url, basic_url_web, basic_url_local As String
basic_url_web = "www.notexistingurl.com"
basic_url_local = "192.168.178.3"
Dim myurl As String
On Error Resume Next
myurl = basic_url_web + "/api/product/read_product.php?id=4"
xmlhttp.Open "GET", myurl, False
xmlhttp.Send
If Err.Number <> 0 then
myurl = basic_url_local + "/api/product/read_product.php?id=4"
xmlhttp.Open "GET", myurl, False
xmlhttp.Send
END IF
On Error GoTo 0 'error checkin restored
But it is not working, actually when I call the xmlhttp.Send function I ever have a -2147467259 error number, and this is the same error number I have if actually there is no error.
If I don't use the "On Error Resume Next", then I have automation error on the .send() function (on an inexistent link), and therefore I cannot work with it.
I'm becoming crazy, I took about only 2-3 hours to do all the connection with the API and everything was working perfectly, but now I'm working since days to find a solution to have a second URL if the first cannot be reached.
Thank you for your suggestions
I have a HTML web page at work that I want to query data from tables into excel 2007. This web page requires I sign on with a password. I sign in with my normal IE7 browser, then I go to DATA -> connections -> my connections and edit the query. This reads the IE7 cookie cache and I re-POST the data to connect to the server's security by clicking "retry" when it says "the web query returned no data". After I do this, the data imports fine.
I can do this just fine and it only needs to be done once a day. Other users of my application find this difficult which leads to my question:
Is there a way to automatically POST this data back with VB? I'm thinking maybe I should use the cookie property of the IE.Document.cookie?
I'm calling the following login script, before I continue with the web query (set reference to XML library). Look around to find some instructions how you can find your POST parameters.
Sub XMLHttpLogin()
Dim i As Integer
Dim sExpr As String
Dim sPar As String, sURL as String
Dim sResp As String
Dim XMLHttp As MSXML2.XMLHTTP60
Set XMLHttp = New MSXML2.XMLHTTP60
sPar = "name=user1&pass=pass1&form_id=form1" 'The parameters to send.
sURL = "http://www.stackoverflow.com"
With XMLHttp
.Open "POST", sURL, True 'Needs asynchronous connection
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
.send (sPar)
i = 0 'wait until data has been downloaded
Do While i = 0
If .readyState = 4 Then
If .Status = 200 Then Exit Do
End If
DoEvents
Loop
sResp = .responseText 'contains source code of website
sExpr = "not-logged-in" 'look for this string in source code
If InStr(1, sResp, sExpr, vbTextCompare) Then
MsgBox "Not logged in. Error in XMLHttpLogin"
End If
End With
End Sub