I have a HTML web page at work that I want to query data from tables into excel 2007. This web page requires I sign on with a password. I sign in with my normal IE7 browser, then I go to DATA -> connections -> my connections and edit the query. This reads the IE7 cookie cache and I re-POST the data to connect to the server's security by clicking "retry" when it says "the web query returned no data". After I do this, the data imports fine.
I can do this just fine and it only needs to be done once a day. Other users of my application find this difficult which leads to my question:
Is there a way to automatically POST this data back with VB? I'm thinking maybe I should use the cookie property of the IE.Document.cookie?
I'm calling the following login script, before I continue with the web query (set reference to XML library). Look around to find some instructions how you can find your POST parameters.
Sub XMLHttpLogin()
Dim i As Integer
Dim sExpr As String
Dim sPar As String, sURL as String
Dim sResp As String
Dim XMLHttp As MSXML2.XMLHTTP60
Set XMLHttp = New MSXML2.XMLHTTP60
sPar = "name=user1&pass=pass1&form_id=form1" 'The parameters to send.
sURL = "http://www.stackoverflow.com"
With XMLHttp
.Open "POST", sURL, True 'Needs asynchronous connection
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
.send (sPar)
i = 0 'wait until data has been downloaded
Do While i = 0
If .readyState = 4 Then
If .Status = 200 Then Exit Do
End If
DoEvents
Loop
sResp = .responseText 'contains source code of website
sExpr = "not-logged-in" 'look for this string in source code
If InStr(1, sResp, sExpr, vbTextCompare) Then
MsgBox "Not logged in. Error in XMLHttpLogin"
End If
End With
End Sub
Related
I have been using the following Excel VBA macro to bring back data from a website. It worked fine until a few days ago when the website stopped supporting IE. Of course the macro just fails now as there is no data on the webpage to bring back to Excel, just a message saying, "Your browser, Internet Explorer, is no longer supported." Is there a way to have the "Get method" (MSXML2.XMLHTTP) use Chrome instead of IE to interact with the website? BTW, my default browser is already set to "Chrome".
Dim html_doc As HTMLDocument ' note: reference to Microsoft HTML Object Library must be set
Sub KS()
' Define product url
KS_url = "https://www.kingsoopers.com/p/r-w-knudsen-just-blueberry-juice/0007468210784"
' Collect data
Set html_doc = New HTMLDocument
Set xml_obj = CreateObject("MSXML2.XMLHTTP")
xml_obj.Open "GET", KS_url, False
xml_obj.send
html_doc.body.innerHTML = xml_obj.responseText
Set xml_obj = Nothing
KS_product = html_doc.getElementsByClassName("ProductDetails-header")(0).innerText
KS_price = "$" & html_doc.getElementsByClassName("kds-Price kds-Price--alternate mb-8")(1).Value
do Stuff
End Sub
The check for this is a basic server check on user agent. Tell it what it wants to "hear" by passing a supported browser in the UA header...(or technically, in this case, just saying the equivalent of: "Hi, I am not Internet Explorer".)
It can be as simple as xml.setRequestHeader "User-Agent", "Chrome". I said basic because you could even pass xml.setRequestHeader "User-Agent", "I am a unicorn", so it is likely an exclusion based list on the server for Internet Explorer.
Option Explicit
Public Sub KS()
Dim url As String
url = "https://www.kingsoopers.com/p/r-w-knudsen-just-blueberry-juice/0007468210784"
Dim html As MSHTML.HTMLDocument, xml As Object
Set html = New MSHTML.HTMLDocument
Set xml = CreateObject("MSXML2.XMLHTTP")
xml.Open "GET", url, False
xml.setRequestHeader "User-Agent", "Mozilla/5.0"
xml.send
html.body.innerHTML = xml.responseText
Debug.Print html.getElementsByClassName("ProductDetails-header")(0).innerText
Debug.Print "$" & html.getElementsByClassName("kds-Price kds-Price--alternate mb-8")(1).Value
Stop
End Sub
Compare that with adding no UA or adding xml.setRequestHeader "User-Agent", "MSIE".
Study the article here by Daniel Pineault and this paragraph:
Feature Browser Emulation
Also note my comment dated 2020-09-13.
Any one of three highlighted part are the value i want to print. I am trying below code
Sub JJ()
Dim IE As New SHDocVw.InternetExplorer
Dim hdoc As MSHTML.HTMLDocument
Dim ha As String
IE.Visible = True
IE.navigate "https://www.nseindia.com/get-quotes/equity?symbol=DIVISLAB"
Do While IE.readyState <> READYSTATE_COMPLETE
Loop
Set hdoc = IE.document
ha = hdoc.getElementById("preOpenFp").innerText
Debug.Print ha
End Sub
But the output is nothing pls help.
The website you're trying to scrape offers a very convenient way to do it. All you need to do is send an HTTP request and get the corresponding JSON response which looks like so:
If you take a look at the network traffic in your browser's developer tools, you'll see the requests that are being sent to the server when the page is being loaded. Among these requests you'll find the following one:
To send this request and get the info you need, you have to do the following:
Option Explicit
Sub nse()
Dim req As New MSXML2.XMLHTTP60
Dim url As String
Dim json As Object
url = "https://www.nseindia.com/api/quote-equity?symbol=DIVISLAB"
With req
.Open "GET", url, False
.send
Set json = JsonConverter.ParseJson(.responseText)
End With
Debug.Print json("preOpenMarket")("IEP")
End Sub
This will print the value of IEP to your immediate window (in this case 2390). You can modify the code to best fit your needs.
To parse a JSON string you will need to add this to your project. Follow the installation instructions in the link and you should be set to go.
You will also need to add the following references to your project (VBE>Tools>References):
Microsoft XML version 6.0
Microsoft Scripting Runtime
Request your help to solve the below issue,I am struck for two days,
vba run time error '- 2146697208 (800c0008 )': The download of the specified resource has failed
This is my code for your reference:
Sub telebot15status155555()
'Program Excel Messeger
' Purpose : Send Message to Telegram
' Author : John Eswin Nizar
' Date : 09 October 2017
Dim objRequest As Object
Dim strChatId As String
Dim strMessage1 As String
Dim strMessage2 As String
Dim strPostData As String
Dim strResponse As String
strChatId = "#messstatus"
strMessage1 = ""
strMessage2 = Now()
'strMessage = "hallo"
strPostData = "chat_id=" & strChatId & "&text=" & strMessage1 & strMessage2
Set objRequest = CreateObject("MSXML2.XMLHTTP")
With objRequest
.Open "POST", "https://api.telegram.org/sendMessage?", False
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
.send (strPostData)
GetSessionId = .responseText
MsgBox GetSessionId
End With
End Sub
My advice here is to perform this API call in Chrome with DevTools open. Or access the page, where this happens, and find the Request on the 'Network' tab. Then look at the headers.
debug.print strPostData is returning:
chat_id=#messstatus&text=2/7/2020 9:40:23 AM
That format does not look correct for an API that is form-urlencoded. You need to see what the website is actually sending and receiving in the DevTools in order to replicate it.
IOW, The resource isn't found because the API is incorrectly formatted. From the base api, it looks like its actually a QueryString too because it ends with ? which might mean you have to url-encode format the POST Data into the QueryString for the main HTTP request. Have a look here: Pass Parameters in VBA HTTP Post Request
I have a nice VBA script that allowed me to download stock ticker information from Yahoo into Excel.
Yahoo have recently changed their web interface and the download commands that used to work have now ceased to do so.
The previous commands had the following structure:-
"http://chart.finance.yahoo.com/table.csv?s=TSCO.L&a=5&b=09&c=2016&d=5&e=26&f=2016&g=d&ignore=.csv"
where the new commands look like:-
"https://query1.finance.yahoo.com/v7/finance/download/TSCO.L?period1=1465456200&period2=1466925000&interval=1d&events=history&crumb=Ns6veY6jrcA"
period1 & period2 are epoch representations of the date, and the 'crumb' I believe is unique to each machine that sends a download request to the yahoo server.
I think this remains the same for a reasonable period of time so it doesn't have to be changed.
If I paste the https request into a browser it works. However the routine that I used to check for the data's existence and to subsequently download the data no longer work.
This is a shortened version of the overall code, if anyone is so good as to try it out, you will probably have to replace my crumb value with your crumb value, this can be found if you follow this link:-
https://uk.finance.yahoo.com/quote/TSCO.L/history?p=TSCO.L and hover your mouse over the 'Download Data' link.
`Function IsResourceAvailable(strUrl As String) As Boolean
Dim objXhr As Object 'MSXML2.XMLHTTP60
Dim strStatus As String
Set objXhr = CreateObject("MSXML2.XMLHTTP") 'New XMLHTTP60
With objXhr
.Open "GET", strUrl, False
.send
strStatus = .Status
End With
'HTTP response of 200 = OK
IsResourceAvailable = (strStatus = "200")
End Function'
'Sub dloadDebug()
Dim strUrl As String
Dim blnAvailable As Boolean
strUrl = "https://query1.finance.yahoo.com/v7/finance/download/TSCO.L?period1=1465456200&period2=1466925000&interval=1d&events=history&crumb=Ns6veY6jrcA"
blnAvailable = IsResourceAvailable(strUrl)
Workbooks.Open Filename:=(strUrl)
End Sub`
Neither of the above now work, can anybody point me in the right direction please?
many thanks
GLW
I want to extract the title of every news item displayed on "http://pib.nic.in/newsite/erelease.aspx?relid=58313" website using Excel VBA. I have written a code using getelementsbyclassname("contentdiv"). But the debugger is showing a error pertaining to that the object doesn't support...I want to extract the information items of every relid..which is there in the URL as well...
Cold scrapes like this are generally handled more efficiently with a XMLHTTP pull. This requires the addition of a few libraries to the VBE's Tools ► References. The code below needs Microsoft XML, v6.0, Microsoft HTML Object library and Microsoft Internet Controls. Might not need the last one but you probably will if you expand the code beyond what is supplied.
Public Const csURL As String = "http://pib.nic.in/newsite/erelease.aspx?relid=×ID×"
Sub scrape_PIBNIC()
Dim htmlBDY As HTMLDocument, xmlHTTP As MSXML2.ServerXMLHTTP60
Dim i As Long, u As String, iDIV As Long
On Error GoTo CleanUp
Set xmlHTTP = New MSXML2.ServerXMLHTTP60
Set htmlBDY = New HTMLDocument
For i = 58313 To 58313
htmlBDY.body.innerHTML = vbNullString
With xmlHTTP
u = Replace(csURL, "×ID×", i)
'Debug.Print u
.Open "GET", u, False
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded; charset=UTF-8"
.send
If .Status <> 200 Then GoTo CleanUp
htmlBDY.body.innerHTML = .responseText
For iDIV = 0 To (htmlBDY.getElementsByClassName("contentdiv").Length - 1)
If CBool(htmlBDY.getElementsByClassName("contentdiv")(iDIV).getElementsByTagName("span").Length) Then
Sheets("Sheet1").Cells(Rows.Count, 1).End(xlUp).Offset(1, 0) = _
htmlBDY.getElementsByClassName("contentdiv")(iDIV).getElementsByTagName("span")(0).innerText
End If
Next iDIV
End With
Next i
CleanUp:
Set htmlBDY = Nothing
Set xmlHTTP = Nothing
End Sub
That should be enough to get you started. The site you are targeting requires that charset=UTF-8 be added to the request. I had no success without it. I strongly suspect that this may have been the source of your object doesn't support error.