I have been using the following Excel VBA macro to bring back data from a website. It worked fine until a few days ago when the website stopped supporting IE. Of course the macro just fails now as there is no data on the webpage to bring back to Excel, Is there a way to have the "Get method" (MSXML2.XMLHTTP)
here is my Code
Public Sub GGGG()
Dim MSX As Object
Dim HTML As HTMLDocument
Dim URL As String
Dim UrlResponse As String
Dim N As Long
Dim sht1, sht2 As Worksheet
' On Error Resume Next
Set MSX = CreateObject("MSXML2.XMLHTTP")
Set HTML = New HTMLDocument
URL = "https://www.justdial.com/Agra/Yogi-General-Store-Opp-Eclave-Satiudum-Sadar-Bazaar/0562P5612-5612-120207212812-H5I2_BZDET"
With MSX
.Open "GET", URL, False
.setRequestHeader "If-Modified-Since", "Sat, 1 Jan 2000 00:00:00 GMT"
.send
UrlResponse = StrConv(.responseBody, vbUnicode)
End With
ActiveCell.Offset(0, 1) = UrlResponse
End Sub
I get response like
Error
An error occurred while processing your request.
Reference #97.ec8a2c31.1621136928.281f3ca8
Please anyone can support me how to get data when IE dose not support
I am not an expert in coding
Okay, try this to get the title and votes from that site using vba in combination with selenium.
Sub FetchInfo()
Dim driver As Object, oTitle As Object
Dim oVotes As Object
Set driver = CreateObject("Selenium.ChromeDriver")
driver.get "https://www.justdial.com/Agra/Yogi-General-Store-Opp-Eclave-Satiudum-Sadar-Bazaar/0562P5612-5612-120207212812-H5I2_BZDET"
Set oTitle = driver.FindElementByCss("span.item > span", Raise:=False, timeout:=10000)
Set oVotes = driver.FindElementByCss("span.rtngsval > span.votes", Raise:=False, timeout:=10000)
Debug.Print oTitle.Text, oVotes.Text
End Sub
When the webpage no longer support IE in future, you can try out web scrape using Google Chrome with new add-in installed, please see following link for the add-in installation adn how to write in VBA. However, it is in my opinion the most simple way to perform your work is to use Uipath free community version, it work for all type of web-browser.
VBA guideline:
https://www.wiseowl.co.uk/vba-macros/videos/vba-scrape-websites/web-scraping-selenium-chrome/
VBA library installation for Selenium:
https://code.google.com/archive/p/selenium-vba/downloads
You probably need to set the Feature Browser Emulation to zero as detailed by Daniel here:
Everything You Never Wanted to Know About the Access WebBrowser Control
That said, your URL fails even when opened in Edge Chromium, so the site may suffer from a general failure.
Related
I have looked around and have not found a way using get in VBA within excel with selenium. I am trying not open the the browser and still get the url info. Does anyone know a way using selenium in vba to do this? For example:
Option Explicit
Sub TestSelenium()
dim MyBrowser As Selenium.ChromeDriver
Set MyBrowser = New Selenium.ChromeDriver
MyBrowser.Start
driver.Get "https://www.google.com/"
End Sub
I have seen way such as with the html object library
Option Explicit
Public Sub test()
'tools > references > Microsoft HTML Object Library
Dim html As MSHTML.HTMLDocument, xhr As Object
Set xhr = CreateObject("MSXML2.XMLHTTP")
Set html = New MSHTML.HTMLDocument
With xhr
.Open "GET", ""https://www.google.com/"", False
.setRequestHeader "User-Agent", "Safari/537.36"
.Send
html.body.innerHTML = .responseText
Debug.Print html.Title
End With
End Sub
However, I am interested in solving the problem using selenium.
Please use headless mode for this use case.
Dim driver As New ChromeDriver, post As Object
With driver
.AddArgument "--headless" ''This is the fix
.get "https://yts.am/browse-movies"
End With
download
https://phantomjs.org/download.html
and add the exe to the correct file path in selenium
Sub HeadlessSelenium()
Dim PJSD As Selenium.PhantomJSDriver
Dim strHTML As String
' Instantiate Selenium through the PhantomJS Driver
Set PJSD = New Selenium.PhantomJSDriver
PJSD.Start
' Navigate to the URL
PJSD.Get "https://www.google.com/"
' Extract the HTML code of the website
strHTML = PJSD.PageSource
' Print the HTML code to the Immediate Window
Debug.Print strHTML
End Sub
I have been using the following Excel VBA macro to bring back data from a website. It worked fine until a few days ago when the website stopped supporting IE. Of course the macro just fails now as there is no data on the webpage to bring back to Excel, just a message saying, "Your browser, Internet Explorer, is no longer supported." Is there a way to have the "Get method" (MSXML2.XMLHTTP) use Chrome instead of IE to interact with the website? BTW, my default browser is already set to "Chrome".
Dim html_doc As HTMLDocument ' note: reference to Microsoft HTML Object Library must be set
Sub KS()
' Define product url
KS_url = "https://www.kingsoopers.com/p/r-w-knudsen-just-blueberry-juice/0007468210784"
' Collect data
Set html_doc = New HTMLDocument
Set xml_obj = CreateObject("MSXML2.XMLHTTP")
xml_obj.Open "GET", KS_url, False
xml_obj.send
html_doc.body.innerHTML = xml_obj.responseText
Set xml_obj = Nothing
KS_product = html_doc.getElementsByClassName("ProductDetails-header")(0).innerText
KS_price = "$" & html_doc.getElementsByClassName("kds-Price kds-Price--alternate mb-8")(1).Value
do Stuff
End Sub
The check for this is a basic server check on user agent. Tell it what it wants to "hear" by passing a supported browser in the UA header...(or technically, in this case, just saying the equivalent of: "Hi, I am not Internet Explorer".)
It can be as simple as xml.setRequestHeader "User-Agent", "Chrome". I said basic because you could even pass xml.setRequestHeader "User-Agent", "I am a unicorn", so it is likely an exclusion based list on the server for Internet Explorer.
Option Explicit
Public Sub KS()
Dim url As String
url = "https://www.kingsoopers.com/p/r-w-knudsen-just-blueberry-juice/0007468210784"
Dim html As MSHTML.HTMLDocument, xml As Object
Set html = New MSHTML.HTMLDocument
Set xml = CreateObject("MSXML2.XMLHTTP")
xml.Open "GET", url, False
xml.setRequestHeader "User-Agent", "Mozilla/5.0"
xml.send
html.body.innerHTML = xml.responseText
Debug.Print html.getElementsByClassName("ProductDetails-header")(0).innerText
Debug.Print "$" & html.getElementsByClassName("kds-Price kds-Price--alternate mb-8")(1).Value
Stop
End Sub
Compare that with adding no UA or adding xml.setRequestHeader "User-Agent", "MSIE".
Study the article here by Daniel Pineault and this paragraph:
Feature Browser Emulation
Also note my comment dated 2020-09-13.
Any one of three highlighted part are the value i want to print. I am trying below code
Sub JJ()
Dim IE As New SHDocVw.InternetExplorer
Dim hdoc As MSHTML.HTMLDocument
Dim ha As String
IE.Visible = True
IE.navigate "https://www.nseindia.com/get-quotes/equity?symbol=DIVISLAB"
Do While IE.readyState <> READYSTATE_COMPLETE
Loop
Set hdoc = IE.document
ha = hdoc.getElementById("preOpenFp").innerText
Debug.Print ha
End Sub
But the output is nothing pls help.
The website you're trying to scrape offers a very convenient way to do it. All you need to do is send an HTTP request and get the corresponding JSON response which looks like so:
If you take a look at the network traffic in your browser's developer tools, you'll see the requests that are being sent to the server when the page is being loaded. Among these requests you'll find the following one:
To send this request and get the info you need, you have to do the following:
Option Explicit
Sub nse()
Dim req As New MSXML2.XMLHTTP60
Dim url As String
Dim json As Object
url = "https://www.nseindia.com/api/quote-equity?symbol=DIVISLAB"
With req
.Open "GET", url, False
.send
Set json = JsonConverter.ParseJson(.responseText)
End With
Debug.Print json("preOpenMarket")("IEP")
End Sub
This will print the value of IEP to your immediate window (in this case 2390). You can modify the code to best fit your needs.
To parse a JSON string you will need to add this to your project. Follow the installation instructions in the link and you should be set to go.
You will also need to add the following references to your project (VBE>Tools>References):
Microsoft XML version 6.0
Microsoft Scripting Runtime
First post here. Tried looking for similar posts but wasn't able to turn up anything.
I'm a little new to VBA. I'm trying to use Excel to navigate to a specific website, click a radio button, type in some text as a search string, and then search on that text. Everything seems fine when I walk through my code, but when I click the search button my search string gets blanked out and I get an error message telling me to enter search criteria. Code below:
Sub FranklinCountyWebsite()
'References: Microsoft Internet Controls, Microsoft HTML Object Library
Dim IE As New SHDocVw.InternetExplorer
Dim HTMLDoc As MSHTML.HTMLDocument
IE.Visible = True
IE.navigate "https://sheriff.franklincountyohio.gov/real-estate/"
Do While IE.readyState <> READYSTATE_COMPLETE
Loop
Set HTMLDoc = IE.document
HTMLDoc.getElementById("ctl00_SheetContentPlaceHolder_c_search1_rblSrchOptions_3").Click
HTMLDoc.getElementById("ctl00_SheetContentPlaceHolder_c_search1_SrchSearchString").Value = "43215"
HTMLDoc.getElementById("ctl00_SheetContentPlaceHolder_c_search1_btnSearch").Click
End Sub
Interestingly, if I go to the Franklin County website and type in the text manually and then hit search, everything works fine. Is there something easy I'm overlooking?
You can try the same using serverxmlhttp request which is way faster than IE. The below script can lead you to the target page where you wished to get data from.
Sub Fetch_Item()
Dim post As Object, qsp$, S$
qsp = "q=searchType%3dZipCode%26searchString%3d43215%26foreclosureType%3d%26sortType%3daddress%26saleDateFrom%3d4%2f30%2f2017+12%3a00%3a00+AM%26saleDateTo%3d10%2f30%2f2018+11%3a59%3a59+PM"
With New ServerXMLHTTP
.Open "GET", "https://sheriff.franklincountyohio.gov/real-estate/results.aspx?" & qsp, False
.setRequestHeader "User-Agent", "Mozilla/5.0"
.send
S = .responseText
End With
With New HTMLDocument
.body.innerHTML = S
Set post = .getElementById("ctl00_SheetContentPlaceHolder_C_searchresults_reSaleSummary_ctl00_lblAddrHeader")
MsgBox post.innerText
End With
End Sub
Output:
155-157 CLEVELAND AVE COLUMBUS, OH 43215 010054688, 010055721
Reference to add to the library:
Microsoft XML, V6.0
Microsoft HTML Object Library
I have a VBA program which reads HTML contents, processes it and outputs the result onto an Excel spreadsheet.
The program runs without issue when the default browser is set to Chrome, however testing the program on a machine where IE 11 is set as the default browser results in a run-time error.
I believe that the issue relates to the following code:-
Dim oDom As Object: Set oDom = CreateObject("htmlFile")
With CreateObject("msxml2.xmlhttp")
.Open "GET", "http://bouch/jira/browse/MAX-1", False
.Send
oDom.body.innerHtml = .responseText
End With
with IE 11, I don't think that the msxml2.xmlhttp is returning anything (or whatever it does return results in subsequent parts of the code issuing the fault).
I've tried msxml2.xmlhttp.3.0 msxml2.xmlhttp.6.0 but to no avail.
Is there a simple answer?
Many thanks in advance.