Zillow web scraping in excel vba (Big issue help) - excel

I am trying to web scrape Zillow. I am currently using web automation however, I cannot search my desire location. The value appears on the search bar, but when it submits it doesn't change it goes back to the same page. It is like the value pre-established stays there even though I change it. Please help me, I've been trying for so many days and I can't get answers.
Zillow's code.----------------------------------------------------------------------------------
<input class="react-autosuggest__input" role="combobox" aria-expanded="false" aria-controls="react-autowhatever-1" aria-owns="react-autowhatever-1" aria-autocomplete="list" aria-label="Search: Suggestions appear below" type="text" placeholder="Address, neighborhood, or ZIP" value="new jersey" autoComplete="off">
Sub zillow()
Dim ie As New SHDocVw.InternetExplorer
Dim doc As MSHTML.HTMLDocument
Dim zillowinput As MSHTML.IHTMLElementCollection
Dim zillowinput2 As MSHTML.IHTMLElementCollection
Dim direc As String
Dim iny As MSHTML.IHTMLElementCollection
Dim inys As MSHTML.IHTMLElement
ie.Visible = True
ie.navigate "https://www.zillow.com/homes/new-jersey_rb/"
Do While ie.readyState <> READYSTATE_COMPLETE Or ie.Busy
Loop`enter code here`
Set doc = ie.document
direc = Range("D5").Value
Application.Wait Now() + #12:00:02 AM#
Set inys = doc.getElementById("srp-search-box")
Set inys = doc.getElementsByTagName("input")(0)
inys.Focus
inys.Value = "35 Krakow St, Garfield, NJ 07026"
inys.Blur
**strong text**
doc.forms(0).submit

The change event in the search box can be fired with SendKeys. You can simulate user input using SendKeys to set value of the search box and press Enter to do the search.
You can refer to the working code below:
Sub zillow()
Dim ie As New SHDocVw.InternetExplorer
Dim doc As MSHTML.HTMLDocument
Dim inys As MSHTML.IHTMLElement
ie.Visible = True
ie.navigate "https://www.zillow.com/homes/new-jersey_rb/"
Do While ie.readyState <> READYSTATE_COMPLETE Or ie.Busy
Loop
Set doc = ie.document
Application.Wait Now() + #12:00:02 AM#
Set inys = doc.getElementsByTagName("input")(0)
inys.Focus
SendKeys ("35 Krakow St, Garfield, NJ 07026")
Application.Wait (Now + TimeValue("00:00:02"))
SendKeys ("{ENTER}")
End Sub
Result in IE:

Related

How to close a web message from IE VBA?

I,ve been dealing with a issue during my web scraping.
My problem is that when I click to "Submit" a form, it pops-up a web message that I've not been able to close.
I´ve already try the sendKeys Method but no success.
here is the link of the web page:
https://www3.bcb.gov.br/CALCIDADAO/publico/corrigirPorIndice.do?method=corrigirPorIndice
Here is a print of the info to put in the webpage
Here is a print of the info to put in the webpage
The message appers when you click the button on the left "Corrigir Valor"
enter image description here
the messege
enter image description here
obs. pressing enter or esc with the keyboard close it, but I`m not been able to do it in the code
here is the code if may be helpful
Sub atualizacao_valores()
Dim data_base As String
Dim data_atualizacao As String
Dim valor_face As String
Dim drp As HTMLFormElement
Dim html As HTMLDocument
Set ie = CreateObject("internetexplorer.application")
ie.navigate "https://www3.bcb.gov.br/CALCIDADAO/publico/exibirFormCorrecaoValores.do?method=exibirFormCorrecaoValores&aba=1"
ie.Visible = True
Do While ie.busy And ie.readyState <> "READYSTATE_COMPLETE"
DoEvents
Loop
data_base_ipcae = "01/2022"
data_atualizacao_ipcae = "12/2022"
valor_face = "100000"
Application.Wait (Now + TimeValue("00:00:02"))
Set html = ie.document
Set drp = html.getElementById("selIndice")
drp.selectedIndex = 4
ie.document.getelementsbytagname("input")(1).Value = data_base_ipcae
ie.document.getelementsbytagname("input")(2).Value = data_atualizacao_ipcae
ie.document.getelementsbytagname("input")(3).Value = valor_face
ie.document.getelementsbyclassname("botao")(0).Click
`here happens the pop up that I cant close
...
end sub
I agree with the suggestion given by Tim Williams.
Adding code below just above the line when you click the button could suppress the Alert().
With html.parentWindow
.execScript "window.alert = function(){return true;};", "JScript"
End With
Your modified code:
Sub atualizacao_valores()
Dim data_base As String
Dim data_atualizacao As String
Dim valor_face As String
Dim drp As HTMLFormElement
Dim html As HTMLDocument
Set ie = CreateObject("internetexplorer.application")
ie.navigate "https://www3.bcb.gov.br/CALCIDADAO/publico/exibirFormCorrecaoValores.do?method=exibirFormCorrecaoValores&aba=1"
ie.Visible = True
Do While ie.busy And ie.readyState <> "READYSTATE_COMPLETE"
DoEvents
Loop
data_base_ipcae = "01/2022"
data_atualizacao_ipcae = "12/2022"
valor_face = "100000"
Application.Wait (Now + TimeValue("00:00:02"))
Set html = ie.document
Set drp = html.getElementById("selIndice")
drp.selectedIndex = 4
ie.document.getelementsbytagname("input")(1).Value = data_base_ipcae
ie.document.getelementsbytagname("input")(2).Value = data_atualizacao_ipcae
ie.document.getelementsbytagname("input")(3).Value = valor_face
With html.parentWindow
.execScript "window.alert = function(){return true;};", "JScript"
End With
ie.document.getelementsbyclassname("botao")(0).Click
End Sub
I have tested this code on my end and it is suppressing the Alert message.
You could test it and let us know your test results.

VBA Excel click on href and fill in form

im trying to get some data from a website by using vba.
The data i want is from this site: https://www.uitvoeringarbeidsvoorwaardenwetgeving.nl/mozard/!suite16.scherm1168?mSel=145576
What i want the code to do is click on the purple bar with the pencil
on it so the screen appears for filters and than fill in a specific time frame in the filters.
When this is done i want to get the data that appears.
Im able to navigate to the site and click on the purple bar so the filter screen appears. but i cant fill in the dates
this is the code i have so far:
Dim IE As New SHDocVw.InternetExplorer
Dim HTMLDoc As MSHTML.HTMLDocument
Dim HTMLInput As MSHTML.IHTMLElement
Dim HTMLAs As MSHTML.IHTMLElementCollection
Dim HTMLA As MSHTML.IHTMLElement
Dim pastDate As MSHTML.IHTMLElement
Dim futuredate As MSHTML.IHTMLElement
IE.Visible = True
IE.Navigate "https://www.uitvoeringarbeidsvoorwaardenwetgeving.nl/mozard/!suite16.scherm1168?mGmr=66"
Do While IE.ReadyState <> READYSTATE_COMPLETE
Loop
Set HTMLDoc = IE.Document
Set HTMLAs = HTMLDoc.getElementsByTagName("a")
For Each HTMLA In HTMLAs
'Debug.Print HTMLA.className, HTMLA.getAttribute("href"), HTMLA.getAttribute("rel"), HTMLA.innerText
If HTMLA.getAttribute("href") = "https://www.uitvoeringarbeidsvoorwaardenwetgeving.nl/mozard/!suite16.scherm1168?mGmr=66#editmodal" Then
HTMLA.Click
Exit For
End If
Next HTMLA
Do While IE.ReadyState <> 4 Or IE.Busy:
DoEvents: Loop
Set HTMLInput = HTMLDoc.getElementById("frm_FKMT_B931_542_823883_dva_id1")
HTMLInput.Value = "01-01-2020" 'THIS GIVES AN ERROR?
The last line of code gives an error and i dont understand why??
This is the HTML code from the website that i want to change the value of:
<input name="FKMT_B931_542_823883_dva" class="datumveld form-control" id="frm_FKMT_B931_542_823883_dva_id1" type="text" pattern="(0[1-9]|1[0-9]|2[0-9]|3[01]).(0[1-9]|1[012]).[0-9]{4}">
Thanks and sorry for the inconvience or poorly asked question, if there is anything else you guys need to now please feel free to ask!
Thank you!!
This is an example to fill the first date field. The IDs seems not very stable.
Beware: There is a pattern for the entered dates
pattern="(0[1-9]|1[0-9]|2[0-9]|3[01]).(0[1-9]|1[012]).[0-9]{4}"
There are some html events. I don't know if it is necessary to trigger them to make the dialog realy work.
Have you checked if the page works in IE?
Sub OpenAndFillForm()
Dim browser As Object
Dim url As String
Dim nodeToClick As Object
Dim nodeForm As Object
Dim nodeFirstDate As Object
url = "https://www.uitvoeringarbeidsvoorwaardenwetgeving.nl/mozard/!suite16.scherm1168?mGmr=66"
Set browser = CreateObject("internetexplorer.application")
browser.Visible = True
browser.navigate url
Do Until browser.readyState = 4: DoEvents: Loop
Set nodeToClick = browser.document.getElementByID("tabel2").getElementsByTagName("a")(0)
nodeToClick.Click
Application.Wait Now + TimeValue("00:00:02")
Set nodeForm = browser.document.getElementByID("tabel12")
Set nodeFirstDate = nodeForm.getElementsByClassName("datumveld")(0)
nodeFirstDate.Value = "31-12-2019"
End Sub

How to use excel vba to click on interactive dialog pop-up?

I am trying to use excel vba to navigate and export data from this website. I am able to click on 2018, 2019 buttons and the setting option, but unable to click the 'export data' option with vba. I attach my code below for your reference.
Option Explicit
Sub GetURLOfFrame()
Dim IE As New SHDocVw.InternetExplorer
Dim webpage As MSHTML.HTMLDocument
IE.Visible = True
IE.navigate "https://www.epa.gov/fuels-registration-reporting-and-compliance-help/rin-trades-and-price-information"
Dim t As Date, ele As Object 'waiting
Const MAX_WAIT_SEC As Long = 10 '<==Adjust wait time
While IE.Busy Or IE.readyState < 4: DoEvents: Wend
t = Timer
Do
DoEvents
On Error Resume Next
Set ele = IE.document.getElementById("show-service-popup-dialog")
If Timer - t > MAX_WAIT_SEC Then Exit Do
On Error GoTo 0
Loop While ele Is Nothing
Application.Wait (Now + TimeValue("0:00:10"))
Set webpage = IE.document
Dim wkb As Workbook, wksRIN As Worksheet, wksFrames As Worksheet
Set wkb = ThisWorkbook
Set wksRIN = wkb.Sheets("RIN Trades and Prices")
Dim Frame As MSHTML.IHTMLElement
Set Frame = webpage.getElementsByTagName("iframe")(1)
Dim URL As String
URL = Frame.getAttribute("src") 'storing the link to the frame
IE.navigate URL 'goes to the iframe site
While IE.Busy Or IE.readyState < 4: DoEvents: Wend
t = Timer
Do
DoEvents
On Error Resume Next
Set ele = IE.document.getElementById("content")
If Timer - t > MAX_WAIT_SEC Then Exit Do
On Error GoTo 0
Loop While ele Is Nothing
Application.Wait (Now + TimeValue("0:00:10"))
Dim htmlarticle As MSHTML.IHTMLElement
Set htmlarticle = webpage.getElementsByTagName("article")(0)
Application.Wait (Now + TimeValue("0:00:02"))
Dim buttons As MSHTML.IHTMLElementCollection
Set buttons = htmlarticle.getElementsByClassName("lui-buttongroup ng-scope")(0).getElementsByTagName("button")
buttons(buttons.Length - 2).Click 'clicking the second last year
buttons(buttons.Length - 1).Click 'clicking the latest year
Application.Wait (Now + TimeValue("0:00:02"))
Dim SettingButton As MSHTML.IHTMLElement
Set SettingButton = htmlarticle.getElementsByClassName("cl-icon cl-icon--cogwheel cl-icon-right-align")(0)
SettingButton.Click
Dim SettingOptions As MSHTML.IHTMLElementCollection, ExportDataButton As MSHTML.IHTMLElement, button As MSHTML.IHTMLElement
Set SettingOptions = webpage.getElementsByClassName("qv-contextmenu ng-scope")(0).getElementsByTagName("li")
Application.Wait (Now + TimeValue("0:00:05"))
Set ExportDataButton = webpage.getElementsByClassName("qv-contextmenu ng-scope")(0).getElementsByTagName("li")(0)
ExportDataButton.Focus
ExportDataButton.Click
IE.Quit
Set IE = Nothing
End Sub
When I run 'ExportDataButton.Click', nothing seems to happen and I am not sure the reason why. It is also not possible to scrape from the data in the table directly as the the values of innertext of the td are dynamic and changes as you scroll down the table.
Unable to click on 'Export Data'
You have done a good job till the point you needed help. A tip at your code. Declare all variables in the head of the macro. That is more clearly and other people who read your code have a better focus.
I think it would be easier to take all values from the html table. But you've done good groundwork. So I completed your macro in the way you started. I commented on the code I added.
Sub GetURLOfFrame()
Dim IE As New SHDocVw.InternetExplorer
Dim webpage As MSHTML.htmlDocument
IE.Visible = True
IE.navigate "https://www.epa.gov/fuels-registration-reporting-and-compliance-help/rin-trades-and-price-information"
Dim t As Date, ele As Object 'waiting
Const MAX_WAIT_SEC As Long = 10 '<==Adjust wait time
While IE.Busy Or IE.readyState < 4: DoEvents: Wend
t = Timer
Do
DoEvents
On Error Resume Next
Set ele = IE.document.getElementById("show-service-popup-dialog")
If Timer - t > MAX_WAIT_SEC Then Exit Do
On Error GoTo 0
Loop While ele Is Nothing
Application.Wait (Now + TimeValue("0:00:10"))
Set webpage = IE.document
Dim wkb As Workbook, wksRIN As Worksheet, wksFrames As Worksheet
Set wkb = ThisWorkbook
Set wksRIN = wkb.Sheets("RIN Trades and Prices")
Dim Frame As MSHTML.IHTMLElement
Set Frame = webpage.getElementsByTagName("iframe")(1)
Dim URL As String
URL = Frame.getAttribute("src") 'storing the link to the frame
IE.navigate URL 'goes to the iframe site
While IE.Busy Or IE.readyState < 4: DoEvents: Wend
t = Timer
Do
DoEvents
On Error Resume Next
Set ele = IE.document.getElementById("content")
If Timer - t > MAX_WAIT_SEC Then Exit Do
On Error GoTo 0
Loop While ele Is Nothing
Application.Wait (Now + TimeValue("0:00:10"))
Dim htmlarticle As MSHTML.IHTMLElement
Set htmlarticle = webpage.getElementsByTagName("article")(0)
Application.Wait (Now + TimeValue("0:00:02"))
Dim buttons As MSHTML.IHTMLElementCollection
Set buttons = htmlarticle.getElementsByClassName("lui-buttongroup ng-scope")(0).getElementsByTagName("button")
buttons(buttons.Length - 2).Click 'clicking the second last year
buttons(buttons.Length - 1).Click 'clicking the latest year
Application.Wait (Now + TimeValue("0:00:02"))
Dim SettingButton As MSHTML.IHTMLElement
Set SettingButton = htmlarticle.getElementsByClassName("cl-icon cl-icon--cogwheel cl-icon-right-align")(0)
SettingButton.Click
Dim SettingOptions As MSHTML.IHTMLElementCollection, ExportDataButton As MSHTML.IHTMLElement, button As MSHTML.IHTMLElement
Set SettingOptions = webpage.getElementsByClassName("qv-contextmenu ng-scope")(0).getElementsByTagName("li")
'Application.Wait (Now + TimeValue("0:00:05")) 'Not needed
'The fifth li-tag is "Export data"
Set ExportDataButton = webpage.getElementsByClassName("qv-contextmenu ng-scope")(0).getElementsByTagName("li")(4)
'There is no focus and no click event
'There are the two events keypress and qv-activate
'We must trigger qv-activate to get the download popup
'(See function TriggerEvent to understand what we do here)
Call TriggerEvent(webpage, ExportDataButton, "qv-activate")
'Give the popup time to raise
Application.Wait (Now + TimeValue("0:00:01"))
'I do it like you here, but it's much better to declare all variables in the head of the macro
Dim lastIsDownloadLink As MSHTML.IHTMLElementCollection
'The download link has the css class ng-binding. 101 elements has these css class
'But the code for the popup is generated after clicking 'Export data' at the end of the HTML document
'So we can be sure that the last element with the CSS class contains our searched link
Set lastIsDownloadLink = webpage.getElementsByClassName("ng-binding")
lastIsDownloadLink(lastIsDownloadLink.Length - 1).Click
'Give the server time to generate the document and the IE to show the download button in the footer
Application.Wait (Now + TimeValue("0:00:03"))
'Attention!
'We have to use SendKeys to perform the download!!!
'As long as the macro is running, keep your fingers away from the mouse and keyboard!!!
'
'I've tryed to use the URLDownloadToFile method from the urlmon.dll but that don't work
'I'm sure the server blocks download questions from other applications than that one who
'has generated the download url with. I tryed it also with FireFox and there comes the text,
'that the requestet resource is not available. But the same link works at the same time in IE.
'You will find the download in the download folder of your system
Application.SendKeys ("%{S}")
'IE.Quit
'Set IE = Nothing
End Sub
This is the procedure to trigger a html event:
Private Sub TriggerEvent(htmlDocument As Object, htmlElementWithEvent As Object, eventType As String)
Dim theEvent As Object
htmlElementWithEvent.Focus
Set theEvent = htmlDocument.createEvent("HTMLEvents")
theEvent.initEvent eventType, True, False
htmlElementWithEvent.dispatchEvent theEvent
End Sub

Excel VBA controlling IE dialog box "Do you want to Open or Save"

I've been searching for how to press save when downloading file with IE and save the file under specific name in a specific location.
This is the screen as fas as I go with vba code
My code looks like this so far:
Sub BrowseToSite()
Dim IE As New SHDocVw.InternetExplorer
Dim HTMLDoc As MSHTML.HTMLDocument
Dim HTMLInput As MSHTML.IHTMLElement
Dim HTMLButtons As MSHTML.IHTMLElementCollection
Dim HTMLButton As MSHTML.IHTMLElement
IE.Visible = True
IE.navigate "cboe.com/delayedquote/quote-table-download"
Do While IE.readyState <> READYSTATE_COMPLETE
Loop
Set HTMLDoc = IE.document
Set HTMLInput = HTMLDoc.getElementById("ContentTop_C005_txtTicker")
HTMLInput.Value = "DJX"
Set HTMLButtons = HTMLDoc.getElementsByClassName("button blue")
HTMLButtons(0).Click
End Sub
All other sources i have looked didn't worked for me, maybe I am missing something. SendKeys didn't worked either which I think is supposed to be simplest way of achieving pressing save button. What code should i use?

VB Click Button Macro Note Working

I am trying to get the results of tracking a package on my screen. In Excel, I have this code, which successfully opens the IE browser and successfully types the tracking number into the input box. What doesn't work is the last step - button click. I get the runtime error 91 - object variable or with block variable not set. Thanks in advance.
Sub TrackUSPS()
Const cURL = "https://tools.usps.com/go/TrackConfirmAction!input.action" 'Enter the web address here
Const trackNum = "9405511899560005266920" 'Tracking number
Dim IE As InternetExplorer
Dim doc As HTMLDocument
Dim LoginForm As HTMLFormElement
Dim TrackInputBox As HTMLInputElement
Dim TrackButton As HTMLInputButtonElement
Dim HTMLelement As IHTMLElement
Dim qt As QueryTable
Set IE = New InternetExplorer
IE.Visible = True
IE.navigate cURL
'Wait for initial page to load
Do While IE.readyState <> READYSTATE_COMPLETE Or IE.Busy: DoEvents: Loop
Set doc = IE.document
'Get the tracking form on the page
Set LoginForm = doc.forms(1)
'Get the tracking input box and populate it
'<input type="text" id="search-text" autocomplete="off" name="searchText" class="default" value="Search USPS.com or Track Packages" onclick="javascript:dojo.byId('search-text').value='';"/>
Set TrackInputBox = LoginForm.elements("search-text")
TrackInputBox.Value = trackNum
'Get the form input button and click it
'<input type="image" id="search-btn" src="/media/images/global/blank.gif" alt="Process Search" />
Set TrackButton = LoginForm.elements("search-btn")
TrackButton.Click
'Wait for the new page to load
Do While IE.readyState <> READYSTATE_COMPLETE Or IE.Busy: DoEvents: Loop
End Sub
This works for me, replace
Set TrackButton = LoginForm.elements("search-btn")
TrackButton.Click
with
IE.document.getelementbyID("search-btn").Click

Resources