im trying to get some data from a website by using vba.
The data i want is from this site: https://www.uitvoeringarbeidsvoorwaardenwetgeving.nl/mozard/!suite16.scherm1168?mSel=145576
What i want the code to do is click on the purple bar with the pencil
on it so the screen appears for filters and than fill in a specific time frame in the filters.
When this is done i want to get the data that appears.
Im able to navigate to the site and click on the purple bar so the filter screen appears. but i cant fill in the dates
this is the code i have so far:
Dim IE As New SHDocVw.InternetExplorer
Dim HTMLDoc As MSHTML.HTMLDocument
Dim HTMLInput As MSHTML.IHTMLElement
Dim HTMLAs As MSHTML.IHTMLElementCollection
Dim HTMLA As MSHTML.IHTMLElement
Dim pastDate As MSHTML.IHTMLElement
Dim futuredate As MSHTML.IHTMLElement
IE.Visible = True
IE.Navigate "https://www.uitvoeringarbeidsvoorwaardenwetgeving.nl/mozard/!suite16.scherm1168?mGmr=66"
Do While IE.ReadyState <> READYSTATE_COMPLETE
Loop
Set HTMLDoc = IE.Document
Set HTMLAs = HTMLDoc.getElementsByTagName("a")
For Each HTMLA In HTMLAs
'Debug.Print HTMLA.className, HTMLA.getAttribute("href"), HTMLA.getAttribute("rel"), HTMLA.innerText
If HTMLA.getAttribute("href") = "https://www.uitvoeringarbeidsvoorwaardenwetgeving.nl/mozard/!suite16.scherm1168?mGmr=66#editmodal" Then
HTMLA.Click
Exit For
End If
Next HTMLA
Do While IE.ReadyState <> 4 Or IE.Busy:
DoEvents: Loop
Set HTMLInput = HTMLDoc.getElementById("frm_FKMT_B931_542_823883_dva_id1")
HTMLInput.Value = "01-01-2020" 'THIS GIVES AN ERROR?
The last line of code gives an error and i dont understand why??
This is the HTML code from the website that i want to change the value of:
<input name="FKMT_B931_542_823883_dva" class="datumveld form-control" id="frm_FKMT_B931_542_823883_dva_id1" type="text" pattern="(0[1-9]|1[0-9]|2[0-9]|3[01]).(0[1-9]|1[012]).[0-9]{4}">
Thanks and sorry for the inconvience or poorly asked question, if there is anything else you guys need to now please feel free to ask!
Thank you!!
This is an example to fill the first date field. The IDs seems not very stable.
Beware: There is a pattern for the entered dates
pattern="(0[1-9]|1[0-9]|2[0-9]|3[01]).(0[1-9]|1[012]).[0-9]{4}"
There are some html events. I don't know if it is necessary to trigger them to make the dialog realy work.
Have you checked if the page works in IE?
Sub OpenAndFillForm()
Dim browser As Object
Dim url As String
Dim nodeToClick As Object
Dim nodeForm As Object
Dim nodeFirstDate As Object
url = "https://www.uitvoeringarbeidsvoorwaardenwetgeving.nl/mozard/!suite16.scherm1168?mGmr=66"
Set browser = CreateObject("internetexplorer.application")
browser.Visible = True
browser.navigate url
Do Until browser.readyState = 4: DoEvents: Loop
Set nodeToClick = browser.document.getElementByID("tabel2").getElementsByTagName("a")(0)
nodeToClick.Click
Application.Wait Now + TimeValue("00:00:02")
Set nodeForm = browser.document.getElementByID("tabel12")
Set nodeFirstDate = nodeForm.getElementsByClassName("datumveld")(0)
nodeFirstDate.Value = "31-12-2019"
End Sub
Related
I am trying to web scrape Zillow. I am currently using web automation however, I cannot search my desire location. The value appears on the search bar, but when it submits it doesn't change it goes back to the same page. It is like the value pre-established stays there even though I change it. Please help me, I've been trying for so many days and I can't get answers.
Zillow's code.----------------------------------------------------------------------------------
<input class="react-autosuggest__input" role="combobox" aria-expanded="false" aria-controls="react-autowhatever-1" aria-owns="react-autowhatever-1" aria-autocomplete="list" aria-label="Search: Suggestions appear below" type="text" placeholder="Address, neighborhood, or ZIP" value="new jersey" autoComplete="off">
Sub zillow()
Dim ie As New SHDocVw.InternetExplorer
Dim doc As MSHTML.HTMLDocument
Dim zillowinput As MSHTML.IHTMLElementCollection
Dim zillowinput2 As MSHTML.IHTMLElementCollection
Dim direc As String
Dim iny As MSHTML.IHTMLElementCollection
Dim inys As MSHTML.IHTMLElement
ie.Visible = True
ie.navigate "https://www.zillow.com/homes/new-jersey_rb/"
Do While ie.readyState <> READYSTATE_COMPLETE Or ie.Busy
Loop`enter code here`
Set doc = ie.document
direc = Range("D5").Value
Application.Wait Now() + #12:00:02 AM#
Set inys = doc.getElementById("srp-search-box")
Set inys = doc.getElementsByTagName("input")(0)
inys.Focus
inys.Value = "35 Krakow St, Garfield, NJ 07026"
inys.Blur
**strong text**
doc.forms(0).submit
The change event in the search box can be fired with SendKeys. You can simulate user input using SendKeys to set value of the search box and press Enter to do the search.
You can refer to the working code below:
Sub zillow()
Dim ie As New SHDocVw.InternetExplorer
Dim doc As MSHTML.HTMLDocument
Dim inys As MSHTML.IHTMLElement
ie.Visible = True
ie.navigate "https://www.zillow.com/homes/new-jersey_rb/"
Do While ie.readyState <> READYSTATE_COMPLETE Or ie.Busy
Loop
Set doc = ie.document
Application.Wait Now() + #12:00:02 AM#
Set inys = doc.getElementsByTagName("input")(0)
inys.Focus
SendKeys ("35 Krakow St, Garfield, NJ 07026")
Application.Wait (Now + TimeValue("00:00:02"))
SendKeys ("{ENTER}")
End Sub
Result in IE:
I got the below vba code from this site. The code will automatically open the site in IE(will be using different url) and will import files as well. When I ran this code,it did not work due to incorrect elementsTagName and maybe InputType as well. What should be the correct codes? I am not sure. The second part is the html codes.
Please help check the codes.
Sub File_Test()
Dim HTMLDoc As MSHTML.HTMLDocument
Dim HTMLButtons As MSHTML.IHTMLElementCollection
Dim HTMLButton As MSHTML.IHTMLElement
Dim btnInput As MSHTML.IHTMLInputElement
Dim ie As Object
Dim pointer As Integer
Set ie = CreateObject("internetexplorer.application")
ie.Visible = True
ie.navigate "http://www.htmlquick.com/reference/tags/input-file.html"
Do While ie.readyState <> READYSTATE_COMPLETE
Loop
Set HTMLDoc = ie.document
Set HTMLButtons = HTMLDoc.getElementsByTagName("Upload Files")
For Each HTMLButton In HTMLButtons
For Each btnInput In HTMLButtons
If btnInput.Type = "button" Then
HTMLButton.Click btnInput.Value = "C:\temp\test.txt"
pointer = 1
Exit For
End If
Next
btnInput
If pointer = 1 Then Exit For
Next
End sub
<button title="Upload Files" class="button button--white xc-action-nav__button ng-binding ng-scope" type="button" loading-key="0" ng-click="setLoading('group1', '0'); " ng-disabled="isLoading('group1', null)"><span class="spinner-transition" ng-class="{'spinner spinner--is-loading': isLoading('group1', '0')}"></span> Upload Files</button>
Some pointers:
This HTMLButton.Click btnInput.Value = "C:\temp\test.txt" should be on two separate lines.
HTMLButton.Click
btnInput.Value = "C:\temp\test.txt"
I think the second line is redundant. Your code is missing the more complex instructions required to interact with the file dialog for inputing the filepath. A subject for a whole other question which has been addressed elsewhere on SO.
The "buttons" are input tag elements and you need to limit to the appropriate three.
Dim uploads As Object, i As Long
Set uploads = ie.document.querySelectorAll("#examples [type=file]")
For i = 0 To uploads.Length-1
uploads.item(i).click
'other code
Next
My advice would be to try and code with the actual url and share the problems you are having with the actual scenario. The code above, IMO, is not fit for purpose for the url given.
I tested it but it is not working. I received Run Time Error '-2147417848 (80010108). THE OBJECT INVOKED HAS DISCONNECTED FROM ITS CLIENTS after I clicked F8 in the Loop line.. Below is the new code as per your suggestion. What is causing this issue? Apologies if I postes my comment here as a simple Add Comment does not allow me to put the codes. Thanks for your time!
Private Sub CommandButton21_Click() Dim ie As Object Dim uploads As Object, i As Long Set ie = CreateObject("internetexplorer.application") ie.Visible = True ie.navigate "http://www.htmlquick.com/reference/tags/input-file.html" Do While ie.readyState <> 4 Loop Set uploads = ie.document.querySelectorAll("#examples [type=file]") For i = 0 To uploads.Length - 1 uploads.Item(i).Click'other code Next End Sub
I have written vba code for entering manufacturer part number in search box of below website and clicking on search icon. It is able enter manufacturer part number in search box and click on search icon, but when "search icon is clicked the text entered in the text box is not picked up". It searches empty data.
'HTML Part for search icon
<em class="fa fa-search" aria-hidden="true" style="color: gray;"></em>
It being almost a month I have tried various different way which was also mentioned on stack overflow, like using "createEvent("keyboardevent")" but nothing worked.
' VBA code
Sub AptivScrapping()
Dim IE As SHDocVw.InternetExplorer
Set IE = New InternetExplorer
IE.Visible = True
IE.navigate "https://ecat.aptiv.com"
Do While IE.readyState < READYSTATE_COMPLETE
Loop
Dim idoc As MSHTML.HTMLDocument
Set idoc = IE.document
idoc.getElementById("searchUserInput").Value = "33188785"
Dim doc_ele As MSHTML.IHTMLElement
Dim doc_eles As MSHTML.IHTMLElementCollection
Set doc_eles = idoc.getElementsByTagName("a")
For Each doc_ele In doc_eles
If doc_ele.getAttribute("ng-click") = "SearchButtonClick(1)" Then
doc_ele.Click
Exit Sub
Else
End If
Next doc_ele
End Sub
The page does an xhr request to retrieve the search results. You can find it in the network tab after clicking submit. This means you can avoid, in this case, the expense of a browser and issue an xhr request. The response is json so you do need a json parser to handle the results.
I would use jsonconverter.bas to parse the json. After installing the code from that link in a standard module called JsonConverter, go to VBE > Tools > References > Add a reference to Microsoft Scripting Runtime
I dimension an array to hold the results. I determine rows from the number of items in the json collection returned and the number of columns from the size of the first item dictionary. I loop the json object, and inner loop the dictionary keys of each dictionary in collection, and populate the array. I write the array out in one go at end which is less i/o expensive.
Option Explicit
Public Sub GetInfo()
Dim json As Object, ws As Worksheet, headers()
Dim item As Object, key As Variant, results(), r As Long, c As Long
Set ws = ThisWorkbook.Worksheets("Sheet1")
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://ecat.aptiv.com/json/eCatalogSearch/SearchProducts?filter=All&options=&pageSize=10&search=33188785", False
.send
Set json = JsonConverter.ParseJson(.responseText)("Products")
End With
headers = json.item(1).keys
ReDim results(1 To json.Count, 1 To UBound(headers) + 1)
For Each item In json
r = r + 1: c = 1
For Each key In item.keys
results(r, c) = item(key)
c = c + 1
Next
Next
With ws
.Cells(1, 1).Resize(1, UBound(headers) + 1) = headers
.Cells(2, 1).Resize(UBound(results, 1), UBound(results, 2)) = results
End With
You can do this instead:
txt = "33188785"
IE.navigate "https://ecat.aptiv.com/feature?search=" & txt
This will take you straight to the Search Result.
Code:
Sub AptivScrapping()
Dim IE As SHDocVw.InternetExplorer
Dim txt As String
Set IE = New InternetExplorer
txt = "33188785"
IE.Visible = True
IE.navigate "https://ecat.aptiv.com/feature?search=" & txt
Do While IE.Busy
Application.Wait DateAdd("s", 1, Now)
Loop
End Sub
This will be faster as You will only have to load one page.
Why that's happening, i am not sure, but seems like the TextBox that is used to input text is not being Activated when adding text automatically to it. It is being activated when we click inside it.
I got the solution for above problem from Mrxel.com below is the link for that post.
https://www.mrexcel.com/forum/excel-questions/1105434-vba-ie-automation-issue-angularjs-input-text-post5317832.html#post5317832
In this case I need to enter the search string character by character and sendKeys and input events inside the loop. Below is the working vba code.
Sub AptivScrapping()
Dim IE As SHDocVw.InternetExplorer
Set IE = New InternetExplorer
IE.Visible = True
IE.navigate "https://ecat.aptiv.com"
Do While IE.readyState < READYSTATE_COMPLETE
Loop
Dim idoc As MSHTML.HTMLDocument
Set idoc = IE.document
IE.document.getElementById("searchUserInput").Focus = True
IE.document.getElementById("searchUserInput").Select
sFieldInput = "33188785"
For s = 1 To Len(sFieldInput)
Application.SendKeys Mid(sFieldInput, s, 1)
While IE.readyState < 4 Or IE.Busy
Application.Wait DateAdd("s", LoopSeconds, Now)
Wend
Next s
IE.document.getElementById("searchUserInput").Focus = False
Dim doc_ele As MSHTML.IHTMLElement
Dim doc_eles As MSHTML.IHTMLElementCollection
Set doc_eles = idoc.getElementsByTagName("a")
For Each doc_ele In doc_eles
If doc_ele.getAttribute("ng-click") = "SearchButtonClick(1)" Then
doc_ele.Click
Exit Sub
Else
End If
Next doc_ele
End Sub
I would like to scrape a website (extract a product price) from a single website page (with XML HTTP request). But before this script should run I need to have selected the correct store first (saved in browser cookie variable or included in any other way/request if possible) since prices are different in different shops.
I have created a working code but it's taking a very long time to run so i assume there must be faster and cleaner :) way. I also needed to include the application to wait for the website to follow the steps.
My current vba code:
runs a HTTP IE request to open the website, and in multiple clicks selects the desired store and saves it in a cookie (like a site user should do)
next the product page is requested with another HTTP IE request and data is extracted. I found out a can't use the XML HTTP request because it won't use the cookie value with the correct store, displaying the correct price.
The price i'm after (in the example below) is E 1,39 instead of E 1,48 (when no cookie value is used and no store is selected).
The cookie value is saved in the cookie "www.jumbo.com/cookie/HomeStore the Content is holding the store tag which is known upfront and could be hardcoded in a request if possible.
Selecting the correct store (and saving it in a browser cookie)
Sub SetStore()
Dim IE As New SHDocVw.InternetExplorer
Dim HTMLDoc As MSHTML.HTMLDocument
Dim HTMLSearchbox As MSHTML.IHTMLElement
Dim HTMLSearchboxes As MSHTML.IHTMLElementCollection
Dim HTMLButton As MSHTML.IHTMLElement
Dim HTMLButtons As MSHTML.IHTMLElementCollection
Dim HTMLSearchButton As MSHTML.IHTMLElement
Dim HTMLSearchButtons As MSHTML.IHTMLElementCollection
Dim HTMLStoreID As MSHTML.IHTMLElement
Dim HTMLStoreIDs As MSHTML.IHTMLElementCollection
Dim HTMLSaveStore As MSHTML.IHTMLElement
Dim HTMLSaveStores As MSHTML.IHTMLElementCollection
'set on False to hide IE screen
IE.Visible = True
'navigate to url with limited content
IE.navigate "https://www.jumbo.com/content/algemene-voorwaarden/"
Do While IE.readyState <> READYSTATE_COMPLETE
Loop
Set HTMLDoc = IE.document
Set HTMLButtons = HTMLDoc.getElementsByTagName("button")
For Each HTMLButton In HTMLButtons
If HTMLButton.getAttribute("data-jum-action") = "openHomeStoreFinder" Then
HTMLButton.Click
Exit For
End If
Next HTMLButton
Application.Wait Now + #12:00:02 AM#
Set HTMLSearchboxes = HTMLDoc.getElementsByTagName("input")
For Each HTMLSearchbox In HTMLSearchboxes
If HTMLSearchbox.getAttribute("id") = "searchTerm__DkKYx4XylsAAAFJktpb2Guy" Then
'input field store name/location to show search results
HTMLSearchbox.Value = "Oosterhout"
Application.Wait Now + #12:00:03 AM#
HTMLSearchbox.Click
Exit For
End If
Next HTMLSearchbox
Set HTMLSearchButtons = HTMLDoc.getElementsByTagName("button")
For Each HTMLSearchButton In HTMLSearchButtons
If HTMLSearchButton.getAttribute("data-jum-filter") = "search" Then
HTMLSearchButton.Click
Exit For
End If
Next HTMLSearchButton
Application.Wait Now + #12:00:05 AM#
Set HTMLStoreIDs = HTMLDoc.getElementsByTagName("li")
For Each HTMLStoreID In HTMLStoreIDs
'oosterhout = YC8KYx4XB88AAAFIDcIYwKxJ
'nieuwegein = 84IKYx4XziUAAAFInSYYwKrH
'vaassen = JYYKYx4XC1oAAAFItvcYwKxJ
'brielle = OG8KYx4XP4wAAAFIlsEYwKxK
If HTMLStoreID.getAttribute("data-jum-store-id") = "YC8KYx4XB88AAAFIDcIYwKxJ" Then
HTMLStoreID.Click
Application.Wait Now + #12:00:03 AM#
Exit For
End If
Next HTMLStoreID
Set HTMLSaveStores = HTMLDoc.getElementsByTagName("button")
For Each HTMLSaveStore In HTMLSaveStores
If HTMLSaveStore.getAttribute("data-jum-action") = "saveHomeStore" Then
HTMLSaveStore.Click
Exit For
End If
Next HTMLSaveStore
'IE.Quit
End Sub
Extracting data from product page (IE HTTP request, working with cookie store value)
Sub GetJumboPriceIE()
Dim IE As New SHDocVw.InternetExplorer
Dim HTMLDoc As MSHTML.HTMLDocument
Dim JumInputs As MSHTML.IHTMLElementCollection
Dim JumInput As MSHTML.IHTMLElement
Dim JumPrice As MSHTML.IHTMLElement
Dim JumboPrice As Double
Dim Price_In_Cents_Tag As String
Dim SKU_tag As String, SKU_url As String
SKU_tag = "173140KST"
SKU_url = "https://www.jumbo.com/lu-bastogne-koeken-original-260g/173140KST/"
IE.Visible = False
IE.navigate SKU_url
Do While IE.readyState <> READYSTATE_COMPLETE
Loop
Set HTMLDoc = IE.document
IE.Quit
Set JumInputs = HTMLDoc.getElementsByTagName("input")
Price_In_Cents_Tag = "PriceInCents_" & SKU_tag
Set JumPrice = HTMLDoc.getElementById(Price_In_Cents_Tag)
JumboPrice = JumPrice.getAttribute("value") / 100
Debug.Print JumboPrice
End Sub
The code above is working but would like to use XML HTTP request code like below (but using the correct store). The price of 1,39 is printed.
Extracting data from product page (using XML HTTP request), but cookie value is not used
Sub GetJumboPriceXML()
Dim XMLReq As New MSXML2.XMLHTTP60
Dim HTMLDoc As New MSHTML.HTMLDocument
Dim JumInputs As MSHTML.IHTMLElementCollection
Dim JumInput As MSHTML.IHTMLElement
Dim JumPrice As MSHTML.IHTMLElement
Dim JumboPrice As Double
Dim Price_In_Cents_Tag As String
Dim SKU_tag As String, SKU_url As String
SKU_tag = "173140KST"
SKU_url = "https://www.jumbo.com/lu-bastogne-koeken-original-260g/173140KST/"
XMLReq.Open "GET", SKU_url, False
XMLReq.send
If XMLReq.Status <> 200 Then
MsgBox "Problem" & vbNewLine & XMLReq.Status & " - " & XMLReq.statusText
Exit Sub
End If
HTMLDoc.body.innerHTML = XMLReq.responseText
Set JumInputs = HTMLDoc.getElementsByTagName("input")
Price_In_Cents_Tag = "PriceInCents_" & SKU_tag
Set JumPrice = HTMLDoc.getElementById(Price_In_Cents_Tag)
JumboPrice = JumPrice.getAttribute("value") / 100
Debug.Print JumboPrice
End Sub
This code is not using the correct store and outputting the price i'm not after (The price 1,48 is printed).
To summarize:
When no store is selected (no cookie set) the following URL now gives the price of €1,48.
I would like the VB script to set the store to “Jumbo Oosterhout Nieuwe Bouwlingstraat” and then scrape a predefined list op product URL’s and extract the prices (URL above gives €1,39).
Then set the store to a different local store “Jumbo Brielle Thoelaverweg” and scrape the identical list of product URL’s. The above URL gives €1,48.
You can select a different store by clicking on the location pin icon at the top right of the page.
Thanks a lot for your help
I've been searching for how to press save when downloading file with IE and save the file under specific name in a specific location.
This is the screen as fas as I go with vba code
My code looks like this so far:
Sub BrowseToSite()
Dim IE As New SHDocVw.InternetExplorer
Dim HTMLDoc As MSHTML.HTMLDocument
Dim HTMLInput As MSHTML.IHTMLElement
Dim HTMLButtons As MSHTML.IHTMLElementCollection
Dim HTMLButton As MSHTML.IHTMLElement
IE.Visible = True
IE.navigate "cboe.com/delayedquote/quote-table-download"
Do While IE.readyState <> READYSTATE_COMPLETE
Loop
Set HTMLDoc = IE.document
Set HTMLInput = HTMLDoc.getElementById("ContentTop_C005_txtTicker")
HTMLInput.Value = "DJX"
Set HTMLButtons = HTMLDoc.getElementsByClassName("button blue")
HTMLButtons(0).Click
End Sub
All other sources i have looked didn't worked for me, maybe I am missing something. SendKeys didn't worked either which I think is supposed to be simplest way of achieving pressing save button. What code should i use?