I am trying to do some web scraping in Excel VBA. Here is the part of the code that I am having trouble with:
IE.Navigate URL
Do
DoEvents
Loop While IE.ReadyState <> 4 Or IE.Busy = True
Set doc = IE.document
After running this doc contains html that still has unexecuted JavasScript in it.
This is the signature of the script that has not been executed:
<SCRIPT type=text/javascript>
goosSearchPage.Initialize(...)...;
</SCRIPT>
I can wait for execution by doing Application.Wait(Now + TimeValue(x)) but that really is not satisfactory, as the amount of time the script takes to execute is quite variable depending on input.
Is there a way to either wait for the script to finish evaluating or to just evaluate the script directly in the doc object?
I found code that does wait for a page to complete. per the notes here, it requires the Microsoft Internet Controls as a reference in your code.
Code reproduced here, just in case the link dies:
'Following code goes into a sheet or thisworkbook class object module
Option Explicit
'Requires Microsoft Internet Controls Reference Library
Dim WithEvents ie As InternetExplorer
Sub start_here()
Set ie = New InternetExplorer
'Here I wanted to show the progress, so setting ie visible
ie.Visible = True
'First URL to go, next actions will be executed in
'Webbrowser event sub procedure - DocumentComplete
ie.Navigate "www.google.com"
End Sub
Private Sub ie_DocumentComplete(ByVal pDisp As Object, URL As Variant)
'pDisp is returned explorer object in this event
'pDisp.Document is HTMLDocument control that you can use
'Following is a choice to follow,
'since there is no do-loop, we have to know where we are by using some reference
'for example I do check the URL and do the actions according to visited URL
'In this sample, we use google entry page, set search terms, click on search button
'and navigate to first found URL
'First condition; after search is made
'Second condition; search just begins
If InStr(1, URL, "www.google.com/search?") > 0 Then
'Open the first returned page
ie.Navigate pDisp.Document.getelementsbytagname("ol")(0).Children(0).getelementsbytagname("a")(0).href
ElseIf InStr(1, URL, "www.google.com") > 0 Then
pDisp.Document.getelementsbyname("q")(0).Value = "VB WebBrowser DocumentComplete Event"
pDisp.Document.getelementsbyname("btnG")(0).Click
End If
End Sub
You actually can evaluate the javascript function with the ie window. But you gotta set up a Callback because the function will be evaluated async.
This post is quite old, but I'll answer this also as a reply to myself, now that I've discovered how to do this.
Simply point to a content you expect to be there after the jQuery script has run, trigger the desired event using JavaScript ran through IE automation, and do a Loop to wait until the desired content appears.
'This will trigger the jQuery event.
Doc.parentWindow.execScript "$('#optionbox').trigger('change')"
'This is the code that will make you wait. It's surprisingly efficient
Do While InStrB(Doc.getElementById("optionbox").innerHTML, "<desired html tag>") = 0
DoEvents
Loop
Related
I am currently having trouble trying to insert text into the website "www.skyvector.com". I've been trying to paste some text in the "Route" field, which appears in a grey box at the top left (usually after clicking 'Flight Plan').
This is the code that I have so far, which has worked for other websites, but strangely not for SkyVector:
Sub test1()
Dim IE As Object
Dim doc As HTMLDocument
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.navigate "http://www.skyvector.com/"
Do While IE.Busy
Application.Wait DateAdd("s", 1, Now)
Loop
Set doc = IE.document
doc.getElementsById("sv_planEditField").Value = "test"
End Sub
Unfortunately, an error appears whenever this line is set to run:
doc.getElementsById("sv_planEditField").Value = "test"
The error is "Run-time error '438': Object doesn't support this property or method".
Been wracking my head for a solution to this, and I couldn't find any solution here as well, specifically for websites that work like SkyVector. I am not exactly sure what the difference is between that and any other website. Thank you very much for your time!
First of all, the method name is not getElementsById(). The name is getElementById() without the s for plural. The reason is, an ID should be only used once in a html document, it's unique.
But if you use the right name you will receive the error that there is no object. The reason here is, there is no element with an ID named sv_planEditField.
So what can you do? You can use another method called getElementsByClassName() because the html line in question is
<input autocomplete="false" spellcheck="false" class="sv_search" autocorrect="off">
The method getElementsByClassName() buids a node collection. Therefore it uses the s for plural. There can be as many elements with the same class name as the developer want. You can get a specific element by it's index like you use it with an array. The clss name sv_search is only once used in the document. A node collections first index is allways 0. So you must use the following line of vba code, instead of yours:
doc.getElementsByClassName("sv_search")(0).Value = "test"
Edit
After reading your question again and understand it ;-) and based on the answer of Sam here is the way you can solve your problem. What you need is a new text node and (I think) to trigger the right event to make the input work for the page. Try it with original data.
Sub test1()
Dim IE As Object
Dim textToEnter As Object
Dim nodeToAppendText As Object
Dim nodeText As Object
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.navigate "http://www.skyvector.com/"
Do While IE.Busy
Application.Wait DateAdd("s", 1, Now)
Loop
'Open overlay to enter data
IE.document.getElementsByClassName("sv_topbarlink")(0).Click
'Click textfield to hide helptext and place curser
IE.document.getElementsByClassName("svfpl_helpmessage")(0).Click
'Create a text node which belongs to the document
Set textToEnter = IE.document.createTextNode("Test")
'Get the node you want to append the new text node
Set nodeToAppendText = IE.document.getElementById("sv_planEditField")
'Append the new text node
Set nodeText = nodeToAppendText.appendChild(textToEnter)
'Not sure if it is necessary to trigger an event
'But there are two events in question:
' First one is input
' Second one is keypress
'You must try how it works
Call TriggerEvent(IE.document, nodeToAppendText, "input")
End Sub
If needed use this method to trigger any event:
Private Sub TriggerEvent(htmlDocument As Object, htmlElementWithEvent As Object, eventType As String)
Dim theEvent As Object
htmlElementWithEvent.Focus
Set theEvent = htmlDocument.createEvent("HTMLEvents")
theEvent.initEvent eventType, True, False
htmlElementWithEvent.dispatchEvent theEvent
End Sub
The element sv_planEditField is not a normal text box. Open it in your browser and inspect it with the developer tools (Press F12). Do this both before and after filling it. You will notice that this is quite different from a standard input. Either recreate the html structure of the filled control or recreate the form submission. Have a look at createElement and appendChild for more information.
I am trying to create a macro that scrapes a cargo tracking website.
But I have to create 4 such macros as each airline has a different website.
I am new to VBA and web scraping.
I have put together a code that works for 1 website. But when I tried to replicate it for another one, I am stuck in the loop. I think it maybe how I am referring to the element, but like I said, I am new to VBA and have no clue about HTML.
I am trying to get the "notified" value in the highlighted line from the image.
IMAGE:"notified" text to be extracted
Below is the code I have written so far that gets stuck in the loop.
Any help with this would be appreciated.
Sub FlightStat_AF()
Dim url As String
Dim ie As Object
Dim nodeTable As Object
'You can handle the parameters id and pfx in a loop to scrape dynamic numbers
url = "https://www.afklcargo.com/mycargo/shipment/detail/057-92366691"
'Initialize Internet Explorer, set visibility,
'call URL and wait until page is fully loaded
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = False
ie.navigate url
Do Until ie.readyState = 4: DoEvents: Loop
'Wait to load dynamic content after IE reports it's ready
'We can do that in a loop to match the point the information is available
Do
On Error Resume Next
Set nodeTable = ie.document.getElementByClassName("block-whisper")
On Error GoTo 0
Loop Until Not nodeTable Is Nothing
'Get the status from the table
MsgBox Trim(nodeTable.getElementsByClassName("fs-12 body-font-bold").innerText)
'Clean up
ie.Quit
Set ie = Nothing
Set nodeTable = Nothing
End Sub
Some basics:
For simple accesses, like the present ones, you can use the get methods of the DOM (Document Object Model). But there is an important difference between getElementByID() and getElementsByClassName() / getElementsByTagName().
getElementByID() searches for the unique ID of a html tag. This is written as the ID attribute to html tags. If the html standard is kept by the page, there is only one element with this unique ID. That's the reason why the method begins with getElement.
If the ID is not found when using the method, VBA throws a runtime error. Therefore the call is encapsulated in the loop from the other answer from me, into switching off and on again the error handling. But in the page from this question there is no ID for the html area in question.
Instead, the required element can be accessed directly. You tried the access with getElementsByClassName(). That's right. But here comes the difference to getElementByID().
getElementsByClassName() and getElementsByTagName() begin with getElements. Thats plural because there can be as many elements with the same class or tag name as you want. This both methods create a html node collection. All html elements with the asked class or tag name will be listet in those collections.
All elements have an index, just like an array. The indexes start at 0. To access a particular element, the desired index must be specified. The two class names fs-12 body-font-bold (class names are seperated by spaces, you can also build a node collection by using only one class name) deliver 2 html elements to the node collection. You want the second one so you must use the index 1.
This is the VBA code for the asked page by using the IE:
Sub FlightStat_AF()
Dim url As String
Dim ie As Object
'You can handle the parameters id and pfx in a loop to scrape dynamic numbers
url = "https://www.afklcargo.com/mycargo/shipment/detail/057-92366691"
'Initialize Internet Explorer, set visibility,
'call URL and wait until page is fully loaded
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = False
ie.navigate url
Do Until ie.readyState = 4: DoEvents: Loop
'Wait to load dynamic content after IE reports it's ready
'We do that with a fix manual break of a few seconds
'because the whole page will be "reload"
'The last three values are hours, minutes, seconds
Application.Wait (Now + TimeSerial(0, 0, 3))
'Get the status from the table
MsgBox Trim(ie.document.getElementsByClassName("fs-12 body-font-bold")(1).innerText)
'Clean up
ie.Quit
Set ie = Nothing
End Sub
Edit: Sub as function
This sub to test the function:
Sub testFunction()
Dim flightStatAfResult As String
flightStatAfResult = FlightStat_AF("057-92366691")
MsgBox flightStatAfResult
End Sub
This is the sub as function:
Function FlightStat_AF(cargoNo As String) As String
Dim url As String
Dim ie As Object
Dim result As String
'You can handle the parameters id and pfx in a loop to scrape dynamic numbers
url = "https://www.afklcargo.com/mycargo/shipment/detail/" & cargoNo
'Initialize Internet Explorer, set visibility,
'call URL and wait until page is fully loaded
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = False
ie.navigate url
Do Until ie.readyState = 4: DoEvents: Loop
'Wait to load dynamic content after IE reports it's ready
'We do that with a fix manual break of a few seconds
'because the whole page will be "reload"
'The last three values are hours, minutes, seconds
Application.Wait (Now + TimeSerial(0, 0, 3))
'Get the status from the table
result = Trim(ie.document.getElementsByClassName("fs-12 body-font-bold")(1).innerText)
'Clean up
ie.Quit
Set ie = Nothing
'Return value of the function
FlightStat_AF = result
End Function
I'm trying to access the document object of a frame with JS using WebBrowser. The code below works perfectly, but I need to use WebBrowser instead.
This is the error I've got:
Run-time error '438'
Object doesn't support this property or method
Any guidance please?
Sub GrabWorkerId1()
Dim objIE As InternetExplorerMedium
Set objIE = New InternetExplorerMedium
objIE.Visible = True
objIE.Navigate "my url"
Do While objIE.Busy = True Or objIE.ReadyState <> 4: DoEvents: Loop
Open "c:\temp\GrabWorkerId2.log" For Output As #3
Write #3, objIE.Document.getElementsbyTagName("iframe")(1).contentDocument.getElementsbyTagName("select")(0).innerhtml
Close 3
Set objIE = Nothing
End Sub
Object doesn't support this property or method
Perhaps the issue is related the Close method and the related method to access the html elements.
First, please try to use F12 developer tools to check whether the website contains the iframe tag and the select element, and check the account. According to your code, the page should contain at least two iframe tag and the iframe content page will contains at least one select element, please verify it. Also, you could add a debugger from the VBA script, and check the innerhtml value, make sure you could get the value.
Second, about the Close method, try to modify the code as below:
Close #3
Besides, please check the following code, I have tested it on my side, it works well (the website contains one iframe tag, and the iframe content page just contains one select element):
Sub GrabWorkerId1()
Dim objIE As Object
Set objIE = CreateObject("InternetExplorer.Application")
objIE.Visible = True
objIE.Navigate "<website url>"
While objIE.ReadyState <> 4
DoEvents
Wend
Open "d:\temp\test.log" For Output As #3
Write #3, objIE.Document.getElementsbyTagName("iframe")(0).contentDocument.getElementsbyTagName("select")(0).innerhtml
Close #3
Set objIE = Nothing
End Sub
I'm trying to pull data from multiple webpages (different stock pages from the same site). I can get the data pulled for the first 3 times the loop is executed but on the 4th iteration it brings up error 91: Object Variable or With block Variable not set up.
I tried moving around the internet explorer opening command so that it opens a new browser at the beginning of each iteration, and closes it at the end of the loop, to make sure the IE object wasn't somehow failing. That didn't work, same issue.
Sub GetStock()
Dim ws As Worksheet: Set ws = ActiveSheet
Dim cellnum As Range: Set cellnum = Range(ActiveCell.Address)
Dim i As Integer
Dim IE As Object
Dim text As String
i = 1
Do Until i > 10
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
cellnum = Range(ActiveCell.Offset(i, 7).Address)
With IE
.navigate cellnum.Value
Do While .Busy And .readyState <> 4: DoEvents: Loop
Sleep 1000
text = .Document.getElementsByClassName("classname")(1).outerText
End With
ws.Cells(i, 12).Value = text
i = i + 1
IE.Quit
Loop
End Sub
The links to the webpage are held within cells, hence the cellnum code. Finds the correct cell, retrieves the webpage within it, then moves on to the cell below it. The code is working perfectly for the first 3 iterations but for some reason fails on the 4th. The error code identifies the "text=.document.getElementsByClassName..." line as the error.
I think your issue is probably due to the element not existing on the webpage. If it does exist, are you sure you are pulling the right element from the collection?
Try running it with
.document.getElementsByClassName("classname")(0).outerText.
If that works then I would suggest looking at how many elements with the class "classname" are on the webpage. While on the other pages you may have 2 or more elements, it could be that on the 3rd page you only have one.
Can you post the webpages you are scraping?
Found the solution! The Sleep.1000 command wasn't providing enough time in all cases, and I guess the code was trying to pull data before a page was available. I thought the loop in there would solve that but I guess not (very new to this). Anyways, I changed it to Sleep.3000 to give my slow internet enough time to catch up and its working like a dream.
Thanks for all the help everyone.
There are many online resources that illustrate using Microsoft Internet Explorer Controls within VBA Excel to perform basic IE automation tasks. These work when the webpage has a basic construct. However, when webpages contain multiple frames they can be difficult to work with.
I need to determine if an individual frame within a webpage has completely loaded. For example, this VBA Excel code opens IE, loads a webpage, loops thru an Excel sheet placing data into the webpage fields, executes search, and then returns the IE results data to Excel (my apologies for omitting the site address).
The target webpage contains two frames:
1) The searchbar.asp frame for search value input and executing search
2) The searchresults.asp frame for displaying search results
In this construct the search bar is static, while the search results change according to input criteria. Because the webpage is built in this manner, the IEApp.ReadyState and IEApp.Busy cannot be used to determine IEfr1 frame load completion, as these properties do not change after the initial search.asp load. Therefore, I use a large static wait time to avoid runtime errors as internet traffic fluctuates. This code does work, but is slow. Note the 10 second wait after the cmdGO statement. I would like to improve the performance by adding solid logic to determine the frame load progress.
How do I determine if an autonomous frame has finished loading?
' NOTE: you must add a VBA project reference to "Internet Explorer Controls"
' in order for this code to work
Dim IEapp As Object
Dim IEfr0 As Object
Dim IEfr1 As Object
' Set new IE instance
Set IEapp = New InternetExplorer
' With IE object
With IEapp
' Make visible on desktop
.Visible = True
' Load target webpage
.Navigate "http://www.MyTargetWebpage.com/search.asp"
' Loop until IE finishes loading
While .ReadyState <> READYSTATE_COMPLETE
DoEvents
Wend
End With
' Set the searchbar.asp frame0
Set IEfr0 = IEapp.Document.frames(0).Document
' For each row in my worksheet
For i = 1 To 9999
' Input search values into IEfr0 (frame0)
IEfr0.getElementById("SearchVal1").Value = Cells(i, 5)
IEfr0.getElementById("SearchVal2").Value = Cells(i, 6)
' Execute search
IEfr0.all("cmdGo").Click
' Wait a fixed 10sec
Application.Wait (Now() + TimeValue("00:00:10"))
' Set the searchresults.asp frame1
Set IEfr1 = IEapp.Document.frames(1).Document
' Retrieve webpage results data
Cells(i, 7) = Trim(IEfr1.all.Item(26).innerText)
Cells(i, 8) = Trim(IEfr1.all.Item(35).innerText)
Next
As #JimmyPena said. it's a lot easier to help if we can see the URL.
If we can't, hopefully this overview can put you in the right direction:
Wait for page to load (IEApp.ReadyState and IEApp.Busy)
Get the document object from the IE object. (done)
Loop until the document object is not nothing.
Get the frame object from the document object.
Loop until the frame object is not nothing.
Hope this helps!
I used loop option to check the field value until its populated like this
Do While IE.Document.getElementById("USERID").Value <> "test3"
IE.Document.getElementById("USERID").Value = "test3"
Loop
this is a Rrrreeally old thread, but I figured I would post my findings, because I came here looking for an answer...
Looking in the locals window, I could see that the "readystate" variable was only "READYSTATE_COMPLETE" for the IE App itself. but for the iframe, it was lowercase "complete"
So I explored this by using a debug.print loop on the .readystate of the frame I was working with.
Dim IE As Object
Dim doc As MSHTML.HTMLDocument
Set doc = IE.Document
Dim iframeDoc As MSHTML.HTMLDocument
Set iframeDoc = doc.Frames("TheFrameIwasWaitingFor").Document
' then, after I had filled in the form and fired the submit event,
Debug.Print iframeDoc.readyState
Do Until iframeDoc.readyState = "complete"
Debug.Print iframeDoc.readyState
DoEvents
Loop
So this will show you line after line of "loading" in the immediate window, eventually showing "complete" and ending the loop. it can be abridged to remove the debug.prints of course.
another thing:
debug.print iframeDoc.readystate ' is the same as...
debug.print doc.frames("TheFrameIwasWaitingFor").Document.readystate
' however, you cant use...
IE.Document.frames("TheFrameIwasWaitingFor").Document.readystate ' for some reason...
forgive me if all of this is common knowledge. I really only picked up VBA scripting a couple days ago...