I'm trying to make a web crawler. I want to evaluate if the web page has loaded.
I tried ie.busy not working well [VBA]. It worked the first time.
I changed my sub to be a function, and calling this function from a new sub I'm facing the trouble again.
Public Enum READYSTATE
READYSTATE_UNINITIALIZED = 0
READYSTATE_LOADING = 1
READYSTATE_LOADED = 2
READYSTATE_INTERACTIVE = 3
READYSTATE_COMPLETE = 4
End Enum
Function CrawlerNasdaq(strCountry As String) As Double
Dim ie As InternetExplorer
Dim strURL As String
Dim x
strURL = "http://www.nasdaqomxnordic.com/indeks/historiske_priser?Instrument=" & strCountry
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = True
ie.Navigate strURL
Do While (ie.Busy Or ie.READYSTATE <> READYSTATE.READYSTATE_COMPLETE)
DoEvents
Loop
Set x = ie.Document.getElementById("historicalTable").getElementsByTagname("tr")(1).getElementsByTagname("td")(6)
CrawlerNasdaq = x.InnerText
ie.Quit
End Function
Sub HentIndex()
Range("H18") = CrawlerNasdaq("DK0016268840")
Range("I18") = CrawlerNasdaq("SE0000337842")
Range("J18") = CrawlerNasdaq("FI0008900212")
End Sub
It is stuck in the loop.
Related
How can I loop the script 1000 times to get the data in different cells and get URL value from different cells?
Private Sub CommandButton1_Click()
Dim ie As InternetExplorer
Dim ht As HTMLDocument
Set ie = New InternetExplorer
ie.Visible = True
ie.navigate Workbooks("FINAL-1").Worksheets("sheet1").Range("A2").Value
Do While ie.readyState <> READYSTATE_COMPLETE
Loop
Set ht = ie.document
Workbooks("FINAL-1").Worksheets("sheet1").Range("C").Value = ht.getElementById("addr1").Value
'Workbooks("FINAL-1").Worksheets("sheet1").Range("C2").Value = ht.getElementById("fullName").Value
'Workbooks("FINAL-1").Worksheets("sheet1").Range("d2").Value = ht.getElementById("addr1").Value
Workbooks("FINAL-1").Worksheets("sheet1").Range("D").Value = ht.getElementById("dob").Value
Workbooks("FINAL-1").Worksheets("sheet1").Range("E").Value = ht.getElementById("applyDte").Value
Workbooks("FINAL-1").Worksheets("sheet1").Range("B").Value = ht.getElementById("idNo").Value
'Workbooks("FINAL-1").Worksheets("sheet1").Range("G2").Value = ht.getElementById("genderMtdtId").Value
End Sub
Private Sub Worksheet_SelectionChange(ByVal Target As Range)
End Sub
Set ie = New InternetExplorer
ie.Visible = True
ie.navigate Workbooks("FINAL-1").Worksheets("sheet1").Range("A2").Value
Do While ie.readyState <> READYSTATE_COMPLETE
Loop
i am struggling on the following: I try to open an URL(Link), become redirected to a new URL and retrieve this new URL into a Cell in my excel worksheet.
I have written the following code but it is not retrieving the new URL and is not quitting the Internet Explorer at the end:
Sub Get_URL()
Dim ISIN As String
Dim Link As String
Dim IE As Object
ISIN = Range("A1").Value
Link = "https://www.finanzen.net/suchergebnis.asp?_search=" & ISIN
Set IE = CreateObject("InternetExplorer.Application")
With IE
.Visible = True
.Navigate Link
End With
Range("A2") = IE.LocationURL
Set IE = Nothing
IE.Quit
End Sub
The ISIN in A1 is KYG875721634.
I would be very glad if some of you guys could find the problem. Thank you very much!
Greetings, Robin
This works. Read the comments please:
Sub Get_URL()
Dim ISIN As String
Dim Link As String
Dim IE As Object
'ISIN = Range("A1").Value
ISIN = "KYG875721634"
Link = "https://www.finanzen.net/suchergebnis.asp?_search=" & ISIN
Set IE = CreateObject("InternetExplorer.Application")
With IE
.Visible = True
.Navigate Link
End With
'You must wait to load the page
Do While IE.readystate <> 4: DoEvents: Loop
'Range("A2") = IE.LocationURL
MsgBox IE.LocationURL
'First quit the IE
'because this step needs the reference ;-)
IE.Quit
Set IE = Nothing
End Sub
There're two issues in your code:
You need to add a wait to load page.
Do Until IE.readyState = 4
DoEvents
Loop
First quit IE. If you first set IE to Nothing then there will be no reference to IE when quit.
IE.Quit
Set IE = Nothing
The final code is like this which can work well:
Sub Get_URL()
Dim ISIN As String
Dim Link As String
Dim IE As Object
ISIN = Range("A1").Value
Link = "https://www.finanzen.net/suchergebnis.asp?_search=" & ISIN
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.Navigate Link
Do Until IE.readyState = 4
DoEvents
Loop
Range("A2") = IE.LocationURL
IE.Quit
Set IE = Nothing
End Sub
Result:
I have written some VBA code in Excel to retrieve the latitude and longitude from a Google Maps URL and paste it into a cell in my worksheet. My problem is in retrieving the URL from internet explorer. Below I have two examples of my code, one macro returns an about:blank as though the object doesn't have the LocationURL property, and the other example seems like it is saving all of my previous searches, so it cycles through all of my previous searches and pastes the very first searches' URL. Example 2 uses a shell suggestion that I found online to reassign the properties to the oIE object. I can get both to slightly work, but neither will do exactly what I need from the macro.
Cell(8,8) is a hyperlink to google maps where I'm searching an address, and Cell(8,9) is where I want to paste the URL after google maps has redirected and has the latitude and longitude in the URL.
Example 1:
Sub CommandButton1_Click()
Dim ie As Object
Dim Doc As HTMLDocument
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = True
ie.Navigate "http://www.google.com/maps?q=" & Range("I7").Value
Do
DoEvents
Loop Until ie.ReadyState = 4
Set Doc = ie.Document
Cells(8, 9).Value = ie.LocationName
End Sub
Example 2:
Sub Macro()
Dim oIE, oShell, objShellWindows, strPath, X
strPath = Cells(8, 8)
Set oIE = CreateObject("InternetExplorer.Application")
'This is to resolve oIE.navigate "about:blank" issue
oIE.Top = 0
oIE.Left = 0
oIE.Width = 500
oIE.Height = 500
oIE.Navigate strPath
Do While oIE.Busy And oIE.ReadyState < 2
DoEvents
Loop
'Reassigning oIE.LocationName & vbCrLf & oIE.LocationURL values after redirect in IE
Set oShell = CreateObject("WScript.Shell")
Set objShellWindows = CreateObject("Shell.Application").Windows
For X = objShellWindows.Count - 1 To 0 Step -1
Set oIE = objShellWindows.Item(X)
If Not oIE Is Nothing Then
If StrComp(oIE.LocationURL, strPath, 1) = 0 Then
Do While oIE.Busy And oIE.ReadyState < 2
DoEvents
Loop
oIE.Visible = 2
Exit For
End If
End If
Cells(8, 9).Value = oIE.LocationURL
Set oIE = Nothing
Next
Set objShellWindows = Nothing
Set oIE = Nothing
End Sub
Thanks,
Andrew
Is this as simple as looping until the document.URL changes? In my timed loop I wait for the string safe=vss in the original page load to disappear.
Option Explicit
Public Sub GetNewURL()
Dim IE As New InternetExplorer, newURL As String, t As Date
Const MAX_WAIT_SEC As Long = 5
With IE
.Visible = True
.navigate2 "http://www.google.com/maps?q=" & "glasgow" '<==Range("I7").Value
While .Busy Or .readyState < 4: DoEvents: Wend
t = Timer
Do
DoEvents
newURL = .document.URL
If Timer - t > MAX_WAIT_SEC Then Exit Do
Loop While InStr(newURL, "safe=vss") > 0
Debug.Print newURL
End With
End Sub
My code is able to interact with the first webpage it brings up, but then as soon as i navigate away from it, my code gives me a '424' object required error. Here is an example.
EX: go to yahoo.com - search for 'Earth' - click on the Yahoo logo to return to the yahoo homepage
Sub InternetPractice()
Dim ie As InternetExplorer
Set ie = New InternetExplorer
ie.Visible = True
ie.navigate "https://www.yahoo.com"
Do While ie.Busy = True Or ie.readyState <> 4: DoEvents: Loop
ie.document.getElementById("uh-search-box").Value = "Earth"
ie.document.getElementById("uh-search-button").Click
Do While ie.Busy = True Or ie.readyState <> 4: DoEvents: Loop
ie.document.getElementById("logo").Click
End Sub
Using VBA with the internet is new to me and any help is appreciated!
EDIT:
I think you are not giving enough time for ie to load... It is working for me.
So you can try this code that gives more time and try to load 3 times before aborting.
Option Explicit
Public Declare Sub Sleep Lib "kernel32" (ByVal dwMilliseconds As Long)
Sub InternetPractice2()
Dim IE As InternetExplorer
Dim HTMLDoc As HTMLDocument
Dim logo As Object
Dim n As Long
Set IE = New InternetExplorer 'Set IE = CreateObject("InternetExplorer.Application")
With IE
.Silent = True
.Visible = True
.Navigate "https://www.yahoo.com"
End With
WaitIE IE, 5000
Set HTMLDoc = IE.Document
HTMLDoc.getElementById("uh-search-box").Value = "Earth"
HTMLDoc.getElementById("uh-search-button").Click
load:
WaitIE IE, 10000
Set logo = HTMLDoc.getElementById("logo")
If Not logo Is Nothing Then
logo.Click
ElseIf logo Is Nothing And n < 4 Then
n = n + 1
GoTo load
End If
WaitIE IE, 5000
'Set ie to Nothing
IE.Quit
Set IE = Nothing
End Sub
Sub WaitIE(IE As Object, Optional time As Long = 250)
'Code from: https://stackoverflow.com/questions/33808000/run-time-error-91-object-variable-or-with-block-variable-not-set
Dim i As Long
Do
Sleep time
Debug.Print CStr(i) & vbTab & "Ready: " & CStr(IE.READYSTATE = 4) & _
vbCrLf & vbTab & "Busy: " & CStr(IE.Busy)
i = i + 1
Loop Until IE.READYSTATE = 4 Or Not IE.Busy
End Sub
Original Answer
Try closing ie in the end of the code and add some delay, because if you step the code with F8, you will see that it works:
Sub InternetPractice2()
Dim ie As InternetExplorer
Set ie = New InternetExplorer
ie.Visible = True
ie.navigate "https://www.yahoo.com"
Do While ie.Busy = True Or ie.READYSTATE <> 4: DoEvents: Loop
ie.Document.getElementById("uh-search-box").Value = "Earth"
ie.Document.getElementById("uh-search-button").Click
'Add delay After .Click
Do While ie.Busy = True Or ie.READYSTATE <> 4: DoEvents: Loop
Application.wait Now+Timevalue("00:00:05")
ie.Document.getElementById("logo").Click
'Set ie to Nothing
Set ie = Nothing
End Sub
Also close all iexplorer.exe on Windows Task Manager.
I'm trying to click a button multiple times to get my page loaded fully. Webpage contains a button SHOW MORE instead of next page so in html coding behind the button remains same.
I am using the following excel-vba code to hit that button. It actually did click that button, but instead of showing the next results it shows the same result over and over again.
Could you please kindly show me how to make it right. Thanks in advance!
**' VARIABLE DECLARATION
Dim IE As Object
Dim county As String
Dim htmlDoc As Object
' CREATING OBJECT
Set IE = CreateObject("internetexplorer.application")
' WEBPAGE NAVIGATION
With IE
.navigate ("http://www.physiofirst.org.uk/find-physio/search-physio.html")
.Visible = True
End With
' WAITING FOR WEBPAGE TO LOAD
Do
DoEvents
Loop Until IE.readystate = 4
' SEARCHING ALL THE THE INDIVIDUAL STATE PHYSICIANS
Set htmlDoc = IE.document
Set searchbarvalue = htmlDoc.getelementsbyclassname("form-control mod-text-display")
i = 0
For Each classSearch In searchbarvalue
searchbarvalue(i).Value = "BRISTOL"
Next classSearch
While IE.busy
DoEvents
Wend
Set buttonclick = htmlDoc.getelementsbyclassname("btn btn-search")
i = 0
For Each buttonsearch In buttonclick
buttonclick(i).Click
Next buttonsearch
While IE.busy
DoEvents
Wend
Do
htmlDoc.getelementbyid("load-more-practice").Click
While IE.busy
DoEvents
Wend
Loop Until htmlDoc.getelementbyid("load-more-practice").Click = True
End Sub**
Try this code that Do Until data-page is = "1", because it starts on 2 and go to 1 when is completely loaded, with all Show More clicked.
Public Declare Sub Sleep Lib "kernel32" (ByVal dwMilliseconds As Long)
Sub test()
Dim IE As Object
Dim county As String
Dim htmlDoc As Object
Dim sURL As String
' CREATING OBJECT
Set IE = CreateObject("internetexplorer.application")
sURL = "http://www.physiofirst.org.uk/find-physio/search-physio.html"
' WEBPAGE NAVIGATION
With IE
.navigate (sURL)
.Visible = True
End With
WaitIE IE, 2000
' SEARCHING ALL THE THE INDIVIDUAL STATE PHYSICIANS
Set htmlDoc = IE.document
Set searchbarvalue = htmlDoc.getElementsByClassName("form-control mod-text-display")
i = 0
For Each classSearch In searchbarvalue
searchbarvalue(i).Value = "BRISTOL"
Next classSearch
WaitIE IE, 1000
Set buttonclick = htmlDoc.getElementsByClassName("btn btn-search")
buttonclick(0).Click
WaitIE IE, 1000
Set ShowMore = htmlDoc.getElementById("load-more-practice")
Do
ShowMore.Click
WaitIE IE, 2000
Loop Until ShowMore.getAttribute("data-page") = 1
'IE.Quit
'Set IE = Nothing
End Sub
Sub WaitIE(IE As Object, Optional time As Long = 250)
'Code from: https://stackoverflow.com/questions/33808000/run-time-error-91-object-variable-or-with-block-variable-not-set
Dim i As Long
Do
Sleep time
Debug.Print CStr(i) & vbTab & "Ready: " & CStr(IE.readyState = 4) & _
vbCrLf & vbTab & "Busy: " & CStr(IE.Busy)
i = i + 1
Loop Until IE.readyState = 4 Or Not IE.Busy
End Sub