How to crawl Googlemaps from VBA Excel? - excel

As a real beginner in coding, I am looking for some help around here !
I would like to extract data from GoogleMaps: let's say, Zip code, from a company input in a cell in excel, through VBA.
As I started coding, I met difficulties at the string level as I can't target the accurate tag (here, I think, but thats the point, span) and effectively extract the zip code on the address line on Googlemaps HTML page :
Private Sub Worksheet_Change(ByVal Target As Range)
If Target.Row = Range("nom").Row And _
Target.Column = Range("nom").Column Then
Dim GM As New InternetExplorer
GM.Visible = True
GM.navigate "https://www.google.fr/maps/#43.3082377,5.4259519,17z?q=" & Range("nom").Value
Do
DoEvents
Loop Until GM.readyState = READYSTATE_COMPLETE
Dim doc As HTMLDocument
Set doc = GM.document
Dim sSPAN As String
sSPAN = Trim(doc.getElementsByTagName("span")(4).innerText)
MsgBox sSPAN
End If
End Sub
Line 13 displays the following errors :
Run Titre Error 91 : object variable or with block variable not set
as I can't select the accurate block.
Following this, I have two questions:
Is it effectively possible to extract data from Google Maps ? (Through VBA or an opensource Google API, extracting from Google sheet (10000-lines excel)
Has someone ever facde this kind of difficulty ? How can I select the accurate block on Googlemaps HTML page?

This is not how I would go about it and API is definitely preferable though not via GoogleMaps unless you are building an application where you can also embed the required map.
You need a proper page load wait, a wait for the element to be present, quit the application at the end, and use a different selector strategy. You also want to implement some logic to extract the post code from address (I am not sure what view you were getting. I used a test value of Enterome):
Option Explicit
Private Sub Worksheet_Change(ByVal Target As Range)
Dim t As Date, gm As New InternetExplorer
Const MAX_WAIT_SEC As Long = 10
If Target.Row = Range("nom").Row And Target.Column = Range("nom").Column Then
gm.Visible = True
gm.navigate "https://www.google.fr/maps/#43.3082377,5.4259519,17z?q=" & Range("nom").Value
While gm.Busy Or gm.readyState <> 4: DoEvents: Wend
Dim elem As Object
t = Timer
Do
On Error Resume Next
Set elem = gm.document.querySelector(".section-info-text")
On Error GoTo 0
If Timer - t > MAX_WAIT_SEC Then Exit Do
Loop While elem Is Nothing
If Not elem Is Nothing Then
MsgBox Trim$(elem.innerText)
'Implement logic to extract post code
End If
gm.Quit
End If
End Sub

Related

Excel VBA IE Object and using dropdown list

I am experimenting with web automation and struggling a bit trying to utilize a drop down list.
My code works up to the point of searching for a company name and hitting "go". On the new page I can't seem to find the right code that selects the group of elements that represents the drop down list. I then want to select "100" entries, but I can't even grab the nodes that represent this list.
I have been browsing multiple different pages on stackoverflow that talk about CSS selectors and looked at tutorials but that doesn't seem to help either. I either end up grabbing nothing, or whatever I grab can't use the getElementsByTagName method, which ultimately I am trying to drill down into the td and select nodes . Not sure what to do with those yet, but I can't even grab them. Thoughts?
(note stopline is just a line that I use a breakpoint on to stop my code)
CSS helper website: https://www.w3schools.com/cssref/trysel.asp
Code:
Option Explicit
Sub test()
On Error GoTo ErrHandle
Dim ie As New InternetExplorer
Dim doc As New HTMLDocument
Dim ws As Worksheet
Dim stopLine As Integer
Dim oSearch As Object, oSearchButton As Object
Dim oForm As Object
Dim oSelect As Object
Dim list As Object
Set ws = ThisWorkbook.Worksheets("Sheet1")
ie.Visible = True
ie.navigate "https://www.sec.gov/edgar/searchedgar/companysearch.html"
Do
DoEvents
Loop Until ie.readyState = READYSTATE_COMPLETE
Set doc = ie.Document
Set oSearch = doc.getElementById("companysearchform")
Set oSearchButton = oSearch.getElementsByTagName("input")(1)
Set oSearch = oSearch.getElementsByTagName("input")(0)
oSearch.Value = "Summit Midstream Partners, LP"
oSearchButton.Click
Do
DoEvents
Loop Until ie.readyState = READYSTATE_COMPLETE
Set doc = ie.Document
Set list = doc.querySelectorAll("td select")
stopLine = 1
Exit Sub
ErrHandle:
MsgBox Err.Number & " - " & Err.Description, vbCritical
Exit Sub
End Sub
td select will return a single node so you only need querySelector. The node has an id so you might as well use the quicker querySelector("#count") to target the parent select. To change the option you can then use SelectedIndex on the parent select, or, target the child option by its value attribute querySelector("[value='100']").Selected = True. You may then need to attach and trigger change/onchange htmlevent to the parent select to register the change.
However, I would simply extract the company CIK from current page then concatenate the count=100 param into the url and .Navigate2 that using following format:
https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0001549922&type=&dateb=&owner=include&count=100&search_text=
You can extract CIK, after initial search company click and wait for page load, with:
Dim cik As String
cik = ie.document.querySelector("[name=CIK]").value
ie.Navigate2 "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=" & cik & "&type=&dateb=&owner=include&count=100&search_text="
Given several params are left blank you can likely shorten to:
"https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=" & cik & "&owner=include&count=100"
If you are unable to get the initial parent select you probably need a timed loop waiting for that element to be present after clicking the search button. An example is shown here in a StackOverflow answer.

How to input values into dropdown box of web page using Excel VBA

I'm trying to operate a website to display desired option chain data with an Excel VBA macro. The website -- CBOE.com -- has an input field for the ticker symbol of the desired option chains. My code has been able to drive that part of the webpage and a default option chain is displayed. It defaults to the most current month that options expire (May 2018 as of this note). From there the user can input other expiration dates for which to have other option chains (for the same symbol) to be retrieved and displayed. This is where my code seems to be breaking down.
Just above the default option chain display is a dropdown input box labeled "Expiration:" where a list of other expiration months can be selected. Once selected, a green Submit button must be clicked to get the specified option chain for the selected expiration month. Alternatively, below the default option chain are explicit filter buttons for expiration months also.
As said, my code gets to the point of specifying the symbol and getting default option chains displayed, but I can't seem to get the dropdown input field for other expiration months to work.
If anyone can see where and how my code is deficient, I'd really appreciate that insight.
Many thanks.
--Mark.
Here is my core code in question:
Sub getmarketdata_V3()
Dim mybrowser As Object, myhtml As String
Dim htmltables As Object, htmltable As Object
Dim htmlrows As Object, htmlrow As Object
Dim htmlcells As Object, htmlcell As Object
Dim xlrow As Long, xlcol As Integer
Dim exitat As Date, symbol As String
Dim flag As Integer
On Error GoTo errhdl
Const myurl = "http://www.cboe.com/delayedquote/quote-table"
symbol = UCase(Trim(Range("ticker").Text))
With Range("ticker").Worksheet
Range(Range("ticker").Offset(1, 0), Cells(Rows.Count, Range("ticker").Column + 13)).ClearContents
End With
Set mybrowser = CreateObject("internetexplorer.application")
mybrowser.Visible = True
mybrowser.navigate myurl
While mybrowser.busy Or mybrowser.readyState <> 4
DoEvents
Wend
With mybrowser.document.all
exitat = Now + TimeValue("00:00:05")
Do
.Item("ctl00$ContentTop$C002$txtSymbol").Value = symbol
.Item("ctl00$ContentTop$C002$btnSubmit").Value = "Submit"
.Item("ctl00$ContentTop$C002$btnSubmit").Click
If Err.Number = 0 Then Exit Do
Err.Clear
DoEvents
If Now > exitat Then Exit Do
Loop
End With
'This With statement is to refresh the mybrowser.document since the prior With statement pulls up a partially new webpage
With mybrowser.document.all
On Error Resume Next
exitat = Now + TimeValue("00:00:05")
'Tried using "ID" label to select desired month--in this case 2018 July is a dropdown option:
'Usind this label seems to blank out the value displayed in the dropdown input box, but does not cause
'any of the options to display nor implant "2018 July" in it either. It just remains blank and no new option
'chain is retrieved.
.Item("ContentTop_C002_ddlMonth").Select
.Item("ContentTop_C002_ddlMonth").Value = "2018 July"
.Item("ContentTop_C002_ddlMonth").Click
'Then tried using "Name" label to select desired month--in this case 2018 July is an option:
' .Item("ctl00$ContentTop$C002$ddlMonth").Value = "2018 July"
' .Item("ctl00$ContentTop$C002$ddlMonth").Click
' .Item("ctl00$ContentTop$C002$btnFilter").Value = "View Chain"
' .Item("ctl00$ContentTop$C002$btnFilter").Click
End With
While mybrowser.busy Or mybrowser.readyState <> 4
DoEvents
Wend
'Remaining logic, except for this error trap logic deals with the option chain results once it has been successfully retrieved.
'For purposes of focus on the issue of not being able to successfully have such a table displayed, that remaining process logic is not
'included here.
errhdl:
If Err.Number Then MsgBox Err.Description, vbCritical, "Get data"
On Error Resume Next
mybrowser.Quit
Set mybrowser = Nothing
Set htmltables = Nothing
End Sub
For your code:
These 2 lines change the month and click the view chain (I tested with symbol FLWS). Make sure you have sufficient delays for page to actually have loaded.
mybrowser.document.querySelector("#ContentTop_C002_ddlMonth").Value = "201809"
mybrowser.document.querySelector("#ContentTop_C002_btnFilter").Click
I found the above sketchy for timings when added into your code so I had a quick play with Selenium basic as well. Here is an example with selenium:
Option Explicit
'Tools > references > selenium type library
Public Sub GetMarketData()
Const URL As String = "http://www.cboe.com/delayedquote/quote-table"
Dim d As ChromeDriver, symbol As String
symbol = "FLWS"
Set d = New ChromeDriver
With d
.Start
.Get URL
Dim b As Object, c As Object, keys As New keys
Set b = .FindElementById("ContentTop_C002_txtSymbol")
b.SendKeys symbol
.FindElementById("ContentTop_C002_btnSubmit").Click
Set c = .FindElementById("ContentTop_C002_ddlMonth")
c.Click
c.SendKeys keys.Down 'move one month down
.FindElementById("ContentTop_C002_btnFilter").Click
Stop '<<delete me later
.Quit
End With
End Sub
Try the below approach, in case you wanna stick to IE. I tried to kick out hardcoded delay from the script. It should get you there. Make sure to fill in the text field with the appropriate ticker from the below script before execution.
There you go:
Sub HandleDropDown()
Const url As String = "http://www.cboe.com/delayedquote/quote-table"
Dim IE As New InternetExplorer, Html As HTMLDocument, post As Object, elem As Object
With IE
.Visible = True
.navigate url
While .Busy Or .readyState <> 4: DoEvents: Wend
Set Html = .document
End With
Do: Set post = Html.getElementById("ContentTop_C002_txtSymbol"): DoEvents: Loop While post Is Nothing
post.Value = "tickername" ''make sure to fill in this box with appropriate symbol
Html.getElementById("ContentTop_C002_btnSubmit").Click
Do: Set elem = Html.getElementById("ContentTop_C002_ddlMonth"): DoEvents: Loop While elem Is Nothing
elem.selectedIndex = 2 ''just select the month using it's dropdown order
Html.getElementById("ContentTop_C002_btnFilter").Click
End Sub
Reference to add to the library:
Microsoft Internet Controls
Microsoft HTML Object Library

runtime error 438 object doesn't support this property or method internet explorer

I'm trying to use a loop to get the data from web to excel sheet. I will attach the sheet and also paste the code here. Please help me with this. Thank you.
Private Sub Worksheet_Change(ByVal Target As Range)
If Target.Row = Range("Number").Row And _
Target.Column = Range("Number").Column Then
Dim IE As New InternetExplorer
IE.Visible = True
IE.navigate "https://www.truecaller.com/search/in/" & Range("Number").Value
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Dim Doc As HTMLDocument
Set Doc = IE.document
Dim sdd As String
sdd = Doc.getElementsByClassName("profile-name").innerText
MsgBox sdd
End If
End Sub
WHEN I USE ABOVE CODE I GET THE BELOW ERROR
Run-time error '438':
Object doesn't support this property or method
.
See how .getElementsByClassName is plural? .getElementsByClassName returns a collection, not a single object. In essence, you are trying to treat an array of integers as a single integer; they are not the same thing.
You cannot simply remove the s; there is no getElementByClassName function. However, you can ask for the first element the same way you specify an individual integer within an array of integers.
sdd = Doc.getElementsByClassName("profile-name")(0).innerText

Edit sharepoint list with macro

I need to change the status and write comments for about 100 lists in SharePoint every week. I tried to automate it. I know how to open them in edit mode with a macro, but I don't know how to change status or how to write a comment with a macro, any ideas?
Here is my code:
Sub TT()
Dim ie(40) As Object, obj As Object
Dim cislo As String
For i = 0 To 40
If Cells(i + 2, 1).Value = "" Then
Exit Sub
End If
Set ie(i) = CreateObject("Internetexplorer.Application")
ie(i).Visible = True
ie(i).Navigate "http://adress of sharepoint list .com"
Do While ie(i).Busy
Loop
Next i
End Sub
These tutorials should give you what you need to know...excellent and well done, they show you how to do it via Listobjects and via SQL
https://www.youtube.com/watch?v=nM-gq3N6f2E
There is a series of 13 videos.

Web scraping with VBA

I would like to create a macro to get real time stock quote from a financial website.
Below is my code.
Private Sub Worksheet_Change(ByVal Target As Range)
Dim stock As Long, rng As Range, quote As String, ie As InternetExplorer, doc As HTMLDocument
Set rng = Range("A1")
stock = rng.Value
Set ie = CreateObject("InternetExplorer.Application")
If Target.Rows = rng.Rows And Target.Columns = rng.Columns Then
ie.navigate "http://www.aastocks.com/en/ltp/rtquote.aspx?symbol=0" & rng.Value
ie.Visible = True
Do
DoEvents
Loop Until ie.readyState = READYSTATE_COMPLETE
Set doc = ie.document
quote = doc.getElementsByTagName("neg bold").innertext
MsgBox quote
End If
End Sub
However when I run the macro, it shows error 91 (Object variable or With block variable not set) in the line
quote = doc.getElementsByTagName("neg bold").innertext
below is the HTML code of the source (the stock price)
<span class="neg bold">1.900</span>
Thanks a lot!
The argument for the getElementsByTagName should be an element name - not a class name. If you are looking to target an element by its class name, you could use the getElementsByClassname method instead
Also, be aware both these methods return elements collections - so unless in this specific instance the collection returned boils down to the single element you are targetting, there may be a bit more work to be done in order to narrow it down to what you are looking for

Resources