Selecting a dropdown list when inserting data from web (VBA) - excel

I want to download some data from a webpage (http://www.debentures.com.br/exploreosnd/consultaadados/sndemumclique/) into an Excel spreadsheet.
After loading this page I have to manually choose an option from the "Código do Ativo" dropdown list, and then choose "Agenda".
Is there a way I can do it automatically via VBA?
For example: selecting "RDVT11" from the "Código do Ativo" dropdown list, selecting "Agenda" and then downloading the data from the table that will appear in the bottom part of the page?
My macro so far:
Private Sub Agenda()
Sheets("Dados").Select
Dim ProductionAddress As String
ProductionAddress = "http://www.debentures.com.br/exploreosnd/consultaadados/sndemumclique/x_pu_historico_r.aspx?"
Dim ie As Object
Set ie = CreateObject("InternetExplorer.Application")
With ie
.Silent = True
.Visible = True
.Navigate ProductionAddress
End With
While ie.ReadyState <> 4 Or ie.Busy: DoEvents: Wend
ie.document.getElementByid("ctl00_ddlAti").Value = "RDVT11|11001110111100001"
While ie.ReadyState <> 4 Or ie.Busy: DoEvents: Wend
Set objButton = ie.document.getElementByid("ctl00_x_agenda_r")
objButton.Focus
objButton.Click
While ie.ReadyState <> 4 Or ie.Busy: DoEvents: Wend
ie.Quit
Set ie = Nothing
End Sub

You need to capture the request that a browser sends when the dropdown is activated. Open up Chrome dev tools and watch the network tab. You will see a POST request to sndemumclique/. This will have some headers and form data. Your code will need to basically replicate this request. Likely, not all of the header and form fields are required but there is not way to know without trying.

Here are all 3 parts. The making two selections and the writing of the table to the sheet.
Notes:
① Making first selection:
To make the RDVT11 selection, I first use the Id of the dropdown to capture the element in a variable with:
Set a = .document.getElementById("ctl00_ddlAti")
Next, I loop the drop down options, using a.getElementsByTagName("Option") to generate the collection which I loop over. When the target selection text is found, I set that option to Selected and exit the loop.
For Each currentOption In a.getElementsByTagName("Option")
If InStr(currentOption.innerText, optionText) > 0 Then
currentOption.Selected = True
Exit For
End If
Next currentOption
② Making Agenda selection:
I then target the agenda option of Sobre e emissão by its id and click it and wait for a refresh of the page:
.document.getElementById("ctl00_x_agenda_r").Click
While .Busy Or .readyState < 4: DoEvents: Wend
③ Getting the table and writing to sheet:
I then target the table that is loaded by its id. This is done within a loop to ensure the table is present:
Do: On Error Resume Next: Set nTable = .document.getElementById("aGENDA"): On Error GoTo 0: DoEvents: Loop While nTable Is Nothing
I finally, loop the rows and columns in the table, writing out to the Activesheet.
Code:
Option Explicit
Public Sub MakeSelectiongGetData()
Dim IE As New InternetExplorer
Const URL = "http://www.debentures.com.br/exploreosnd/consultaadados/sndemumclique/"
Const optionText As String = "RDVT11"
Application.ScreenUpdating = False
With IE
.Visible = True
.navigate URL
While .Busy Or .readyState < 4: DoEvents: Wend
Dim a As Object
Set a = .document.getElementById("ctl00_ddlAti")
Dim currentOption As Object
For Each currentOption In a.getElementsByTagName("Option")
If InStr(currentOption.innerText, optionText) > 0 Then
currentOption.Selected = True
Exit For
End If
Next currentOption
.document.getElementById("ctl00_x_agenda_r").Click
While .Busy Or .readyState < 4: DoEvents: Wend
Dim nTable As HTMLTable
Do: On Error Resume Next: Set nTable = .document.getElementById("aGENDA"): On Error GoTo 0: DoEvents: Loop While nTable Is Nothing
Dim nRow As Object, nCell As Object, r As Long, c As Long
With ActiveSheet
Dim nBody As Object
Set nBody = nTable.getElementsByTagName("tbody")(0).getElementsByTagName("tr")
.Cells(1, 1) = nBody(0).innerText
For r = 2 To nBody.Length - 1
Set nRow = nBody(r)
For Each nCell In nRow.Cells
c = c + 1: .Cells(r + 1, c) = nCell.innerText
Next nCell
c = 0
Next r
End With
.Quit
End With
Application.ScreenUpdating = True
End Sub
Data on page (sample)
Code output (sample):

Related

Scraping using VBA

i am trying to extract one figure from a gov website, I have done a lot of googling and I am kinda lost for ideas, my code below returns a figure but it isnt the figure I want to get and I am not entirely sure why.
I want to subtract the figure from the 'Cases by Area (Whole Pandemic)' table 'Upper tier LA' section and 'Southend on Sea' Case number.
https://coronavirus.data.gov.uk/details/cases
I stole this code from online somewhere and tried to replicate with my class number I found within F12 section on the site.
Sub ExtractLastValue()
Set objIE = CreateObject("InternetExplorer.Application")
objIE.Top = 0
objIE.Left = 0
objIE.Width = 800
objIE.Height = 600
objIE.Visible = True
objIE.Navigate ("https://coronavirus.data.gov.uk/details/cases")
Do
DoEvents
Loop Until objIE.readystate = 4
MsgBox objIE.document.getElementsByClassName("sc-bYEvPH khGBIg govuk-table__cell govuk-table__cell--numeric ")(0).innerText
Set objIE = Nothing
End Sub
Data comes from the official API and returns a json response dynamically on that page when you click the Upper Tier panel.
Have a look and play with the API guidance
here:
https://coronavirus.data.gov.uk/details/developers-guide
You can make a direct xhr request by following the guidance in the API documentation and then using a json parser to handle the response. For your request it would be something like the following:
https://coronavirus.data.gov.uk/api/v1/data?filters=areaName=Southend-on-Sea&areaType=utla&latestBy=cumCasesByPublishDate&structure=
{"date":"date", "areaName":"areaName","cumCasesByPublishDate":"cumCasesByPublishDate",
"cumCasesByPublishDateRate":"cumCasesByPublishDateRate"}
XHR:
A worked example using jsonconverter.bas as the json parser
Option Explicit
Public Sub GetCovidNumbers()
Dim http As Object, json As Object
Set http = CreateObject("MSXML2.XMLHTTP")
With http
.Open "GET", "https://coronavirus.data.gov.uk/api/v1/data?filters=areaName=Southend-on-Sea&areaType=utla&latestBy=cumCasesByPublishDate&structure={""date"":""date"",""areaName"":""areaName"",""cumCasesByPublishDate"":""cumCasesByPublishDate"",""cumCasesByPublishDateRate"":""cumCasesByPublishDateRate""}", False
.setRequestHeader "User-Agent", "Mozilla/5.0"
.send
Set json = JsonConverter.ParseJson(.responseText)("data")(1)
End With
With ActiveSheet
Dim arr()
arr = json.Keys
.Cells(1, 1).Resize(1, UBound(arr) + 1) = arr
arr = json.Items
.Cells(2, 1).Resize(1, UBound(arr) + 1) = arr
End With
End Sub
Json library (Used in above solution):
I use jsonconverter.bas. Download raw code from here and add to standard module called JsonConverter . You then need to go VBE > Tools > References > Add reference to Microsoft Scripting Runtime. Remove the top Attribute line from the copied code.
Internet Explorer:
You could do a slower, more complicated, internet explorer solution where you need to select the utla option when present, then select from the table the desired value:
Option Explicit
Public Sub GetCovidNumbers()
'Tools references Microsoft Internet Controls and Microsoft HTML Object Library
Dim ie As SHDocVw.InternetExplorer, t As Date, ele As Object
Const MAX_WAIT_SEC As Long = 10
Set ie = New SHDocVw.InternetExplorer
With ie
.Visible = True
.Navigate2 "https://coronavirus.data.gov.uk/details/cases"
While .Busy Or .ReadyState <> READYSTATE_COMPLETE: DoEvents: Wend
t = Timer 'timed loop for element to be present to click on (to get utla)
Do
On Error Resume Next
Set ele = .Document.querySelector("#card-cases_by_area_whole_pandemic [aria-label='Upper tier LA']")
On Error GoTo 0
If Timer - t > MAX_WAIT_SEC Then Exit Do
Loop While ele Is Nothing
If ele Is Nothing Then Exit Sub
ele.Click
While .Busy Or .ReadyState <> READYSTATE_COMPLETE: DoEvents: Wend
Dim table As MSHTML.HTMLTable, datetime As String, result()
Set table = .Document.querySelector("table[download='cumCasesByPublishDate,cumCasesByPublishDateRate']")
datetime = .Document.querySelector("time").getAttribute("datetime")
result = GetDataForUtla("Southend-on-Sea", datetime, table)
With ActiveSheet
.Cells(1, 1).Resize(1, 4) = Array("Datetime", "Area", "Cases", "Rate per 100,000 population")
.Cells(2, 1).Resize(1, UBound(result) + 1) = result
End With
.Quit
End With
End Sub
Public Function GetDataForUtla(ByVal utla As String, ByVal datetime As String, ByVal table As MSHTML.HTMLTable) As Variant
Dim row As MSHTML.HTMLTableRow, i As Long
For Each row In table.Rows
If InStr(row.outerHTML, utla) > 0 Then
Dim arr(4)
arr(0) = datetime
For i = 0 To 2
arr(i + 1) = row.Children(i).innerText
Next
GetDataForUtla = arr
Exit Function
End If
Next
GetDataForUtla = Array("Not found")
End Function
References:
https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors
https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelector

VBA code to copy table data from webpage into Excel

I wish to copy data from a table on a webpage into Excel using VBA code but didn't get anything on the Excel sheet :(.
I have tried to put together some VBA code from different sources. Here is my code:
Sub CopyWebData()
Dim IE As Object
On Error Resume Next
Application.DisplayAlerts = False
Set IE = CreateObject("InternetExplorer.Application")
With IE
.Visible = True
.navigate "https://eresearch.fidelity.com/eresearch/evaluate/fundamentals/earnings.jhtml?tab=details&symbols=GOOG"
Do Until .readyState = 4: DoEvents: Loop
End With
Dim idoc As MSHTML.HTMLDocument
Dim elem As MSHTML.IHTMLElement
Set idoc = IE.document
Set elem = idoc.getElementsByClassName("layout-outer-table-width")(0).innerText
Sheets("Sheet1").Activate
Range("A1:A1000") = "" ' erase previous data
Range("A1").Select
Range("A1").Value = elem
End Sub
This is a password-protected webpage and I have logged in so I can see the webpage has been successfully pulled out by the VBA code. However, the data in the table on this webpage failed to be copied into excel - I saw nothing on the destination worksheet.
As you can see, I used code .getElementsByClassName("layout-outer-table-width") since I used Chrome's "Inspect" function to check the webpage and found that when the mouse was hovering over the statements:
...<table cellspacing="0" cellpadding="0" border="0" class="layout-outer-table-width"> == $0
<tbody>...</tbody>
</table>
part of the webpage covering the table I need was shaded. I then coded in the class name "layout-outer-table-width". However, as I said, I didn't see anything appearing on the Excel sheet.
Any instruction would be much appreciated!
If after the earnings detail table you need a different selector. I am showing a css selector for that table. Your current selector (class), and index 0, is matching on a breadcrumb (nav tree). That class is also not correct for selecting the table on the page.
.earningsHistoryTable-Cont table
I cannot test this but you may also want a timed loop for table to be present
Option Explicit
'VBE > Tools > References:
' Microsoft Internet Controls
Public Sub RetrieveInfo()
Dim IE As InternetExplorer, hTable As Object, clipboard As Object, t As Date
Const MAX_WAIT_SEC As Long = 5
Set clipboard = GetObject("New:{1C3B4210-F441-11CE-B9EA-00AA006B1A69}")
Set IE = New InternetExplorer
With IE
.Visible = True
.Navigate2 "https://eresearch.fidelity.com/eresearch/evaluate/fundamentals/earnings.jhtml?tab=details&symbols=GOOG"
While .Busy Or .readyState < 4: DoEvents: Wend
With .document
.querySelector("#userId").Value = "xyz"
.querySelector("#password").Value = "123456"
.querySelector("form").submit
End With
While .Busy Or .readyState < 4: DoEvents: Wend
t = Timer 'timed loop for details table to be present
Do
On Error Resume Next
Set hTable = IE.document.querySelector(".earningsHistoryTable-Cont table")
On Error GoTo 0
If Timer - t > MAX_WAIT_SEC Then Exit Do
Loop While hTable Is Nothing
If Not hTable Is Nothing Then 'use clipboard to copy paste
clipboard.SetText hTable.outerHTML
clipboard.PutInClipboard
ThisWorkbook.Worksheets("Sheet1").Range("A1").PasteSpecial
End If
End With
End Sub

VBA to insert data in Search box IE

When i insert Few words in searchbox, Its fetching related data.
I need to select first option from it.
There is one website "https://indiarailinfo.com/"
When i search "ADI" in from station box, system fetching related station having "ADI" in their name?. First option always showing very close match to it.
How can i select First Option from it using vba code
Dim ie As Object
Set ie = CreateObject("InternetExplorer.Application")
ie.navigate "https://indiarailinfo.com/"
While ie.readyState <> 4: DoEvents: Wend
ie.Visible = True
ie.document.querySelector("[placeholder='from station']").Value = "ADI"
HTML Codes can be available from that site
It's Bring First Answer in Dropdown like "ADI/Ahmedabad Junction"
How can i get this answer in selected"
Kindly Suggest
Automation purists won't like using javascript to execute but I will use here for IE to trigger dropdown. If I was going pure route I would use selenium.
Option Explicit
Public Sub MakeSelection()
Dim ie As InternetExplorer, t As Date, dropdown1 As Object
Set ie = New InternetExplorer
Const MAX_WAIT_SEC As Long = 5
With ie
.Visible = True
.Navigate2 "https://indiarailinfo.com/"
While .Busy Or .readyState < 4: DoEvents: Wend
With .document.querySelector("[placeholder='from station']")
.Focus
.Value = "ADI"
ie.document.parentWindow.execScript "document.querySelector('[placeholder^=from]').click();"
End With
t = Timer
Do
DoEvents
On Error Resume Next
Set dropdown1 = .document.querySelectorAll(".icol span")
On Error GoTo 0
If Timer - t > MAX_WAIT_SEC Then Exit Do
Loop While dropdown1.Length = 0
If dropdown1.Length > 0 Then
dropdown1.item(0).Click
End If
Stop
.Quit
End With
End Sub
For automation purists using selenium basic
Option Explicit
Public Sub MakeSelection()
Dim d As WebDriver
Set d = New ChromeDriver
Const Url = "https://indiarailinfo.com/"
With d
.Start "Chrome"
.get Url
.FindElementByCss("[placeholder='from station']").SendKeys "ADI"
.FindElementByCss(".icol span").Click
Stop
.Quit
End With
End Sub

Looping through a row and copying each cell In a specific procedure

What I have to do is use Excel VBA to:
login to Amazon Seller
open a workbook
loop through a column to get an order number
put it in the search box
hit the search button
go to the order page and extract the data
then have the extracted data go back into a specified column in
another Excel workbook
The loop and order number parts are what I'm currently stumped on. I've figured out this much code as of this moment:
Sub MyAmazonSeller()
Dim MyHTML_Element As IHTMLElement
Dim MyURL As String
Dim oSignInLink As HTMLLinkElement
Dim oInputEmail As HTMLInputElement
Dim oInputPassword As HTMLInputElement
Dim oInputSigninButton As HTMLInputButtonElement
'InputSearchOrder will be the destination for order numbers taken from the workbook
Dim InputSearchOrder As HTMLInputElement
Dim InputSearchButton As HTMLInputButtonElement
Dim IE As InternetExplorer
Dim AAOrder As Workbook
Dim AAws As Worksheet
MyURL = "https://sellercentral.amazon.com/gp/homepage.html"
Set IE = New InternetExplorer
' Open the browser and navigate.
With IE
.Silent = True
.Navigate MyURL
.Visible = True
Do
DoEvents
Loop Until .ReadyState = READYSTATE_COMPLETE
End With
' Get the html document.
Set HTMLDoc = IE.Document
' See if you have the sign in link is because you are in the main
' page
Set oSignInLink = HTMLDoc.getElementById("signin-button-container")
If Not oSignInLink Is Nothing Then
oSignInLink.Click
Do
DoEvents
Loop Until IE.ReadyState = READYSTATE_COMPLETE
End If
' Get the email field and the next button
Set oInputEmail = HTMLDoc.getElementById("username")
Set oInputPassword = HTMLDoc.getElementById("password")
' Click the button and wait
oInputEmail.Value = "xxxxxx#xxxxxx.net"
' Get the password field and the sign in button
Set oInputPassword = HTMLDoc.getElementById("password")
Set oInputSigninButton = HTMLDoc.getElementById("sign-in-button")
' Click the button and wait
oInputPassword.Value = "xxxxxxxx"
oInputSigninButton.Click
Do
DoEvents
Loop Until IE.ReadyState = READYSTATE_COMPLETE
Application.Wait (Now + TimeValue("0:00:05"))
Set AAOrder = Application.Workbooks.Open("Z:\Employee Folders\Employee\trackingnumber_sample_spreadsheet.xls")
Set AAws = AAws.Worksheets("PrimeOrdersWithNoFulfillmentRe")
Set InputSearchOrder = HTMLDoc.getElementById("sc-search-field")
'What I'm currently stuck on
InputSearchOrder.Value = "001-7163923-7572632"
Set InputSearchButton = HTMLDoc.getElementsByClassName("sc-search-button")(0)
InputSearchButton.Click
Do
DoEvents
Loop Until IE.ReadyState = READYSTATE_COMPLETE
'Was able to add this snippet, but I'm getting an error 13, most likely with
'my e variable. I'm basically trying to do a loop within a loop, extracting 5
'pieces of data and sticking them back into their respective columns in the
'original Excel sheet. The problem comes when scraping the HTML. I'm basically
'trying to get text in the tables which have a few levels and it's frustrating
'me to no end.
With HTMLDoc
Set elems = HTMLDoc.getElementsByTagName("td")
For Each e In elems
If e.innerText Like "*1Z*" Then
Range("D2").Value = e.innerText
End If
Next e
End With
Err_Clear:
If Err <> 0 Then
Err.Clear
Resume Next
End If
End Sub

Web-scraping on intranet

I wrote a VBA code to scrape data from my company's intranet.
Issues:
The below error occurs:
Run-time error '91':
object variable or with block variable not set
It happens on:
myPoints = Trim(Doc.getElementsByName("price")(0).getAttribute("value"))
When I debug it and run line by line, it can retrieve all the values.
Input and Output:
I input multiple product ID on column B and retrieve data on column C:
Column B = product ID
Column C = price
HTML:
<td id="myPower_val_9" style="visibility: visible;">
<input type="text" disabled="disabled" value="300" name="price"></input>
</td>
VBA:
Sub Button1_Click()
Dim ie As Object
Dim r As Integer
Dim myPoints As String
Dim Doc As HTMLDocument
Set ie = New InternetExplorerMedium
For r = 2 To Range("B65535").End(xlUp).Row
With ie
.Visible = 0
.navigate "www.example.com/product/" & Cells(r, "B").Value
Do Until .readyState = 4
DoEvents
Loop
End With
Set Doc = ie.document
myPoints = Trim(Doc.getElementsByName("price")(0).getAttribute("value"))
Cells(r, "C").Value = myPoints
Next r
End Sub
Have I missed an error handler?
You need to wait for the document to be fully rendered and the DOM available before accessing any elements. ie.ReadyState changes to READYSTATE_COMPLETE once the page connects and starts loading. The reason that your code works when debugging is that in the couple of seconds it takes for you to start working with the debugger, the page finishes loading.
With ie
.Visible = True
.Navigate "www.example.com/product/" & Cells(r, "B").Value
Do Until .ReadyState = READYSTATE_COMPLETE
DoEvents
Loop
Do Until .Document.ReadyState = "complete"
DoEvents
Loop
End With
I would also recommend that you make the ie Window visible, at least while you're developing. Once you've got your functionality complete and debugging, you can make the window invisible. Keep in mind if you forget to close your invisible IE windows when your code finishes, your users will end up with runaway iexplore.exe processes.
If you only want to ignore the error and continue with the next iteration, use this modified code:
Sub Button1_Click()
Dim ie As Object
Dim r As Integer
Dim myPoints As String
Dim Doc As HTMLDocument
Set ie = New InternetExplorerMedium
For r = 2 To Range("B65535").End(xlUp).Row
With ie
.Visible = 0
.navigate "www.example.com/product/" & Cells(r, "B").Value
Do Until .readyState = 4
DoEvents
Loop
End With
Set Doc = ie.document
'Edit:
myPoints = ""
On Error Resume Next
myPoints = Trim(Doc.getElementsByName("price")(0).getAttribute("value"))
On Error Goto 0
Cells(r, "C").Value = myPoints
Next r
End Sub
You could also loop until element is set (add a timeout clause as well)
Dim a As Object
Do
DoEvents
On Error Resume Next
Set a = Doc.getElementsByName("price")
On Error GoTo 0
Loop While a Is Nothing

Resources