I want to copy into Excel 3 tracking information tables that website generates when I track a parcel. I want to do it through Excel VBA. I can write a loop and generate this webpage for various tracking numbers. But I am having a hard time copying tables - the top table, travel history and shipments track table. Any solution? In my vba code last 3 lines below are giving an error :( - run time error '438' Object doesn't support this property or error.
Sub final()
Application.ScreenUpdating = False
Set ie = CreateObject("InternetExplorer.Application")
my_url = "https://www.fedex.com/fedextrack/index.html?tracknumbers=713418602663&cntry_code=us"
With ie
.Visible = True
.navigate my_url
Do Until Not ie.Busy And ie.readyState = 4
DoEvents
Loop
End With
ie.document.getElementById("detailsBody").Value
ie.document.getElementById("trackLayout").Value
ie.document.getElementById("detail").Value
End Sub
.Value is not a method available in that context. also, you will want to assign the return value of the method call to a variable. Also, you should declare your variables :)
I made some modifications and include one possible way of getting data from one of the tables. YOu may need to reformat the output using TextToColumns or similar, since it prints each row in a single cell.
I also notice that when I execute this, the tables have sometimes not finished loading and the result will be an error unless you put in a suitable Wait or use some other method to determine when the data has fully loaded on the webpage. I use a simple Application.Wait
Option Explicit
Sub final()
Dim ie As Object
Dim my_url As String
Dim travelHistory As Object
Dim history As Variant
Dim h As Variant
Dim i As Long
Application.ScreenUpdating = False
Set ie = CreateObject("InternetExplorer.Application")
my_url = "https://www.fedex.com/fedextrack/index.html?tracknumbers=713418602663&cntry_code=us"
With ie
.Visible = True
.navigate my_url
'## I modified this logice a little bit:
Do While .Busy And .readyState <> 4
DoEvents
Loop
End With
'## Here is a simple method wait for IE to finish, you may need a more robust solution
' For assistance with that, please ask a NEW question.
Application.Wait Now() + TimeValue("0:00:10")
'## Get one of the tables
Set travelHistory = ie.Document.GetElementByID("travel-history")
'## Split teh table to an array
history = Split(travelHistory.innerText, vbLf)
'## Iterate the array and write each row to the worksheet
For Each h In history
Range("A1").Offset(i).Value = h
i = i + 1
Next
ie.Quit
Set ie = Nothing
End Sub
Related
I am experimenting with web automation and struggling a bit trying to utilize a drop down list.
My code works up to the point of searching for a company name and hitting "go". On the new page I can't seem to find the right code that selects the group of elements that represents the drop down list. I then want to select "100" entries, but I can't even grab the nodes that represent this list.
I have been browsing multiple different pages on stackoverflow that talk about CSS selectors and looked at tutorials but that doesn't seem to help either. I either end up grabbing nothing, or whatever I grab can't use the getElementsByTagName method, which ultimately I am trying to drill down into the td and select nodes . Not sure what to do with those yet, but I can't even grab them. Thoughts?
(note stopline is just a line that I use a breakpoint on to stop my code)
CSS helper website: https://www.w3schools.com/cssref/trysel.asp
Code:
Option Explicit
Sub test()
On Error GoTo ErrHandle
Dim ie As New InternetExplorer
Dim doc As New HTMLDocument
Dim ws As Worksheet
Dim stopLine As Integer
Dim oSearch As Object, oSearchButton As Object
Dim oForm As Object
Dim oSelect As Object
Dim list As Object
Set ws = ThisWorkbook.Worksheets("Sheet1")
ie.Visible = True
ie.navigate "https://www.sec.gov/edgar/searchedgar/companysearch.html"
Do
DoEvents
Loop Until ie.readyState = READYSTATE_COMPLETE
Set doc = ie.Document
Set oSearch = doc.getElementById("companysearchform")
Set oSearchButton = oSearch.getElementsByTagName("input")(1)
Set oSearch = oSearch.getElementsByTagName("input")(0)
oSearch.Value = "Summit Midstream Partners, LP"
oSearchButton.Click
Do
DoEvents
Loop Until ie.readyState = READYSTATE_COMPLETE
Set doc = ie.Document
Set list = doc.querySelectorAll("td select")
stopLine = 1
Exit Sub
ErrHandle:
MsgBox Err.Number & " - " & Err.Description, vbCritical
Exit Sub
End Sub
td select will return a single node so you only need querySelector. The node has an id so you might as well use the quicker querySelector("#count") to target the parent select. To change the option you can then use SelectedIndex on the parent select, or, target the child option by its value attribute querySelector("[value='100']").Selected = True. You may then need to attach and trigger change/onchange htmlevent to the parent select to register the change.
However, I would simply extract the company CIK from current page then concatenate the count=100 param into the url and .Navigate2 that using following format:
https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0001549922&type=&dateb=&owner=include&count=100&search_text=
You can extract CIK, after initial search company click and wait for page load, with:
Dim cik As String
cik = ie.document.querySelector("[name=CIK]").value
ie.Navigate2 "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=" & cik & "&type=&dateb=&owner=include&count=100&search_text="
Given several params are left blank you can likely shorten to:
"https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=" & cik & "&owner=include&count=100"
If you are unable to get the initial parent select you probably need a timed loop waiting for that element to be present after clicking the search button. An example is shown here in a StackOverflow answer.
I have some questions reagrding an Excel VBA program that I want to build.
Basically it's pretty easy. I want to access the following website https://coronavirus.jhu.edu/map.html
and extract the Confirmed Cases by Country/Region/Sovereignty (it's the table on the very left of the dashborad) and paste the values in excel.
I know all the basic stuff on how to setup an internetexplorer instance and scraping the page by tags, classes, ids etc.
But I think in this sceanrio I cannot use the basic things. I guess it's pretty tricky actually.
The information I am looking for is within some tags. But I cannot get their textcontent when I use the getelementsbytagname("strong") approach.
Could someone help me in this case?
I am grateful for any hints, advices and solutions.
Below you'll find the start of my code.
Best
Simon
Sub test()
Dim ie As InternetExplorer
Dim html As HTMLDocument
Dim i As Integer
Dim obj_coll As IHTMLElementCollection
Dim obj As HTMLObjectElement
Set ie = New InternetExplorer
ie.Visible = False
ie.navigate "https://coronavirus.jhu.edu/map.html"
Do Until ie.readyState = READYSTATE_COMPLETE
DoEvents
Loop
Debug.Print "Successfully connected with host"
Set html = ie.document
Set obj_coll = html.getElementsByTagName("strong")
For Each obj In obj_coll
Debug.Print obj.innerText
Next obj
ie.Quit
End Sub
You can use the iframe url direct to navigate to. You then need a timed wait to ensure the data has loaded within that iframe. I would then collect nodeLists via faster css selectors. As the nodeLists (one for figures and the other for locations) are the same length you will only need a single loop to index into both lists to get rows of data.
Option Explicit
Public Sub GetCovidFigures()
Dim ie As SHDocVw.InternetExplorer
Set ie = New SHDocVw.InternetExplorer
Dim t As Date
Const MAX_WAIT_SEC As Long = 30
With ie
.Visible = True
.Navigate2 "https://www.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6"
Do
DoEvents
Loop While .Busy Or .readyState <> READYSTATE_COMPLETE
t = Timer
Do
If Timer - t > MAX_WAIT_SEC Then Exit Sub
Loop While .document.querySelectorAll(".feature-list strong").Length = 0
Dim figures As Object, location As Object, results(), i As Long
Set figures = .document.querySelectorAll("h5 strong")
Set location = .document.querySelectorAll("h5 span:last-child")
ReDim results(1 To figures.Length, 1 To 2)
For i = 0 To figures.Length - 1
results(i + 1, 1) = figures.item(i).innerText
results(i + 1, 2) = location.item(i).innerText
Next
.Quit
End With
ActiveSheet.Cells(1, 1).Resize(UBound(results, 1), UBound(results, 2)) = results
End Sub
Consider how frequently you want this. There are large numbers of APIs popping up to supply this data which you could instead issue faster xhr requests to. Additionally, you could simply take the source data in csv form from github here. *Files after Feb 1 (UTC): once a day around 23:59 (UTC). There is a rest API visible in dev tools network tab that is frequently supplying new data in json format which is used to update the page. That can be accessed via Python + requests or R + httr modules for example. I suspect this endpoint is not intended to be hit so look for public APIs.
I'm quite new to VBA so please bear with me. I've been trying to create an automation to fill in username and password and login to a site (to start with) but I've been having trouble trying to click on the submit button. Scoured the internet and learnt a whole bunch of things but I didnt find anything that seems to work. The page loads and fills in the details and nothing happens when I run the code below.
Would greatly appreciate some help with this. Thanks in advance as always!
Sub worldcheck()
Dim lastrow As Long
Dim IE As Object
Dim cel As Range
Dim post As Object
Dim ws As Worksheet
Dim element As Object
Set ws = Sheets("sheet1")
Set IE = CreateObject("internetexplorer.application")
lastrow = ws.Range("B" & ws.Rows.Count).End(xlUp).Row
IE.Visible = True
IE.Navigate "https://www.world-check.com/frontend/login/"
Do While IE.busy
DoEvents
Loop
Application.Wait (Now + TimeValue("0:00:2"))
IE.document.getElementbyID("username").Value = ws.Range("D2")
IE.document.getElementbyID("password").Value = ws.Range("D3")
IE.document.getElementbyClass("button").click
Do While IE.busy
DoEvents
Loop
End Sub
Nothing else happens? You should be getting an error message at the very least as you are trying to use a non existent method (VBA Run-time error 438 Object doesn't support this property or method) . The method is getElementsByClassName - note the s indicating it returns a collection and the ending is ClassName. You would then need to index into that collection before attempting to access the Click method
As there is only a single element with classname button you can use a faster css class selector (this is also faster than using a type selector of form; likewise, you can use the faster css equivalent of getElementById for the other two DOM elements). document.querySelector stops at the first match so is also more efficient.
Finally, rather than hard coded waits use proper page load waits as shown below:
Option Explicit
Public Sub WorldCheck()
Dim ie As Object
Set ie = CreateObject("InternetExplorer.Application")
With ie
.Visible = True
.Navigate2 "https://www.world-check.com/frontend/login/"
While .busy Or .readystate <> 4: DoEvents: Wend
With .document
.querySelector("#username").Value = "ABCDEF" ' equivalent to .getElementbyID("username").Value = "ABCDEF"
.querySelector("#password").Value = "GHIJKL" '.getElementbyID("password").Value = "GHIJKL"
.querySelector(".button").Click
End With
While .busy Or .readystate <> 4: DoEvents: Wend
Stop '<== delete me later
.Quit
End With
End Sub
I'm trying to operate a website to display desired option chain data with an Excel VBA macro. The website -- CBOE.com -- has an input field for the ticker symbol of the desired option chains. My code has been able to drive that part of the webpage and a default option chain is displayed. It defaults to the most current month that options expire (May 2018 as of this note). From there the user can input other expiration dates for which to have other option chains (for the same symbol) to be retrieved and displayed. This is where my code seems to be breaking down.
Just above the default option chain display is a dropdown input box labeled "Expiration:" where a list of other expiration months can be selected. Once selected, a green Submit button must be clicked to get the specified option chain for the selected expiration month. Alternatively, below the default option chain are explicit filter buttons for expiration months also.
As said, my code gets to the point of specifying the symbol and getting default option chains displayed, but I can't seem to get the dropdown input field for other expiration months to work.
If anyone can see where and how my code is deficient, I'd really appreciate that insight.
Many thanks.
--Mark.
Here is my core code in question:
Sub getmarketdata_V3()
Dim mybrowser As Object, myhtml As String
Dim htmltables As Object, htmltable As Object
Dim htmlrows As Object, htmlrow As Object
Dim htmlcells As Object, htmlcell As Object
Dim xlrow As Long, xlcol As Integer
Dim exitat As Date, symbol As String
Dim flag As Integer
On Error GoTo errhdl
Const myurl = "http://www.cboe.com/delayedquote/quote-table"
symbol = UCase(Trim(Range("ticker").Text))
With Range("ticker").Worksheet
Range(Range("ticker").Offset(1, 0), Cells(Rows.Count, Range("ticker").Column + 13)).ClearContents
End With
Set mybrowser = CreateObject("internetexplorer.application")
mybrowser.Visible = True
mybrowser.navigate myurl
While mybrowser.busy Or mybrowser.readyState <> 4
DoEvents
Wend
With mybrowser.document.all
exitat = Now + TimeValue("00:00:05")
Do
.Item("ctl00$ContentTop$C002$txtSymbol").Value = symbol
.Item("ctl00$ContentTop$C002$btnSubmit").Value = "Submit"
.Item("ctl00$ContentTop$C002$btnSubmit").Click
If Err.Number = 0 Then Exit Do
Err.Clear
DoEvents
If Now > exitat Then Exit Do
Loop
End With
'This With statement is to refresh the mybrowser.document since the prior With statement pulls up a partially new webpage
With mybrowser.document.all
On Error Resume Next
exitat = Now + TimeValue("00:00:05")
'Tried using "ID" label to select desired month--in this case 2018 July is a dropdown option:
'Usind this label seems to blank out the value displayed in the dropdown input box, but does not cause
'any of the options to display nor implant "2018 July" in it either. It just remains blank and no new option
'chain is retrieved.
.Item("ContentTop_C002_ddlMonth").Select
.Item("ContentTop_C002_ddlMonth").Value = "2018 July"
.Item("ContentTop_C002_ddlMonth").Click
'Then tried using "Name" label to select desired month--in this case 2018 July is an option:
' .Item("ctl00$ContentTop$C002$ddlMonth").Value = "2018 July"
' .Item("ctl00$ContentTop$C002$ddlMonth").Click
' .Item("ctl00$ContentTop$C002$btnFilter").Value = "View Chain"
' .Item("ctl00$ContentTop$C002$btnFilter").Click
End With
While mybrowser.busy Or mybrowser.readyState <> 4
DoEvents
Wend
'Remaining logic, except for this error trap logic deals with the option chain results once it has been successfully retrieved.
'For purposes of focus on the issue of not being able to successfully have such a table displayed, that remaining process logic is not
'included here.
errhdl:
If Err.Number Then MsgBox Err.Description, vbCritical, "Get data"
On Error Resume Next
mybrowser.Quit
Set mybrowser = Nothing
Set htmltables = Nothing
End Sub
For your code:
These 2 lines change the month and click the view chain (I tested with symbol FLWS). Make sure you have sufficient delays for page to actually have loaded.
mybrowser.document.querySelector("#ContentTop_C002_ddlMonth").Value = "201809"
mybrowser.document.querySelector("#ContentTop_C002_btnFilter").Click
I found the above sketchy for timings when added into your code so I had a quick play with Selenium basic as well. Here is an example with selenium:
Option Explicit
'Tools > references > selenium type library
Public Sub GetMarketData()
Const URL As String = "http://www.cboe.com/delayedquote/quote-table"
Dim d As ChromeDriver, symbol As String
symbol = "FLWS"
Set d = New ChromeDriver
With d
.Start
.Get URL
Dim b As Object, c As Object, keys As New keys
Set b = .FindElementById("ContentTop_C002_txtSymbol")
b.SendKeys symbol
.FindElementById("ContentTop_C002_btnSubmit").Click
Set c = .FindElementById("ContentTop_C002_ddlMonth")
c.Click
c.SendKeys keys.Down 'move one month down
.FindElementById("ContentTop_C002_btnFilter").Click
Stop '<<delete me later
.Quit
End With
End Sub
Try the below approach, in case you wanna stick to IE. I tried to kick out hardcoded delay from the script. It should get you there. Make sure to fill in the text field with the appropriate ticker from the below script before execution.
There you go:
Sub HandleDropDown()
Const url As String = "http://www.cboe.com/delayedquote/quote-table"
Dim IE As New InternetExplorer, Html As HTMLDocument, post As Object, elem As Object
With IE
.Visible = True
.navigate url
While .Busy Or .readyState <> 4: DoEvents: Wend
Set Html = .document
End With
Do: Set post = Html.getElementById("ContentTop_C002_txtSymbol"): DoEvents: Loop While post Is Nothing
post.Value = "tickername" ''make sure to fill in this box with appropriate symbol
Html.getElementById("ContentTop_C002_btnSubmit").Click
Do: Set elem = Html.getElementById("ContentTop_C002_ddlMonth"): DoEvents: Loop While elem Is Nothing
elem.selectedIndex = 2 ''just select the month using it's dropdown order
Html.getElementById("ContentTop_C002_btnFilter").Click
End Sub
Reference to add to the library:
Microsoft Internet Controls
Microsoft HTML Object Library
The below function opens Oanda.com and takes the currency conversion rate between USD and a input (another currency). The equation will be filled down with another macro to span 48 rows. The macro as it stands, will open 48 Internet Explorer Windows and close them after extracting the data point (1 at a time as the equation updates down the column). This process is tedious and the below method seems to be more efficient on every front but I cannot figure out how to implement:
Is there a way to amend this to first check for existing Internet Explorer window, If one is open, simply use that window to go to the domain (which is variable here) and extract data. If a window is not open, then open one. I do not know procedures to search for programs running in the background.
The main goal is to speed up the equation when it is executed by fill down. Any suggestions welcome.
Clarity Edit: I created the UDF to help me build a table. Currency tickers (USD, EUR, GBP, etc) will span Column A down to row 48. Column B needs to show the corresponding conversion rate matched against 1 USD (down to row 48). The UDF below does as intended but i'm seeking an alternative, more efficient, way to do this.
Option Explicit
Public Function ConvertUSD(ConvertWhat As String) As Double
'References
' Microsoft XML, vs.0
' Microsoft Internet Controls
' Microsoft HTML Object Library.
Dim IE As New InternetExplorer
'IE.Visible = True
IE.Navigate "https://www.oanda.com/currency/converter?quote_currency=USD&base_currency=" & ConvertWhat
Do
DoEvents
Loop Until IE.ReadyState = ReadyState_Complete
Dim Doc As HTMLDocument
Set Doc = IE.Document
Dim Ans As String
Ans = Trim(Doc.getElementsByTagName("tbody")(2).innerText)
Dim AnsExtract As Variant
AnsExtract = Split(Ans, " ")
ConvertUSD = AnsExtract(4)
IE.Quit
End Function
Try this:
Sub MainSub()
Dim IE As InternetExplorer
Set IE = New InternetExplorer
'
Dim x As Long
Dim Currencies As Variant
Currencies = Array("GBP", "EUR", "JPY", "HKD")
'
For x = LBound(Currencies) To UBound(Currencies)
Debug.Print "1 USD = " & ConvertUSD(Currencies(x), IE) & " " & Currencies(x)
Next x
IE.Quit ' Quit here instead
Set IE = Nothing
End Sub
Public Function ConvertUSD(ByVal ConvertWhat As String, IE As InternetExplorer) As Double
'References
' Microsoft XML, vs.0
' Microsoft Internet Controls
' Microsoft HTML Object Library.
' Dim IE As New InternetExplorer ' Commented out here
IE.Navigate "https://www.oanda.com/currency/converter?quote_currency=USD&base_currency=" & ConvertWhat
Do
DoEvents
Loop Until IE.ReadyState = ReadyState_Complete
Dim Doc As HTMLDocument
Set Doc = IE.Document
Dim Ans As String
Ans = Trim(Doc.getElementsByTagName("tbody")(2).innerText)
Dim AnsExtract As Variant
AnsExtract = Split(Ans, " ")
ConvertUSD = AnsExtract(4)
' IE.Quit ' Don't quit here
End Function
Your problem is that you keep opening a new IE every time the function is called. However if you open one before calling the function, you'll be able to re-use it as needed - and only quit after you've finished.