Get a single value from a table with no ID with VBA - excel

I am developing a web bot that scrapes the importation taxes from different countries customs website, and I have a problem retrieving the value I want from the following site : http://www.aduanet.gob.pe/itarancel/arancelS01Alias , using the test value 3303000000 next to CODIGO. The value I want to retrieve is the 6% next to "Ad / Valorem", but the table it is in has no ID properties nor class or something relevant to get directly to it or at least near to it. I have been trying to use .parent and .child methods, but without success. My code so far is as follows:
Function Peru(partida As String) As String
'Open IE
Set objIE = New InternetExplorer
objIE.Visible = True
objIE.navigate "http://www.aduanet.gob.pe/itarancel/arancelS01Alias"
'Load sub
Cargar
'Navigate further into the website (Im using partida = 3303000000)
For Each box In objIE.document.getElementsByTagName("input")
If box.Name = "cod_partida" Then
box.Value = partida
Exit For
End If
Next
For Each boton In objIE.document.getElementsByTagName("input")
If boton.Value = "Consultar" Then
boton.Click
Exit For
End If
Next
'Get the 6% value (This part is the one I cant figure out)
End Function

This is how you can get the data from that page. It was needed to switch two iframes from that page to reach the required content.
Sub Aduanet_Info()
Dim IE As New InternetExplorer, html As HTMLDocument
Dim elem As Object, frm As Object, frm1 As Object
With IE
.Visible = False
.navigate "http://www.aduanet.gob.pe/itarancel/arancelS01Alias"
Do While .readyState <> READYSTATE_COMPLETE: Loop
Set html = .document
End With
html.getElementsByTagName("input")(0).Value = "3303000000"
html.getElementsByTagName("input")(3).Click
Application.Wait Now + TimeValue("00:00:05")
Set frm = html.getElementsByClassName("autoHeight")(0).contentWindow.document
Set frm1 = frm.getElementsByClassName("autoHeight")(1).contentWindow.document
For Each elem In frm1.getElementsByTagName("td")
If InStr(elem.innerText, "Valorem") > 0 Then MsgBox elem.NextSibling.NextSibling.innerText: Exit For
Next elem
IE.Quit
End Sub
Output:
6%

Related

When the search button is clicked using vba the text entered in search box is not seen by web page

I have written vba code for entering manufacturer part number in search box of below website and clicking on search icon. It is able enter manufacturer part number in search box and click on search icon, but when "search icon is clicked the text entered in the text box is not picked up". It searches empty data.
'HTML Part for search icon
<em class="fa fa-search" aria-hidden="true" style="color: gray;"></em>
It being almost a month I have tried various different way which was also mentioned on stack overflow, like using "createEvent("keyboardevent")" but nothing worked.
' VBA code
Sub AptivScrapping()
Dim IE As SHDocVw.InternetExplorer
Set IE = New InternetExplorer
IE.Visible = True
IE.navigate "https://ecat.aptiv.com"
Do While IE.readyState < READYSTATE_COMPLETE
Loop
Dim idoc As MSHTML.HTMLDocument
Set idoc = IE.document
idoc.getElementById("searchUserInput").Value = "33188785"
Dim doc_ele As MSHTML.IHTMLElement
Dim doc_eles As MSHTML.IHTMLElementCollection
Set doc_eles = idoc.getElementsByTagName("a")
For Each doc_ele In doc_eles
If doc_ele.getAttribute("ng-click") = "SearchButtonClick(1)" Then
doc_ele.Click
Exit Sub
Else
End If
Next doc_ele
End Sub
The page does an xhr request to retrieve the search results. You can find it in the network tab after clicking submit. This means you can avoid, in this case, the expense of a browser and issue an xhr request. The response is json so you do need a json parser to handle the results.
I would use jsonconverter.bas to parse the json. After installing the code from that link in a standard module called JsonConverter, go to VBE > Tools > References > Add a reference to Microsoft Scripting Runtime
I dimension an array to hold the results. I determine rows from the number of items in the json collection returned and the number of columns from the size of the first item dictionary. I loop the json object, and inner loop the dictionary keys of each dictionary in collection, and populate the array. I write the array out in one go at end which is less i/o expensive.
Option Explicit
Public Sub GetInfo()
Dim json As Object, ws As Worksheet, headers()
Dim item As Object, key As Variant, results(), r As Long, c As Long
Set ws = ThisWorkbook.Worksheets("Sheet1")
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://ecat.aptiv.com/json/eCatalogSearch/SearchProducts?filter=All&options=&pageSize=10&search=33188785", False
.send
Set json = JsonConverter.ParseJson(.responseText)("Products")
End With
headers = json.item(1).keys
ReDim results(1 To json.Count, 1 To UBound(headers) + 1)
For Each item In json
r = r + 1: c = 1
For Each key In item.keys
results(r, c) = item(key)
c = c + 1
Next
Next
With ws
.Cells(1, 1).Resize(1, UBound(headers) + 1) = headers
.Cells(2, 1).Resize(UBound(results, 1), UBound(results, 2)) = results
End With
You can do this instead:
txt = "33188785"
IE.navigate "https://ecat.aptiv.com/feature?search=" & txt
This will take you straight to the Search Result.
Code:
Sub AptivScrapping()
Dim IE As SHDocVw.InternetExplorer
Dim txt As String
Set IE = New InternetExplorer
txt = "33188785"
IE.Visible = True
IE.navigate "https://ecat.aptiv.com/feature?search=" & txt
Do While IE.Busy
Application.Wait DateAdd("s", 1, Now)
Loop
End Sub
This will be faster as You will only have to load one page.
Why that's happening, i am not sure, but seems like the TextBox that is used to input text is not being Activated when adding text automatically to it. It is being activated when we click inside it.
I got the solution for above problem from Mrxel.com below is the link for that post.
https://www.mrexcel.com/forum/excel-questions/1105434-vba-ie-automation-issue-angularjs-input-text-post5317832.html#post5317832
In this case I need to enter the search string character by character and sendKeys and input events inside the loop. Below is the working vba code.
Sub AptivScrapping()
Dim IE As SHDocVw.InternetExplorer
Set IE = New InternetExplorer
IE.Visible = True
IE.navigate "https://ecat.aptiv.com"
Do While IE.readyState < READYSTATE_COMPLETE
Loop
Dim idoc As MSHTML.HTMLDocument
Set idoc = IE.document
IE.document.getElementById("searchUserInput").Focus = True
IE.document.getElementById("searchUserInput").Select
sFieldInput = "33188785"
For s = 1 To Len(sFieldInput)
Application.SendKeys Mid(sFieldInput, s, 1)
While IE.readyState < 4 Or IE.Busy
Application.Wait DateAdd("s", LoopSeconds, Now)
Wend
Next s
IE.document.getElementById("searchUserInput").Focus = False
Dim doc_ele As MSHTML.IHTMLElement
Dim doc_eles As MSHTML.IHTMLElementCollection
Set doc_eles = idoc.getElementsByTagName("a")
For Each doc_ele In doc_eles
If doc_ele.getAttribute("ng-click") = "SearchButtonClick(1)" Then
doc_ele.Click
Exit Sub
Else
End If
Next doc_ele
End Sub

VBA code to copy table data from webpage into Excel

I wish to copy data from a table on a webpage into Excel using VBA code but didn't get anything on the Excel sheet :(.
I have tried to put together some VBA code from different sources. Here is my code:
Sub CopyWebData()
Dim IE As Object
On Error Resume Next
Application.DisplayAlerts = False
Set IE = CreateObject("InternetExplorer.Application")
With IE
.Visible = True
.navigate "https://eresearch.fidelity.com/eresearch/evaluate/fundamentals/earnings.jhtml?tab=details&symbols=GOOG"
Do Until .readyState = 4: DoEvents: Loop
End With
Dim idoc As MSHTML.HTMLDocument
Dim elem As MSHTML.IHTMLElement
Set idoc = IE.document
Set elem = idoc.getElementsByClassName("layout-outer-table-width")(0).innerText
Sheets("Sheet1").Activate
Range("A1:A1000") = "" ' erase previous data
Range("A1").Select
Range("A1").Value = elem
End Sub
This is a password-protected webpage and I have logged in so I can see the webpage has been successfully pulled out by the VBA code. However, the data in the table on this webpage failed to be copied into excel - I saw nothing on the destination worksheet.
As you can see, I used code .getElementsByClassName("layout-outer-table-width") since I used Chrome's "Inspect" function to check the webpage and found that when the mouse was hovering over the statements:
...<table cellspacing="0" cellpadding="0" border="0" class="layout-outer-table-width"> == $0
<tbody>...</tbody>
</table>
part of the webpage covering the table I need was shaded. I then coded in the class name "layout-outer-table-width". However, as I said, I didn't see anything appearing on the Excel sheet.
Any instruction would be much appreciated!
If after the earnings detail table you need a different selector. I am showing a css selector for that table. Your current selector (class), and index 0, is matching on a breadcrumb (nav tree). That class is also not correct for selecting the table on the page.
.earningsHistoryTable-Cont table
I cannot test this but you may also want a timed loop for table to be present
Option Explicit
'VBE > Tools > References:
' Microsoft Internet Controls
Public Sub RetrieveInfo()
Dim IE As InternetExplorer, hTable As Object, clipboard As Object, t As Date
Const MAX_WAIT_SEC As Long = 5
Set clipboard = GetObject("New:{1C3B4210-F441-11CE-B9EA-00AA006B1A69}")
Set IE = New InternetExplorer
With IE
.Visible = True
.Navigate2 "https://eresearch.fidelity.com/eresearch/evaluate/fundamentals/earnings.jhtml?tab=details&symbols=GOOG"
While .Busy Or .readyState < 4: DoEvents: Wend
With .document
.querySelector("#userId").Value = "xyz"
.querySelector("#password").Value = "123456"
.querySelector("form").submit
End With
While .Busy Or .readyState < 4: DoEvents: Wend
t = Timer 'timed loop for details table to be present
Do
On Error Resume Next
Set hTable = IE.document.querySelector(".earningsHistoryTable-Cont table")
On Error GoTo 0
If Timer - t > MAX_WAIT_SEC Then Exit Do
Loop While hTable Is Nothing
If Not hTable Is Nothing Then 'use clipboard to copy paste
clipboard.SetText hTable.outerHTML
clipboard.PutInClipboard
ThisWorkbook.Worksheets("Sheet1").Range("A1").PasteSpecial
End If
End With
End Sub

Cannot get the text inside a <p> tag using VBA

I have the following URL
https://www.wavemotion.gr/shop/smartphone-accessories/easy-one-touch-wireless-fast-charging-mount
I am trying to get the availability of the product by using the following
For i = 2 To lastrow
mylink = wks.Cells(i, 2).Value
ie.Navigate mylink
While ie.Busy Or ie.ReadyState < 4: DoEvents: Wend
t = Timer
Do
DoEvents
On Error Resume Next
Set instock = ie.Document.querySelector(".stock.in-stock").innerText
If instock Is Nothing Then
Set availability = ie.Document.querySelector(".stock.out-of-stock").innerText
Else
Set availability = instock
End If
wks.Cells(i, "D") = availability
If Timer - t > MAX_WAIT_SEC Then Exit Do
On Error GoTo 0
Loop
Next i
But I get allways nothing on
Set instock = ie.Document.querySelector(".stock.in-stock").innerText
I checked the query on
https://try.jsoup.org/
It is working
What I am doing wrong here? There is not any id to target only class name
<p class="stock in-stock">Διαθέσιμο</p>
So, what's happening here is that you're trying to Set string datatype innerText to object variable instock. The reason it's returning Nothing is because your On Error Resume Next statement is suppressing the error message. If you took that out and ran it, you would get a Type Mismatch. What you'd need to do is split it into a line that assigns the object to the object variable and then a line that reads the innerText of the assigned object.
Set instock = ie.Document.querySelector(".stock.in-stock")
If instock Is Nothing Then
Set availability = ie.Document.querySelector(".stock.out-of-stock")
Else
Set availability = instock
End If
wks.Cells(i, "D") = availability.innerText
There is a better, faster way. Use xmlhttp and parse that info out of the json stored in one of the script tags. If issuing large numbers of requests you may need to add a wait every x number of requests in case of throttling/blocking. Note: You can use the same approach with InternetExplorer and thus remove many of your lines of code, though you have another library (.bas) dependancy.
You need to install jsonconverter.bas from here and go vbe > tools > references > and add a reference to Microsoft Scripting Runtime
Option Explicit
Public Sub GetStocking()
Dim json As Object, html As HTMLDocument
Set html = New HTMLDocument
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://www.wavemotion.gr/shop/smartphone-accessories/easy-one-touch-wireless-fast-charging-mount", False
.send
html.body.innerHTML = StrConv(.responseBody, vbUnicode)
End With
Set json = JsonConverter.ParseJson(html.querySelector("script[type='application/ld+json']").innerHTML)
Debug.Print json("offers")("availability")
End Sub
This is what the entire json contains:
Internet Explorer version:
Option Explicit
Public Sub GetInfo()
Dim ie As New InternetExplorer, i As Long, s As String, scripts As Object, json As Object
With ie
.Visible = False
.Navigate2 "https://www.wavemotion.gr/shop/smartphone-accessories/easy-one-touch-wireless-fast-charging-mount"
While .Busy Or .readyState < 4: DoEvents: Wend
Set scripts = .document.querySelectorAll("script[type='application/ld+json']")
For i = 0 To scripts.Length - 1
s = scripts.item(i).innerHTML
If InStr(s, "availability") > 0 Then
Set json = JsonConverter.ParseJson(s)
Exit For
End If
Next
.Quit
If Not json Is Nothing Then Debug.Print json("offers")("availability")
End With
End Sub

VBA to insert data in Search box IE

When i insert Few words in searchbox, Its fetching related data.
I need to select first option from it.
There is one website "https://indiarailinfo.com/"
When i search "ADI" in from station box, system fetching related station having "ADI" in their name?. First option always showing very close match to it.
How can i select First Option from it using vba code
Dim ie As Object
Set ie = CreateObject("InternetExplorer.Application")
ie.navigate "https://indiarailinfo.com/"
While ie.readyState <> 4: DoEvents: Wend
ie.Visible = True
ie.document.querySelector("[placeholder='from station']").Value = "ADI"
HTML Codes can be available from that site
It's Bring First Answer in Dropdown like "ADI/Ahmedabad Junction"
How can i get this answer in selected"
Kindly Suggest
Automation purists won't like using javascript to execute but I will use here for IE to trigger dropdown. If I was going pure route I would use selenium.
Option Explicit
Public Sub MakeSelection()
Dim ie As InternetExplorer, t As Date, dropdown1 As Object
Set ie = New InternetExplorer
Const MAX_WAIT_SEC As Long = 5
With ie
.Visible = True
.Navigate2 "https://indiarailinfo.com/"
While .Busy Or .readyState < 4: DoEvents: Wend
With .document.querySelector("[placeholder='from station']")
.Focus
.Value = "ADI"
ie.document.parentWindow.execScript "document.querySelector('[placeholder^=from]').click();"
End With
t = Timer
Do
DoEvents
On Error Resume Next
Set dropdown1 = .document.querySelectorAll(".icol span")
On Error GoTo 0
If Timer - t > MAX_WAIT_SEC Then Exit Do
Loop While dropdown1.Length = 0
If dropdown1.Length > 0 Then
dropdown1.item(0).Click
End If
Stop
.Quit
End With
End Sub
For automation purists using selenium basic
Option Explicit
Public Sub MakeSelection()
Dim d As WebDriver
Set d = New ChromeDriver
Const Url = "https://indiarailinfo.com/"
With d
.Start "Chrome"
.get Url
.FindElementByCss("[placeholder='from station']").SendKeys "ADI"
.FindElementByCss(".icol span").Click
Stop
.Quit
End With
End Sub

Excel VBA: Check all check boxes on Web Page

I'm trying to check all the checkboxes available on a webpage via VBA since the name convention doesnt appear to be one in which I can be selective. However I cannot seem to get anything to work. I can login to the website and navigate to the section of the website I want but cannot cross this hurdle. Any help would be greatly appreciate. Below is the source code from the webpage.
<li data-product-family="30yr"
data-product-amortizationTerm="30"
data-product-type="Conventional"
data-product-amortizationType="Fixed"
>
<label>
<input type="checkbox"
value="154232"
class="product-Conventional product-item"
data-authorized-remittance-types="ActualActual "
/>30-Year Fixed Rate - 110k Max Loan Amount</label>
</li>
VBA I attempted to write (edited)... code I'm using presently:
Public Sub TestIE()
Dim IE As Object
Dim aNodeList As Object, i As Long
' Create InternetExplorer Object
Set IE = CreateObject("InternetExplorer.Application")
' You can uncoment Next line To see form results
IE.Visible = False
' Send the form data To URL As POST binary request
IE.Navigate "https://"
' Statusbar
Application.StatusBar = "Page is loading. Please wait..."
' Wait while IE loading...
Do While IE.Busy
Application.Wait DateAdd("s", 1, Now)
Loop
IE.Visible = True
Set aNodeList = IE.document.querySelectorAll("input[type=checkbox]")
If aNodeList Is Nothing Then Exit Sub
For i = 0 To aNodeList.Length
aNodeList.Item(i).Checked = True
Next i
End Sub
You can try to get a nodeList of the checkboxes with:
IE.document.querySelectorAll("input[type=checkbox]")
You can traverse the nodeList along its .Length property.
E.g.
Dim aNodeList As Object, i As Long
Set aNodeList = IE.document.querySelectorAll("input[type=checkbox]")
If aNodeList Is Nothing Then Exit Sub
For i = 0 To aNodeList.Length -1
On Error Resume Next
aNodeList.item(i).Checked = True
On Error GoTo 0
Next i

Resources