Loop through NextSibling DIV tagname Selenium VBA - excel

Question

In my HTML page, at this link https://pastebin.com/nu0dLvch
There are elements which have the id "DetailSection1" .. And I am trying to loop through the div tags which is after this id (the tags are five in count)
This is my try but didn't work for me
Dim v, post As Object, sibling As Object, i As Long
Set post = .FindElementsByCss("#DetailSection1")
For i = 1 To post.Count
'Debug.Print post.Item(i).Attribute("outerHTML")
Set sibling = post.Item(i).NextSibling
Select Case sibling.NodeType
Case 3
v = sibling.NodeValue
Case 1
v = sibling.innerText
End Select
Debug.Print v
Next i
How can I loop through the five tags of DIV after this element of id "#DetailSection1"?
This is what I got as Arabic characters (Green is what I got while the yellow part is the correct characters)

Answer 1

1) If you want to use your logic then you will need to loop the nextElementSiblings until empty string is returned (rather than hardcode loop to 5). n.b. I am using nextElementSibling as I want the next sibling element node (node type 1).
W3C Element Traversal Specification
2.4. nextElementSibling:
Accessing this attribute of an element must return a reference to the sibling node of that element which most
immediately follows that element in document order, and which is of
nodeType 1, as an Element object. If the element on which this
attribute is accessed does not have any following sibling nodes, or if
none of those following sibling nodes are element nodes, then this
attribute must return null.
This is an ie version, as I can test that, but should be easy to translate. Selenium Basic does not expose nextElementSibling method so you can still use HTMLDocument and querySelector to access from page html, via transfer from pageSource of webdriver (assuming no non-trivial html changes with MSHTML.HTMLDocument), or switch to nextSibling traversal; and add in nodeType = 1 test during loop.
Option Explicit
Public Sub TEST()
Dim ie As New InternetExplorer
ie.Visible = True
'ie.Document.Charset = "utf-8" ''< may be require for handling Arabic chars. Not required with my settings.
ie.Navigate2 "file:///C:/Users/<User>/Desktop/Test.html"
While ie.Busy Or ie.readyState <> READYSTATE_COMPLETE: DoEvents: Wend
Dim startNodes As Object, node As Object, firstNode As Boolean
Dim i As Long, c As Long, r As Long
Set startNodes = ie.document.querySelectorAll("#DetailSection1")
r = 1
For i = 0 To startNodes.Length - 1 'you could determine number of child divs to get num columns then use step 5 loop or mod to write out in rows and cols
Set node = Nothing
firstNode = True
c = 1
Do
If firstNode Then
Set node = startNodes.item(i).nextElementSibling
Else
Set node = node.nextElementSibling
End If
If node.innerText <> vbNullString Then
c = c + 1 'you may need backwards loop to reverse output columns
ActiveSheet.Cells(r, c) = Trim$(node.innerText)
End If
firstNode = False
Loop Until node.innerText = vbNullString
r = r + 1
Next
ie.Quit
End Sub
2) You could dynamically pick up class and use a combination of css attribute selectors to hopefully correctly target nodes of interest. I have read in your html from file via IE. It is shaky due to reliance on attributes and relationships. It is all nested tables with few distinguishing features and likely dynamic attribute values. If not dynamic, then hardcode value for targetClass as ad66b5fc2d-4b59-45e6-b104-e14dfb5b1dac-0.
Option Explicit
Public Sub Test()
Dim ie As New InternetExplorer
ie.Visible = True
ie.Navigate2 "file:///C:/Users/User/Desktop/Test.html"
While ie.Busy Or ie.readyState <> READYSTATE_COMPLETE: DoEvents: Wend
Dim startNodes As Object, i As Long, targetClass As String, cssSelector As String
targetClass = "." & ie.document.querySelector("#DetailSection1").nextElementSibling.className ' "." & ie.document.querySelector("div[style*='center']").className
cssSelector = targetClass & "[style*='center'], " & targetClass & "[style*='center'] ~ div[style*='text-align']"
Set startNodes = ie.document.querySelectorAll(cssSelector)
For i = 0 To startNodes.Length - 2 'you could determine number of child divs to get num columns then use step 5 loop or mod to write out in rows and cols
ActiveSheet.Cells(i + 1, 1) = startNodes.item(i).innerText
Next
ie.Quit
End Sub
Untested selenium translation (can't test so written from my memory (eek!):
Dim startNodes As Object, i As Long, targetClass As String, cssSelector As String
targetClass = "." & .FindElementByCss("div[style*='center']").Attribute("class")
cssSelector = targetClass & "[style*='center'], " & targetClass & "[style*='center'] ~ div[style*='text-align']"
Set startNodes = .FindElementsByCss(cssSelector)
For i = 0 To startNodes.Count - 2 'you could determine number of child divs to get num columns then use step 5 loop or mode to write out in rows and cols
ActiveSheet.Cells(i + 1, 1) = startNodes.item(i).Text
Next
You can explore whether you can get targetClass from (nodeType = 1):
.FindElementByCss("#DetailSection1").nextSibling.className
.FindElementByCss("#DetailSection1").nextSibling.Attribute("class") '<== I think this
I'm sorry I can't test. I'm not sure those versions will work but would be good to know.
Ref:
https://stackoverflow.com/a/62366101/6241235 # أبو عائشة ورقية ومحمد

Answer 2

In fact, I am very satisfied with the solutions presented by QHarr. But I am eager to discover and learn new skills, so I am trying on my side after studying well what QHarr presented and this is my try-on selenium
Sub Test()
Dim bot As New ChromeDriver, a(1 To 1000, 1 To 5), post As Object, i As Long, j As Long
With bot
.AddArgument "--headless"
.Get "file:///C:\Sample.html"
Set post = .FindElementsByCss("#DetailSection1")
If post.Count > 0 Then
For i = 1 To post.Count
For j = 1 To 5
a(i, j) = Application.WorksheetFunction.Clean(Trim$(post.Item(i).FindElementsByXPath("following-sibling::div")(j).Text))
Next j
Next i
ActiveSheet.Range("A1").Resize(post.Count, UBound(a, 2)).Value = a
End If
End With
End Sub
** Note: I welcome any new and other ideas as I am eager to learn about different approaches.