Error 91 Using For Loop with XMLHttpRequest VBA - excel

I am trying to scrape a site for the contents of various HTML tags. I am using an array of html tags against an array of URLs.
Sub pArse(UserForm4_HiddenList_Text)
Dim http As New XMLHTTP60
'Dim html As New HTMLDocument
Dim hero_true_Val As Integer
Dim down_var As Integer
Dim bill_array_redim
Dim element_tag As String
Dim address_count As Integer
hero_true_Val = Application.WorksheetFunction.CountIf(Range("B:B"), True)
down_var = 1
bill_array_redim = Split(UserForm4_HiddenList_Text, Chr(10))
ReDim address_array(hero_true_Val)
For Z = 2 To 6
If Sheets("resource").Cells(Z, 2).Value <> "False" Then
address_count = address_count + 1
address_array(address_count) = Sheets("resource").Cells(Z, 1).Value
End If
Next Z
Sheets("Sheet1").Select
For url_stack = 0 To UBound(bill_array_redim)
Sheets("sheet1").Cells(down_var, 1) = bill_array_redim(url_stack)
For what_to_check = 1 To address_count
With http
.Open "GET", bill_array_redim(url_stack), False
.send
Do While http.readyState <> 4
DoEvents
Loop
End With
Dim html As New HTMLDocument
With html
.body.innerHTML = http.responseText
' the next line is where the error occurs
Sheets("Sheet1").Cells(down_var, 2).Value = .querySelector(address_array(what_to_check)).innerText
End With
down_var = down_var + 1
Next what_to_check
Next url_stack
UserForm4.Hide
End Sub
A user clicks a command button which then feeds in an array of URLs (UserForm4_HiddenList_Text). The code then checks for the existence of the word "true" in the sheet named "resource." This generates an array of terms to check for.
the problem I'm having is that on the second iteration of the what_to_check loop, I'm getting
error 91 : Object or with block variable not set
I'm unsure of what the issue is here and it only seems to occur if I pass in two html tags rather than one.
edit: thanks for replying Cindy. The error location is bolded. The message is "Object or with block variable not set."

Turns out I was being given an error I either didn't fully understand or wasn't very clear. The issue was that the response didn't have the tag inside it that I was looking for as it wasn't present on the site. By using:
With html
.body.innerHTML = http.responseText
If .body.contains(.querySelector(tag_array(tag_array_index_no)))
Then
Sheets(1).Range(Cells(down_var, 2).Address).Value =
tag_array(tag_array_index_no)
Sheets(1).Range(Cells(down_var, 3).Address).Value =
.querySelector(tag_array(tag_array_index_no)).innerText
Else
Sheets(1).Range(Cells(down_var, 2).Address).Value =
tag_array(tag_array_index_no)
Sheets(1).Range(Cells(down_var, 3).Address).Value =
tag_array(tag_array_index_no) + " not set."
End If
End With
I'm able to check if the tag exists in the response text.

Related

Webscraping of product prices and specs

I'm new to the VBA world and i was trying to create a Excel file to get the cheapest price and name tag on a website. I've created one file that goes through the whole search list and gathers each product URL and places on a spreadsheet. My challenge now is in making this second code work. It starts out getting the information without any issues but then, after 10 or more URLs the macro gives a bug and it starts repeating the information on all the following cells.. Is there a way i can make the code run slower so it doesn't get into this problem?
I'll list here my code and a sample of the URLs im scraping..
here's the code i've been using:
Sub test()
Dim URL As String
Set ie = CreateObject("internetexplorer.application")
For i = 2 To 300
URL = Cells(i, 1).Value
ie.navigate URL
ie.Visible = False
Do While ie.busy And ie.readystate <> "readystate_complete"
DoEvents
Loop
Cells(i, 3) = ie.document.getElementsByTagName("h1")(0).innerText
Cells(i, 4) = ie.document.getElementsByTagName("strong")(0).innerText
Next i
ie.Quit
MsgBox "acabou"
End Sub
<https://www.zoom.com.br/notebook/notebook-vaio-vjf157f11x-b0211s-intel-core-i5-8250u-15-6-8gb-ssd-256-gb-windows-10-8-geracao>
<https://www.zoom.com.br/notebook/notebook-samsung-chromebook-plus-intel-celeron-3965y-12-2-4gb-emmc-32-gb-chrome-os-touchscreen>
<https://www.zoom.com.br/notebook/notebook-dell-xps-7390-intel-core-i7-10710u-13-3-16gb-ssd-512-gb-windows-10-touchscreen>
<https://www.zoom.com.br/notebook/notebook-dell-i15-3583-a5-intel-core-i7-8565u-15-6-8gb-hd-2-tb-windows-10-8-geracao>
<https://www.zoom.com.br/notebook/notebook-lenovo-b330-intel-core-i5-8250u-15-6-4gb-hd-1-tb-windows-10-8-geracao>
<https://www.zoom.com.br/notebook/notebook-dell-i15-7580-a20-intel-core-i7-8550u-15-6-8gb-hd-1-tb-geforce-mx150-windows-10>
<https://www.zoom.com.br/notebook/notebook-dell-i14-3480-u30-intel-core-i5-8265u-14-4gb-hd-1-tb-linux-8-geracao>
<https://www.zoom.com.br/notebook/macbook-pro-apple-muhn2bz-intel-core-i5-13-3-8gb-ssd-128-gb-tela-de-retina>
<https://www.zoom.com.br/notebook/notebook-multilaser-pc150-amd-a4-9120-14-2gb-emmc-32-gb-windows-10>
<https://www.zoom.com.br/notebook/notebook-samsung-np930qaa-kw1br-intel-core-i7-8550u-13-3-8gb-ssd-256-gb-windows-10-touchscreen>
<https://www.zoom.com.br/notebook/notebook-acer-a515-51g-58vh-intel-core-i5-7200u-15-6-8gb-hd-1-tb-geforce-940mx>
<https://www.zoom.com.br/notebook/notebook-multilaser-pc222-intel-celeron-dual-core-13-3-4gb-emmc-64-gb-windows-10>
<https://www.zoom.com.br/notebook/notebook-acer-pt515-51-788a-intel-core-i7-9750h-15-6-32gb-ssd-1-tb-geforce-rtx-2070-windows-10>
<https://www.zoom.com.br/notebook/notebook-acer-a315-53-53ak-intel-core-i5-7200u-15-6-4gb-hd-1-tb-windows-10-7-geracao>
<https://www.zoom.com.br/notebook/notebook-dell-i15-5584-m40-intel-core-i7-8565u-15-6-8gb-hd-2-tb-geforce-mx130-windows-10>
<https://www.zoom.com.br/notebook/notebook-acer-a315-41g-r21b-amd-ryzen-5-2500u-15-6-8gb-hd-1-tb-radeon-535-windows-10>
<https://www.zoom.com.br/notebook/notebook-positivo-master-n2140-intel-core-i3-7020u-14-4gb-hd-500-gb-windows-10-7-geracao>
<https://www.zoom.com.br/notebook/notebook-multilaser-pc101-intel-atom-14-1gb-ssd-32-gb-windows-10>
<https://www.zoom.com.br/notebook/notebook-lenovo-b330-intel-core-i5-8250u-15-6-8gb-hd-1-tb-windows-10-8-geracao>
<https://www.zoom.com.br/notebook/notebook-acer-an515-51-77fh-intel-core-i7-7700hq-15-6-8gb-hd-1-tb-geforce-gtx-1050-windows-10>
<https://www.zoom.com.br/notebook/notebook-dell-i15-3583-a2yp-intel-core-i5-8265u-15-6-4gb-optane-16-gb-hd-1-tb-windows-10>
<https://www.zoom.com.br/notebook/notebook-asus-g531gt-intel-core-i7-9750h-15-6-16gb-ssd-512-gb-geforce-gtx-1650-windows-10>
<https://www.zoom.com.br/notebook/notebook-vaio-fit-15s-intel-core-i3-7100u-15-6-4gb-hd-1-tb-windows-10-home>
<https://www.zoom.com.br/notebook/notebook-samsung-s50-intel-core-i7-7500u-13-3-8gb-ssd-256-gb-windows-10-style>
<https://www.zoom.com.br/notebook/notebook-lenovo-b330-intel-core-i3-7020u-15-6-4gb-ssd-120-gb-windows-10-7-geracao>
First of all:
Always declare all variables. To force this, always write Option Explicit as the first line in each module. This way, especially typos in variable names are immediately detected by the compiler.
Here is what to do about your problem:
IE is sometimes a real diva. For example, it doesn't like to have to process URLs in the same instance in quick succession. Therefore it is advisable to kick it out of memory and restart it for each new URL.
To restart it quickly, the deletion of coockies, the cache, etc. must not be set in its settings. Otherwise automation errors will occur.
Try this macro. With the given URLs it works:
Option Explicit
Sub test()
Dim URL As String
Dim ie As Object
Dim i As Long
For i = 2 To 300
If i > 14 Then
ActiveWindow.SmallScroll down:=1
End If
URL = ActiveSheet.Cells(i, 1).Value
Set ie = CreateObject("internetexplorer.application")
ie.navigate URL
ie.Visible = False
Do While ie.readystate <> 4: DoEvents: Loop
ActiveSheet.Cells(i, 3) = ie.document.getElementsByTagName("h1")(0).innerText
ActiveSheet.Cells(i, 4) = ie.document.getElementsByTagName("strong")(0).innerText
ie.Quit
Set ie = Nothing
Next i
MsgBox "acabou"
End Sub
Try this
Sub GetPrices()
Dim html As MSHTML.HTMLDocument, r As Long
For r = 1 To 4
Set html = GetHTML(Cells(r, 1).Value)
Cells(r, 3).Value = html.querySelector("h1.product-name").innerText
Cells(r, 4).Value = Replace(Replace(html.querySelector(".product-price").innerText, "a partir de ", vbNullString), ":( ", "")
Set html = Nothing
Next r
End Sub
Function GetHTML(ByVal sURL As String) As HTMLDocument
Dim http As MSXML2.XMLHTTP60, html As MSHTML.HTMLDocument
Set http = New MSXML2.XMLHTTP60
Set html = New MSHTML.HTMLDocument
With http
.Open "Get", sURL, False
.send
html.body.innerHTML = .responseText
End With
Set GetHTML = html
End Function
In python I will do :
You should first import library
from time import sleep
Two functions available :
this function let you sleep before execute every 3 seconds
time.sleep(3)
this function let you random sleep between every execute from 1 to 3 seconds
sleep(randint(1, 3)
Note: take aware about :
cookies because sometimes you need to post request to scrape some id before get.
the syntax of your header and set correctly origin & referrer parameters.

Get pictures links from Google Search

I am trying to get the pictures links from a searched link through google and this is my try
Sub Test()
Const sURL As String = "https://www.google.com.eg/search?q=baby&sxsrf=ALeKk01tyfvvxyYjaC0YctjxaY0RlvPnuw:1586804351129&source=lnms&tbm=isch&sa=X&ved=2ahUKEwjB77TtiuboAhUl5uAKHR5KA2wQ_AUoAXoECBQQAw&biw=1280&bih=881"
Dim http As MSXML2.XMLHTTP60, html As MSHTML.HTMLDocument
Set http = New MSXML2.XMLHTTP60
Set html = New MSHTML.HTMLDocument
With http
.Open "Get", sURL, False
.send
html.body.innerHTML = .responseText
End With
Dim post As Object, i As Long
Set post = html.querySelectorAll(".mM5pbd .bRMDJf")
For i = 0 To post.Length - 1
Debug.Print post.Item(i).innerHTML
Next i
Stop
End Sub
First I got the post.Length only 20 while I expect about 300
Second I can't get the correct link for the picture as it seems it is base64 encrypted or something similar (I am not sure)
How can I get the real links for the picture and get all the links for all the pictures related?
I think it is solved for one point
Set post = html.querySelectorAll("a.VFACy.kGQAp")
For i = 0 To post.Length - 1
Debug.Print post.Item(i).href
Next i
But how to get all the links instead of the 20 links only?
** The links are not totally right, for example I got this link
https://www.fool.com/taxes/2018/03/27/are-you-having-a-baby-here-are-the-tax-breaks-you.aspx
While the correct link is
https://g.foolcdn.com/editorial/images/466737/new-parents-holding-newborn-baby-mom-dad-father-mother.jpg
** I tried using IE
Sub TestIE()
Dim ie As New InternetExplorer
Dim lastrow As Long
Dim i As Long
Dim j As Long
lastrow = Range("A" & Rows.Count).End(xlUp).Row
For i = 2 To lastrow
ie.Visible = True
ie.navigate "https://www.google.com.eg/search?q=baby&sxsrf=ALeKk01tyfvvxyYjaC0YctjxaY0RlvPnuw:1586804351129&source=lnms&tbm=isch&sa=X&ved=2ahUKEwjB77TtiuboAhUl5uAKHR5KA2wQ_AUoAXoECBQQAw&biw=1280&bih=881"
While ie.Busy Or ie.readyState < 4: DoEvents: Wend
'querySelectorAll("a.VFACy.kGQAp")
Dim post As Object
Set post = ie.document.querySelectorAll("a.VFACy.kGQAp")
For j = 0 To post.Length - 1
Debug.Print post.Item(i).innerHTML
Next j
Next
End Sub
But in results I got the same innerhtml for all
<div class="sMi44c lNHeqe"><div class="WGvvNb" dir="ltr">Baby colic - Wikipedia</div><div class="fxgdke"><span dir="ltr">en.wikipedia.org</span></div>
</div>
Is using QuerySelectorAll different when dealing with IE?
** Another try
Dim post As Object
Set post = ie.document.querySelectorAll(".bRMDJf img")
Dim r As Long
For j = 0 To post.Length - 1
r = r + 1
Cells(r, 1).Value = post.Item(i).getAttribute("src")
Next j
Now I got 100 but not the links, it is base64 encryption for the pictures, moreover I found out the output is the same for all the pictures. I could decrypt the pictures but the quality is low .. and I got only 100 .. How can I increase the number of results and get the correct links?

Web Scraping: Button clicking and help navigating through paths

I am trying to scrape some doctor names and addresses from the website: https://albertafindadoctor.ca/find-a-doc/directory
I am trying to solve the following issue:
Once on the doctor's toggle, I want to pull 4 pieces of data from the entire page, not just the first 25 displayed.
While the code works for the initial webpage, it only pulls the first 25 pieces of data. There are a significant number of other pages that I still need to pull (3822 different doctors).
Unfortunately, I'm at a loss on how to navigate and pull from these different pages. When I inspect elements to see how to navigate between pages a see matrix changing so I'm not sure if that has something to do with it?
Option Explicit
Sub GetAlbertaDoctors()
Dim objIE As InternetExplorer
Dim clinicEle As Object
Dim clinicName As String
Dim clinicAddress As String
Dim clinicCategory As String
Dim doctorName As String
Dim y As Integer
Set objIE = New InternetExplorer
objIE.Visible = True
objIE.Navigate "https://albertafindadoctor.ca/find-a-doc/directory"
While objIE.Busy = True Or objIE.readyState <> 4: DoEvents: Wend
Application.Wait (Now + TimeValue("0:00:5"))
objIE.Document.getElementsByClassName("physician-toggle")(0).Click
Application.Wait (Now + TimeValue("0:00:5"))
y = 2
For Each clinicEle In objIE.Document.getElementsByClassName("clinic")
clinicCategory = clinicEle.getElementsByClassName("pcn")(0).innerText
clinicName = clinicEle.getElementsByClassName("clinic-name")(0).innerText
doctorName = clinicEle.getElementsByTagName("h3")(0).innerText
clinicAddress = clinicEle.getElementsByClassName("address")(0).innerText
Sheets("Sheet2").Range("A" & y).Value = clinicCategory
Sheets("Sheet2").Range("B" & y).Value = clinicName
Sheets("Sheet2").Range("C" & y).Value = doctorName
Sheets("Sheet2").Range("D" & y).Value = clinicAddress
y = y + 1
Next
objIE.Quit
End Sub
When I run this, I get the error 91 "Object variable or With block variable not set" on the clicking line:
objIE.Document.getElementsByClassName("physician-toggle active")(0).Click
You don't need to loop all pages. You can use the browser to get to that page and click on Doctors if required. After that, grab the number of results and then mimic the xhr request the page makes for listings - which is returned as json. Alter the query string the page makes i.e. the parameter for limit to get all listings. Use a json parser (I use jsonconverter - instructions in the code for installation) to parse out your info.
There is a proper page load wait and a couple of loops to ensure elements are present. These should really be timed loops. See loop format here.
I add an additional test to ensure you do not attempt to click Doctors when it is not required to do so.
Not all listings has all info hence the On Error Resume Next paired with On Error GoTo 0. Looks like you may be able to build a dictionary to fill in some of the blank values based on existing paired values (or using ids present in json object).
I store all results in an array and write out in one go.
Option Explicit
'VBE > Tools > References:
' Microsoft Internet Controls
' Microsoft Scripting Runtime
'Download and add to standard module called jsonconverter from https://github.com/VBA-tools/VBA-JSON/blob/master/JsonConverter.bas
Public Sub GetListings()
Dim ie As InternetExplorer, s As String, json As Object, newUrl As String
Set ie = New InternetExplorer
With ie
.Visible = True
.Navigate2 "https://albertafindadoctor.ca/find-a-doc/directory"
While .Busy Or .readyState < 4: DoEvents: Wend
With .document.querySelector("[data-cp-option=physician]")
If Not .className = "physician-toggle active" Then .Click
End With
Dim resultsInfo() As String, numResults As Long, ele As Object
Do
On Error Resume Next
Set ele = .document.querySelector(".paginator")
On Error GoTo 0
Loop While ele Is Nothing
Do
Loop While .document.querySelector(".paginator").innerText = vbNullString
resultsInfo = Split(Trim$(.document.querySelector(".paginator").innerText), "of ")
.Quit
End With
numResults = resultsInfo(UBound(resultsInfo))
newUrl = "https://albertafindadoctor.ca/search/directory/physicians?page=1&limit=" & numResults & "&with[]=pcn&with[]=clinics&with[]=languages&with[]=specialties"
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", newUrl, False
.send
Set json = JsonConverter.ParseJson(.responseText)("items")
End With
Dim row As Object, results(), r As Long, headers(), ws As Worksheet, key As Variant
headers = Array("clinicCategory", "clinicName", "doctorName", "clinicAddress")
Set ws = ThisWorkbook.Worksheets("Sheet1")
ReDim results(1 To json.Count, 1 To UBound(headers) + 1)
For Each row In json
r = r + 1
On Error Resume Next
For Each key In row.keys
Select Case key
Case "clinical_name"
results(r, 3) = row(key)
Case "pcn"
results(r, 1) = row(key)("name")
Case "clinics"
results(r, 2) = row(key)(1)("name")
results(r, 4) = Join$(Array(row(key)(1)("street_address"), row(key)(1)("city"), row(key)(1)("province"), row(key)(1)("postal_code")), ", ")
End Select
Next
On Error GoTo 0
Next
With ws
.Cells(1, 1).Resize(1, UBound(headers) + 1) = headers
.Cells(2, 1).Resize(UBound(results, 1), UBound(results, 2)) = results
End With
End Sub
Sample output:
Reading:
querySelector
json
css selectors
arrays and arrays2

web scraping using excel and VBA

i wrote my VBA code in excel sheet as below but it is not scrape data for me and also i don't know why please any one help me. it gave me reullt as "click her to read more" onlyi want to scrape enitre data such as first name last name state zip code and so on
Sub extractTablesData()
Dim IE As Object, obj As Object
Dim myState As String
Dim r As Integer, c As Integer, t As Integer
Dim elemCollection As Object
Set IE = CreateObject("InternetExplorer.Application")
myState = InputBox("Enter the city where you wish to work")
With IE
.Visible = True
.navigate ("http://www.funeralhomes.com/go/listing/Search? name=&city=&state=&country=USA&zip=&radius=")
While IE.readyState <> 4
DoEvents
Wend
For Each obj In IE.document.all.item("state").Options
If obj.innerText = myState Then
obj.Selected = True
End If
Next obj
IE.document.getElementsByValue("Search").item.Click
Do While IE.Busy: DoEvents: Loop
ThisWorkbook.Sheets("Sheet1").Range("A1:K1500").ClearContents
Set elemCollection = IE.document.getElementsByTagName("TABLE")
For t = 0 To (elemCollection.Length - 1)
For r = 0 To (elemCollection(t).Rows.Length - 1)
For c = 0 To (elemCollection(t).Rows(r).Cells.Length - 1)
ThisWorkbook.Worksheets(1).Cells(r + 1, c + 1) = elemCollection(t).Rows(r).Cells(c).innerText
Next c
Next r
Next t
End With
Set IE = Nothing
End Sub
Using the same URL as the answer already given you could alternatively select with CSS selectors to get the elements of interest, and use split to get just the names and address parts from the text. We can also do away with the browser altogether to get faster results from first results page.
Business name:
You can get the name with the following selector (using paid listing example):
div.paid-listing .listing-title
This selects (sample view)
Try
Address info:
The associated descriptive information can be retrieved with the selector:
div.paid-listing .address-summary
And then using split we can parse this into just the address information.
Code:
Option Explicit
Public Sub GetTitleAndAddress()
Dim oHtml As HTMLDocument, nodeList1 As Object, nodeList2 As Object, i As Long
Const URL As String = "http://www.funeralhomes.com/go/listing/ShowListing/USA/New%20York/New%20York"
Set oHtml = New HTMLDocument
With CreateObject("WINHTTP.WinHTTPRequest.5.1")
.Open "GET", URL, False
.send
oHtml.body.innerHTML = .responseText
End With
Set nodeList1 = oHtml.querySelectorAll("div.paid-listing .listing-title")
Set nodeList2 = oHtml.querySelectorAll("div.paid-listing .address-summary")
With Worksheets("Sheet3")
.UsedRange.ClearContents
For i = 0 To nodeList1.Length - 1
.Range("A" & i + 1) = nodeList1.Item(i).innerText
.Range("B" & i + 1) = Split(nodeList2.Item(i).innerText, Chr$(10))(0)
Next i
End With
End Sub
Example output:
Yeah, without an API, this can be very tricky at best, and very inconsistent at worst. For now, you can try the script below.
Sub DumpData()
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
URL = "http://www.funeralhomes.com/go/listing/ShowListing/USA/New%20York/New%20York"
'Wait for site to fully load
IE.Navigate2 URL
Do While IE.Busy = True
DoEvents
Loop
RowCount = 1
With Sheets("Sheet1")
.Cells.ClearContents
RowCount = 1
For Each itm In IE.document.all
If itm.classname Like "*free-listing*" Or itm.classname Like "*paid-listing*" Then
.Range("A" & RowCount) = itm.classname
.Range("B" & RowCount) = Left(itm.innertext, 1024)
RowCount = RowCount + 1
End If
Next itm
End With
End Sub
You probably want some kind of input box to capture the city and state and radius from the user, or capture those variable in cells in your worksheet.
Notice, the '%20' is a space character.
I got this idea from a friend of mine, Joel, a long time ago. That guy is great!

VBA - Excel - Automation Error Unspecified Error

So I ran into a slight stumbling block and hopefully here someone can help me. In short, I need to visit a string of webpages (the list of the names on each page are already input, that code works fine). As my code visits each page, I need to pull back information. Unfortunately, there's a problem - it can't even make it through the "A" list before I get "Automation Error Unspecified Error" and it's never at the same spot.
I've tried the "normal" steps to fix this. I've installed the VB 6 Controls and I've unregistered and re-registered mscomctl.ocx, and including On Error Resume Next (which doesn't do anything).
It usually reaches over 100 cases before it dies (randomly as I said earlier). And AFTER the error pops up, when I try to re-run it (with or without changes) and it errors on the first one. If I restart my computer it will let me try again (for whatever reason) but it still doesn't finish.
Is the code too complex and I need to reduce it? I can probably find a way to make it only run for each letter at a time (run all A's, then do B's, etc) but I still can't even get it to complete the letter A.
I noticed in another thread someone had suggested instead of using IE to swap to xmlhttp - is that a fix for this? Is the problem that this script is too long? What exactly am I doing wrong here?
Sub Lookup()
Range("AI1").Value = "Unique ID"
Range("AJ1").Value = "Name"
Range("AK1").Value = "Birth Year"
Range("AL1").Value = "Title"
Range("AM1").Value = "State"
Range("AN1").Value = "Position"
Range("AO1").Value = "Country"
Range("AP1").Value = "Appointed"
Range("AQ1").Value = "Credentials"
Range("AR1").Value = "Terminations"
Dim i As Integer
For i = 1 To 26
If i = 24 Then
Range("X:X").End(xlUp).Select
ActiveCell.Value = ""
Else
Dim ic As String
ic = LCase(ConvertToLetter(i))
Range(ic & "5000").End(xlUp).Select
Dim J As Integer
J = ActiveCell.Row
Dim k As Integer
For k = 2 To J
Range(ic & k).Select
Dim Lookup As String
Lookup = ActiveCell.Value
Dim IE As Variant
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = False
IE.navigate "http://history.state.gov/departmenthistory/people/" & Lookup
Do
DoEvents
Loop Until IE.readyState = READYSTATE_COMPLETE
Dim Doc As HTMLDocument
Set Doc = IE.document
Dim Italics As Integer
Italics = 0
Dim EachA As Integer
For EachA = 64 To 100
Dim Position As String
Position = Doc.getElementsByTagName("a")(EachA).innerText
If Position = "Home" Then
Exit For
Else
Dim NameBY As String
NameBY = Doc.getElementsByTagName("h2")(1).innerText
Dim TitleST As String
TitleST = Doc.getElementsByTagName("p")(1).innerText
Range("AJ" & "90000").End(xlUp).Offset(1, 0).Select
ActiveCell.Value = NameBY
TitleState = Split(TitleST, vbLf)
ActiveCell.Offset(0, 2).Value = TitleState(0)
On Error GoTo 1037
ActiveCell.Offset(0, 3).Value = TitleState(1)
On Error GoTo 1037
1037
ActiveCell.Offset(0, 4).Select
ActiveCell.Value = Position
Dim EachLi As Integer
EachLi = EachA - 1
If Doc.getElementsByTagName("li").Item(EachLi + Italics).innerHTML Like "<em>*" Then
Italics = Italics + 1
Else
End If
Dim JobList As String
JobList = Doc.getElementsByTagName("li")(EachLi + Italics).innerText
Dim Job() As String
Job() = Split(JobList, vbLf)
Dim JCount As Integer
For JCount = LBound(Job) To UBound(Job)
ActiveCell.Offset(0, 1).Select
ActiveCell.Value = Job(JCount)
Next JCount
End If
Next EachA
Next k
End If
Next i
End Sub
One thing I notice is that you're continually creating new IE objects inside the loop, and you're never destroying them or setting to Nothing. It's pointless, expensive, and possibly a source of error to be creating 100+ instances of IE.
I think it will probably help to create a single instance of IE initially, and then use that same object inside the loop to navigate the desired URLs.
So instead of this:
Dim IE As Variant
Set IE = CreateObject("InternetExplorer.Application")
Do this:
Dim IE as Object
If IE Is Nothing Then Set IE = CreateObject("InternetExplorer.Application")

Resources