My match function is taking too long (3 hours!!), need another recommendation - excel

Question

As the title says, match function taking too long. One spreadsheet is 100,000 rows long and it has a bunch of securities that i need to make sure are on another spreadsheet which has 800,000 rows. Below is the code:
FYI i am average in code building so i am pretty rudimentary in terms of laying out my arguments.
Option Explicit
'a lot of dims
StartTime = Timer
Set ShVar = ThisWorkbook.Worksheets("in1")
With wnewwqr
Set OutShVar = wnewwqr.Worksheets("First Sheet")
Set RngConcat = OutShVar.Range("B:B")
Set RngConcatISIN = OutShVar.Range("A:A")
Set OutShVar1 = wnewwqr.Worksheets("Second Sheet")
Set RngConcat1 = OutShVar1.Range("B:B")
Set RngConcatISIN1 = OutShVar1.Range("A:A")
End With
With ShVar
lastrow = .Cells(.Rows.Count, "H").End(xlUp).Row
End With
For i = 2 To lastrow
With ShVar
If .Range("O" & i).Value = "" Then
.Range("P" & i & ":Q" & i).Value = "No Security" 'Checking for no securities
Else
If Not IsError(Application.Match(.Range("O" & i).Value, RngConcat, 0)) Then
.Range("P" & i).Value = "US" ' writing US when it finds a US security in the confidential workbook
Else
.Range("P" & i).Value = "Not a US Security"
End If
End If
If .Range("P" & i).Value = "Not a US Security" Then
If Not IsError(Application.Match(.Range("O" & i).Value, RngConcat1, 0)) Then 'Only searching for securities if the first vlookup resulted in nothing and then it would go into the second sheet
.Range("Q" & i).Value = "US"
Else
.Range("Q" & i).Value = .Range("P" & i).Value
End If
End If
End With
Next i
SecondsElapsed = Round(Timer - StartTime, 2)
'Notify user in seconds
MsgBox "This code ran successfully in " & SecondsElapsed & " seconds", vbInformation
End Sub
Update:
I have turned everything to variant and now using find function but still not that fast as i would have hoped. Took 14 mins approx. to do a trial run of 2000 rows. And i have to do this on 90,000 rows
Option Explicit
Sub something
Dim lastrow As Long
Dim OutShVar As Worksheet
Dim ShVar As Worksheet
Dim WhatCell As Range
Dim i As Long
Dim TaskID As Variant
Dim confidentialfp As String
Dim confidential As String
Dim wconfidential As Workbook
Dim x As Variant
Set ShVar = ThisWorkbook.Worksheets("in1")
With ShVar
lastrow = .Cells(.Rows.Count, "H").End(xlUp).Row
End With
confidential = "confidential_2018-03-01 (Consolidated).xlsx"
Set wconfidential = Workbooks(confidential)
With wconfidential
Set OutShVar = .Worksheets("First Sheet")
End With
With ShVar
For i = 1 To lastrow
TaskID = ShVar.Range("O" & i).Value
Set x = .Range("A" & i)
Set WhatCell = OutShVar.Range("B:B").Find(TaskID, lookat:=xlWhole)
On Error Resume Next
x.Offset(0, 7).Value = WhatCell.Offset(0, 1)
Next i
End With
End Sub

Answer 1

I'm not sure you're quite getting ScottCraner's point. What he's saying is you should read all of your reference values (ie the big list of securities) into a couple of arrays, and you should write your output values to another array. You'd then write the entire output array to the sheet in one command.
It might also be worth you converting your list of securities to a Collection as that has a very fast 'look-up' capability. There'd be ways of making this much faster, for example by sorting the securities, but you'd need to get into some mathematics for that.
In the example below, this skeleton code shows how it might be done. You should be aware that I didn't bother splitting the two securities lists into two collections, so you'd want to do that yourself if you needed it. I've also put all my test sheets on the same workbook, so adjust the worksheet qualifiers as needed:
Option Explicit
Sub RunMe()
Dim securities As Collection
Dim testSheet As Worksheet
Dim testItems As Variant
Dim i As Long
Dim exists As Boolean
Dim output() As Variant
'Read the first list of securities into the collection.
PopulateColumnCollection _
ThisWorkbook.Worksheets("First Sheet"), _
"B", _
securities
'Read the second list of securities into the collection.
'I've used the same collection in this example, you'll need
'to create two if you want separate columns in your output.
PopulateColumnCollection _
ThisWorkbook.Worksheets("Second Sheet"), _
"B", _
securities
'Read the test items into an array.
Set testSheet = ThisWorkbook.Worksheets("in1")
With testSheet
testItems = RangeTo2DArray(.Range( _
.Cells(2, "O"), _
.Cells(.Rows.Count, "O").End(xlUp)))
End With
'Prepare your output array.
'I've just used one column for output. If you want two then
'you'll need to resize the second dimension.
ReDim output(1 To UBound(testItems, 1), 1 To 1)
'Populate the output array based on the presence of
'a matching security.
For i = 1 To UBound(testItems, 1)
If IsEmpty(testItems(i, 1)) Then
output(i, 1) = "No Security"
Else
exists = False: On Error Resume Next
exists = securities(CStr(testItems(i, 1))): On Error GoTo 0
output(i, 1) = IIf(exists, "US", "Not a US Security")
End If
Next
'Write the output array to your sheet.
testSheet.Cells(2, "P").Resize(UBound(output, 1), UBound(output, 2)).Value = output
End Sub
Private Function RangeTo2DArray(rng As Range) As Variant
'Helper function to read range values into an array.
Dim v As Variant
Dim arr(1 To 1, 1 To 1) As Variant
v = rng.Value2
If Not IsArray(v) Then
arr(1, 1) = v
RangeTo2DArray = arr
Else
RangeTo2DArray = v
End If
End Function
Private Sub PopulateColumnCollection(ws As Worksheet, columnIndex As String, col As Collection)
'Helper sub to read a column of values into a collection.
Dim rng As Range
Dim v As Variant
Dim i As Long
With ws
Set rng = .Range( _
.Cells(1, columnIndex), _
.Cells(.Rows.Count, columnIndex).End(xlUp))
End With
v = RangeTo2DArray(rng)
If col Is Nothing Then Set col = New Collection
On Error Resume Next 'this avoids duplicates.
For i = 1 To UBound(v, 1)
col.Add True, CStr(v(i, 1))
Next
End Sub