VBA Excel 2-Dimensional Arrays - excel

I was trying to find out how to declare a 2-Dimensional array but all of the examples I have found so far are declared with set integers. I'm trying to create a program that will utilize two 2-Dimensional arrays and then perform simple operations on those arrays (such as finding difference or percent). The arrays are populated by numbers in Excel sheets (one set of numbers is on Sheet1 and another set is on Sheet2, both sets have the same number of rows and columns).
Since I don't know how many rows or columns there are I was going to use variables.
Dim s1excel As Worksheet
Dim s2excel As Worksheet
Dim s3excel As Worksheet
Dim firstSheetName As String
Dim secondSheetName As String
Dim totalRow As Integer
Dim totalCol As Integer
Dim iRow As Integer
Dim iCol As Integer
Set s1excel = ThisWorkbook.ActiveSheet
' Open the "Raw_Data" workbook
Set wbs = Workbooks.Open(file_path & data_title)
wbs.Activate
ActiveWorkbook.Sheets(firstSheetName).Select
Set s2excel = wbs.ActiveSheet
' Find totalRow, totalColumn (assumes there's values in Column A and Row 1 with no blanks)
totalRow = ActiveSheet.Range("A1").End(xlDown).Row
totalCol = ActiveSheet.Range("A1").End(xlToRight).Column
Dim s2Array(totalRow, totalCol)
Dim s3Array(totalRow, totalCol)
For iRow = 1 To totalRow
For iCol = 1 To totalCol
s2Array(iRow, iCol) = Cells(iRow, iCol)
Next iCol
Next iRow
ActiveWorkbook.Sheets(secondSheetName).Select
Set s3excel = wbs.ActiveSheet
For iRow = 1 To totalRow
For iCol = 1 To totalCol
s3Array(iRow, iCol) = Cells(iRow, iCol)
Next iCol
Next iRow
When I attempt to run this I get a compile-time error at the Dim s2Array(totalRow, totalCol) saying that a constant expression is required. The same error occurs if I change it to Dim s2Array(1 To totalRow, 1 To totalCol). Since I don't know what the dimensions are from the get go I can't declare it like Dim s2Array(1, 1) because then I'll get an out-of-bounds exception.
Thank you,
Jesse Smothermon

In fact I would not use any REDIM, nor a loop for transferring data from sheet to array:
dim arOne()
arOne = range("A2:F1000")
or even
arOne = range("A2").CurrentRegion
and that's it, your array is filled much faster then with a loop, no redim.

You need ReDim:
m = 5
n = 8
Dim my_array()
ReDim my_array(1 To m, 1 To n)
For i = 1 To m
For j = 1 To n
my_array(i, j) = i * j
Next
Next
For i = 1 To m
For j = 1 To n
Cells(i, j) = my_array(i, j)
Next
Next
As others have pointed out, your actual problem would be better solved with ranges. You could try something like this:
Dim r1 As Range
Dim r2 As Range
Dim ws1 As Worksheet
Dim ws2 As Worksheet
Set ws1 = Worksheets("Sheet1")
Set ws2 = Worksheets("Sheet2")
totalRow = ws1.Range("A1").End(xlDown).Row
totalCol = ws1.Range("A1").End(xlToRight).Column
Set r1 = ws1.Range(ws1.Cells(1, 1), ws1.Cells(totalRow, totalCol))
Set r2 = ws2.Range(ws2.Cells(1, 1), ws2.Cells(totalRow, totalCol))
r2.Value = r1.Value

Here's A generic VBA Array To Range function that writes an array to the sheet in a single 'hit' to the sheet. This is much faster than writing the data into the sheet one cell at a time in loops for the rows and columns... However, there's some housekeeping to do, as you must specify the size of the target range correctly.
This 'housekeeping' looks like a lot of work and it's probably rather slow: but this is 'last mile' code to write to the sheet, and everything is faster than writing to the worksheet. Or at least, so much faster that it's effectively instantaneous, compared with a read or write to the worksheet, even in VBA, and you should do everything you possibly can in code before you hit the sheet.
A major component of this is error-trapping that I used to see turning up everywhere . I hate repetitive coding: I've coded it all here, and - hopefully - you'll never have to write it again.
A VBA 'Array to Range' function
Public Sub ArrayToRange(rngTarget As Excel.Range, InputArray As Variant)
' Write an array to an Excel range in a single 'hit' to the sheet
' InputArray must be a 2-Dimensional structure of the form Variant(Rows, Columns)
' The target range is resized automatically to the dimensions of the array, with
' the top left cell used as the start point.
' This subroutine saves repetitive coding for a common VBA and Excel task.
' If you think you won't need the code that works around common errors (long strings
' and objects in the array, etc) then feel free to comment them out.
On Error Resume Next
'
' Author: Nigel Heffernan
' HTTP://Excellerando.blogspot.com
'
' This code is in te public domain: take care to mark it clearly, and segregate
' it from proprietary code if you intend to assert intellectual property rights
' or impose commercial confidentiality restrictions on that proprietary code
Dim rngOutput As Excel.Range
Dim iRowCount As Long
Dim iColCount As Long
Dim iRow As Long
Dim iCol As Long
Dim arrTemp As Variant
Dim iDimensions As Integer
Dim iRowOffset As Long
Dim iColOffset As Long
Dim iStart As Long
Application.EnableEvents = False
If rngTarget.Cells.Count > 1 Then
rngTarget.ClearContents
End If
Application.EnableEvents = True
If IsEmpty(InputArray) Then
Exit Sub
End If
If TypeName(InputArray) = "Range" Then
InputArray = InputArray.Value
End If
' Is it actually an array? IsArray is sadly broken so...
If Not InStr(TypeName(InputArray), "(") Then
rngTarget.Cells(1, 1).Value2 = InputArray
Exit Sub
End If
iDimensions = ArrayDimensions(InputArray)
If iDimensions < 1 Then
rngTarget.Value = CStr(InputArray)
ElseIf iDimensions = 1 Then
iRowCount = UBound(InputArray) - LBound(InputArray)
iStart = LBound(InputArray)
iColCount = 1
If iRowCount > (655354 - rngTarget.Row) Then
iRowCount = 655354 + iStart - rngTarget.Row
ReDim Preserve InputArray(iStart To iRowCount)
End If
iRowCount = UBound(InputArray) - LBound(InputArray)
iColCount = 1
' It's a vector. Yes, I asked for a 2-Dimensional array. But I'm feeling generous.
' By convention, a vector is presented in Excel as an arry of 1 to n rows and 1 column.
ReDim arrTemp(LBound(InputArray, 1) To UBound(InputArray, 1), 1 To 1)
For iRow = LBound(InputArray, 1) To UBound(InputArray, 1)
arrTemp(iRow, 1) = InputArray(iRow)
Next
With rngTarget.Worksheet
Set rngOutput = .Range(rngTarget.Cells(1, 1), rngTarget.Cells(iRowCount + 1, iColCount))
rngOutput.Value2 = arrTemp
Set rngTarget = rngOutput
End With
Erase arrTemp
ElseIf iDimensions = 2 Then
iRowCount = UBound(InputArray, 1) - LBound(InputArray, 1)
iColCount = UBound(InputArray, 2) - LBound(InputArray, 2)
iStart = LBound(InputArray, 1)
If iRowCount > (65534 - rngTarget.Row) Then
iRowCount = 65534 - rngTarget.Row
InputArray = ArrayTranspose(InputArray)
ReDim Preserve InputArray(LBound(InputArray, 1) To UBound(InputArray, 1), iStart To iRowCount)
InputArray = ArrayTranspose(InputArray)
End If
iStart = LBound(InputArray, 2)
If iColCount > (254 - rngTarget.Column) Then
ReDim Preserve InputArray(LBound(InputArray, 1) To UBound(InputArray, 1), iStart To iColCount)
End If
With rngTarget.Worksheet
Set rngOutput = .Range(rngTarget.Cells(1, 1), rngTarget.Cells(iRowCount + 1, iColCount + 1))
Err.Clear
Application.EnableEvents = False
rngOutput.Value2 = InputArray
Application.EnableEvents = True
If Err.Number <> 0 Then
For iRow = LBound(InputArray, 1) To UBound(InputArray, 1)
For iCol = LBound(InputArray, 2) To UBound(InputArray, 2)
If IsNumeric(InputArray(iRow, iCol)) Then
' no action
Else
InputArray(iRow, iCol) = "" & InputArray(iRow, iCol)
InputArray(iRow, iCol) = Trim(InputArray(iRow, iCol))
End If
Next iCol
Next iRow
Err.Clear
rngOutput.Formula = InputArray
End If 'err<>0
If Err <> 0 Then
For iRow = LBound(InputArray, 1) To UBound(InputArray, 1)
For iCol = LBound(InputArray, 2) To UBound(InputArray, 2)
If IsNumeric(InputArray(iRow, iCol)) Then
' no action
Else
If Left(InputArray(iRow, iCol), 1) = "=" Then
InputArray(iRow, iCol) = "'" & InputArray(iRow, iCol)
End If
If Left(InputArray(iRow, iCol), 1) = "+" Then
InputArray(iRow, iCol) = "'" & InputArray(iRow, iCol)
End If
If Left(InputArray(iRow, iCol), 1) = "*" Then
InputArray(iRow, iCol) = "'" & InputArray(iRow, iCol)
End If
End If
Next iCol
Next iRow
Err.Clear
rngOutput.Value2 = InputArray
End If 'err<>0
If Err <> 0 Then
For iRow = LBound(InputArray, 1) To UBound(InputArray, 1)
For iCol = LBound(InputArray, 2) To UBound(InputArray, 2)
If IsObject(InputArray(iRow, iCol)) Then
InputArray(iRow, iCol) = "[OBJECT] " & TypeName(InputArray(iRow, iCol))
ElseIf IsArray(InputArray(iRow, iCol)) Then
InputArray(iRow, iCol) = Split(InputArray(iRow, iCol), ",")
ElseIf IsNumeric(InputArray(iRow, iCol)) Then
' no action
Else
InputArray(iRow, iCol) = "" & InputArray(iRow, iCol)
If Len(InputArray(iRow, iCol)) > 255 Then
' Block-write operations fail on strings exceeding 255 chars. You *have*
' to go back and check, and write this masterpiece one cell at a time...
InputArray(iRow, iCol) = Left(Trim(InputArray(iRow, iCol)), 255)
End If
End If
Next iCol
Next iRow
Err.Clear
rngOutput.Text = InputArray
End If 'err<>0
If Err <> 0 Then
Application.ScreenUpdating = False
Application.Calculation = xlCalculationManual
iRowOffset = LBound(InputArray, 1) - 1
iColOffset = LBound(InputArray, 2) - 1
For iRow = 1 To iRowCount
If iRow Mod 100 = 0 Then
Application.StatusBar = "Filling range... " & CInt(100# * iRow / iRowCount) & "%"
End If
For iCol = 1 To iColCount
rngOutput.Cells(iRow, iCol) = InputArray(iRow + iRowOffset, iCol + iColOffset)
Next iCol
Next iRow
Application.StatusBar = False
Application.ScreenUpdating = True
End If 'err<>0
Set rngTarget = rngOutput ' resizes the range This is useful, *most* of the time
End With
End If
End Sub
You will need the source for ArrayDimensions:
This API declaration is required in the module header:
Private Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" _
(Destination As Any, _
Source As Any, _
ByVal Length As Long)
...And here's the function itself:
Private Function ArrayDimensions(arr As Variant) As Integer
'-----------------------------------------------------------------
' will return:
' -1 if not an array
' 0 if an un-dimmed array
' 1 or more indicating the number of dimensions of a dimmed array
'-----------------------------------------------------------------
' Retrieved from Chris Rae's VBA Code Archive - http://chrisrae.com/vba
' Code written by Chris Rae, 25/5/00
' Originally published by R. B. Smissaert.
' Additional credits to Bob Phillips, Rick Rothstein, and Thomas Eyde on VB2TheMax
Dim ptr As Long
Dim vType As Integer
Const VT_BYREF = &H4000&
'get the real VarType of the argument
'this is similar to VarType(), but returns also the VT_BYREF bit
CopyMemory vType, arr, 2
'exit if not an array
If (vType And vbArray) = 0 Then
ArrayDimensions = -1
Exit Function
End If
'get the address of the SAFEARRAY descriptor
'this is stored in the second half of the
'Variant parameter that has received the array
CopyMemory ptr, ByVal VarPtr(arr) + 8, 4
'see whether the routine was passed a Variant
'that contains an array, rather than directly an array
'in the former case ptr already points to the SA structure.
'Thanks to Monte Hansen for this fix
If (vType And VT_BYREF) Then
' ptr is a pointer to a pointer
CopyMemory ptr, ByVal ptr, 4
End If
'get the address of the SAFEARRAY structure
'this is stored in the descriptor
'get the first word of the SAFEARRAY structure
'which holds the number of dimensions
'...but first check that saAddr is non-zero, otherwise
'this routine bombs when the array is uninitialized
If ptr Then
CopyMemory ArrayDimensions, ByVal ptr, 2
End If
End Function
Also: I would advise you to keep that declaration private. If you must make it a public Sub in another module, insert the Option Private Module statement in the module header. You really don't want your users calling any function with CopyMemoryoperations and pointer arithmetic.

For this example you will need to create your own type, that would be an array. Then you create a bigger array which elements are of type you have just created.
To run my example you will need to fill columns A and B in Sheet1 with some values. Then run test(). It will read first two rows and add the values to the BigArr. Then it will check how many rows of data you have and read them all, from the place it has stopped reading, i.e., 3rd row.
Tested in Excel 2007.
Option Explicit
Private Type SmallArr
Elt() As Variant
End Type
Sub test()
Dim x As Long, max_row As Long, y As Long
'' Define big array as an array of small arrays
Dim BigArr() As SmallArr
y = 2
ReDim Preserve BigArr(0 To y)
For x = 0 To y
ReDim Preserve BigArr(x).Elt(0 To 1)
'' Take some test values
BigArr(x).Elt(0) = Cells(x + 1, 1).Value
BigArr(x).Elt(1) = Cells(x + 1, 2).Value
Next x
'' Write what has been read
Debug.Print "BigArr size = " & UBound(BigArr) + 1
For x = 0 To UBound(BigArr)
Debug.Print BigArr(x).Elt(0) & " | " & BigArr(x).Elt(1)
Next x
'' Get the number of the last not empty row
max_row = Range("A" & Rows.Count).End(xlUp).Row
'' Change the size of the big array
ReDim Preserve BigArr(0 To max_row)
Debug.Print "new size of BigArr with old data = " & UBound(BigArr)
'' Check haven't we lost any data
For x = 0 To y
Debug.Print BigArr(x).Elt(0) & " | " & BigArr(x).Elt(1)
Next x
For x = y To max_row
'' We have to change the size of each Elt,
'' because there are some new for,
'' which the size has not been set, yet.
ReDim Preserve BigArr(x).Elt(0 To 1)
'' Take some test values
BigArr(x).Elt(0) = Cells(x + 1, 1).Value
BigArr(x).Elt(1) = Cells(x + 1, 2).Value
Next x
'' Check what we have read
Debug.Print "BigArr size = " & UBound(BigArr) + 1
For x = 0 To UBound(BigArr)
Debug.Print BigArr(x).Elt(0) & " | " & BigArr(x).Elt(1)
Next x
End Sub

Related

Faster way to insert rows and copy data

I need to find values and test few conditions and insert row into an Excel sheet(file is heavy 65 MB). I have 7 such sheets where I need to insert data. And the reference basedata sheet inside the same file is 75k+ rows(wsSrcREDW)
My code runs really slow. Can someone please suggest faster algorithm. Thanks
Edit: the part that runs really slow is not the array assignments but the insertion of row loop in the end. It takes more than 5 mins to find new accounts and insert information.
Dim Curr() As String
For Each c In wsSrcREDW.Range("J2:J" & lrow1).Cells
ReDim Preserve Curr(2 To c.Row)
Curr(c.Row) = c.Value
Next c
Dim Entity() As String
For Each c In wsSrcREDW.Range("C2:C" & lrow1).Cells
ReDim Preserve Entity(2 To c.Row)
Entity(c.Row) = c.Value
Next c
Dim M9() As String
For Each c In wsSrcREDW.Range("F2:F" & lrow1).Cells
ReDim Preserve M9(2 To c.Row)
M9(c.Row) = c.Value
Next c
''' ECL Wback
Set wsECLWMBB = wbREDWMBB.Sheets("ECL WBack")
lrowECLWOrg = wsECLWMBB.Range("A" & Rows.Count).End(xlUp).Row
Dim I7() As String
For Each c In wsSrcREDW.Range("S2:S" & lrow1).Cells
ReDim Preserve I7(2 To c.Row)
I7(c.Row) = c.Value
Next c
For i = 2 To UBound(I7)
Set c = wsECLWMBB.Range("B2:B" & lrowECLWOrg).Find(I7(i))
If c Is Nothing And Entity(i) = "MIB" Then
lrowECLW = wsECLWMBB.Range("A" & Rows.Count).End(xlUp).Row
wsECLWMBB.Range("A" & (lrowECLW + 1)).EntireRow.Insert
wsECLWMBB.Range("A" & (lrowECLW + 1)).Value = M9(i)
wsECLWMBB.Range("B" & (lrowECLW + 1)).Value = I7(i)
wsECLWMBB.Range("C" & (lrowECLW + 1)).Value = Curr(i)
wsECLWMBB.Range("D" & (lrowECLW + 1)).Formula = "=MID(B" & (lrowECLW + 1) & ",1,7)"
End If
Next i
Use a variant array. Fill it and write the whole array in one operation. The following code should do it.
Option Explicit
Sub TEST()
Dim dataSrcEDW As Variant, dataECLWMBB As Variant, dataNew As Variant
Dim wsSrcREDW As Worksheet, wsECLWMBB As Worksheet
Dim colEntity As Long, colCurr As Long, colM9 As Long, colI7 As Long
Dim iSrcRow As Long, iTargetRow As Long, iNewRow As Long
Dim bFound As Boolean
Dim rgNew As Range
dataSrcEDW = wsSrcREDW.Range("A1").CurrentRegion ' Retrives all the source data
dataECLWMBB = wsECLWMBB.Range("A1").CurrentRegion ' Retrieves all the target data
ReDim dataNew(0, 1 To 4) ' This will contain the new rows you are adding at the end of wsECLWMBB
' Identify the columnns of interest
colCurr = Asc("J") - 64: colEntity = Asc("C") - 64: colM9 = Asc("F") - 64: colI7 = Asc("S") - 64
For iSrcRow = 2 To UBound(dataSrcEDW, 1) ' Scane through the source
bFound = False
If dataSrcEDW(iSrcRow, colEntity) = "MIB" Then
For iTargetRow = 2 To UBound(dataECLWMBB, 1)
If dataSrcEDW(iSrcRow, colI7) = dataECLWMBB(iTargetRow, 2) Then
bFound = True
Exit For
End If
Next
If Not bFound Then ' Check if this is a duplicate add
For iNewRow = 1 To UBound(dataNew, 1)
If dataSrcEDW(iSrcRow, colI7) = dataNew(iNewRow, 2) Then
bFound = True
Exit For
End If
Next
End If
If Not bFound Then
dataNew = AddRowToArray(dataNew)
iNewRow = UBound(dataNew, 1)
dataNew(iNewRow, 1) = dataSrcEDW(iSrcRow, colM9)
dataNew(iNewRow, 2) = dataSrcEDW(iSrcRow, colI7)
dataNew(iNewRow, 3) = dataSrcEDW(iSrcRow, colCurr)
dataNew(iNewRow, 4) = "=MID(B" & UBound(dataECLWMBB, 1) + iNewRow & ",1,7)"
End If
End If
Next
' Write out the new rows
If UBound(dataNew, 1) > 0 Then
Set rgNew = wsECLWMBB.Range("A" & UBound(dataECLWMBB, 1) + 1).Resize(UBound(dataNew, 1), UBound(dataNew, 2))
rgNew = dataNew
End If
End Sub
Public Function AddRowToArray(vArray) As Variant
' Can't do a redim preserve on a multi dimensional array. Add a row manually.
Dim vNewArray As Variant, iRow As Long, iCol As Long
ReDim vNewArray(1 To UBound(vArray, 1) + 1, 1 To UBound(vArray, 2))
For iRow = 1 To UBound(vArray, 1)
For iCol = 1 To UBound(vArray, 2)
vNewArray(iRow, iCol) = vArray(iRow, iCol)
Next
Next
AddRowToArray = vNewArray
End Function

Excel VBA Passing Variables

I need to pass the variables max, min, and their respective locations to another sub where it will format each max and min in their respective column. I am trying to create an array that will store the locations and the values but its not working.
I was told to first identify the number of columns used and the number of rows, which is the beginning.
Rows = wsData.UsedRange.Rows.Count
Columns = wsData.UsedRange.Col.Count
j = 1
ReDim Min(j)
With wsData.Range("A3:A19")
For j = 1 To 19 'colum
Min(j) = WorksheetFunction.Min(Range(.Offset(1, j), .Offset(Row, j)))
Max = WorksheetFunction.Max(Range(.Offset(1, j), .Offset(Row, j)))
Min(j) = Min
j = j + 1
ReDim Preserve Min(j) 'saves variables
Next 'next column
End With
The code below uses the ActiveSheet which you need to change to reference the worksheet for your data. Additionally, it assumes that your data starts with Row 1. The code looks at each column in the range and stores the minimum/maximum (it does not account for multiple cells which may share the min or max value) value found in the column as well as the cell's address, in an array and then passes the array to two different subs, one which simply displays the information in a message and one which formats the the background color of the cells. This code does not perform any kind of error handling, but should get you where you want to go.
the line Option Explicit requires that all of the variables be defined using a Dim statement
the line Option Base 1 makes the default starting point for arrays 1 instead of 0
Option Explicit
Option Base 1
Sub GatherData()
Dim iRows As Long
Dim iCols As Long
Dim j As Long
Dim iMin() As Variant
Dim iMax() As Variant
Dim R As Range
iRows = ActiveSheet.UsedRange.Rows.Count
iCols = ActiveSheet.UsedRange.Columns.Count
ReDim iMin(iCols, 2)
ReDim iMax(iCols, 2)
For j = 1 To iCols
Set R = Range(Cells(1, j), Cells(iRows, j)).Find(WorksheetFunction.Min(Range(Cells(1, j), Cells(iRows, j))), LookIn:=xlValues)
iMin(j, 1) = R.Value
iMin(j, 2) = R.Address
Set R = Range(Cells(1, j), Cells(iRows, j)).Find(WorksheetFunction.Max(Range(Cells(1, j), Cells(iRows, j))), LookIn:=xlValues)
iMax(j, 1) = R.Value
iMax(j, 2) = R.Address
Next j
ListMinMax iMax(), True
ListMinMax iMin(), False
FormatMinMax iMax, "green"
FormatMinMax iMin, "yellow"
Set R = Nothing
End Sub
Sub ListMinMax(ByRef Arr() As Variant, ByVal MinMax As Boolean)
Dim strOutput As String
Dim i As Long
If MinMax = True Then
strOutput = "Maximums:" & vbCrLf & vbCrLf
Else
strOutput = "Minimums:" & vbCrLf & vbCrLf
End If
For i = 1 To UBound(Arr, 1)
strOutput = strOutput & "Cell: " & Arr(i, 2) & " = " & Arr(i, 1) & vbCrLf
Next i
MsgBox strOutput, vbOKOnly
End Sub
Sub FormatMinMax(ByRef Arr() As Variant, ByVal BGColor As String)
Dim i As Long
Select Case UCase(BGColor)
Case "GREEN"
For i = 1 To UBound(Arr, 1)
ActiveSheet.Range(Arr(i, 2)).Interior.Color = vbGreen
Next i
Case "YELLOW"
For i = 1 To UBound(Arr, 1)
ActiveSheet.Range(Arr(i, 2)).Interior.Color = vbYellow
Next i
Case Else
MsgBox "Invalid Option", vbCritical
End Select
End Sub
======================================================================
The code below does away with the need for the arrays and formats the color of the min/max values as it finds them
Sub GatherData2()
Dim iRows As Long
Dim iCols As Long
Dim j As Long
Dim R As Range
iRows = ActiveSheet.UsedRange.Rows.Count
iCols = ActiveSheet.UsedRange.Columns.Count
For j = 1 To iCols
Set R = Range(Cells(1, j), Cells(iRows, j)).Find(WorksheetFunction.Min(Range(Cells(1, j), Cells(iRows, j))), LookIn:=xlValues)
R.Interior.Color = vbYellow
Set R = Range(Cells(1, j), Cells(iRows, j)).Find(WorksheetFunction.Max(Range(Cells(1, j), Cells(iRows, j))), LookIn:=xlValues)
R.Interior.Color = vbGreen
Next j
Set R = Nothing
End Sub

How to increase the performance of a partial match lookup function?

The current performance of this function is to slow, currently I am working with a list of 500+ item codes on sheet1. The function searches in a range of 200 000 + items on sheet2 to find all matches including partial matches. This means that we include a wildcards before and after the lookup criteria to find all matches.
Currently it takes over 15 mins to complete. Is there a better method to do this? To get this under 5 mins?
Function ConcatIf(ByVal compareRange As Range, ByVal xCriteria As Variant, _
Optional ByVal stringsRange As Range, Optional Delimiter As String) As String
Application.Calculation = xlCalculationManual
Application.ScreenUpdating = False
Application.DisplayStatusBar = False
Application.EnableEvents = False
ActiveSheet.DisplayPageBreaks = False
Dim i As Long, j As Long, criteriaMet As Boolean
Set compareRange = Application.Intersect(compareRange, _
compareRange.Parent.UsedRange)
If compareRange Is Nothing Then Exit Function
If stringsRange Is Nothing Then Set stringsRange = compareRange
Set stringsRange = compareRange.Offset(stringsRange.Row - _
compareRange.Row, stringsRange.Column - compareRange.Column)
For i = 1 To compareRange.Rows.Count
For j = 1 To compareRange.Columns.Count
If (Application.CountIf(compareRange.Cells(i, j), _
xCriteria)= 1) Then
ConcatIf = ConcatIf & Delimiter & _
CStr(stringsRange.Cells(i, j))
End If
Next j
Next i
ConcatIf = Mid(ConcatIf, Len(Delimiter) + 1)
Application.Calculation = xlCalculationAutomatic
Application.ScreenUpdating = True
Application.DisplayStatusBar = True
Application.EnableEvents = True
ActiveSheet.DisplayPageBreaks = True
End Function
Example:
+500 ITEM CODES
Sheet1:
BCD
CDF
XLMH
XPT
ZPY
200 000 + FULL ITEM CODES
Sheet2:
FDBCDGH
HSGDBCDSU
GFD-CDFGDTR
SBGCDFHUD
GKJYCDFFDS
DDFGFDXLMHGFD
SDGXLMHSDFS
SDGVSDXLMHFAMN
DDDSXPTDFGFD
JUYXPTFADS
DDDFFZPYDGDFDF
Outcome should be:
Sheet1:
COLUMN A - COLUMN B
BCD - FDBCDGH,HSGDBCDSU
CDF - GFD-CDFGDTR,SBGCDFHUD,GKJYCDFFDS
XLMH - DDFGFDXLMHGFD,SDGXLMHSDFS,SDGVSDXLMHFAMN
XPT - DDDSXPTDFGFD,JUYXPTFADS
ZPY - DDDFFZPYDGDFDF
To use the following code you will need to add a reference to Microsoft Scripting Runtime. This uses two arrays and compiles the data in a dictionary. This can then be written back to your sheet. The code currently writes the results back to the immediate window which can be displayed using Ctrl+G or View->Immediate Window
Public Sub demo()
Dim compArr As Variant, strArr As Variant
Dim strDict As Dictionary
Dim i As Long
Dim Delimiter As String: Delimiter = "; "
Dim key
' Set data to arrays. This assumes your data is in column A
With Sheets("Sheet1")
' Application.Transpose is a trick to convert the range to a 1D array (otherwise a 2D array will be created)
compArr = Application.Transpose(.Range(.Cells(1, 1), .Cells(.Cells(.Rows.Count, 1).End(xlUp).Row, 1)))
End With
With Sheets("Sheet2")
strArr = Application.Transpose(.Range(.Cells(1, 1), .Cells(.Cells(.Rows.Count, 1).End(xlUp).Row, 1)))
End With
' Initiate dictionary
Set strDict = New Dictionary
' Loop through all the values you wish to find
For i = LBound(compArr) To UBound(compArr)
' Tests if value exists
If Not strDict.Exists(compArr(i)) Then
' Adds value to dictionary and uses filter on string array to get similar matches.
' Join is used to convert the resulting array into a string
strDict.Add key:=compArr(i), Item:=Join(Filter(strArr, compArr(i), True), Delimiter)
End If
Next i
' Read back results
For Each key In strDict.Keys
Debug.Print key, strDict(key)
Next key
End Sub
To maintain all of your current functionality and useability regarding the size of your dataset, this should work for you and be faster than the original code. When I timed it, I used 400,000 full item codes and applied the concatif formula on sheet 1 for 1000 partial matches and it completed all cell calculations in under 9 minutes.
Public Function CONCATIF(ByVal arg_rCompare As Range, _
ByVal arg_vCriteria As Variant, _
Optional ByVal arg_rStrings As Range, _
Optional ByVal arg_sDelimiter As String = vbNullString _
) As Variant
Dim aData As Variant
Dim aStrings As Variant
Dim aCriteria As Variant
Dim vString As Variant
Dim vCriteria As Variant
Dim aResults() As String
Dim ixResult As Long
Dim i As Long, j As Long
If arg_rStrings Is Nothing Then Set arg_rStrings = arg_rCompare
If arg_rStrings.Rows.Count <> arg_rCompare.Rows.Count _
Or arg_rStrings.Columns.Count <> arg_rCompare.Columns.Count Then
CONCATIF = CVErr(xlErrRef)
Exit Function
End If
If arg_rCompare.Cells.Count = 1 Then
ReDim aData(1 To 1, 1 To 1)
aData(1, 1) = arg_rCompare.Value
Else
aData = arg_rCompare.Value
End If
If arg_rStrings.Cells.Count = 1 Then
ReDim aStrings(1 To 1, 1 To 1)
aStrings(1, 1) = arg_rStrings.Value
Else
aStrings = arg_rStrings.Value
End If
If IsArray(arg_vCriteria) Then
aCriteria = arg_vCriteria
ElseIf TypeName(arg_vCriteria) = "Range" Then
If arg_vCriteria.Cells.Count = 1 Then
ReDim aCriteria(1 To 1)
aCriteria(1) = arg_vCriteria.Value
Else
aCriteria = arg_vCriteria.Value
End If
Else
ReDim aCriteria(1 To 1)
aCriteria(1) = arg_vCriteria
End If
ReDim aResults(1 To arg_rCompare.Cells.Count)
ixResult = 0
For i = LBound(aData, 1) To UBound(aData, 1)
For j = LBound(aData, 2) To UBound(aData, 2)
For Each vCriteria In aCriteria
If aData(i, j) Like vCriteria Then
ixResult = ixResult + 1
aResults(ixResult) = aStrings(i, j)
End If
Next vCriteria
Next j
Next i
If ixResult > 0 Then
ReDim Preserve aResults(1 To ixResult)
CONCATIF = Join(aResults, arg_sDelimiter)
Else
CONCATIF = vbNullString
End If
Erase aData: aData = vbNullString
Erase aCriteria: aCriteria = vbNullString
Erase aResults
End Function

Optimize macro for millions of calculations

I'm matching ids on separate files, if a match happens the row on the source gets retrieved to the other file. I did a FOR statement for both files to scan each row, the source workbook has over 27000 rows and the other about 8000, if I understand right that is 216M+ calculations until the end of the loops. I've implemented screenUpdating = False and xlCalculationManual. But here am I, I've waited about 30 minutes and there is no sign of the code finishing (both VBA editor and Excel are "not responding").
For filaIndiceFuente = 2 To filaFuenteUltima
criterioFuente = planillaFuente.Range("A" & filaIndiceFuente).Value
For filaIndiceDestino = 2 To filaDestinoUltima
' filaIndiceDestino = filaIndiceDestino + 1
If planillaDestino.Range("A" & filaIndiceDestino).Value = criterioFuente Then
'CELLS GET TO THE OTHER FILE HERE
End If
Next filaIndiceDestino
Next filaIndiceFuente
On test files I implemented the code and it runs almost instantly with positive results. If you could hint me other ways of improving my code I'll be thankful.
Usually when I have a large dataset that I'm iterating through for matches, I find that using a Dictionary is faster even than a .Find() operation or iterating through every row.
I would try something like
Dim dict As New Scripting.Dictionary
For filaIndiceFuente = 2 To filaFuenteUltima
dict.Add CStr(planillaFuente.Range("A" & filaIndiceFuente).Value), filaIndiceFuente '<- this will act as a pointer to the row where your match data is
Next filaIndiceFuente
For filaIndiceDestino = 2 To filaDestinoUltima
If dict.Exists(CStr(planillaDestino.Range("A" & filaIndiceDestino).Value)) Then
'CELLS GET TO THE OTHER FILE HERE
End If
Next filaIndiceDestino
Set dict = Nothing
I would probably take it a step further, load the data into arrays, then loop through the arrays. The indice will be off by 1 due the offset on reading the array data. There is a bit of fluff in the loadscp routine, I built it for reuse. I suspect you won’t need the status bar.
Dim scpFuente As scripting.dictionary
Dim arrFuente As variant
Dim arrDest As variant
Arrfuente = planillaFuente.range(“a2”).resize(filaFuenteUltima-1,1).value
ArrDest = planillaDestino.range(“a2”).resize(filaDestinaUltima-1,1).value
Set scpFuente = loadscp(arrfuente)
For filaIndiceDestino = lbound(arrDest,1) to ubound(arrDest,1)
' filaIndiceDestino = filaIndiceDestino + 1
If scpFuente.exists(arrdest(filaindicedestino,1)) Then
'CELLS GET TO THE OTHER FILE HERE
End If
Next filaIndiceDestino
The loadscp function:
Public Function Loadscp(ByVal varList As Variant, Optional ByVal intCol As Integer = 1, _
Optional ByVal intCols As Integer = 1, Optional ByVal strDelim As String = ".") As Scripting.Dictionary
Dim scpList As Scripting.Dictionary
Dim arrVals As Variant
Dim lngLastRow As Long
Dim lngRow As Long
Dim intABSCol As Integer
Dim intColCurr As Integer
Dim strVal As String
Dim intRngCol As Integer
Set Loadscp = New Scripting.Dictionary
Loadscp.CompareMode = vbTextCompare
intABSCol = Abs(intCol)
If IsArray(varList) Then
arrVals = varList
ElseIf TypeName(varList) = "Range" Then
intRngCol = varList.Column
lngLastRow = LastRow(varList.Parent, intCol)
If lngLastRow > varList.Row Then
arrVals = varList.Offset(1, intABSCol - 1).Resize(lngLastRow - varList.Row, 1)
End If
ElseIf TypeName(varList) = "Dictionary" Then
Set scpList = varList
ReDim arrVals(1 To scpList.Count, 1 To 1)
For lngRow = 1 To scpList.Count
arrVals(lngRow, 1) = scpList.Keys(lngRow - 1)
Next lngRow
End If
If IsArray(arrVals) Then
For lngRow = LBound(arrVals, 1) To UBound(arrVals, 1)
strVal = arrVals(lngRow, intCol)
For intColCurr = intCol + 1 To intCol + intCols - 1
strVal = strVal & strDelim & arrVals(lngRow, intColCurr)
Next intColCurr
If Not Loadscp.Exists(strVal) Then
Loadscp.Item(strVal) = lngRow
End If
Next lngRow
End If
End Function
First I would add Application.Statusbar value to control how long it is running
Second I would add an exit for if a value is found in the inner loop to prevent unneccessary steps in the loop like :
For filaIndiceFuente = 2 To filaFuenteUltima
criterioFuente = planillaFuente.Range("A" & filaIndiceFuente).Value
if filaIndiceFuente mod 50 = 0 then
**Application.statusbar = filaIndiceFuente**
end if
For filaIndiceDestino = 2 To filaDestinoUltima
' filaIndiceDestino = filaIndiceDestino + 1
If planillaDestino.Range("A" & filaIndiceDestino).Value = criterioFuente Then
'CELLS GET TO THE OTHER FILE HERE
**exit for**
End If
Next filaIndiceDestino
Next filaIndiceFuente
Application.statusbar = ""
You can have the statusbar info inside the inner loop
For filaIndiceFuente = 2 To filaFuenteUltima
criterioFuente = planillaFuente.Range("A" & filaIndiceFuente).Value
For filaIndiceDestino = 2 To filaDestinoUltima
' filaIndiceDestino = filaIndiceDestino + 1
if filaIndiceDestino mod 50 = 0 then
**Application.statusbar = filaIndiceFuente & " - " & filaIndiceDestino **
end if
If planillaDestino.Range("A" & filaIndiceDestino).Value = criterioFuente Then
'CELLS GET TO THE OTHER FILE HERE
**exit for**
End If
Next filaIndiceDestino
Next filaIndiceFuente
Application.statusbar = ""
I do not see a way to make comparsion faster, but maybe some other has a better idea. See this as a first step to identify the reason for taking a long time.
First sort the planillaDest range ascending by column A, then:
Dim lookupRange As Range
Set lookupRange = planillaDestino.Range("A2:A" & filaDestinoUltima)
For filaIndiceFuente = 2 To filaFuenteUltima
criterioFuente = planillaFuente.Cells(filaIndiceFuente, "A").Value
Dim matchRow As Long
matchRow = Application.WorksheetFunction.Match(criterioFuente, lookupRange, 1)
If lookupRange.Cells(matchRow, 1).Value = criterioFuente Then
'CELLS GET TO THE OTHER FILE HERE
' If row to move from planillaFuente to planillaDest, then:
planillaDest.Cells(matchRow + 1, "P").Value = planillaFuente.Cells(filaIndiceFuente, "D").Value
End If
Next filaIndiceFuente

How to receive all combinations of all columns?

I am trying to get all row combinations of all columns (say 8 columns). The following vba macro can do that but I get an error that says data overload:
Option Explicit
Const sTitle As String = "shg Cartesian Product"
Sub CartesianProduct()
' shg 2012, 2013
' Choose one from col A, one from col B, ...
Dim rInp As Range
Dim avInp As Variant ' ragged input list
Dim nCol As Long ' # columns in list
Dim rOut As Range ' output range
Dim iCol As Long ' column index
Dim iRow As Long ' row index
Dim aiCum() As Long ' cum count of arrangements from right to left
Dim aiCnt() As Long ' count of items in each column
Dim iArr As Long ' arrangement number
Dim avOut As Variant ' output buffer
Application.ScreenUpdating = False
Set rInp = Range("rgnInp")
If VarType(rInp.Value) = vbEmpty Then
MsgBox Prompt:="No input!", _
Buttons:=vbOKOnly, _
Title:=sTitle
Exit Sub
End If
Set rInp = rInp.CurrentRegion
If rInp.Columns.Count < 2 Or rInp.Rows.Count < 2 Then
MsgBox Prompt:="Must have more than one row and more than one columns!", _
Buttons:=vbOKOnly, _
Title:=sTitle
Exit Sub
End If
With rInp
.Style = "Input"
avInp = .Value
nCol = .Columns.Count
Set rOut = .Resize(1).Offset(.Rows.Count + 1)
Range(rOut.Offset(-1, -1), Cells(Rows.Count, Columns.Count)).Clear
End With
ReDim aiCum(1 To nCol + 1)
ReDim aiCnt(1 To nCol)
aiCum(nCol + 1) = 1
For iCol = nCol To 1 Step -1
For iRow = 1 To UBound(avInp, 1)
If IsEmpty(avInp(iRow, iCol)) Then Exit For
aiCnt(iCol) = aiCnt(iCol) + 1
Next iRow
aiCum(iCol) = aiCnt(iCol) * aiCum(iCol + 1) <------ This is where it says error is
Next iCol
If aiCum(1) > Rows.Count - rOut.Row + 1 Then
MsgBox Prompt:=Format(aiCum(1), "#,##0") & _
" is too many rows!", _
Buttons:=vbOKOnly, Title:=sTitle
Exit Sub
End If
ReDim avOut(1 To aiCum(1), 1 To nCol)
For iArr = 1 To aiCum(1)
For iCol = 1 To nCol
avOut(iArr, iCol) = avInp((Int((iArr - 1) * aiCnt(iCol) / aiCum(iCol))) Mod aiCnt(iCol) + 1, iCol)
Next iCol
Next iArr
With rOut.Resize(aiCum(1), nCol)
.NumberFormat = "#"
.Value = avOut
.Style = "Code"
.Cells(1, 0).Value = 1
.Cells(2, 0).Value = 2
.Cells(1, 0).Resize(2).AutoFill .Columns(0)
End With
ActiveWindow.FreezePanes = False
rOut.EntireColumn.AutoFit
ActiveSheet.UsedRange
Beep
End Sub
Is there away to adjust for this? I also want it to not bring back the same values for a row. So lets say that two columns had the exact same data. If column A has lets say Ice cream, cake, and cookies and so does Column B, I don't want Row 1 to have cookies in column B if it is already picked in Column A.

Resources