Nested loops causing Excel crash - excel

I am attempting to run a VBA macro that iterates down about 67,000 rows with 100 columns in each row. For each of the cells in these rows, the value is compared against a column with 87 entries in another sheet. There are no errors noted when the code is run but Excel crashes every time. The odd thing is that the code seems to work; I have it set to mark each row in which a match is found and it does so before crashing. I have attempted to run it many times and it has gotten through between 800 and 11,000 rows before crashing, depending on the attempt.
My first suspect was memory overflow due to the volume of calculations but my system shows CPU utilization at 100% and memory usage around 50% while running this code:
Sub Verify()
Dim codes As String
Dim field As Object
For i = 2 To Sheets("DSaudit").Rows.Count
For Each field In Sheets("Dsaudit").Range(Cells(i, 12), Cells(i, 111))
r = 1
While r <= 87
codes = ThisWorkbook.Sheets("287 Denominator CPT").Cells(r, 1).Value
If field = codes Then
Cells(i, 112).Value = "True"
r = 88
Else
r = r + 1
End If
Wend
Next field
i = i + 1
Next i
End Sub
It should also be noted that I am still very new to VBA so it's likely I've made some sort of egregious rookie mistake. Can I make some alterations to this code to avoid a crash or should I scrap it and take a more efficient approach?

When ever possible iterate variant arrays. This limits the number of times vba needs to access the worksheet.
Every time the veil between vba and Excel is pierced cost time. This only pierces that veil 3 times not 9,031,385,088
Sub Verify()
With Sheets("DSaudit")
'Get last row of Data
Dim lastrow As Long
lastrow = .Cells(.Rows.Count, 12).End(xlUp).Row 'if column 12 ends before the last row of data change to column that has them all.
'Load Array with input Values
Dim rng As Variant
rng = .Range(.Cells(2, 12), .Cells(lastrow, 111)).Value
'Create output array
Dim outpt As Variant
ReDim outpt(1 To UBound(rng, 1), 1 To 1)
'Create Match array
Dim mtch As Variant
mtch = Worksheets("287 Denominator CPT").Range("A1:A87").Value
'Loop through first dimension(Row)
Dim i As Long
For i = LBound(rng, 1) To UBound(rng, 1)
'Loop second dimension(Column)
Dim j As Long
For j = LBound(rng, 2) To UBound(rng, 2)
'Loop Match array
Dim k As Long
For k = LBound(mtch, 1) To UBound(mtch, 1)
'If eqaul set value in output and exit the inner loop
If mtch(k, 1) = rng(i, j) Then
outpt(i, 1) = "True"
Exit For
End If
Next k
'If filled true then exit this for
If outpt(i, 1) = "True" Then Exit For
Next j
Next i
'Assign the values to the cells.
.Cells(2, 112).Resize(UBound(outpt, 1), 1).Value = outpt
End With
End Sub

Related

Excel VBA ListBox in User Form Populate data from Sheet Range, add row by row after evaluating for a condition

I am trying to write a VBA code where I want to populate DATA from a worksheet Range A to AQ spanning over multiple Rows. AQ contains Value "Open" or "Closed". I want to get the rows where AQ value is closed. I tried using the AutoFilter. This is working fine to an extent. But I have to use 2 For loops. One for Each Row and another for Each Column to populate Row wise, column by column into the list box
My Code as follows:
Note : Actual contents start from 6th Row where 6 contains the headers and data starts from 7th Row
Dim i As Long
Dim rowRange As Range
Dim AllData(1 To 1000, 1 To 43) As String
lstRecords.ColumnCount = 43
Set shDSR = mydata1.Sheets("DSR")
last_Row = shDSR.Cells(Rows.Count, 1).End(xlUp).Row
shDSR.AutoFilterMode = False
shDSR.Range("A6:AQ" & last_Row).AutoFilter Field:=43, Criteria1:="CLOSED"
Set rng = shDSR.Range("A6:AQ" & last_Row).SpecialCells(xlCellTypeVisible)
Dim filtrRow() As String
Dim rowCnt As Long
'Me.lstRecords.Clear
rowCnt = 0
If rng.Count > 0 Then
Me.lstRecords.Clear
Me.lstRecords.ColumnCount = rng.Columns.Count
For Each Row In rng.Rows
Me.lstRecords.AddItem
rowCnt = rowCnt +1
filterRow = Range(Row.Address)
'Me.lstRecords.List() = filterRow ''This throws error Type Mismatch so not using
For i = 1 To Row.Columns.Count
AllData(rowCnt, i) = Row.Cells(1, i).Value ''Move to Array
Me.lstRecords.List(rowCnt - 1, i - 1) = filterRow(1, i)'Buggy error when i = 11
Next
Next
'' Following segment works. Add data to Array and then populate ListBox from Array
Me.lstRecords.List() = AllData
Else
MsgBox "No data matches the filter criteria."
End If
Above Code has both approaches
a) Trying to load directly from excel Range (actually using filterRow, but can also directly use range with same issue). But, this approach stops always when i=11 with Invalid property error. I tried changing the data contents etc still same issue
Another Issue when Not taking the array based approach, only one line is added, so in affect only last line is available in the list box
b) Using the AllData array. I load all the row data (matching criteria) into the array and finally populate the listbox from array. THIS WORKS. But I do not like this approach
Can some one please point out where it is going wrong.
Thanks in advance
Problem is that filters create a non contiguous range consisting of areas which you have to iterate separately.
Option Explicit
Sub demo()
Dim mydata1 As Workbook, shDSR As Worksheet
Dim rng As Range, a As Range, r As Range
Dim last_row As Long, n As Long
Dim i As Long, rowCnt As Long
Dim ListData() As String
' change this
Set mydata1 = ThisWorkbook
Set shDSR = mydata1.Sheets("DSR")
With shDSR
.AutoFilterMode = False
last_row = .Cells(.Rows.Count, "AQ").End(xlUp).Row
.Range("A6:AQ" & last_row).AutoFilter Field:=43, Criteria1:="CLOSED"
Set rng = .Range("A6:AQ" & last_row).SpecialCells(xlCellTypeVisible)
.AutoFilterMode = False
End With
' clear listbox
With Me.lstRecords
.Clear
.ColumnCount = rng.Columns.Count
End With
'iterate areas and rows to count visible rows
For Each a In rng.Areas
n = n + a.Rows.Count
Next
rowCnt = 0
If n > 1 Then
' size array
ReDim ListData(1 To n, 1 To rng.Columns.Count)
' fill array
For Each a In rng.Areas
For Each r In a.Rows
rowCnt = rowCnt + 1
For i = 1 To UBound(ListData, 2)
ListData(rowCnt, i) = r.Cells(1, i).Value ''Move to Array
Next
Next
Next
' populate ListBox from Array
Me.lstRecords.List() = ListData
Else
MsgBox "No data matches the filter criteria."
End If
End Sub

Merging Data from multiple sheets that contains formulas/cell references

Hi all! I have posted the same question previously but this one time is more explained with screenshots of the file.
I have 85 sheets (Sheet 1 screenshot for reference) and a specified range for each sheet (I12:N42). But this range consists formulas & cell references. What I am trying to do is:
Copying the data from all the 85 sheets with this range (I12:N42) except if "Qty = 0".
Pasting the copied data by PasteValues only to a master sheet.
PS: I tried to do the same thing using Power Query but it is quite slow so maybe VBA code will work faster on this.
Appreciate you guys!
Please, try the next code. It assumes that there are no other sheets except the master one and the mentioned 85 to copy from. If others, except them adding new conditions where the master one is skipped:
Sub copyNonZeroRowsInMaster()
Dim sh As Worksheet, shM As Worksheet, rng As Range, lastRow As Long, boolOK As Boolean
Dim arrRows, arrCopy, arr, arrSlice, count0 As Long, i As Long, k As Long
lastRow = 2 'the initial row where to paste
Set shM = ActiveWorkbook.Sheets("MASTER") 'please, use here the appropriate sheet name
shM.Range("A2:F10000").ClearContents
For Each sh In ActiveWorkbook.Sheets
If sh.Name <> shM.Name And sh.Name <> "TRACKING" Then 'if other sheets needs to be excepted, add them in the condition
Set rng = sh.Range("K13:K42") 'the range being the reference for non zero values
arrCopy = sh.Range("I13:N42").Value 'place all the range to be processed in an array, to make code faster
count0 = Application.CountIf(rng, 0) 'count the zero values (even from formulas) to redim in the next row
arr = rng.Value 'place the reference range in an array (also, to make the code faster)
If rng.Count - count0 = 0 Then GoTo OverProcessing
ReDim arrRows(1 To rng.Count - count0, 1 To 1) 'redim the array to keep the row numbers without 0 in K:K
k = 1: boolOK = False 'initialize the variable based on what the array keeping the rows to be copied is loaded
For i = 1 To UBound(arr) 'iterate beteen the array elements
If arr(i, 1) <> 0 Then
arrRows(k, 1) = i: k = k + 1 'fill the rows to be copied number in the array
boolOK = True
End If
Next i
If Not boolOK Then GoTo OverProcessing 'if there are only zero in all processed K:K range
arrSlice = Application.Index(arrCopy, arrRows, Array(1, 2, 3, 4, 5, 6)) 'create a slice array keeping only the non zero rows
'drop the slice array content at once:
shM.Range("A" & lastRow).Resize(IIf(k = 2, UBound(arrRows), UBound(arrSlice)), 6).Value = arrSlice
lastRow = shM.Range("A" & shM.Rows.Count).End(xlUp).Row + 1 'recalculate the last empty row
End If
OverProcessing:
Next
MsgBox "Ready..."
End Sub
The code is not tested (except the principle of working) and it should be very fast. Please, send some feedback after testing it.

fastest way to process 115 million cells?

I have been given a work task where im to find and replace 8 digits numbers with a corresponding new values coming from a 2 column table....basically a vlookup then replace the old value with a new one...
The challenge im facing is.... the 2 column table is 882k rows, and the cells im trying to replace is about 120 million (41,000 rows x 3000 columns)...
I tried running a vba code i found somewhere...
Option Explicit
Sub Replace_Overwrite()
Dim LRow As Long, i As Long
Dim varSearch As Variant
With Sheets("Sheet2")
LRow = .Cells(.Rows.Count, 1).End(xlUp).Row
varSearch = .Range("A2:B" & LRow)
End With
With Sheets("Sheet1").UsedRange
For i = LBound(varSearch) To UBound(varSearch)
.Replace what:=varSearch(i, 1), replacement:=varSearch(i, 2), lookat:=xlWhole
Next
End With
End Sub
I tried using this and it ran it for 8 hours and my work laptop crashed....
I'm not sure anymore if this is still possible with MS Excel alone...
I wonder if anyone can help me with a code that can process it.. i can leave my system open over the weekend if its stable and does work.. it only has 8GB ram btw, running excel 2013...
To speed up things, do as much as possible in memory and minimize the interaction between VBA and Excel (as this makes things really slow).
The following attempt reads the lookup-list into a dictionary and then processes the data column by column.
I did a test, creating 880.000 lookup rows and 40.000 x 100 cells of data. Building the dictionary took less than a minute, processing the columns took 3-4 seconds per column. I added a logic that after every 10 columns, the whole workbook is saved, that increased the processing time but ensures that after a crash you can more or less continue where you left (the yellow color tells you where, just replace the 1 in for col=1 with the column where you want to restart).
I have added some DoEvents, that in theory slows down the process a little bit. Advantage is that you can see the output of the debug.print and the whole Excel process is not displayed as unresponsive in the task manager.
To build the dictionary, I read the complete data into an array at once (if you are not familiar with Dictionaries: You need to add a reference to the Microsoft Scripting Runtime).
Function createDict() As Dictionary
Dim d As New Dictionary
Dim rowCount As Long
Dim list()
Debug.Print Now, "Read data from Lookup sheet"
With ThisWorkbook.Sheets(1)
rowCount = .Cells(.Rows.Count, 1).End(xlUp).row
list = .Range("A1:B" & rowCount).Value
End With
Debug.Print Now, "Build dictionary."
Dim row As Long
For row = 1 To UBound(list)
If Not d.Exists(list(row, 1)) Then d.Add list(row, 1), list(row, 2)
If row Mod 1000 = 0 Then DoEvents
Next row
Set createDict = d
End Function
As said, replacing the data is done column by column. Again, I read the whole column at once into an array, do the replace on this array and then write it back to the sheet.
Sub replaceAll()
Application.ScreenUpdating = False
Application.Calculation = xlCalculationManual
Dim d As Dictionary
Set d = createDict
Dim row As Long, col As Long
Dim rowCount As Long, colCount As Long
With ThisWorkbook.Sheets(2)
rowCount = .Cells(.Rows.Count, 1).End(xlUp).row
colCount = .Cells(1, .Columns.Count).End(xlToLeft).Column
For col = 1 To colCount
Debug.Print Now & "processing col " & col
DoEvents
Dim data
data = .Range(.Cells(1, col), .Cells(rowCount, col))
For row = 1 To rowCount
If d.Exists(data(row, 1)) Then data(row, 1) = d(data(row, 1))
Next row
.Range(.Cells(1, col), .Cells(rowCount, col)) = data
.Cells(1, col).Interior.Color = vbYellow
If col Mod 10 = 0 Then ThisWorkbook.Save
Next
End With
Application.ScreenUpdating = True
Application.Calculation = xlCalculationAutomatic
End Sub
One remark: You should consider to use a database for such amount of data.

How to delete a row if it contains a string?

I have only one column of data. I need to write a macro that would go through all the values and delete all rows that contain the word "paper".
A B
1 678
2 paper
3 3
4 09
5 89
6 paper
The problem is that the number of rows is not fixed. Sheets may have different number of rows.
Here is another simple macro that will remove all rows with non-numeric values in column A (besides row 1).
Sub DeleteRowsWithStringsInColumnA()
Dim i As Long
With ActiveSheet '<~~ Or whatever sheet you may want to use the code for
For i = .Cells(.Cells(.Rows.Count, 1).End(xlUp).Row, 1).Row To 2 Step -1 '<~~ To row 2 keeps the header
If IsNumeric(.Cells(i, 1).Value) = False Then .Cells(i, 1).EntireRow.Delete
Next i
End With
End Sub
If you're confident that the rows in question would always contain "paper" specifically and never any other string, you should match based on the value paper rather than it being a string. This is because, particularly in Excel, sometimes you may have numbers stored as strings without realizing it--and you don't want to delete those rows.
Sub DeleteRowsWithPaper()
Dim a As Integer
a = 1
Do While Cells(a, 1) <> ""
If Cells(a, 1) = "paper" Then
Rows(a).Delete Shift:=xlUp
'Row counter should not be incremented if row was just deleted
Else
'Increment a for next row only if row not deleted
a = a + 1
End If
Loop
End Sub
The following is a flexible macro that allows you to input a string or number to find and delete its respective row. It is able to process 1.04 million rows of simple strings and numbers in 2.7 seconds.
Sub DeleteRows()
Dim Wsht As Worksheet
Dim LRow, Iter As Long
Dim Var As Variant
Var = InputBox("Please specify value to find and delete.")
Set Wsht = ThisWorkbook.ActiveSheet
LRow = Wsht.Cells(Rows.Count, 1).End(xlUp).Row
StartTime = Timer
Application.ScreenUpdating = False
With Wsht
For Iter = LRow To 1 Step -1
If InStr(.Cells(Iter, 1), Var) > 0 Then
.Cells(Iter, 1).EntireRow.Delete
End If
Next Iter
End With
Application.ScreenUpdating = True
Debug.Print Timer - StartTime
End Sub

Faster way to Find and Copy Values

Hello, I am doing a macro that copy the values on the columns, VALUES1, VALUES2, VALUES3 if it is not blank when the ARTICLE is the same.
I would have the first spreadsheet and I want the macro to return the second Spreadsheet.
I have managed how to make it:
Sub test()
Dim i, last, j, x As Integer
Dim R As Range
last = Sheets("List2").Range("A100000").End(xlUp).Row - 2
For i = 0 To last
Set R = Sheets("List2").Range("A2")
If Not WorksheetFunction.CountIf(Sheets("List2").Columns(1), _
Sheets("List2").Range("A2").Offset(i, 0).Value) = 0 Then
For j = 1 To WorksheetFunction.CountIf(Sheets("List2").Columns(1), _
Sheets("List2").Range("A2").Offset(i, 0).Value)
Set R = Sheets("List2").Columns(1).Find(Sheets("List2").Range("A2"). _
Offset(i, 0).Value, R, LookAt:=xlWhole)
For x = 0 To 2
If Not Sheets("List2").Range("B2").Offset(i, x).Value = "" Then
R.Offset(0, "1" + x).Value = Sheets("List2"). _
Range("B2").Offset(i, x).Value
End If
Next x
Next j
End If
Next i
End Sub
but the problem it takes too long, 'cause I have around 10.000 Rows and 20 Columns, and besides the Spreadsheet is not in order, so it could be to has a disorder, something like (A, B, B, A, ...)
Is there any way to make it faster o better???
Thanks a lot. Themolestones.
Here is a very easy solution with formulas to your problem:
Sheet2!A1=Sheet1!A1
Sheet2!B1=SUMIF(Sheet1!$A:$A,Sheet2!$A1,Sheet1!B:B)
Sheet2!C1=SUMIF(Sheet1!$A:$A,Sheet2!$A1,Sheet1!C:C)
Sheet2!D1=SUMIF(Sheet1!$A:$A,Sheet2!$A1,Sheet1!D:D)
Put these formulas in the cells left of the = and copy down. You really need only the first two, because you can copy the second also to the right.
You need Sheet1 to be sorted by article.
That's it.
Of course, there might be occasions, when it is just necessary to implement this with VBA. Usually the fastest way to handle large amounts of cells with VBA, is to use array-copies of your ranges. Using worksheet-functions and looping through single cell references slows you down heavily.
Edit:
This would be my VBA solution
Public Sub Demo()
Dim arrRange() As Variant
Dim arrRangeResult() As Variant
Dim i As Long
Dim j As Long
Dim copyVal As Variant
Dim copyCond As Variant
Dim copyCol As Long
'create two copies of the origin data
arrRange = Range("A:D")
arrRangeResult = Range("A:D")
'loop through first data-copy, downwards through the articles
For i = LBound(arrRange, 1) + 1 To UBound(arrRange, 1)
'stop loop, if no article was found
If arrRange(i, 1) = "" Then Exit For
'store current article ID
copyCond = arrRange(i, 1)
'loop sideways through value-columns
For j = LBound(arrRange, 2) + 1 To UBound(arrRange, 2)
'store value & column, when found
If arrRange(i, j) <> "" Then
copyVal = arrRange(i, j)
copyCol = j
Exit For
End If
Next j
'loop through output array and paste value
For j = LBound(arrRangeResult, 1) + 1 To UBound(arrRangeResult, 1)
If arrRangeResult(j, 1) = copyCond Then
'paste-down found value to all occurences of article
arrRangeResult(j, copyCol) = copyVal
ElseIf arrRangeResult(j, 1) = "" Then
'early stop, when no article ID is found
Exit For
End If
Next j
Next i
'create output
Range("K:N") = arrRangeResult
End Sub

Resources