VBA for each loop with 64K+ ListRows (ouf of memory) - excel

I'm running a VBA for each loop through an Excel table (Listobject) which checks if a file exists based on a given path. My table has expanded though and has 68K Listrows. After launching the code, it quickly gives an error Run-time-error '7': Out of memory
It runs OK with 63K lines (done within 5 minutes) and based on googling there appears to be something called "64K segment boundary". Is this what's affecting my code to run since it really feels like it buffers the row count at first and then bounces back w/o starting to actually run anything. Is there an easy workaround for this without the need to split up my dataset into multiple batches? Frankly, I was quite surprised that 64K limits would still be a thing in Excel in 2021.
Running it on 64bit Excel 2019, but no luck with Office365 either.
Sub CheckFiles()
Dim Headers As ListObject
Dim lstrw As ListRow
Dim strFileName As String
Dim strFileExists As String
Application.ScreenUpdating = False
Set ws = ThisWorkbook.Sheets("Import")
Set Headers = ws.ListObjects("Import")
For Each lstrw In Headers.ListRows
strFileName = lstrw.Range(7)
strFileExists = Dir(strFileName)
If strFileExists = "" Then
lstrw.Range(4) = "not found"
Else
lstrw.Range(4) = "exists"
End If
Next lstrw
Set ws = Nothing
Set Headers = Nothing
Application.ScreenUpdating = True
End Sub

Avoid Accessing the Worksheet
Since you cannot avoid looping, you better do it in the computer's memory, i.e. rather through the elements of an array than through the cells of a range.
The code is still slow, about 10s for 200k rows on my machine, but that's because of Dir.
Note how easy (one line only, when the range contains more than one cell) and how fast (a split second) it is to write (copy) a range to an array (Data = rg.Value) and write (copy) the array back to a range (rg.Value = Data).
Adjust the values in the constants section.
Option Explicit
Sub CheckFiles()
Const wsName As String = "Import" ' Worksheet Name
Const tblName As String = "Import" ' Table Name
Const cCol As Long = 7 ' Criteria Column
Const dCol As Long = 4 ' Destination Column
Dim wb As Workbook: Set wb = ThisWorkbook
Dim ws As Worksheet: Set ws = wb.Worksheets(wsName)
Dim Headers As ListObject: Set Headers = ws.ListObjects(tblName)
Dim Data As Variant ' Data Array
With Headers.ListColumns(cCol).DataBodyRange
If .Rows.Count = 1 Then
ReDim Data(1 To 1, 1 To 1): Data = .Value
Else
Data = .Value
End If
End With
Dim r As Long ' Array Row Counter (DataBodyRange Row Counter)
Dim FileName As String ' File Name Retrieved by Dir
For r = 1 To UBound(Data, 1)
FileName = Dir(CStr(Data(r, 1)))
If Len(FileName) = 0 Then
Data(r, 1) = "not found"
Else
Data(r, 1) = "exists"
End If
Next r
Headers.ListColumns(dCol).DataBodyRange.Value = Data
End Sub

Thank you all! A few takeaways. While obviously trying to write as efficient code as possible, any reasonable performance here is acceptable. With that said, for each loop took approx 5 minutes to run with 63K lines, meawhile it was done in about 15 seconds by the code I accepted as an answer by #VBasic2008 - without capacity problems either.
The only problem I had with this particular code was it being somewhat new approach for me, so possibly building on it in the future needs some dedication in looking deeper into it - but it sure looks efficient. I also put together a regular for ... to loop which also didn't run into problems with 68K lines and would steer between rows and columns with offset function.
Clearly faster than for each as #Pᴇʜ suggested but took approx 2x as long as the array method (30 seconds or so).
Sub CheckFiles_2()
Dim strFileName, strFileExists As String
Dim ws As Worksheet
Dim Headers As ListObject
Dim result As String
Dim counter, RowCount As Long
Application.ScreenUpdating = False
Set ws = ThisWorkbook.Sheets("Import")
Set Headers = ws.ListObjects("Import")
RowCount = Headers.ListRows.Count
For counter = 1 To RowCount
strFileName = Range("anchorCell").Offset(counter, 3)
strFileExists = Dir(strFileName)
If strFileExists = "" Then
result = "not found"
Else
result = "exists"
End If
Range("anchorCell").Offset(counter, 0) = result
Next counter
Set ws = Nothing
Set Headers = Nothing
Application.ScreenUpdating = True
End Sub

Related

Is there a way to reassign a Range variable to a different range?

I am very new to VBA, having started programming it yesterday. I am writing a data processing program which requires keeping track of two cells, one on each spreadsheet. The code which reproduces the errors I am experiencing is below. When I call the sub moveCell() in sub Processor(), nothing happens to DIRow and DIColumn, and the code spits out error 1004 at the line indicated. I have tried using DICell = DICell.Offset(), but it returns the same error.
How can I redefine a Range variable to be a different cell?
'<<Main Processor Code>>'
Sub Processor()
Dim PDRow As Integer
Dim PDColumn As Integer
Dim DIRow As Integer
Dim DIColumn As Integer
PDRow = 1
PDColumn = 1
DIRow = 1
DIColumn = 1
Dim PDCell As Range
Dim DICell As Range
Set PDCell = Worksheets("Processed Data").Cells(PDRow, PDColumn)
Set DICell = Worksheets("Data Input").Cells(DIRow, DIColumn)
Call moveCell(2, 0, "Data Input")
End Sub
'<<Function which moves the cell which defines the range>>'
Sub moveCell(r As Integer, c As Integer, sheet As String)
If sheet = "Processed Data" Then
PDRow = PDRow + r
PDColumn = PDColumn + c
Set PDCell = Worksheets("Data Input").Cells(PDRow, PDColumn)
ElseIf sheet = "Data Input" Then
DIRow = DIRow + r '<<<<<<This line does nothing to DIRow's value
DIColumn = DIColumn + c
Set DICell = Worksheets("Data Input").Cells(DIRow, DIColumn) '<<<<<<This line causes error 1004
End If
End Sub
As far as I can tell, you could instead use a quick Function instead. There doesn't seem to be any difference in your If statement results in the moveCell() function, except which worksheet you're using.
We can make this simpler by referring to the Range you're passing to moveCell.
Option Explicit ' forces you to declare all variables
Sub something()
Dim PDCell As Range
Set PDCell = Worksheets("Processed Data").Cells(1, 1)
Dim DICell As Range
Set DICell = Worksheets("Data Input").Cells(1, 1)
PDCell.Select ' can remove
Set PDCell = moveCell(2, 0, PDCell, PDCell.Worksheet.Name)
PDCell.Select ' can remove
Worksheets(DICell.Worksheet.Name).Activate ' can remove
DICell.Select ' can remove
Set DICell = moveCell(5, 0, DICell, DICell.Worksheet.Name)
DICell.Select ' can remove
End Sub
Function moveCell(rowsToMove As Long, colsToMove As Long, cel As Range, ws As String) As Range
Set moveCell = Worksheets(ws).Cells(cel.Row + rowsToMove, cel.Column + colsToMove)
End Function
I've included some rows you don't need (which I've marked with a comment afterwards), but that will show you how the routine works. You can step through with F8 to help see it step-by-step.
Edit: Although, you don't need a separate function at all. Just use OFFSET().
Set PDCell = ...whatever originally
Set PDCell = PDCell.Offset([rows],[cols])

how to optimize for each loop in vba

I need to classify each row of a range accordingly with another range. The script works just fine. But it takes too much time even if it has no more than 300 rows. E.g. 298 rows take more than 2 minutes.
In order to achieve the classification, the script was built with a for each loop inside another one. All is done in the same worksheet called WSSeg. I tried to use all the good practices that I know of.
Option Explicit
Sub Input_Classification()
Application.ScreenUpdating = False
Application.EnableEvents = False
Dim TBLClassification As ListObject
Dim TBLReference As ListObject
Dim rClassificationCell As Range
Dim rClassification As Range
Dim rReferenceCell As Range
Dim rReference As Range
Set TBLClassification = WSSeg.ListObjects("TBClass")
Set rClassification = TBL.ListColumns(4).DataBodyRange
Set TBLReference = WSSeg.ListObjects("TBResumo")
Set rReference = TBL.ListColumns(4).DataBodyRange
For Each rClassificationCell In rClassification
For Each rReferenceCell In rReference
If rClassificationCell.Offset(0, -1).Value <= rReferenceCell.Value Then
rClassificationCell.Value = rReferenceCell.Value
End If
Next rReferenceCell
Next rClassificationCell
Application.EnableEvents = True
Application.ScreenUpdating = True
End Sub
I expect the run time code to be shorter. I don't know if I have to use another logic system. Thanks in advance.
Tried to modify the code, it takes only 0.04 Secs with two tables of around 500 rows.
Tried to keep the replacement logic same as the original, But may please check the same, as i am little confused about the same. If find otherwise, please modify them to your need. Also Could not understand the what is TBL in cases with both the tables and assumed the obvious.
Option Explicit
Sub Input_Classification()
Dim WSSeg As Worksheet
Dim TBLClassification As ListObject
Dim TBLReference As ListObject
Dim rClassification As Range
Dim SrcArr As Variant, TrgArr As Variant, SrcCel As Variant
Dim i As Long, Tm As Double
Set WSSeg = ThisWorkbook.Sheets("Sheet1")
Tm = Timer
Set TBLClassification = WSSeg.ListObjects("TBClass")
Set rClassification = TBLClassification.ListColumns(3).DataBodyRange.Resize(TBLClassification.DataBodyRange.Rows.Count, 2)
TrgArr = rClassification.Value
Set TBLReference = WSSeg.ListObjects("TBResumo")
SrcArr = TBLReference.ListColumns(4).DataBodyRange.Value
For i = 1 To UBound(TrgArr, 1)
For Each SrcCel In SrcArr
If TrgArr(i, 1) <= SrcCel Then
TrgArr(i, 2) = SrcCel
End If
Next SrcCel
Next i
rClassification.Value = TrgArr
Debug.Print "Seconds taken " & Timer - Tm
End Sub
Since I personally don't prefer to keep calculations, event processing and screen updating off (in normal cases) i haven't added that standard lines. However you may use these standard techniques, depending on the working file condition.

Why does using Rows.Count only find the first 12 rows of data?

I'm trying to find the rows with data in my source data sheet and then copy some of the columns into various places in my destination worksheet using VBA. I have successfully done this for a list with 12k lines but when I do some test data, it only copies the first 12 rows out of 19 rows of data....
Sub Header_Raw()
Dim dataBook As Workbook
Dim Header_Raw As Worksheet, Header As Worksheet
Dim dataSource As Range, dataDest As Range
Dim sourceDataRowCount As Integer, index As Integer
Set dataBook = Application.ThisWorkbook
Set sheetSource = dataBook.Sheets("Header_Raw")
Set sheetDest = dataBook.Sheets("Header")
Set dataSource = sheetSource.Range("B4", _
sheetSource.Range("J90000").End(xlUp))
sourceDataRowCount = dataSource.Rows.Count
Set dataDest = sheetDest.Range("B13", "B" & _
sourceDataRowCount)
For index = 1 To sourceDataRowCount
dataDest(index, 1).Value = dataSource(index, 1).Value
dataDest(index, 2).Value = dataSource(index, 2).Value
Next index
End Sub
If you can help tell me what I have done wrong, that would be great
thanks
Julie
Make your life a bit easier with simple debugging. Run the following:
Sub HeaderRaw()
'Dim all the variables here
Set dataBook = Application.ThisWorkbook
Set SheetSource = dataBook.Sheets("Header_Raw")
Set sheetDest = dataBook.Sheets("Header")
Set dataSource = SheetSource.Range("B4", SheetSource.Range("J90000").End(xlUp))
SheetSource.Activate
dataSource.Select
End Sub
Now you will see what is your dataSource, as far as it is selected. Probably it is not what you expect.

VBa Looping through RecordSets is too slow for my program

I have a RecordSet loop inside another RecordSet loop. It'd work well if it didn't take 45 secs for the .OpenRecordSet to run, and the table it'll open has 445k registers.
The reason for the inside loop is because I need to filter results obtained from another RecordSet, and then get these new results and compare.
Would it be better to use other methods, or other way? Is there another way to get specific data from a table(a faster way, of course)? Should I try multithreading?
Since people might need my code:
Private Sub btnGetQ_Click()
Dim tabEQ As DAO.Recordset: Dim tabT7 As DAO.Recordset: Dim tabPesqC As DAO.Recordset: Dim PesqCqdf As DAO.QueryDef
Dim index As Integer: Dim qtdL As Long: Dim qtdL2 As Long
Dim arrC() As String: Dim arrC2() As String: Dim arrC3() As String
Set tabEQ = dbC.OpenRecordset("EQuery", dbOpenSnapshot)
Set tabT7 = dbC.OpenRecordset("T7Query", dbOpenSnapshot)
If Not tabEQ.EOF Then
tabEQ.MoveFirst
qtdL = tabEQ.RecordCount - 1
ReDim arrC(qtdL): ReDim arrC2(qtdL)
If Not tabT7.EOF Then
tabT7.MoveFirst: index = 0
Do Until tabT7.EOF
arrC(index) = tabT7.Fields("CCO"): arrC2(index) = tabT7.Fields("CCE")
Set PesqCqdf = dbC.QueryDefs("pesqCCO")
PesqCqdf.Parameters("CCO") = arrC(index)
Set tabPesqC = PesqCqdf.OpenRecordset(dbOpenSnapshot)
qtdL2 = tabPesqConj.RecordCount - 1
If qtdL2 > 0 Then
ReDim arrC3(qtdL2)
Dim i As Integer
For i = 0 To UBound(arrC3)
arrC3(i) = tabPesqC.Fields("CCE")
tabPesqC.MoveNext
Next
End If
On Error GoTo ERROR_TabT7
index = index + 1: tabT7.MoveNext
Loop
End If
ERROR_TabT7:
Set tabT7 = Nothing
End If
If IsObject(tabEQ) Then Set tabEQ = Nothing
End Sub
I created tables linked with what i wanted :/

Replace text in a cell

I have a sheet that has names, SSNs and 4 columns filled with the following values: S, MB, B.
For said columns I wish to replace S with the number 4, MB with the number 3 and B with the number 2.
Sub replace()
Dim str1, str2, str3, filename, pathname As String
Dim i As Integer
str1 = "MB"
str2 = "B"
str3 = "S"
filename = "p"
pathname = ActiveWorkbook.Path
i = 1
Do While filename <> ""
Set wb = Workbooks.Open(pathname & filename + i)
DoWork wb
wb.Close SaveChanges:=True
filename = Dir()
Loop
End Sub
Sub DoWork(wb As Workbook)
With wb
End With
End Sub
In the function DoWork, how do I create a loop to replace each of the values?
I mostly agree with Michael--to learn the most, you should get started on your own, and come back with more specific questions. However, I am looking to reach 50 rep so I will pander to you. But do please try to go through the code and understand it.
Your name suggests you are a programmer, so the concepts we make use of should be familiar. I like to work from the inside out, so here goes:
here are my variables:
Dim sFileName As String
Dim sFilePath As String
Dim wbBook As Excel.Workbook
Dim i As Double
Dim wsSheet As Excel.Worksheet
Dim rRange As Range
Dim c As Range
Dim dReplace As Double
At the core, you'll want a select case statement to read each cell and decide what the new value should be. Then you will assign the new value to the cell:
Select Case c.value 'c being the cell we are currently examining
Case "S"
dReplace = 4
Case "M"
dReplace = 3
Case "B"
dReplace = 2
Case Else
'Assuming you should only encounter the above values,
'then anything else is an error
'.assert false will stop the code, or you can replace
'with more refined error handling
Debug.Assert False
End Select
c.value = dReplace
Around that you'll want a for each loop to define the current cell and iterate through all the cells in the range you specify for that particular worksheet:
set rRange = wsSheet.Range("C2:E5000") 'Customize to your range
for each c in rRange.Cells
'...
next
Next level up is the for next loop to iterate through all the worksheets in the current file:
For i = 1 To 30
If wbBook.Sheets(i).Name = "" Then
GoTo NextOne
End If
Set wsSheet = wbBook.Sheets(i)
'...
NextOne:
Next i
The if then statement at the top prevents an error if there are fewer than 30 worksheets in a workbook. If the number of sheets per file varies then this will be useful, if the number is fixed, just adjust the loop to stop and the right spot. Of course, this assumes your workbooks have information on multiple sheets. If not skip the loop altogether.
I'm sure many will criticize my use of goto, but since VBA loops lack a continue command, this is the workaround I employ.
Around that you'll want another iterator to loop through your multiple files. Assuming they are all in the same folder, you can use the Dir() function to grab the file names one-by-one. You give it the file path and (optionally) the file type, and it will return the first file name it finds that meets your cirteria. Run it again and it returns the second file name, etc. Assign that to a string variable, then use the file path plus the file name to open the workbook. Use a do loop to keep going until runs out of files:
sFilePath = "C:\Your File Path Here\"
sFileName = Dir(sFilePath & "*.xlsx")
Do Until sFileName = ""
Set wbBook = Workbooks.Open(sFilePath & sFileName)
'...
wbBook.Save
wbBook.Close
sFileName = Dir()
Loop
Now Put it all together:
Sub ReplaceLetterCodewithNumberCode()
Dim sFileName As String
Dim sFilePath As String
Dim wbBook As Excel.Workbook
Dim i As Double
Dim wsSheet As Excel.Worksheet
Dim rRange As Range
Dim c As Range
Dim dReplace As Double
Application.ScreenUpdating = False
sFilePath = "C:\Your File Path Here\"
sFileName = Dir(sFilePath & "*.xlsx")
Do Until sFileName = ""
Set wbBook = Workbooks.Open(sFilePath & sFileName)
For i = 1 To 30
If wbBook.Sheets(i).Name = "" Then
GoTo NextOne
End If
Set wsSheet = wbBook.Sheets(i)
Set rRange = wsSheet.Cells("C2:E5000") 'Customize to your range. Assumes the range will be the same
For Each c In rRange.Cells
Select Case c.value 'c being the cell we are currently examining
Case "S"
dReplace = 4
Case "M"
dReplace = 3
Case "B"
dReplace = 2
Case Else
'Assuming you should only encounter the above values,
'then anything else is an error
'.assert false will stop the code, or you can replace
'with more refined error handling
Debug.Assert False
End Select
c.value = dReplace
Next
NextOne:
Next i
wbBook.Save
wbBook.Close
sFileName = Dir()
Loop
'Clean up
Set wbBook = Nothing
Set wsSheet = Nothing
Set rRange = Nothing
Set c = Nothing
Application.ScreenUpdating = True
End Sub
I'll provide a high level explanation of this; implementation will be up to you. You'll start with a crawler to open all of these files one by one (a google search should help you with this).
I'm not exactly sure how your sheets are organized but the general idea is to open each sheet and perform the action, so you'll need a list of filenames/paths or do it sequentially. Then once inside the file assuming the structure is the same of each you'll grab the column and input the appropriate value then save and close the file.
If you're looking for how to open the VBA editor go to options and enable the Developer tab.
This is a good beginner project and while you may struggle you'll learn a lot in the process.

Resources