How to speed up a data file import in Excel VBA - excel

Question

Update as of June 11, 2019: I still haven’t figured out why practically all of my delay happens in those two lines, but current status is that I put up with the delay. So far, I have about 6000 rows of data in the master document, and an import process takes about 20 seconds regardless of how many rows I import.
—
I have a "master document" and I import data from lots and lots of little documents all day long. I admit I'm not a super-genius here, and a lot of my coding habits come from doing it "old school" so there may be "Excel ways" that I don't know (but want to learn!).
The issue I'm seeing is how much time a data file import can take.
When I started the tool out, data imports took only a few seconds.
Now that I have about 3500 rows of data, data imports take about 15-20 seconds. It doesn't matter if I am importing one row or a hundred rows. I expect this to keep going up. By the time I get to 7000 rows or 10,000 rows, I expect it to become unbearable.
By using message boxes (remember: "old school"), I've been able to narrow the speed bottleneck down to two lines of code. Between "Step 1" and "Step 2" is about 30% of my delay, and between "Step 2" and "Step 3" is about 70% of my delay.
I've included the whole sub below to make sure I'm not missing something obvious, but I made sure to UNINDENT my message boxes so you can go r-i-g-h-t to the code I suspect. Also, I included the entire sub because usually one of the first responses is “can you show the whole sub so I have better context?”
Thank you kindly for any thoughts or suggestions you might have. :)
Private Sub Btn_ImportDataFiles_Click()
' Search the current worksheet and assign the next TransactionID
Dim TransactionCounter As Integer
Dim TransactionID As Long ' This is the next available Transaction ID
TransactionID = Application.WorksheetFunction.Max(Range("a:a")) + 1
' open the file and import the data
Dim customerBook As Workbook
Dim filter As String
Dim caption As String
Dim customerFilename As String
Dim customerWorkbook As Workbook
Dim targetWorkbook As Workbook
' make weak assumption that active workbook is the target
Set targetWorkbook = Application.ActiveWorkbook
' get the customer workbook
filter = "Text files (*.xlsx),*.xlsx"
caption = "Please Select an input file "
customerFilename = Application.GetOpenFilename(filter, , caption)
If customerFilename <> "False" Then
' If they have uploaded the file before, let them know.
' If they want to keep uploading it, no harm done,
' but no need to stupidly add data that is already present.
' Select the archive sheet
Sheets("Upload_Archive").Select
Dim FileNameHunt As String
Dim cell As Range
Dim ContinueUpload As Boolean
ContinueUpload = True
FileNameHunt = Mid(customerFilename, InStrRev(customerFilename, "\") + 1)
Columns("A:A").Select
Set cell = Selection.Find(what:=FileNameHunt, after:=ActiveCell, LookIn:=xlFormulas, lookat:=xlWhole, searchorder:=xlByRows, searchdirection:=xlNext, MatchCase:=False, searchformat:=False)
If cell Is Nothing Then ' Add the new filename to the archive
Sheets("Upload_Archive").Select
Rows(1).Insert shift:=xlDown
Range("a1:a1").Value = FileNameHunt
Sheets("MasterSheet").Select
Application.Cells.Font.Name = "Calibri Light"
Application.Cells.Font.Size = "8"
Application.Cells.Font.Bold = False
Else
response = MsgBox("This data file has previously been uploaded. " & vbCrLf & "Do you want to cancel this upload?" & vbCrLf & vbCrLf & "Pressing [yes] will cancel the process." & vbCrLf & "Pressing [no] will continue with the file upload" & vbCrLf & "and add the data to the tracking sheet.", vbYesNo)
If response = vbYes Then
ContinueUpload = False
Sheets("MasterSheet").Select
Exit Sub
End If
End If ' If cell Is Nothing Then...
If ContinueUpload = True Then
' Continue with data upload procedure
Sheets("MasterSheet").Select
Set customerWorkbook = Application.Workbooks.Open(customerFilename)
' Copy data from customer to target workbook
Dim targetSheet As Worksheet
Set targetSheet = targetWorkbook.Worksheets(1)
Dim sourceSheet As Worksheet
Set sourceSheet = customerWorkbook.Worksheets(1)
Dim ImportRecordCount As Integer
ImportRecordCount = sourceSheet.Range("B1")
Dim ReconciliationID As String
ReconciliationID = ""
If sourceSheet.Range("E3") = "Removed from Depot" Then ReconciliationID = "1"
MsgBox ("Step 1")
targetSheet.Range("A1").EntireRow.Offset(1).Resize(ImportRecordCount).Insert shift:=xlDown ' Add the blank rows
MsgBox ("Step 2")
targetSheet.Range("B2:AB" & ImportRecordCount + 1).Value = sourceSheet.Range("A3:AA" & ImportRecordCount + 2).Value ' Bring in the big pile of data
MsgBox ("Step 3")
targetSheet.Range("AJ2:AJ" & ImportRecordCount + 1).Value = ReconciliationID ' To help with reconciling shipments
targetSheet.Range("AK2:AK" & ImportRecordCount + 1).Value = ReconciliationID ' To help with deployment timing
'targetSheet.Range("AI2:AI" & ImportRecordCount + 1).Value = "=COUNTIFS($D:$D, D2, $F:$F, F2)" ' This is the helper formula for identifying duplicates (deprecated, but I'm saving the code)
For TransactionCounter = 2 To ImportRecordCount + 1 ' Create and add the new Transaction ID values
targetSheet.Range("a" & TransactionCounter) = TransactionID + ImportRecordCount - TransactionCounter + 1
Next
' Close customer workbook
customerWorkbook.Close
' Format the sheet properly
Application.Cells.Font.Name = "Calibri Light"
Application.Cells.Font.Size = "8"
Application.Cells.Font.Bold = False
Application.Range("1:1").Font.Size = "10"
Application.Range("1:1").Font.Bold = True
' Query the User -- delete the file?
If MsgBox("Delete the local client-generated data file?" & vbCrLf & vbCrLf & "(this will NOT affect your email)", vbYesNo, "Confirm") = vbYes Then
Kill customerFilename
' MsgBox ("File: " & vbCrLf & customerFilename & vbCrLf & "has been deleted.")
End If
End If ' If ContinueUpload = True Then
End If ' If customerFilename <> "False" Then
End Sub
edit
I edited your original question to highlight things I found as suspect. These are things I felt are worth pointing out to you. I shaved everything else out as to focus on these particular issue. Review them and do soem research to see if you can find yourself in a better situation.
MsgBox ("Step 2")
'Ive never moved large amounts of data using this method. Ive always just used arrays. I have moved smaller bits of data though.
' I suspect that this might take a moment if the data set is large. Again use arrays to grab the data and move it.
' Edward says “This step takes about 70% of my delay — even if bringing in only a single line of data.”
targetSheet.Range("B2:AB" & ImportRecordCount + 1).Value = sourceSheet.Range("A3:AA" & ImportRecordCount + 2).Value ' Bring in the big pile of data
MsgBox ("Step 3")
' this loop is probably your main culprit of your performance issue.
' Edward says “Nope, this flies by. It is not the issue at all. I have verified this already.”
' Learn how to construct an array of data on the fly and then learn how to dump the entire array to
' sheet using a simple method.
For TransactionCounter = 2 To ImportRecordCount + 1 ' Create and add the new Transaction ID values
targetSheet.Range("a" & TransactionCounter) = TransactionID + ImportRecordCount - TransactionCounter + 1
Next

Answer 1

It looks like you have a lot of good things going here. A few things that I saw that could potentially be changed to improve your performance.
First, between "Step 1" and "Step 2": In my experience, adding rows takes longer than using rows that already exist. It looks like you are basically "pushing" everything down to make room for the new data, such that the newly entered data is at the top and the oldest data is at the bottom. (Correct me if I am wrong on any of this.) If you were to simply add the data to the end of the sheet, you would probably see some performance improvements, although I don't know how big of an improvement it would be.
Second, between "Step 2" and "Step 3": I have found that using .Value2 as opposed to .Value can give you some performance improvements, and the larger the data the bigger the improvement. This has a down side - Value2 does not retain any of the formatting that might be present, meaning that the number type (date, accounting, etc) does not pull over correctly. If this is something that you do not need, then you can use Value2.
Finally, other methods: When I run extensive macros, I always try to do everything I can to get a performance boost. You can get slight boosts across the board by using tricks like turning off screen updating (Application.ScreenUpdating = False), just be sure to turn it back on at the end of the macro.
I hope that this helps you figure it out! If all else fails, you can do it once or twice by hand to remember how much faster it is using the macro! Haha. Good Luck!

Answer 2

Have you tried using .value2? In some scenarios it might bring you better performance. Check some performance comparatives here: https://fastexcel.wordpress.com/2011/11/30/text-vs-value-vs-value2-slow-text-and-how-to-avoid-it/
It's difficult seeing where's the issue without having access to the original sheets. Maybe the issue is with the data itself instead of your VBA code and sometimes you might need to clean your source data of the heavy stuff and then add it again if needed.
You could also look into doing some parts with Python but I guess that's out of the question if you don't want to add additional software layers to your solution.

Answer 3

Try adding this at the beginning and end of your script. Just be sure to set everything back to TRUE!!
Application.ScreenUpdating = False
Application.DisplayAlerts = False
...CODE HERE...
Application.ScreenUpdating = True
Application.DisplayAlerts = True