Best way to read a very large array into excel sheet - excel

Question

I have to import a number of text files into excel and add each text file to a new sheet. The number of lines on some files are in excess of 350,000. Loops take so long that it's not really user friendly. I've tried using this to read the data in quickly
Dim arrLines() As String
Dim lineValue As String
lineValue = ts.ReadAll
DoEvents
arrLines() = Split(lineValue, vbCrLf)
Dim Destination As Range
Set Destination = Worksheets(WorksheetName).Range("A2")
Set Destination = Destination.Resize(UBound(arrLines), 1)
Destination.Value = Application.Transpose(arrLines)
but this results in every value AFTER line 41243 simply having a value of "#N/A". I was thinking to use a Application.Index to split up the array into smaller arrays, but you need to give the index function an array of lines that you want to compose the new array, and that would mean creating a loop to run through the numbers 1-41000, then 41001-82000, etc. At the point i'm doing a loop to create the arrays it's not really faster. looping through the file line by line is similarly too slow. What's a good way of reading in a such a large number of lines without ending up with the missing values?

Answer 1

You could use and automate the 'Data' -> 'From Text/CSV' wizard of Excel.
Using the Macro recorder you end up with this, which should be a good start:
ActiveWorkbook.Queries.Add Name:="MyFile", Formula:="let" & Chr(13) & "" & Chr(10) & " Source = Table.FromColumns({Lines.FromBinary(File.Contents(""C:\Path\MyFile.txt""), null, null, 1252)})" & Chr(13) & "" & Chr(10) & "in" & Chr(13) & "" & Chr(10) & " Source"
ActiveWorkbook.Worksheets.Add
With ActiveSheet.ListObjects.Add(SourceType:=0, Source:="OLEDB;Provider=Microsoft.Mashup.OleDb.1;Data Source=$Workbook$;Location=""MyFile"";Extended Properties=""""", Destination:=Range("$A$1")).QueryTable
.CommandType = xlCmdSql
.CommandText = Array("SELECT * FROM [MyFile]")
.RowNumbers = False
.FillAdjacentFormulas = False
.PreserveFormatting = True
.RefreshOnFileOpen = False
.BackgroundQuery = True
.RefreshStyle = xlInsertDeleteCells
.SavePassword = False
.SaveData = True
.AdjustColumnWidth = True
.RefreshPeriod = 0
.PreserveColumnInfo = True
.ListObject.DisplayName = "MyFile"
.Refresh BackgroundQuery:=False
End With

Answer 2

Copy Text Files to Excel
Credits to simple-solution for suggesting (in the comments) to open the text files with Workbooks.Open.
The Code
Sub CopyTextFilesToExcel()
' Search Folder Path
Const cStrPath As String _
= "D:\Excel\MyDocuments\StackOverflow\"
Const cStrExt As String = "*.txt" ' File Extension
Const cFolderPicker As Boolean = False ' True to enable FolderPicker
Dim wb As Workbook ' Current File
Dim strPath As String ' Path of Search Folder (Incl. "\" at the end.)
Dim strFileName As String ' Current File Name
With Application
.ScreenUpdating = False
.DisplayAlerts = False
End With
On Error GoTo ProcedureExit
' Determine Search Path ("\" Issue)
If cFolderPicker Then
With Application.FileDialog(msoFileDialogFolderPicker)
If .Show = False Then Exit Sub
strPath = .SelectedItems(1) & "\"
End With
Else
If Right(cStrPath, 1) <> "\" Then
strPath = cStrPath & "\"
Else
strPath = cStrPath
End If
End If
' Determine first Current File Name.
strFileName = Dir(strPath & cStrExt)
With ThisWorkbook ' Target Workbook
' Loop through files in folder.
Do While strFileName <> ""
' Create a reference to the Current File.
Set wb = Workbooks.Open(cStrPath & strFileName)
' Copy first worksheet of Current File after the last sheet
' (.Sheets.Count) in Target Workbook.
wb.Worksheets(1).Copy After:=.Worksheets(.Sheets.Count)
' Close Current File without saving changes (False).
wb.Close False
' Find next File(name).
strFileName = Dir()
Loop
End With
MsgBox "All files copied!"
ProcedureExit:
With Application
.ScreenUpdating = True
.DisplayAlerts = True
End With
End Sub

Answer 3

Mathieu Guindon had EXACTLY the solution I was hoping for. Eliminating the transpose has solved the issue with the #N/A values. Thank you!
Edit:
The code just loops the arrayed data a second time into a two dimensional array and then posts it to the range without the transpose effect. It's a little slower than the old way (taking about two minutes or so longer), but it's still pretty fast and produces the results I want. Code is as follows:
lineValue = ts.ReadAll
DoEvents
arrLines() = Split(lineValue, vbCrLf)
Dim arrBetween() As Variant
ReDim arrBetween(UBound(arrLines), 0)
LoopLength = UBound(arrLines) - 1
For i = 0 To LoopLength
arrBetween(i, 0) = arrLines(i)
DoEvents
If i Mod 2500 = 0 Or i = LoopLength Then
Application.StatusBar = "Importing " & WorksheetName & " " & (i) & " ."
End If
Next i
Dim Destination As Range
Set Destination = Worksheets(WorksheetName).Range("A2:A" & UBound(arrLines))
Destination.Value = arrBetween