Inefficient excel code breaks after few thousand lines of data - excel

Question

I am fairly new to Excel and VBA. I wrote a code that separates a line of data into multiple sections which then adds headers, colors, and plots.
The problem is when I have many lines of data. My code runs just fine when I have about 4000 lines of data, but I get say about 10000 lines, Excel freezes and does not respond anymore. The code is fairly long and I do expect anyone to read the entire thing.
My doubts are that excel does not respond and crashes because there is a watch-dog timer that times the execution of the code and if it does not receive anything back then it crashes. This is only a guess.
Here is a few lines of the actual data that I need to filter and everything.
2017:06:29T14:12:11,0,1013,00,156,-0.112,12.751,000,000,38,34,33,1014,00,202,-0.102,12.734,000,000,38,35,33,1015,00,174,-0.105,12.755,000,000,37,35,33,1008,00,156,-0.110,12.741,000,000,37,35,33,
2017:06:29T14:12:12,0,1013,00,157,-0.102,12.758,000,000,38,34,33,1014,00,203,-0.105,12.744,000,000,38,35,33,1015,00,175,-0.103,12.757,000,000,37,35,33,1008,00,157,-0.107,12.757,000,000,37,35,33,
2017:06:29T14:12:13,0,1013,00,158,-0.113,12.737,000,000,38,34,33,1014,00,204,-0.094,12.760,000,000,38,35,33,1015,00,176,-0.117,12.748,000,000,37,35,33,1008,00,158,-0.109,12.744,000,000,37,35,33,
2017:06:29T14:12:14,0,1013,00,159,-0.103,12.753,000,000,38,34,33,1014,00,205,-0.103,12.720,000,000,38,35,33,1015,00,177,-0.108,12.732,000,000,37,35,33,1008,00,159,-0.110,12.758,000,000,37,35,33,
2017:06:29T14:12:15,0,1013,00,160,-0.112,12.757,000,000,38,34,33,1014,00,206,-0.095,12.734,000,000,38,35,33,1015,00,178,-0.118,12.729,000,000,37,35,33,1008,00,160,-0.115,12.755,000,000,37,35,33,
I am open to any suggestions and more than happy to learn. Thank you for your time and help in advance.
Sub SeparateData()
'Author: Me
'Date: July 13, 2017
'Purpose: This macro take the data in the worksheet and separates the data in a readable fashion for the user.
' This macro also plots and reports any errors that it has caught both in separate sheets named accordingly.
'Define variables
Dim i As Variant
Dim j As Variant
Dim k As Variant
Dim data As Variant
Dim data2 As Variant
Dim count As Variant
Dim shiftDown As Variant
Dim monitorNum As Variant
Dim errorCount As Variant
Dim battChart As ChartObject
Dim currChart As ChartObject
Dim tempChart As ChartObject
'Stop the alerts so we can erase the sheets peacefully
Application.DisplayAlerts = False
'Erase the extra sheets
Sheets("Sheet2").Delete
Sheets("Sheet3").Delete
'Turn on the alerts in case something else happened
Application.DisplayAlerts = True
'Rename the first sheet
ActiveSheet.Name = "Data"
'Create a new sheet for the plots
Sheets.Add.Name = "Plots"
'Create a new sheet for the errors
Sheets.Add.Name = "Errors"
'Activate the first sheet for data processing
Worksheets("Data").Activate
'Enter the number of monitors
monitorNum = 4
'Variable to shift down the data so that te headers will fit (recommended 2)
shiftDown = 2
'Variable to count the number of errors the program thinks occured
errorCount = 0
'Count how many data point there are in the sheet
count = Cells(1, 1).CurrentRegion.Rows.count
'Iterate through the points separating the Data
For i = 0 To count - 1
'First separate the date from the rest
data = Cells(count - i, 1).Value
data = Split(data, "T")
For j = 0 To UBound(data)
Cells(count - i + shiftDown, j + 1).Value = data(j)
Next j
'Now separate the rest of the data
data2 = data(1)
data2 = Split(data2, ",")
For j = 0 To UBound(data2)
Cells(count - i + shiftDown, j + 2).Value = data2(j)
Next j
For k = 0 To monitorNum - 1
'Check for voltage error
If Cells(count - i + shiftDown, (k * 10) + 8).Value > 20 Or IsNumeric(Cells(count - i + shiftDown, (k * 10) + 8).Value) = False Then
'increment the number of errors found
errorCount = errorCount + 1
'Activate the Errors sheet for error recording
Worksheets("Errors").Activate
'Save the row number and the monitor number where the error was founf
Cells(errorCount, 1).Value = "Voltage error in row"
Cells(errorCount, 2).Value = count - i + shiftDown
Cells(errorCount, 3).Value = "in column"
Cells(errorCount, 4).Value = (k * 10) + 8
Cells(errorCount, 5).Value = "in Monitor"
Cells(errorCount, 6).Value = k + 1
Cells(errorCount, 7).Value = "The recorded data was"
Sheets("Data").Cells(count - i + shiftDown, (k * 10) + 8).Copy Cells(errorCount, 8)
'Autofit all the columns
Cells(1, 1).CurrentRegion.EntireColumn.AutoFit
'Activate the first sheet for data processing
Worksheets("Data").Activate
'Clear the contents of the error
Cells(count - i + shiftDown, (k * 10) + 8).ClearContents
End If
'Check for current error
If Cells(count - i + shiftDown, (k * 10) + 7).Value > 80 Or IsNumeric(Cells(count - i + shiftDown, (k * 10) + 7).Value) = False Then
'increment the number of errors found
errorCount = errorCount + 1
'Activate the Errors sheet for error recording
Worksheets("Errors").Activate
'Save the row number and the monitor number where the error was founf
Cells(errorCount, 1).Value = "Current error in row"
Cells(errorCount, 2).Value = count - i + shiftDown
Cells(errorCount, 3).Value = "in column"
Cells(errorCount, 4).Value = (k * 10) + 7
Cells(errorCount, 5).Value = "in Monitor"
Cells(errorCount, 6).Value = k + 1
Cells(errorCount, 7).Value = "The recorded data was"
Sheets("Data").Cells(count - i + shiftDown, (k * 10) + 7).Copy Cells(errorCount, 8)
'Autofit all the columns
Cells(1, 1).CurrentRegion.EntireColumn.AutoFit
'Activate the first sheet for data processing
Worksheets("Data").Activate
'Clear the contents of the error
Cells(count - i + shiftDown, (k * 10) + 7).ClearContents
End If
'Check for temperature error
If Cells(count - i + shiftDown, (k * 10) + 13).Value > 80 Or IsNumeric(Cells(count - i + shiftDown, (k * 10) + 13).Value) = False Then
'increment the number of errors found
errorCount = errorCount + 1
'Activate the Errors sheet for error recording
Worksheets("Errors").Activate
'Save the row number and the monitor number where the error was founf
Cells(errorCount, 1).Value = "Temperature error in row"
Cells(errorCount, 2).Value = count - i + shiftDown
Cells(errorCount, 3).Value = "in column"
Cells(errorCount, 4).Value = (k * 10) + 13
Cells(errorCount, 5).Value = "in Monitor"
Cells(errorCount, 6).Value = k + 1
Cells(errorCount, 7).Value = "The recorded data was"
Sheets("Data").Cells(count - i + shiftDown, (k * 10) + 13).Copy Cells(errorCount, 8)
'Autofit all the columns
Cells(1, 1).CurrentRegion.EntireColumn.AutoFit
'Activate the first sheet for data processing
Worksheets("Data").Activate
'Clear the contents of the error
Cells(count - i + shiftDown, (k * 10) + 13).ClearContents
End If
Next k
Next i
'Erase the data that has been duplicated
For i = 1 To shiftDown
Cells(i, 1).Value = ""
Next i
'Write and color the headers
'For the Date
Range(Cells(shiftDown - 1, 1), Cells(shiftDown, 1)).Merge
Range(Cells(shiftDown - 1, 1), Cells(shiftDown, 1)).Value = "Date"
Range(Cells(shiftDown - 1, 1), Cells(count + shiftDown, 1)).Interior.Color = RGB(200, 190, 150)
'For the Time
Range(Cells(shiftDown - 1, 2), Cells(shiftDown, 2)).Merge
Range(Cells(shiftDown - 1, 2), Cells(shiftDown, 2)).Value = "Time"
Range(Cells(shiftDown - 1, 2), Cells(count + shiftDown, 2)).Interior.Color = RGB(150, 140, 80)
'For the Key Switch
Range(Cells(shiftDown - 1, 3), Cells(shiftDown, 3)).Merge
Range(Cells(shiftDown - 1, 3), Cells(shiftDown, 3)).Value = "Key Switch"
Range(Cells(shiftDown - 1, 3), Cells(count + shiftDown, 3)).Interior.Color = RGB(200, 200, 0)
For i = 1 To monitorNum
Range(Cells(shiftDown - 1, ((i - 1) * 10) + 4), Cells(shiftDown - 1, (i * 10) + 3)).Merge
Range(Cells(shiftDown - 1, ((i - 1) * 10) + 4), Cells(shiftDown - 1, (i * 10) + 3)).Value = "Monitor " & i
'color the headers
If i Mod 4 = 0 Then
Range(Cells(shiftDown - 1, ((i - 1) * 10) + 4), Cells(shiftDown - 1, (i * 10) + 3)).Interior.Color = RGB(100, 255, 100)
ElseIf i Mod 3 = 0 Then
Range(Cells(shiftDown - 1, ((i - 1) * 10) + 4), Cells(shiftDown - 1, (i * 10) + 3)).Interior.Color = RGB(255, 100, 10)
ElseIf i Mod 2 = 0 Then
Range(Cells(shiftDown - 1, ((i - 1) * 10) + 4), Cells(shiftDown - 1, (i * 10) + 3)).Interior.Color = RGB(100, 100, 255)
Else
Range(Cells(shiftDown - 1, ((i - 1) * 10) + 4), Cells(shiftDown - 1, (i * 10) + 3)).Interior.Color = RGB(255, 75, 75)
End If
Next i
For i = 0 To monitorNum - 1
'Monitor ID
Cells(shiftDown, 1 + (i * 10) + 3).Value = "MONITOR_NUM"
'Monitor status
Cells(shiftDown, 2 + (i * 10) + 3).Value = "MONITOR_STATUS"
'Heart Beat count
Cells(shiftDown, 3 + (i * 10) + 3).Value = "HB_COUNT"
'For Current
Cells(shiftDown, 4 + (i * 10) + 3).Value = "CURRENT"
Range(Cells(shiftDown, 4 + (i * 10) + 3), Cells(count + shiftDown, 4 + (i * 10) + 3)).Interior.Color = RGB(240, 150, 150)
'For Voltage
Cells(shiftDown, 5 + (i * 10) + 3).Value = "VOLTAGE"
Range(Cells(shiftDown, 5 + (i * 10) + 3), Cells(count + shiftDown, 5 + (i * 10) + 3)).Interior.Color = RGB(110, 160, 180)
'State of Charge
Cells(shiftDown, 6 + (i * 10) + 3).Value = "SOC"
'State of Health
Cells(shiftDown, 7 + (i * 10) + 3).Value = "SOH"
'Chip temperature
Cells(shiftDown, 8 + (i * 10) + 3).Value = "TEMP_CHP"
'Internal temperature
Cells(shiftDown, 9 + (i * 10) + 3).Value = "TEMP_INT"
'For Temperature of the terminal
Cells(shiftDown, 10 + (i * 10) + 3).Value = "TEMP_EXT"
Range(Cells(shiftDown, 10 + (i * 10) + 3), Cells(count + shiftDown, 10 + (i * 10) + 3)).Interior.Color = RGB(255, 190, 0)
Next i
'Add borders all around the data
Cells(shiftDown, 1).CurrentRegion.Borders.LineStyle = xlContinuous
'Autofit all the columns
Cells(shiftDown, 1).CurrentRegion.EntireColumn.AutoFit
'Plotting
'Activate the first sheet for data plotting
Worksheets("Data").Activate
'Add a new plot
Set battChart = Sheets("Plots").ChartObjects.Add(0, 0, 1200, 300)
'Plot the battery data
With battChart.Chart
.SetSourceData Source:=Sheets("Data").Range(Cells(5, 8), Cells(count + shiftDown, 8))
.SeriesCollection(1).Name = "Battery 1"
.ChartWizard Title:="Voltage", HasLegend:=True, CategoryTitle:="Time (s)", ValueTitle:="Voltage (V)", Gallery:=xlXYScatterLinesNoMarkers
For i = 2 To monitorNum
.SeriesCollection.NewSeries
.SeriesCollection(i).Values = Sheets("Data").Range(Cells(5, ((i - 1) * 10) + 8), Cells(count + shiftDown, ((i - 1) * 10) + 8))
.SeriesCollection(i).Name = "Battery " & i
Next i
End With
'Add a new plot
Set currChart = Sheets("Plots").ChartObjects.Add(0, 300, 1200, 300)
'Plot the current data
With currChart.Chart
.SetSourceData Source:=Sheets("Data").Range(Cells(5, 7), Cells(count + shiftDown, 7))
.SeriesCollection(1).Name = "Battery 1"
.ChartWizard Title:="Current", HasLegend:=True, CategoryTitle:="Time (s)", ValueTitle:="Current (A)", Gallery:=xlXYScatterLinesNoMarkers
For i = 2 To monitorNum
.SeriesCollection.NewSeries
.SeriesCollection(i).Values = Sheets("Data").Range(Cells(5, ((i - 1) * 10) + 7), Cells(count + shiftDown, ((i - 1) * 10) + 7))
.SeriesCollection(i).Name = "Battery " & i
Next i
End With
'Add a new plot
Set tempChart = Sheets("Plots").ChartObjects.Add(0, 600, 1200, 300)
'Plot the current data
With tempChart.Chart
.SetSourceData Source:=Sheets("Data").Range(Cells(5, 13), Cells(count + shiftDown, 13))
.SeriesCollection(1).Name = "Battery 1"
.ChartWizard Title:="Temperature", HasLegend:=True, CategoryTitle:="Time (s)", ValueTitle:="Temperature (F)", Gallery:=xlXYScatterLinesNoMarkers
For i = 2 To monitorNum
.SeriesCollection.NewSeries
.SeriesCollection(i).Values = Sheets("Data").Range(Cells(5, ((i - 1) * 10) + 13), Cells(count + shiftDown, ((i - 1) * 10) + 13))
.SeriesCollection(i).Name = "Battery " & i
Next i
End With
'Indicate that the macro has finished its job
Beep
MsgBox "Data separation is complete. There were " & errorCount & " errors found."
End Sub

Answer 1

All of your Worksheets("x").Activate are totally unnecessary, are slowing your code significantly and are begging for inexplicable errors later when you forget to activate the correct sheet or your bored user starts clicking around during execution because it's taking too long. Declare some Worksheet variables and work with those.
Dim DataSheet as Worksheet
ActiveSheet.Name = "Data"
Set DataSheet = ActiveSheet
Dim PlotSheet as Worksheet
Set PlotSheet as Worksheets.Add
Plotsheet.Name = "Plots"
Dim ErrorSheet as Worksheet
Set ErrorSheet = Worksheets.Add
ErrorSheet.Name = "Errors"
count = Datasheet.Cells(1, 1).CurrentRegion.Rows.count
'GET RID OF THIS EVERYWHERE!!! Worksheets("Errors").Activate
'Save the row number and the monitor number where the error was founf
With ErrorSheet
.Cells(errorCount, 1).Value = "Voltage error in row"
.Cells(errorCount, 2).Value = count - i + shiftDown
.Cells(errorCount, 3).Value = "in column"
.Cells(errorCount, 4).Value = (k * 10) + 8
.Cells(errorCount, 5).Value = "in Monitor"
.Cells(errorCount, 6).Value = k + 1
.Cells(errorCount, 7).Value = "The recorded data was"
'Note subtle change here:
DataSheet.Cells(count - i + shiftDown, (k * 10) + 8).Copy .Cells(errorCount, 8)
'Note: explicitly setting "datasheet" as the destination and using the "With" to save some typing on the ".Cells" call.
'You could explicitly type the "ErrorSheet" to make it more clear
'an even better version is:
.cells(errorCount, 8) = DataSheet.Cells(count - i + shiftDown, (k * 10) + 8)
End With
Continue to do that everywhere. Future you will appreciate current you...
Every time you're doing a Sheet("x").Activate eliminate that line and explicitly add a reference to the appropriate worksheet variable you declared earlier.
Every time you have an unqualified Sheets or Cells or Range call, make it an explicit reference by prepending the appropriate worksheet variable. Future you will appreciate the fact that you can see exactly what worksheet you're referencing. Sure, there may be some extra typing involved, but that extra typing significantly reduces the chance of inserting very subtle and difficult to find bugs.
Using .Copy is very slow for single cells. It does gain a speed advantage if you're copying large blocks of cells in a single go (somewhere in the neighborhood of 3-5k cells in a single copy statement vs a loop through setting individual cell values).
As Uri Goren pointed out setting Application.Calculation = False will definitely improve your speed. I'd recommend not setting Application.ScreenUpdating = False until after your code is 100% functional and not generating any errors. Once you're at that point, it's a great thing to do.
At this point in your code you may want to add the indicated line:
'Iterate through the points separating the Data
For i = 0 To count - 1
'Add this line:
Application.StatusBar = "Separating points #" & i
Put a similar message just inside the top of each of your large loops. You'll likely see that your code isn't hanging, just taking a long time to process. Plus you'll have an update that your user can watch so (s)he'll know it isn't hung and is still doing something.
At the end of your code put:
Application.StatusBar = ""
To clear the message so you get your normal Excel StatusBar functionality returned.

Answer 2

Add these two lines at the beginning of the subroutine:
Application.ScreenUpdating = False
Application.Calculation = xlCalculationManual
And these 2 lines before the end of the subroutine
Application.Calculation = xlCalculationAutomatic
Application.ScreenUpdating = True
It should speed up your code significantly