There is a report in Oracle that I dump into Excel but it has terrible formatting. There are thousands of individual accounts and each account has one or more lines of data. I need to do intensive data manipulation on the report to whip it into shape such that there is any utility to it whatsoever. This is the way the report is laid out (the "Rows of data" lines vary for each record).
Header1
Header2
Row of data
Row of data
Row of data
----------------------
summary data
summary data
The problem is that these records can not be filtered by the headers because the data is stacked vertically. So, after I have stripped out all the extraneous rows that I don't want, leaving a delimiting blank row between each record (a primitive method of enabling exiting the inner loop), I run a very simple VBA routine I created. For each 'Row of Data', the headers print to a column to the right on the same row.
I segment the data into two or more sets because there might be over 60K lines and this is to prevent a run time error "6".
Row of data Header1 Header2
Row of data Header1 Header2
Row of data Header1 Header2
----------------------
summary data
summary data
The following routine used to run at lightening speed - less than 30 seconds, now after a change from Office 2016 to Office 365 desktop, the same routine runs painfully slow. It can take a half hour to run one segment. I am baffled. Can someone tell me what might be causing this and what I can change with this routine to make it run faster?
Sub UnifyRowData()
Dim i As Integer
Dim TtlRows As Integer
TtlRows = Cells(Rows.Count, 1).End(xlUp).Row
i = 0
Do Until i > TtlRows
i = i + 1
Heading1 = Cells(i, 1)
Heading2 = Cells(i + 1, 1)
Heading3 = Cells(i + 2, 1)
Do
Cells(i, 3).Value = Heading1
Cells(i, 4).Value = Heading2
Cells(i, 5).Value = Heading3
i = i + 1
Loop Until IsEmpty(Cells(i, 1))
Loop
End Sub
`
I don't know what I can change. I've read about screen updating causing long run times but someone would need to explain to me why that would slow down this routine that used to run at lightening speed.
This should be faster, using an array to work on the data:
Sub UnifyRowData()
Dim i As Long, ws As Worksheet, h1, h2, h3
Dim TtlRows As Long, arr, rngData As Range
Set ws = ActiveSheet
'get the data range (including additional columns to be populated)
Set rngData = ws.Range("A1", ws.Cells(ws.Rows.Count, 1).End(xlUp)).Resize(, 5)
arr = rngData.Value 'read as array
TtlRows = UBound(arr, 1) '# of rows of data
i = 0
Do While i <= TtlRows
i = i + 1
h1 = arr(i, 1)
h2 = arr(i + 1, 1)
h3 = arr(i + 2, 1)
Do
arr(i, 3) = h1
arr(i, 4) = h2
arr(i, 5) = h3
i = i + 1
If i > TtlRows Then Goto done 'exit loops if at end of array
Loop Until Len(arr(i, 1)) = 0
Loop
done:
rngData.Value = arr 'write the array back to the sheet
End Sub
Related
In my quest to improve the quality of life at work, I've searched for an answer and wound up borrowing this code (posted my current attempt at bottom of the post) to extract differences between two worksheets. While it returns the basic information, it is less QoL change than my current method, which, while it works most of the time, still fails. The current method is as follows:
=IF(COUNTIFS(New!$H:$H, Old!$H2, New!$C:$C, Old!$C2,New!$B:$B, Old!$B2)<1, Old!$H2, "")
This code spans across several columns to populate the appropriate information (appointment time, date, patient name, patient ID, notes, etc). This goes on a sheet called "Removed", and I have one for "Added" where New and Old are reversed.
I attempted to modify the borrowed code to paste entire rows instead of just one column, but I seem to be failing at every turn, mainly because I am new to VBA and do not have a full grasp of it yet. Changing the first For loop to:
For i = LBound(valsM, 1) To UBound(valsM, 1)
If IsError(Application.Match(valsM(i, 1), valsQ, 0)) Then
mm = mm + 1
Worksheets("Old").Cells(i).EntireRow.Copy Destination:= Worksheets("New").Cells(mm, 1)
End If
Next i
is obviously the incorrect way, and I suspect it's due to the whole thing being based on arrays. What must I change in the script to accommodate 16 columns of information that must be moved over to appropriate pages? Bonus would be putting them all on one page and appending a 17th column Q that indicates removed or added. Appreciate the help.
Sub YouSuckAtVBA()
Dim i As Long, mm As Long
Dim valsM As Variant, valsQ As Variant, valsMM As Variant
With Worksheets("New")
valsM = .Range(.Cells(1, "A"), .Cells(.Rows.Count, "A").End(xlUp)).Value2
End With
With Worksheets("Old")
valsQ = .Range(.Cells(1, "H"), .Cells(.Rows.Count, "H").End(xlUp)).Value2
End With
ReDim valsMM(1 To (UBound(valsM, 1) + UBound(valsQ, 1)), 1 To 2)
mm = 1
valsMM(mm, 1) = "value"
valsMM(mm, 2) = "missing from"
For i = LBound(valsM, 1) To UBound(valsM, 1)
If IsError(Application.Match(valsM(i, 1), valsQ, 0)) Then
mm = mm + 1
Worksheets("Old").Cells(i).EntireRow.Copy Destination:=Worksheets("New").Cells(mm, 1)
End If
Next i
For i = LBound(valsQ, 1) To UBound(valsQ, 1)
If IsError(Application.Match(valsQ(i, 1), valsM, 0)) Then
mm = mm + 1
Worksheets("New").Cells(i).EntireRow.Copy Destination:=Worksheets("Old").Cells(mm, 1)
End If
Next i
valsMM = helperResizeArray(valsMM, mm)
With Worksheets("Test")
With .Cells(.Rows.Count, "A").End(xlUp).Offset(1, 0)
.Resize(UBound(valsMM, 1), UBound(valsMM, 2)) = valsMM
End With
End With
End Sub
Function helperResizeArray(vals As Variant, x As Long)
Dim arr As Variant, i As Long
ReDim arr(1 To x, 1 To 2)
For i = LBound(arr, 1) To UBound(arr, 1)
arr(i, 1) = vals(i, 1)
arr(i, 2) = vals(i, 2)
Next i
helperResizeArray = arr
End Function
If you have Office 365 you can use the new Filter-Function
The screenshot shows the formulas using a very basic example.
"Table old" and "Table new" are created via "Insert > Table" therefore it is possible to reference the column names within the formula instead of B or D
I have a worksheet, and I want to be able to go through one column (O) to find the different values, then go to another column (U) and count whether the sting is paper or electronic. Then, I want to be able to take the total of paper/electronic stings from U with each instance in O (source) and put it into the following table on a different sheet with VBA.
Due to the sensitivity of the data, I quickly made a table with basically what I mean. Pretend A is O and B is U.
And I want the output in this table, or if there is a better way to present the data:
I've tried making a pivot table, but it simply counts each instance of the paper/electronic string in the sheet, and I need to cross reference the values in O with U.
Here is the formula what you desire. Remember that we need to change source value and Fillining medium value in each row. you can see from the image that in formula for Source A values are "A" and "Paper" for paper count and "A" and "Electronic" for electronic count. the formulas for Source A are written at the bottom of the table and formula for Source C you can See from formula Bar. This is to show you the change you need to make in formula for each source.
if you have excel 365 you can just use the unique/countifs function. For simplicity I assume your data is in col A & B
To get the unique values (source) col E:
=UNIQUE(A:A)
To count (manually add "paper" as header in col F:
=COUNTIFS(A:A;E2;B:B;$F$1)
Do the same for the other values.
EDIT:
Anything can be done in code:
Option Explicit
Sub DictUniqueFinal()
Dim arr, arr2, arrH, j As Long, dict As Object, id As String
'setup some arrays
arrH = Split("Source, Paper, Electronic", ",")
arr = Sheet1.Range("A1").CurrentRegion.Offset(1, 0).Value2 'load source without headers
ReDim arr2(1 To UBound(arr), 1 To 3)
'setup the dict
Set dict = CreateObject("Scripting.Dictionary") 'create dictionary lateB
For j = 1 To UBound(arr) - 1 'traverse source
id = arr(j, 1)
If Not dict.Exists(id) Then 'create key
If arr(j, 2) = "paper" Then
dict.Add id, 1 & "," & 0
Else
dict.Add id, 0 & "," & 1
End If
Else 'update key
If arr(j, 2) = "paper" Then
dict(id) = Split(dict(id), ",")(0) + 1 & "," & Split(dict(id), ",")(1)
Else
dict(id) = Split(dict(id), ",")(0) & "," & Split(dict(id), ",")(1) + 1
End If
End If
Next j
'build final array
ReDim arr2(0 To dict.Count - 1, 1 To 3)
For j = 0 To dict.Count - 1
arr2(j, 1) = dict.Keys()(j)
arr2(j, 2) = Split(dict.Items()(j), ",")(0)
arr2(j, 3) = Split(dict.Items()(j), ",")(1)
Next j
'dump to sheet
With Sheet2
.Range(.Cells(1, 1), .Cells(1, UBound(arrH) + 1)).Value2 = arrH
.Range(.Cells(2, 1), .Cells(UBound(arr2) + 2, UBound(arr2, 2))).Value2 = arr2
End With
End Sub
It's a bit long and I had to hard code, but I found a solution, thanks to #AnmolKumar I looked in to Countif and found this:
ws2.Range("F15").Value2 = Excel.WorksheetFunction.CountIfs, _
(ws3.Range("O1:O" & lstRow2), "A", ws3.Range("U1:U" & lstRow2), "Paper")
I'll just have to do it for each different section
I have existing script that does a major chunk of what I need. The script (from here: https://www.extendoffice.com/documents/excel/4054-excel-duplicate-rows-based-on-cell-value.html) basically inserts and then copies rows of data X number of times, where X is one of the fields in the table. It works well and the referenced page shows examples of the start and end points.
But when I run the script in Excel I go from ~2,000 lines in my table to ~40,000 lines. I need to modify all the duplicated rows (incremental dates) and so I am now attemting to also include new data into the table while the script runs that will allow me to change data in the duplicated rows... for example I can use the duplicate number 1, 2, 3, 4 and some simple formulas to change dates relative to a start point.
I expect that I will need some additional code inserted into the routine that will add data into a nominated column and do the auto incrementing from 1.
Having zero actual VBA skillz, ive no idea how to tackle the second part of my problem with the code I already have. Any help would be totally awesome !!
Sub CopyData()
'Updateby Extendoffice 20160922
Dim xRow As Long
Dim VInSertNum As Variant
xRow = 1
Application.ScreenUpdating = False
Do While (Cells(xRow, "A") <> "")
VInSertNum = Cells(xRow, "D")
If ((VInSertNum > 1) And IsNumeric(VInSertNum)) Then
Range(Cells(xRow, "A"), Cells(xRow, "D")).Copy
Range(Cells(xRow + 1, "A"), Cells(xRow + VInSertNum - 1, "D")).Select
Selection.Insert Shift:=xlDown
xRow = xRow + VInSertNum - 1
End If
xRow = xRow + 1
Loop
Application.ScreenUpdating = False
End Sub
Try this code below, I used the same sample data on the link you provided. However on this code I created 2 worksheets, one for the raw data to be processed and one for the duplicate output including the increment of dates and duplicate number.
Sub duplicateData()
Dim rSH As Worksheet
Set rSH = ThisWorkbook.Sheets("RAW") 'Your raw data
Dim oSH As Worksheet
Set oSH = ThisWorkbook.Sheets("OUTPUT") 'Output data on another sheet
x = 2
For a = 2 To rSH.Range("A" & Rows.Count).End(xlUp).Row
For b = 1 To rSH.Cells(a, 4).Value '4 is the column of duplicate times
If b = 1 Then
For c = 1 To 4 'Number of your column
oSH.Cells(x, c).Value = rSH.Cells(a, c).Value
Next c
oSH.Cells(x, 5) = 1 'First instance, 5 is the column number of duplicate counter
Else
For c = 1 To 4 'Number of your column
oSH.Cells(x, c).Value = rSH.Cells(a, c).Value
Next c
oSH.Cells(x, 3).Value = CDate(oSH.Cells(x - 1, 3).Value) + 1 '3 is the column number of date to increment
oSH.Cells(x, 5).Value = CInt(oSH.Cells(x - 1, 5).Value) + 1 '5 is the column number of duplicate counter
End If
x = x + 1 'Increment Output row number
Next b
Next a
End Sub
I run a performance database and have gotten stuck with a way to track repeat offenders.
In a Results sheet is all the data, I want to create a macro that goes through the results, filters column C to each staff number and count how many times they have a "Fail" result in column D.
If they have a count of 2 or above I want the sheet to copy their name in column B and staff no in column c to the first available line in a different sheet called "Flagged" with the fail count in a 3rd column.
My data runs from rows b8 to b10008 and I have 300 staff who could be assessed
Thanks in advance!
Set up your source data as a table (Ctrl+T with cell in range selected). Add a helper column with the formula:
=SUMPRODUCT(--([Fail/Pass]="FAIL"),--([Staff No]=[#[Staff No]]))>=2=SUMPRODUCT(--(D:D="FAIL"),--(C:C=[#[Staff No]]))>=2
Create your pivottable, Alt+N+V, using compact report layout, and add your helper column to the page field and filter on True. Add name and staff No to the row fields and remove subtotals.
As it is an Excel table you can add more rows and the formula will autofill down. You then just refresh the pivottable to update your flagged list.
Data:
Fields:
Compact design layout and no subtotals.
I would recommend to make use of arrays and loop your data that way, it should be nearly instant (comparing to looping in the sheet itself).
Keep in mind this is not fully tested, but it should get you pretty close to what you are trying to achieve:
Sub flagged()
Dim arrData As Variant, arrFails As Variant
Dim failCnt As Long, i As Long, j As Long, x As Long, lastRow As Long
Dim shResults As Worksheet, shFails As Worksheet
Set shResults = ActiveWorkbook.Sheets("Results")
Set shFlagged = ActiveWorkbook.Sheets("Flagged")
ReDim arrFails(0 To 300, 0 To 2)
arrData = shResults.Range("B8:D10008").Value
For i = LBound(arrData) To UBound(arrData)
For j = LBound(arrData) To UBound(arrData)
If arrData(i, 2) = arrData(j, 2) Then
If arrData(i, 3) = "FAIL" Then
failCnt = failCnt + 1
End If
If failCnt >= 2 Then
arrFails(x, 0) = arrData(i, 1)
arrFails(x, 1) = arrData(i, 2)
arrFails(x, 2) = failCnt
x = x + 1
End If
End If
Next j
failCnt = 0
Next i
For i = LBound(arrFails) To UBound(arrFails)
If arrFails(i, 0) <> "" Then
lastRow = shFlagged.Cells(1, j).End(xlDown).Row
For j = 1 To 3
shFlagged.Cells(lastRow + 1, j) = arrFails(i, j)
Next j
End If
Next i
End Sub
EDIT: changed the size of the dimension to accommodate 3 columns. Also I've initially done this to look for sorted data by staff number, but given is not that much data, that doesn't matter much, so I've edited out the code accordingly.
RawData is an excel report drawn from an employee database. (Tried to attach the workbook but didn't see how to do that). RawData contains multiple, unwanted duplicate items for some employees. I'm told this is because of a Cartesian join in the employee database that creates the RawData report. Whether or not that's the case, I have no control over how the RawData report is produced. It is what it is.
I need to clean up the RawData report so that the end product looks like the CorrectedView tab, which I corrected manually. RawData can, at times, be several thousand rows so automating the clean-up would be a huge help.
The structure of RawData is in five groupings of columns: Employee Basic Info (cols A-E), Education (cols F-H), Awards (cols I-L), Certifications (cols M-Q) and Accomplishments (cols R-T). In the CorrectedView, what I did was:
Removed the duplicates for each employee in each of the five column sections
Moved the remaining data for each employee upward so that each employee's info begins on his/her first row
Removed any blank rows created between employees after doing #2 above.
I'm looking for a way to automate the process. I have some code (shown below) that accomplishes #1 for the Basic Info section but that's as far as I can get. Thanks for any help.
Sub DelSame()
Dim LastRow As Long, i As Long
LastRow = Cells(Rows.Count, 1).End(xlUp).Row
For i = LastRow To 3 Step -1
If Cells(i, 1).Value = Cells(i - 1, 1).Value Then Rows(i).Range("a1:e1").ClearContents
Next i
End Sub
You pretty much have it... use AND for multiple criteria:
Dim LastRow As Long, i As Long
LastRow = Cells(Rows.Count, 1).End(xlUp).Row
For i = LastRow To 3 Step -1
If Cells(i, 1).Value = Cells(i - 1, 1).Value AND Cells(i, 2).Value = Cells(i - 1, 2).Value AND Cells(i, 3).Value = Cells(i - 1, 3).Value Then
Rows(i).Delete
End If
Next i
Edit1:
So, the above suits the first grouping of columns... now for the hard part.
You can use AND sections for ALL columns, so you truly don't get any duplicates between rows (should actually fit the bill, as to not accidentally remove any items).
To get more involved, before you remove any rows, you will want to start storing values to more appropriately work with each grouping of columns, such that you work with rows a to b (untested code).
Dim a as Long, b as Long, i as Long, lr as Long
lr = cells(rows.count,1).end(xlup).row
For i = lr to 3 step -1
If cells(i,1).value = cells(i+1,1).value then
If a = 0 then
a = i + 1
End If
Else
If a > 0 AND b = 0 then
b = i + 1
End If
End If
If b > 0 AND a > 0 Then
'perform narrowed actions on range(cells(a,1),cells(b,1))
a = 0 'resets for next grouping
b = 0 'resets for next grouping
End If
Next i