Compare columns return maximum power query - excel

I have data from multiple suppliers which I wish to compare. The data shown in the image below has been previously transformed via a series of steps using power query. The final step was to pivot the Supplier column (in this example consisting of X,Y,Z) so that these new columns can be compared and the maximum value is returned.
How can I compare the values in columns X, Y and Z to do this? Importantly, X Y and Z arent necessarily the only suppliers. If I Add Say A as a new supplier to the original data, a new column A will be generated and I wish to include this in the comparison so that at column at the end outputs the highest value found for each row. So reading from the top down it would read in this example: 3,3,1,1,5,0.04,10 etc.
Thanks
Link to file https://onedrive.live.com/?authkey=%21AE_6NgN3hnS6MpA&id=8BA0D02D4869CBCA%21763&cid=8BA0D02D4869CBCA
M Code:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Material", type text}, {"Residual Solvents", type text}, {"RMQ", type text}, {"COA", type text}, {"Technical Data Sheet", type text}}),
//Replace Time and null with blank
#"Replaced Value" = Table.ReplaceValue(#"Changed Type","00:00:00","",Replacer.ReplaceText,{"Material", "RMQ", "Residual Solvents", "Technical Data Sheet", "COA"}),
#"Replaced Value1" = Table.ReplaceValue(#"Replaced Value",null,"",Replacer.ReplaceValue,{"Material", "RMQ", "Residual Solvents", "Technical Data Sheet", "COA"}),
//Trims all whitespace from user
#"Power Trim" = Table.TransformColumns(#"Replaced Value1",{{"Material", #"PowerTrim", type text}, {"Residual Solvents", #"PowerTrim", type text}, {"RMQ", #"PowerTrim", type text}, {"COA", #"PowerTrim", type text}, {"Technical Data Sheet",#"PowerTrim", type text}}),
//Unpivot to develop a single column of solvent/metals/date data
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Power Trim", {"Material", "Supplier"}, "Attribute", "Value"),
//split into rows by line feed
#"Split Column by Delimiter" = Table.ExpandListColumn(Table.TransformColumns(#"Unpivoted Other Columns",
{{"Value", Splitter.SplitTextByDelimiter("#(lf)", QuoteStyle.Csv), let itemType = (type nullable text) meta [Serialized.Text = true] in type {itemType}}}), "Value"),
#"Trimmed Text" = Table.TransformColumns(#"Split Column by Delimiter",{{"Value", Text.Trim, type text}}),
//filter out the blank rows
#"Filtered Rows" = Table.SelectRows(#"Trimmed Text", each ([Value] <> "" and [Value] <> "Not Provided")),
//Add custom column for separating the tables
#"Added Custom" = Table.AddColumn(#"Filtered Rows", "Custom", each try Date.FromText([Value]) otherwise
if [Value] = "Heavy Metals" or [Value] = "Residual Solvents" or [Value] = "Other" then [Value] else null),
#"Changed Type1" = Table.TransformColumnTypes(#"Added Custom",{{"Custom", type text}}),
#"Filled Down" = Table.FillDown(#"Changed Type1",{"Custom"}),
//Filter the value and custom columns to remove contaminant type from Value column and remove dates from Custom column
#"Filtered Rows1" = Table.SelectRows(#"Filled Down", each ([Custom] = "Heavy Metals" or [Custom] = "Residual Solvents") and ([Value] <> "Heavy Metals" and [Value] <> "Residual Solvents")),
//split substance from amount
#"Split Column by Delimiter1" = Table.SplitColumn(#"Filtered Rows1", "Value",
Splitter.SplitTextByEachDelimiter({" "}, QuoteStyle.Csv, true), {"Substance", "Amount"}),
//Filter for Solvents Table
#"Filtered Rows2" = Table.SelectRows(#"Split Column by Delimiter1", each ([Custom] = "Heavy Metals")),
#"Changed Type2" = Table.TransformColumnTypes(#"Filtered Rows2",{{"Amount", type number}}),
//Group by Material and Substance, then extract the Max contaminant and Source
#"Grouped Rows" = Table.Group(#"Changed Type2", {"Substance","Material", "Supplier"}, {
{"Amount", each List.Max([Amount]), type number},
{"Source", (t) => t[Attribute]{List.PositionOf(t[Amount],List.Max(t[Amount]))}, type text}
}),
#"Sorted Rows" = Table.Sort(#"Grouped Rows",{{"Substance", Order.Ascending}}),
//PIVOT to compare suppliers
#"Pivoted Column" = Table.Pivot(#"Sorted Rows", List.Distinct(#"Sorted Rows"[Supplier]), "Supplier", "Amount", List.Sum)
in
#"Pivoted Column"

Add an Index Column starting with zero (0).
Add a Custom Column:
=List.Max(
Record.ToList(
Table.SelectColumns(#"Added Index",
List.RemoveFirstN(
Table.ColumnNames(#"Pivoted Column"),3)){[Index]}))
Then remove the Index column
Algorithm
Generate a list of the relevant column names to sum.
We exclude the first three column names from this list
Note that we refer to the step PRIOR to adding the Index column for the list of column names. If we referred to the actual previous step where we added the Index column, we'd also have to remove the Last column name
Select the relevant columns
{[Index]} will return a record corresponding to the Index number.
Convert the record to a list and use List.Max

Related

Column rows to column headers in Power Query

I have a data set where I would like to have a table with Unique IDs in one Column A and from Column B the rows from a "Input" table above with a different rows as a column headers. In Column A are IDs (unique) and Column B has different rows that have to be in columns but matching values on the rest of the columns.. see on screenshot.
Third Column C is a just observational column that gives info what kind of data type should be there (it can be avoided in this case).
I though I was going to solve it "easily" with Pivot/Unpivot+Transponse method in Power Query but no way....I can get it in one row like in "Output" table..
The dummy data is in link below.
https://docs.google.com/spreadsheets/d/1qKeVj9nJF1usBk-OUZPJfpRqSQnOTCvr/edit?usp=sharing&ouid=101738555398870704584&rtpof=true&sd=true
Merge the value columns into a single column before the pivot, eg
let
Source = Excel.Workbook(File.Contents("C:\Users\david\Downloads\Test1.xlsx"), null, false),
Sheet1_sheet = Source{[Item="Sheet1",Kind="Sheet"]}[Data],
FilterNullAndWhitespace = each List.Select(_, each _ <> null and (not (_ is text) or Text.Trim(_) <> "")),
#"Added Custom" = Table.AddColumn(Sheet1_sheet, "IsEmptyRow", each try List.IsEmpty(FilterNullAndWhitespace(Record.FieldValues(_))) otherwise false),
#"Added Index" = Table.AddIndexColumn(#"Added Custom", "Index", -1),
#"Added Custom1" = Table.AddColumn(#"Added Index", "Section", each if [IsEmptyRow] then -1 else if try #"Added Index"[IsEmptyRow]{[Index]} otherwise true then [Index] else null),
#"Removed Blank Rows" = Table.SelectRows(#"Added Custom1", each not [IsEmptyRow]),
#"Filled Down" = Table.FillDown(#"Removed Blank Rows", {"Section"}),
#"Grouped Rows" = Table.Group(#"Filled Down", {"Section"}, {{"Rows", each _}}, GroupKind.Local),
#"Selected Group" = #"Grouped Rows"[Rows]{1},
#"Removed Columns" = Table.RemoveColumns(#"Selected Group", {"IsEmptyRow", "Index", "Section"}),
#"Promoted Headers" = Table.PromoteHeaders(#"Removed Columns", [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"ObjectID", Int64.Type}, {"Feld", type text}, {"Datentyp", type text}, {"boolValue", type text}, {"dateValue", type date}, {"intValue", Int64.Type}, {"stringValue", type text}, {"longStringValue", type text}, {"referencedObjectId", Int64.Type}}),
#"Inserted Merged Column" = Table.AddColumn(#"Changed Type", "Merged", each Text.Combine({[boolValue], Text.From([dateValue], "en-US"), Text.From([intValue], "en-US"), [stringValue], [longStringValue], Text.From([referencedObjectId], "en-US")}, ""), type text),
#"Removed Columns1" = Table.RemoveColumns(#"Inserted Merged Column",{"Datentyp", "boolValue", "dateValue", "intValue", "stringValue", "longStringValue", "referencedObjectId"}),
#"Pivoted Column" = Table.Pivot(#"Removed Columns1", List.Distinct(#"Removed Columns1"[Feld]), "Feld", "Merged")
in
#"Pivoted Column"

How do I remove duplicates WITHIN a row using Power Query?

I have data that looks like this:
Wire
Point1
Point2
Point3
Point4
Point5
Point6
A
WP1
WP1
WP2
WP2
B
WP3
WP4
WP3
WP4
C
WP5
WP5
WP6
WP7
WP6
WP7
(note the varying lengths of each row, and the duplicates)
I would like to have the end result be:
Wire
Point1
Point2
Point3
A
WP1
WP2
B
WP3
WP4
C
WP5
WP6
WP7
Duplicates removed, and blank spaces removed.
This would be VERY similar to the =UNIQUE() function, but that is not available in power query.
It's a lot easier to work with columns, so I'd recommend unpivoting the Point columns, removing duplicates, and then putting it into the shape you want.
Here's a full query you can past into your Advanced Editor to look at each step more closely:
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WclTSUQoPMEQijYAkBIHYsTrRSk5gtjGYNIGzYWpMwGqcwWxTJNIMTJqjsGNjAQ==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Wire = _t, Point1 = _t, Point2 = _t, Point3 = _t, Point4 = _t, Point5 = _t, Point6 = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Wire", type text}, {"Point1", type text}, {"Point2", type text}, {"Point3", type text}, {"Point4", type text}, {"Point5", type text}, {"Point6", type text}}),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Changed Type", {"Wire"}, "Attribute", "Value"),
#"Filtered Rows" = Table.SelectRows(#"Unpivoted Other Columns", each ([Value] <> "")),
#"Removed Duplicates" = Table.Distinct(#"Filtered Rows", {"Wire", "Value"}),
#"Grouped Rows" = Table.Group(#"Removed Duplicates", {"Wire"}, {{"Point", each Text.Combine([Value],","), type text}}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Grouped Rows", "Point", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv), {"Point1", "Point2", "Point3"})
in
#"Split Column by Delimiter"
Unpivot
Group by Wire
Aggregate into sorted List of Unique Points
Calculate Max number of items in all the Lists to use in the later Column Splitter
Extract the List of points into semicolon separated string
Split into new columns
M Code
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source, List.Transform(Table.ColumnNames(Source), each {_, Text.Type})),
//Unpivot
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Changed Type", {"Wire"}, "Attribute", "Value"),
#"Removed Columns" = Table.RemoveColumns(#"Unpivoted Other Columns",{"Attribute"}),
//Group by Wire
//Aggregate by sorted, unique list of Points for each Wire
#"Grouped Rows" = Table.Group(#"Removed Columns", {"Wire"}, {
{"Point", each List.Sort(List.Distinct([Value]))}}),
//Calculate the Max unique Points for any Wire (for subsequent splitting
maxPoints = List.Max(List.Transform(#"Grouped Rows"[Point], each List.Count(_))),
//Extract the List values into a semicolon separated list
#"Extracted Values" = Table.TransformColumns(#"Grouped Rows",
{"Point", each Text.Combine(List.Transform(_, Text.From), ";"), type text}),
//Then split into new columns using the semicolon delimiter
#"Split Column by Delimiter" = Table.SplitColumn(#"Extracted Values", "Point",
Splitter.SplitTextByDelimiter(";", QuoteStyle.Csv),maxPoints)
in
#"Split Column by Delimiter"

Arranging values in a specific order using power query

I wish to clean up a table I have been working on removing complex formulas and opting to sort my data using PQE.
Below is a made up example of the data I am working with.
I wish to have the output column alternate between material and sub values akin to headers and sub headers in a book.
M Code so far:
let
Source = Excel.CurrentWorkbook(){[Name="Table3"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Material ", type text}, {"Sub ", type text}, {"CAS", type text}}),
#"Trimmed Text" = Table.TransformColumns(#"Changed Type",{{"Material ", Text.Trim, type text}, {"Sub ", Text.Trim, type text}}),
#"Cleaned Text" = Table.TransformColumns(#"Trimmed Text",{{"Sub ", Text.Clean, type text}}),
#"Replaced Value" = Table.ReplaceValue(#"Cleaned Text","-","",Replacer.ReplaceText,{"Sub "}),
#"Merged Columns" = Table.CombineColumns(#"Replaced Value",{"Material ", "Sub "},Combiner.CombineTextByDelimiter(" ", QuoteStyle.None),"Merged"),
#"Split Column by Delimiter" = Table.ExpandListColumn(Table.TransformColumns(#"Merged Columns", {{"Merged", Splitter.SplitTextByDelimiter(" ", QuoteStyle.Csv), let itemType = (type nullable text) meta [Serialized.Text = true] in type {itemType}}}), "Merged"),
#"Changed Type1" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"Merged", type text}})
in
#"Changed Type1"
The problem with this code is that it doesn't properly list the materials when combined and I also don't know how to get the CAS numbers to be correctly sorted.
If anyone has any thoughts how to achieve the desired output it would be appreciated.
You can get your output from your input using the same technique I showed you in your last, similar question.
create two lists of the columns; then List.Zip to combine them.
The trick is that for List 1, (eg output column 1), you may need to add the contents of the Material column at the top of the Sub column list (or replace the - if that's all there is; and, if that is the case, add a - at the start of the CAS list; so things will line up at the end.
M Code
let
Source = Excel.CurrentWorkbook(){[Name="Table13"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Material", type text}, {"Sub", type text}, {"CAS", type text}}),
//combine the columns
#"Added Custom" = Table.AddColumn(#"Changed Type", "Custom", each
let
L1 = if [Sub] = "-" then {[Material]}
else List.Combine({{[Material]},Text.Split([Sub],"#(lf)")}),
L2 = if [Sub] = "-" then {[CAS]}
else List.Combine({{"-"},Text.Split([CAS],"#(lf)")})
in
List.Zip({L1,L2})),
#"Removed Columns1" = Table.RemoveColumns(#"Added Custom",{"Material", "Sub", "CAS"}),
//split the combined columns
#"Expanded Custom" = Table.ExpandListColumn(#"Removed Columns1", "Custom"),
#"Extracted Values" = Table.TransformColumns(#"Expanded Custom",
{"Custom", each Text.Combine(List.Transform(_, Text.From), ";"), type text}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Extracted Values",
"Custom", Splitter.SplitTextByDelimiter(";", QuoteStyle.Csv), {"Material", "CAS"})
in
#"Split Column by Delimiter"

Is there a Power Query function that seperates a comma seperate list into value columns?

Let's say that I have a column that has a comma seperated list of values in each row
Row ID | Symptoms
1 | Vomiting,Diarrhoea
2 | Diarrhoea
3 | Cough
Would it be possible to transform this into the following in Power Query?:
Row ID | Symptoms - Cough | Symptoms - Diarrhoea | Symptom - Vomiting
1 | No | Yes | Yes
2 | No | Yes | No
3 | Yes| No | No
Unfortunately could not a find a specific question and answers that tackles my challenge on Stack Overflow and tried looking up multiple videos on YouTube with limited success.
I suggest:
Split the symptom column by comma
Select the Row ID column and unpivot other columns
Add a custom column consisting of the word "Yes"
Pivot on the Value column (containing the symptoms) and Don't Aggregate
Replace the null with "No"
The code as written should adjust as you add/remove rows or symptoms from your orginal data table.
If you need to prefix all of the symptom columns with Symptoms - , that can be done with a few more lines of code.
M Code
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Row ID", Int64.Type}, {"Symptoms", type text}}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Changed Type", "Symptoms", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv)),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Split Column by Delimiter", {"Row ID"}, "Attribute", "Value"),
#"Removed Columns" = Table.RemoveColumns(#"Unpivoted Other Columns",{"Attribute"}),
#"Trimmed Text" = Table.TransformColumns(#"Removed Columns",{{"Value", Text.Trim, type text}}),
#"Added Custom" = Table.AddColumn(#"Trimmed Text", "Custom", each "Yes"),
#"Pivoted Column" = Table.Pivot(#"Added Custom", List.Distinct(#"Added Custom"[Value]), "Value", "Custom"),
#"Replaced Value" = Table.ReplaceValue(#"Pivoted Column",null,"No",Replacer.ReplaceValue,Table.ColumnNames(#"Pivoted Column"))
in
#"Replaced Value"
EDIT: Provoked by #AlexisOlson comment, I added a few lines of code to both alphabetize the symptom list, and also to prefix each symptom name with the string Symptoms - .
I also added numerous comments to the code, which you will also be able to see in the Applied Steps window.
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Row ID", Int64.Type}, {"Symptoms", type text}}),
//Split the different symptoms into separate columns
#"Split Column by Delimiter" = Table.SplitColumn(#"Changed Type", "Symptoms", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv)),
//Unpivot other than "Row ID" so as to have all the symptoms in a single column
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Split Column by Delimiter", {"Row ID"}, "Attribute", "Value"),
//Remove the Attribute column
#"Removed Columns" = Table.RemoveColumns(#"Unpivoted Other Columns",{"Attribute"}),
//Trim the symptoms column to get rid of any leading spaces that might be there
#"Trimmed Text" = Table.TransformColumns(#"Removed Columns",{{"Value", Text.Trim, type text}}),
//Sort symptoms alphabetically
#"Sorted Rows" = Table.Sort(#"Trimmed Text",{{"Value", Order.Ascending}}),
//Create a new column with the "Yes"'s
#"Added Custom" = Table.AddColumn(#"Sorted Rows", "Custom", each "Yes"),
//Pivot on the symptoms column to create new table with each symptom as a column header
#"Pivoted Column" = Table.Pivot(#"Added Custom", List.Distinct(#"Added Custom"[Value]), "Value", "Custom"),
#"Changed Type1" = Table.TransformColumnTypes(#"Pivoted Column",{{"Cough", type text}, {"Diarrhoea", type text}, {"Headache", type text}, {"Myalgia", type text}, {"Vomiting", type text}}),
//Replace the resultant nulls with "No"'s
#"Replaced Value" = Table.ReplaceValue(#"Changed Type1",null,"No",Replacer.ReplaceValue,Table.ColumnNames(#"Changed Type1")),
firstColName = Table.ColumnNames(#"Replaced Value"){0},
//Prefix each column header with "Symptoms - " except first
rename = Table.TransformColumnNames(#"Replaced Value", each if _ <> firstColName then "Symptoms - " & _ else _)
in
rename

How to split excel columns into multiple rows

I get exports from an event registration form, where each order(row) contains order date, purchase price, and info for between 1-4 registrants. Each registrant has columns for first name, last name, and date of birth. What I need is to split up these rows such that each individual person is on their own row, with first, last, and DOB on the row, ideally along with the other order information duplicated too.
This may be confusing so I have mockup date for the form I get exports in, and the form I'd like to convert them to.
What I get:
https://drive.google.com/file/d/1xt64Re2CTlbQnHuRy1dQaFeNV5OmETfW/view?usp=sharing
What I want:
https://drive.google.com/file/d/156WmiQ4Tx4JGB5FYLTqHBuVmJqi20pRZ/view?usp=sharing
I found this tutorial which seem very close to what I want using Excel Power Query, but it is describing a method for when the multiple items are in one column separated by comma. My situation is a bit different and I can't make it work.
Any ideas?
Below PowerQuery works on sample data set you had, when loaded into range Table1
Written dynamically for any number of First/Last/DOB sets of columns; You had 4 this would handle any number of them
Paste into Home ... Advanced Editor...
a bit long winded to separate the steps and make it visible
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Order Date", type date}}),
// Get names of all First, Last and DOB fields
ColumnsToCombine1 = List.Select(Table.ColumnNames(#"Changed Type"), each Text.Contains(_, "(First)")),
ColumnsToCombine2 = List.Select(Table.ColumnNames(#"Changed Type"), each Text.Contains(_, "(Last)")),
ColumnsToCombine3 = List.Select(Table.ColumnNames(#"Changed Type"), each Text.Contains(_, "DOB")),
//Convert to Text, so we can merge columns; for date fields, convert to date first
#"Changed Type1" = Table.TransformColumnTypes(#"Changed Type", (List.Transform(ColumnsToCombine3 ,each {_,type date}))),
#"Changed Type2" = Table.TransformColumnTypes (#"Changed Type1",List.Transform(Table.ColumnNames(#"Changed Type1"), each {_, type text})),
// convert all First, Last and DOB fields into three comma separated columns
#"Combined1" = Table.AddColumn(#"Changed Type2" , "First", each Text.Combine(Record.FieldValues(Record.SelectFields(_,ColumnsToCombine1)),", ")),
#"Combined2" = Table.AddColumn(#"Combined1", "Last", each Text.Combine(Record.FieldValues(Record.SelectFields(_,ColumnsToCombine2)),", ")),
#"Combined3" = Table.AddColumn(#"Combined2", "DOB", each Text.Combine(Record.FieldValues(Record.SelectFields(_,ColumnsToCombine3)),", ")),
#"Removed Other Columns" = Table.SelectColumns(Combined3,{"ID", "Order Date", "Quantity", "Payment", "First", "Donate", "Last", "DOB"}),
//expand all comma separated columns into their own rows
TableTransform = Table.Combine(List.Transform(List.Transform(Table.ToRecords(#"Removed Other Columns" ), (x) => List.Transform(Record.ToList(x), each Text.Split(_,","))), each Table.FromColumns(_, Table.ColumnNames(#"Removed Other Columns" )))),
#"Reordered Columns" = Table.ReorderColumns(TableTransform,{"ID", "Order Date", "Quantity", "First", "Last", "DOB", "Payment", "Donate"}),
#"Filled Down" = Table.FillDown(#"Reordered Columns",{"ID", "Order Date", "Quantity", "Payment", "Donate"}),
#"Changed Type3" = Table.TransformColumnTypes(#"Filled Down",{{"DOB", type date}, {"Order Date", type date}, {"Payment", type number}, {"Donate", type number}, {"Quantity", type number}, {"First", type text}, {"Last", type text}})
in #"Changed Type3"
Alternate version, hard coded to 4 tables, based on Ron's
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
colNames = Table.ColumnNames(Source),
Table1 = Table.Skip(Table.DemoteHeaders(Table.SelectColumns(Source,List.RemoveMatchingItems(colNames , List.Select(colNames, each Text.Contains(_, "-2") or Text.Contains(_, "-3") or Text.Contains(_, "-4"))))),1),
Table2 = Table.Skip(Table.DemoteHeaders(Table.SelectColumns(Source,List.RemoveMatchingItems(colNames , List.Select(colNames, each Text.Contains(_, "-1") or Text.Contains(_, "-3") or Text.Contains(_, "-4"))))),1),
Table3 = Table.Skip(Table.DemoteHeaders(Table.SelectColumns(Source,List.RemoveMatchingItems(colNames , List.Select(colNames, each Text.Contains(_, "-1") or Text.Contains(_, "-2") or Text.Contains(_, "-4"))))),1),
Table4 = Table.Skip(Table.DemoteHeaders(Table.SelectColumns(Source,List.RemoveMatchingItems(colNames , List.Select(colNames, each Text.Contains(_, "-1") or Text.Contains(_, "-2") or Text.Contains(_, "-3"))))),1),
combined = Table1 & Table2 & Table3 & Table4,
#"Filtered Rows" = Table.SelectRows(combined, each ([Column4] <> null)),
#"Sorted Rows" = Table.Sort(#"Filtered Rows",{{"Column1", Order.Ascending}}),
#"Renamed Columns" = Table.RenameColumns(#"Sorted Rows",{{"Column2", "Order Date"}, {"Column3", "Quantity"}, {"Column4", "First"}, {"Column5", "Last"}, {"Column6", "DOB"}, {"Column7", "Payment"}, {"Column8", "Donate"}, {"Column1", "ID"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"ID", type number}, {"Quantity", type number}, {"Payment", type number}, {"Donate", type number}, {"Order Date", type date}, {"DOB", type date}, {"First", type text}, {"Last", type text}})
in #"Changed Type"
Here is another PQ method to accomplish this.
I post it as a PQ beginner, and I have no idea whether this method or the first method posted is more efficient and/or scalable
It is still a work in progress, as I have fixed column names in some of the steps, and I want to make that more generalized eventually. But this problem of repeating groups of columns comes up from time to time.
Source Data
This method:
Reorder the columns to put the "common columns" together, which, in this case, would be the first three and the last two.
Demote the header row and transpose the table
Process the first column so as to create a proper list of what will be eventual column headers.
Split this table into multiple tables where
the first table represents the common columns
the other tables represent each repeating group of columns
Combine the first table with each of the other tables
Combine the resulting tables
Transpose back to the original column/row orientation, and promote the first row to headers.
Clean up by removing nulls and sorting
Reorder the columns to put the first three and last two back where they were initially.
Note: You'll need to change Table1 in the code to the actual table name, if it is different
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
colNames = Table.ColumnNames(Source),
commonTableColumnCount = 5,
colOrder = List.Combine({List.FirstN(colNames,3),List.LastN(colNames,2),List.Range(colNames,3,List.Count(colNames)-commonTableColumnCount)}),
#"Reordered Columns" = Table.ReorderColumns(Source,colOrder),
//Demote Headers and transpose
transpose1 = Table.Transpose(Table.DemoteHeaders(#"Reordered Columns")),
//Split up Column 1 to create multiple matched names
split = Table.SplitColumn(transpose1, "Column1", Splitter.SplitTextByDelimiter("-", QuoteStyle.Csv), {"Column1", "Column1.1"}),
#"Split Column by Delimiter" = Table.SplitColumn(split, "Column1.1", Splitter.SplitTextByEachDelimiter({" "}, QuoteStyle.Csv, true), {"Column1.1.1", "Column1.1.2"}),
#"Removed Columns" = Table.RemoveColumns(#"Split Column by Delimiter",{"Column1.1.1"}),
#"Merged Columns" = Table.CombineColumns(#"Removed Columns",{"Column1", "Column1.1.2"},Combiner.CombineTextByDelimiter(" ", QuoteStyle.None),"Column1"),
commonTable = Table.Split(#"Merged Columns",commonTableColumnCount){0},
record = Table.ToRecords(commonTable),
//repeated tables
//rem CommonTables Rows and split
data = Table.Split(Table.RemoveRows(#"Merged Columns",0,commonTableColumnCount),3),
//insert common rows into each table
data1 = Table.InsertRows(data{0},0,record),
//get final order of column names
colNames2 = Table.Column(data1,"Column1"),
finalOrder = List.Combine({List.FirstN(colNames2,3),List.LastN(colNames2,List.Count(colNames2)-commonTableColumnCount), List.Range(colNames2,3,2)}),
custom1 = List.Transform(data, each Table.InsertRows(_,0,record)),
//Transpose and combine each Table
custom2 = List.Transform(custom1, each(Table.PromoteHeaders(Table.Transpose(_),[PromoteAllScalars=true]))),
#"Converted to Table" = Table.FromList(custom2, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandTableColumn(#"Converted to Table","Column1",colNames2),
//need to do this manually
#"Changed Type" = Table.TransformColumnTypes(#"Expanded Column1",{{"ID ", Int64.Type}, {"Order Date ", type date}, {"Quantity ", Int64.Type}, {"Payment ", Currency.Type}, {"Donate ", Currency.Type}, {"Name (First)", type text}, {"Name (Last)", type text}, {"DOB ", type date}}),
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each ([#"Name (Last)"] <> null)),
#"Sorted Rows" = Table.Sort(#"Filtered Rows",{"ID ", Order.Ascending}),
#"Reordered Columns1" = Table.ReorderColumns(#"Sorted Rows", finalOrder)
in
#"Reordered Columns1"
Reordered Table

Resources