Dynamically pivot n rows with new row for multiple values - excel

I wish to transform the input to the output shown below:
However for Columns D and K there are multiple values for D and K which causes and error:
M Code to replicate the above:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", Int64.Type}}),
#"Pivoted Column" = Table.Pivot(#"Changed Type", List.Distinct(#"Changed Type"[Column1]), "Column1", "Column2")
in
#"Pivoted Column"
I have also attempted to add an index column so that each data point is unique but this leads to further problems.
So far I have actually grouped the data so that everything is contained in a single cell:
Current M Code:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type1" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", type text}}),
#"Grouped Rows" = Table.Group(#"Changed Type1", {"Column1"}, {{"Column2", each Text.Combine([Column2],"#(lf)"), type text}}),
#"Pivoted Column" = Table.Pivot(#"Grouped Rows", List.Distinct(#"Grouped Rows"[Column1]), "Column1", "Column2")
in
#"Pivoted Column"
However this requires the data to be split which is fine but I want to do this dynamically for n Columns i..e A-B or A-AZ and this kind of shifts the problems so that I have to dynamically Split n columns.
Input data:
Column1 Column2
A 1
B 1
C 2
D 3
D 3
D 1
E 2
F 1
G 2
H 1
I 2
J 3
K 1
K 2
L 1
M 2
N 3

A not unusual problem solved by a custom function. Check the link in the credits for an explanation:
Custom Function
//credit: Cam Wallace https://www.dingbatdata.com/2018/03/08/non-aggregate-pivot-with-multiple-rows-in-powerquery/
//Rename: fnPivotAll
(Source as table,
ColToPivot as text,
ColForValues as text)=>
let
PivotColNames = List.Buffer(List.Distinct(Table.Column(Source,ColToPivot))),
#"Pivoted Column" = Table.Pivot(Source, PivotColNames, ColToPivot, ColForValues, each _),
TableFromRecordOfLists = (rec as record, fieldnames as list) =>
let
PartialRecord = Record.SelectFields(rec,fieldnames),
RecordToList = Record.ToList(PartialRecord),
Table = Table.FromColumns(RecordToList,fieldnames)
in
Table,
#"Added Custom" = Table.AddColumn(#"Pivoted Column", "Values", each TableFromRecordOfLists(_,PivotColNames)),
#"Removed Other Columns" = Table.RemoveColumns(#"Added Custom",PivotColNames),
#"Expanded Values" = Table.ExpandTableColumn(#"Removed Other Columns", "Values", PivotColNames)
in
#"Expanded Values"
Main Code
let
//change next line to reflect your actual data source
Source = Excel.CurrentWorkbook(){[Name="Table24"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", Int64.Type}}),
pivot = fnPivotAll(#"Changed Type","Column1","Column2")
in
pivot
Results from your data

This seems to work fine (a) group and add index inside group (b) expand (c) pivot
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Grouped Rows" = Table.Group(Source, {"Column1"}, {{"data", each Table.AddIndexColumn(_, "Index", 0, 1, Int64.Type), type table}}),
#"Expanded data" = Table.ExpandTableColumn(#"Grouped Rows", "data", {"Column2", "Index"}, {"Column2", "Index"}),
#"Pivoted Column" = Table.Pivot(#"Expanded data", List.Distinct(#"Expanded data"[Column1]), "Column1", "Column2"),
#"Removed Columns" = Table.RemoveColumns(#"Pivoted Column",{"Index"})
in #"Removed Columns"

Related

Excel Unpivot Multiple Columns to a Single Column

What I want to achieve:
As the title says, is there any way to convert my table structure? I've tried using Power Query but it didn't work. Any kind of help is greatly appreciated. Thanks!
So, I was trying to pivot type and type value, but it seems impossible if I maintain the current table structure since it would cause duplicates when I wanted to aggregate on type.
Should I remake the table structure or there is any way to get around this problem?
Thanks in advance!
In Power Query, the following is adaptable to any number of type/value column pairs.
Unpivot all except the ID column
Add a custom column to define if the unpivoted value is a Type or a Type Value
Add an Index column and then do an integer/divide by 2 so things will sort in the desired order
Pivot with no aggregation, using a custom function as the "built-in" function will error with multiple items.
Custom function to Pivot with No aggregation
Rename as noted in comments
//credit: Cam Wallace https://www.dingbatdata.com/2018/03/08/non-aggregate-pivot-with-multiple-rows-in-powerquery/
//Rename: fnPivotAll
(Source as table,
ColToPivot as text,
ColForValues as text)=>
let
PivotColNames = List.Buffer(List.Distinct(Table.Column(Source,ColToPivot))),
#"Pivoted Column" = Table.Pivot(Source, PivotColNames, ColToPivot, ColForValues, each _),
TableFromRecordOfLists = (rec as record, fieldnames as list) =>
let
PartialRecord = Record.SelectFields(rec,fieldnames),
RecordToList = Record.ToList(PartialRecord),
Table = Table.FromColumns(RecordToList,fieldnames)
in
Table,
#"Added Custom" = Table.AddColumn(#"Pivoted Column", "Values", each TableFromRecordOfLists(_,PivotColNames)),
#"Removed Other Columns" = Table.RemoveColumns(#"Added Custom",PivotColNames),
#"Expanded Values" = Table.ExpandTableColumn(#"Removed Other Columns", "Values", PivotColNames)
in
#"Expanded Values"
Regular Query
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{
{"ID", type text}, {"Type 1", type text}, {"Type 1 Value", Int64.Type}, {"Type 2", type text}, {"Type 2 Value", Int64.Type}}),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Changed Type", {"ID"}, "Attribute", "Value"),
#"Added Custom" = Table.AddColumn(#"Unpivoted Other Columns", "Custom",
each if Text.EndsWith([Attribute],"Value") then "Value" else "Type"),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Attribute"}),
#"Added Index" = Table.AddIndexColumn(#"Removed Columns", "Index", 0, 1, Int64.Type),
#"Inserted Integer-Division" = Table.AddColumn(#"Added Index", "Integer-Division", each Number.IntegerDivide([Index], 2), Int64.Type),
#"Removed Columns1" = Table.RemoveColumns(#"Inserted Integer-Division",{"Index"}),
Pivot = fnPivotAll(#"Removed Columns1","Custom","Value"),
#"Removed Columns2" = Table.RemoveColumns(Pivot,{"Integer-Division"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Removed Columns2",{{"Type", type text}, {"Value", Int64.Type}})
in
#"Changed Type1"
More generically to stack vertically in powerquery while keeping certain columns
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
base_columns=1, groupsof=2, //stack them
Combo = List.Transform(List.Split(List.Skip(Table.ColumnNames(Source),base_columns),groupsof), each List.FirstN(Table.ColumnNames(Source),base_columns) & _),
#"Added Custom" =List.Accumulate(Combo, #table({"Column1"}, {}),(state,current)=> state & Table.Skip(Table.DemoteHeaders(Table.SelectColumns(Source, current)),1)),
#"Rename"=Table.RenameColumns(#"Added Custom",List.Zip({Table.ColumnNames(#"Added Custom"),List.FirstN(Table.ColumnNames(Source),base_columns+groupsof)}))
in #"Rename"
What seems to be fastest method of those I've tested
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
leading=1, groupsof=2,
#"Added Custom" = Table.AddColumn(Source, "Custom", each List.Split( List.RemoveFirstN(Record.ToList( _),leading), groupsof) ),
#"Added Custom0" = Table.AddColumn(#"Added Custom", "Custom0", each Text.Combine(List.FirstN(Record.ToList(_),leading),"|")),
#"Removed Other Columns" = Table.SelectColumns(#"Added Custom0",{"Custom0", "Custom"}),
#"Expanded Custom" = Table.ExpandListColumn( #"Removed Other Columns", "Custom"),
#"Extracted Values" = Table.TransformColumns(#"Expanded Custom", {"Custom", each Text.Combine(List.Transform(_, Text.From), "|"), type text}),
#"Merged Columns" = Table.CombineColumns(#"Extracted Values",{"Custom0", "Custom"},Combiner.CombineTextByDelimiter("|", QuoteStyle.None),"Custom"),
#"Split Column by Delimiter" = Table.SplitColumn(#"Merged Columns", "Custom", Splitter.SplitTextByDelimiter("|", QuoteStyle.Csv), List.FirstN(Table.ColumnNames(Source),leading+groupsof))
in #"Split Column by Delimiter"
There is probably a better way, but if you first concat the 4 columns with specific unique delimiter to split on later in a custom column, you have a work-around in PQ:
let
Source = Excel.CurrentWorkbook(){[Name="Table2"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"ID", type text}, {"Type1", type text}, {"Type1 Val", Int64.Type}, {"Type2", type text}, {"Type2 Val", Int64.Type}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "Custom1", each [Type1]&"|"&Number.ToText([Type1 Val])&"$"&[Type2]&"|"&Number.ToText([Type2 Val])),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Type1", "Type1 Val", "Type2", "Type2 Val"}),
#"Split Column by Delimiter" = Table.ExpandListColumn(Table.TransformColumns(#"Removed Columns", {{"Custom1", Splitter.SplitTextByDelimiter("$", QuoteStyle.Csv), let itemType = (type nullable text) meta [Serialized.Text = true] in type {itemType}}}), "Custom1"),
#"Changed Type1" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"Custom1", type text}}),
#"Split Column by Delimiter1" = Table.SplitColumn(#"Changed Type1", "Custom1", Splitter.SplitTextByEachDelimiter({"|"}, QuoteStyle.Csv, false), {"Custom1.1", "Custom1.2"}),
#"Changed Type2" = Table.TransformColumnTypes(#"Split Column by Delimiter1",{{"Custom1.1", type text}, {"Custom1.2", Int64.Type}})
in
#"Changed Type2"
Just in case you tagged 'Excel-Formula' and you have access to ms365:
Formula in H1:
=REDUCE({"ID","Type","Val"},ROW(A2:A5),LAMBDA(X,Y,VSTACK(X,INDEX(A:E,Y,{1,2,3}),INDEX(A:E,Y,{1,4,5}))))
Or formula:
=SORT(VSTACK(A2:C5,HSTACK(A2:A5,D2:E5)))

Column rows to column headers in Power Query

I have a data set where I would like to have a table with Unique IDs in one Column A and from Column B the rows from a "Input" table above with a different rows as a column headers. In Column A are IDs (unique) and Column B has different rows that have to be in columns but matching values on the rest of the columns.. see on screenshot.
Third Column C is a just observational column that gives info what kind of data type should be there (it can be avoided in this case).
I though I was going to solve it "easily" with Pivot/Unpivot+Transponse method in Power Query but no way....I can get it in one row like in "Output" table..
The dummy data is in link below.
https://docs.google.com/spreadsheets/d/1qKeVj9nJF1usBk-OUZPJfpRqSQnOTCvr/edit?usp=sharing&ouid=101738555398870704584&rtpof=true&sd=true
Merge the value columns into a single column before the pivot, eg
let
Source = Excel.Workbook(File.Contents("C:\Users\david\Downloads\Test1.xlsx"), null, false),
Sheet1_sheet = Source{[Item="Sheet1",Kind="Sheet"]}[Data],
FilterNullAndWhitespace = each List.Select(_, each _ <> null and (not (_ is text) or Text.Trim(_) <> "")),
#"Added Custom" = Table.AddColumn(Sheet1_sheet, "IsEmptyRow", each try List.IsEmpty(FilterNullAndWhitespace(Record.FieldValues(_))) otherwise false),
#"Added Index" = Table.AddIndexColumn(#"Added Custom", "Index", -1),
#"Added Custom1" = Table.AddColumn(#"Added Index", "Section", each if [IsEmptyRow] then -1 else if try #"Added Index"[IsEmptyRow]{[Index]} otherwise true then [Index] else null),
#"Removed Blank Rows" = Table.SelectRows(#"Added Custom1", each not [IsEmptyRow]),
#"Filled Down" = Table.FillDown(#"Removed Blank Rows", {"Section"}),
#"Grouped Rows" = Table.Group(#"Filled Down", {"Section"}, {{"Rows", each _}}, GroupKind.Local),
#"Selected Group" = #"Grouped Rows"[Rows]{1},
#"Removed Columns" = Table.RemoveColumns(#"Selected Group", {"IsEmptyRow", "Index", "Section"}),
#"Promoted Headers" = Table.PromoteHeaders(#"Removed Columns", [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"ObjectID", Int64.Type}, {"Feld", type text}, {"Datentyp", type text}, {"boolValue", type text}, {"dateValue", type date}, {"intValue", Int64.Type}, {"stringValue", type text}, {"longStringValue", type text}, {"referencedObjectId", Int64.Type}}),
#"Inserted Merged Column" = Table.AddColumn(#"Changed Type", "Merged", each Text.Combine({[boolValue], Text.From([dateValue], "en-US"), Text.From([intValue], "en-US"), [stringValue], [longStringValue], Text.From([referencedObjectId], "en-US")}, ""), type text),
#"Removed Columns1" = Table.RemoveColumns(#"Inserted Merged Column",{"Datentyp", "boolValue", "dateValue", "intValue", "stringValue", "longStringValue", "referencedObjectId"}),
#"Pivoted Column" = Table.Pivot(#"Removed Columns1", List.Distinct(#"Removed Columns1"[Feld]), "Feld", "Merged")
in
#"Pivoted Column"

Merging and formating Excel Worksheets with power query

I'm trying to format an excel file using Power Query in order to be able to pivot it, but I haven't been able to do it right.
I'm merging many worksheets from different workbooks into one. Every sheet has the data of the specific workbook in the first 3 rows (column 1= Title; and Column 2= Value), and then I have the table well formatted starting in row 5 (with headers and all, but different amount of rows each)
How can I transform the data in the first 3 rows of every sheet into columns, so I can get a Pivotable table?
Here an example of what I get when merging 2 files.
Based on the picture I assume you would like to transform such kind of file
to something like that
I used the following M-Code for that
let
Source = Table.FromColumns({Lines.FromBinary(File.Contents("D:\tmp\Files\file1.txt"), null, null, 1252)}),
#"Kept First Rows" = Table.FirstN(Source,3),
#"Split Column by Delimiter" = Table.SplitColumn(#"Kept First Rows", "Column1", Splitter.SplitTextByDelimiter("#(tab)", QuoteStyle.Csv), {"Col1", "Col2"}),
step = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"Col1", type text}, {"Col2", type text}}),
#"Transposed Table" = Table.Transpose(step),
#"Promoted Headers" = Table.PromoteHeaders(#"Transposed Table", [PromoteAllScalars=true]),
Hdr = Table.TransformColumnTypes(#"Promoted Headers",{{"Nummer", type text}, {"Id", type text}}),
Custom1 = Source,
#"Removed Top Rows" = Table.Skip(Custom1,4),
#"Split Column by Delimiter1" = Table.SplitColumn(#"Removed Top Rows", "Column1", Splitter.SplitTextByDelimiter("#(tab)", QuoteStyle.Csv), {"Column1.1", "Column1.2", "Column1.3", "Column1.4"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Split Column by Delimiter1",{{"Column1.1", type text}, {"Column1.2", type text}, {"Column1.3", type text}, {"Column1.4", type text}}),
#"Promoted Headers1" = Table.PromoteHeaders(#"Changed Type1", [PromoteAllScalars=true]),
#"Changed Type2" = Table.TransformColumnTypes(#"Promoted Headers1",{{"F1", Int64.Type}, {"F2", Int64.Type}, {"F3", Int64.Type}, {"F4", Int64.Type}}),
#"Added Custom" = Table.AddColumn(#"Changed Type2", "Nummer", each Hdr[Nummer]),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "ID", each Hdr[Id]),
#"Added Custom2" = Table.AddColumn(#"Added Custom1", "Export", each Hdr[Export]),
#"Expanded Export" = Table.ExpandListColumn(#"Added Custom2", "Export"),
#"Expanded ID" = Table.ExpandListColumn(#"Expanded Export", "ID"),
#"Expanded Nummer" = Table.ExpandListColumn(#"Expanded ID", "Nummer"),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded Nummer",{{"Export", type date}})
in
#"Changed Type"

POWERBI, DAX : concatenate strings, split, and keep substrings only once

I try to do the following:
I have a column with strings, each can have several substrings separated by a delimiter ":"
I need to Concatenate the column strings (I do a Filter here to keep only interresting lines)
Then split according to the delimiter ":"
keep the substrings only once, if they are repeated.
Example:
ColumnHeader
AA:BB:CC
BB:DD
DD:AA:EE
EE:AA:DD:BB
BB:EE
...
Expected result would be a unique string:
"AA:BB:CC:DD:EE"
How would you do this in DAX to fill a new column ?
I expected to find for/while loops in DAX like in Python ... but failed.
I Tried this:
List =
VAR SIn = ""
VAR SOut = ""
VAR Cursor = 0
VAR SList =
CONCATENATEX(
FILTER(ATable, ATable[Name] = CTable[Name]),
[ColumnHeader],
":")
VAR pos1 = FIND(":", SList, Cursor, len(SList))
VAR pos2 = FIND(":", SList, pos1, len(SList))
VAR elem = TRIM(MID(SList, pos1+1, pos2-pos1))
// following is not good but is what I would like to do:
VAR SOut = CONCATENATE(SOut, elem)
VAR SList = MID(SList, pos2, len(SList)-pos2)
VAR Cursor = pos2
// I need to loop ... but how ? ... as no for/while loops are possibles ?
Thanks for your help.
=====================================
I manage to tackle this thanks to the answers below.
I will still give a bigger data set for a better understanding of the global problem:
I have 2 tables:
TABLE_BY_ELEMENT
KEY GROUP LIST KEY_DATA
1 G1 AA:BB:FF 11
2 G1 CC:AA 22
3 G1 FF:DD:AA 33
4 G1 CC:DD:AA 44
5 G2 CC:FF:GG 55
6 G2 BB:AA 66
TABLE_BY_GROUP
GROUP GROUP_DATA
G1 1111
G2 2222
And I want to view the data like this:
RESULT_BY_GROUP
GROUP GROUP_DATA NewList
G1 111 AA:BB:FF:CC:DD
G2 222 CC:FF:GG:BB:AA
and also:
RESULT_ELEMENT
KEY LIST KEY_DATA
1 AA:BB:FF 11
2 CC:AA 22
3 FF:DD:AA 33
4 CC:DD:AA 44
5 CC:FF:GG 55
6 BB:AA 66
I hope is is easier to understand with this.
This isn't something DAX is suited for well. If you need to use DAX to make it into a dynamic measure, then you'll probably need to reshape your data to be more usable. For example,
ID ColumnHeader
1 AA
1 BB
1 CC
2 BB
2 DD
3 DD
3 AA
3 EE
...
You can do this split in the query editor using the Split Column > By Delimiter tool and choosing to split on the colon and expand into rows.
Once it's in this more usable format, you can work with it in DAX like this:
List = CONCATENATEX( VALUES('Table'[ColumnHeader]), 'Table'[ColumnHeader], ":" )
Borrowing logic from here, it's possible to do this purely in DAX, but I don't recommend this route.
List =
VAR LongString =
CONCATENATEX ( VALUES ( 'Table1'[ColumnHeader] ), Table1[ColumnHeader], ":" )
VAR StringToPath =
SUBSTITUTE ( LongString, ":", "|" )
VAR PathToTable =
ADDCOLUMNS (
GENERATESERIES ( 1, LEN ( StringToPath ) ),
"Item", PATHITEM ( StringToPath, [Value] )
)
VAR GroupItems =
FILTER (
SUMMARIZE ( PathToTable, [Item] ),
NOT ISBLANK ( [Item] )
)
RETURN
CONCATENATEX ( GroupItems, [Item], ":" )
Let your table looks like below-
Now try this below Advance Editor code in the Power Query Editor-
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WcnS0cnKycnZWitWJVgKyXFzALBcXK6CMqyuY4+oK4gCFnJxgykAysQA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [ColumnHeader = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"ColumnHeader", type text}}),
//--NEW STEPS STARTS FROM HERE
#"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 1, 1, Int64.Type),
#"Reordered Columns" = Table.ReorderColumns(#"Added Index",{"Index", "ColumnHeader"}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Reordered Columns", "ColumnHeader", Splitter.SplitTextByDelimiter(":", QuoteStyle.Csv), {"ColumnHeader.1", "ColumnHeader.2", "ColumnHeader.3", "ColumnHeader.4"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"ColumnHeader.1", type text}, {"ColumnHeader.2", type text}, {"ColumnHeader.3", type text}, {"ColumnHeader.4", type text}}),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Changed Type1", {"Index"}, "Attribute", "Value"),
#"Removed Columns" = Table.RemoveColumns(#"Unpivoted Other Columns",{"Attribute", "Index"}),
#"Removed Duplicates" = Table.Distinct(#"Removed Columns"),
#"Sorted Rows" = Table.Sort(#"Removed Duplicates",{{"Value", Order.Ascending}}),
#"Added Index1" = Table.AddIndexColumn(#"Sorted Rows", "Index", 1, 1, Int64.Type),
#"Reordered Columns1" = Table.ReorderColumns(#"Added Index1",{"Index", "Value"}),
#"Pivoted Column" = Table.Pivot(Table.TransformColumnTypes(#"Reordered Columns1", {{"Index", type text}}, "en-US"), List.Distinct(Table.TransformColumnTypes(#"Reordered Columns1", {{"Index", type text}}, "en-US")[Index]), "Index", "Value", List.Max),
#"Merged Columns" = Table.CombineColumns(#"Pivoted Column",{"1", "2", "3", "4", "5"},Combiner.CombineTextByDelimiter(":", QuoteStyle.None),"Merged")
in
#"Merged Columns"
Here is the final output-
Here is code from Power Query Editor considering GROUP BY-
Create a new table RESULT_BY_GROUP with this below code-
let
Source = TABLE_BY_ELEMENT,
#"Removed Columns" = Table.RemoveColumns(Source,{"KEY", "KEY_DATA"}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Removed Columns", "LIST", Splitter.SplitTextByDelimiter(":", QuoteStyle.Csv), {"LIST.1", "LIST.2", "LIST.3"}),
#"Changed Type" = Table.TransformColumnTypes(#"Split Column by Delimiter",{{"LIST.1", type text}, {"LIST.2", type text}, {"LIST.3", type text}}),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Changed Type", {"GROUP"}, "Attribute", "Value"),
#"Removed Columns1" = Table.RemoveColumns(#"Unpivoted Other Columns",{"Attribute"}),
#"Removed Duplicates" = Table.Distinct(#"Removed Columns1"),
#"Sorted Rows" = Table.Sort(#"Removed Duplicates",{{"GROUP", Order.Ascending}, {"Value", Order.Ascending}}),
#"Grouped Rows" = Table.Group(#"Sorted Rows", {"GROUP"}, {{"all", each _, type table [GROUP=nullable text, Value=text]}}),
#"Added Custom" = Table.AddColumn(#"Grouped Rows", "NewList", each [all][Value]),
#"Extracted Values" = Table.TransformColumns(#"Added Custom", {"NewList", each Text.Combine(List.Transform(List.Sort(_), Text.From), ":"), type text}),
#"Removed Columns2" = Table.RemoveColumns(#"Extracted Values",{"all"}),
#"Merged Queries" = Table.NestedJoin(#"Removed Columns2", {"GROUP"}, TABLE_BY_GROUP, {"GROUP"}, "TABLE_BY_GROUP", JoinKind.LeftOuter),
#"Expanded TABLE_BY_GROUP" = Table.ExpandTableColumn(#"Merged Queries", "TABLE_BY_GROUP", {"GROUP_DATA "}, {"TABLE_BY_GROUP.GROUP_DATA "}),
#"Renamed Columns" = Table.RenameColumns(#"Expanded TABLE_BY_GROUP",{{"TABLE_BY_GROUP.GROUP_DATA ", "GROUP_DATA"}}),
#"Changed Type1" = Table.TransformColumnTypes(#"Renamed Columns",{{"GROUP", type text}, {"NewList", type text}, {"GROUP_DATA", Int64.Type}})
in
#"Changed Type1"
Here is the final output-
You can easily visualize your second requirement for table RESULT_ELEMENT using your base table TABLE_BY_ELEMENT

Transpose table partially

I have a CSV-file containing information about some production batches. When loaded into Excels PowerQuery editor, the table looks like this:
Batch Date RawMaterial1 RawMaterial2 RawMaterial3 Amount1 Amount2 Amount3
123 01.01.2020 Fe Cr Ni 70 19 11
234 01.02.2020 Fe Cr Ni 72 17 9
To make this table more readable, I'm looking for a way to transpose it just partially to transform it into a format like this:
Batch Date RawMaterials Amounts
123 01.01.2020 Fe 70
Cr 19
Ni 11
234 01.02.2020 Fe 72
Cr 17
Ni 11
Is there a way to realize this with PowerQueryM alone?
This can be done quite a bit more simply:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],,
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Batch", Int64.Type}, {"Date", type date}, {"RawMaterial1", type text}, {"RawMaterial2", type text}, {"RawMaterial3", type text}, {"Amount1", Int64.Type}, {"Amount2", Int64.Type}, {"Amount3", Int64.Type}}),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Changed Type", {"Batch", "Date"}, "Attribute", "Value"),
#"Split Column by Character Transition" = Table.SplitColumn(#"Unpivoted Other Columns", "Attribute", Splitter.SplitTextByCharacterTransition((c) => not List.Contains({"0".."9"}, c), {"0".."9"}), {"Type", "Index"}),
#"Pivoted Column" = Table.Pivot(#"Split Column by Character Transition", List.Distinct(#"Split Column by Character Transition"[Type]), "Type", "Value")
in
#"Pivoted Column"
Unpivot all but the first two columns.
Split the Attribute column into the text part and index part (in the GUI: Transform > Split Column > By Non-Digit to Digit).
Pivot back on the text part column (choose Don't Aggregate in the Pivot Column Advanced options).
This seems to work
Unpivots all columns but first two
Duplicate the data column
Change the column type to number to force an error on the non numerical columns
Change all errors into something recognizable, like 999999999999
Filter based on that into two tables, and add an index to each table
Merge the two tables together
Add new column, using index to see if Batch is same as prior row to eliminate duplicates
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"Date", "Batch"}, "Attribute", "Value"),
#"Duplicated Column" = Table.DuplicateColumn(#"Unpivoted Other Columns", "Value", "Value - Copy"),
#"Changed Type" = Table.TransformColumnTypes(#"Duplicated Column",{{"Value - Copy", type number}}),
#"Replaced Errors" = Table.ReplaceErrorValues(#"Changed Type", {{"Value - Copy", 999999999999999}}),
#"Filtered Rows" = Table.SelectRows(#"Replaced Errors", each ([#"Value - Copy"] = 999999999999999)),
#"Filtered Rows2" = Table.SelectRows(#"Replaced Errors", each ([#"Value - Copy"] <> 999999999999999)),
Index1 = Table.AddIndexColumn(#"Filtered Rows", "Index", 0, 1),
Index2 = Table.AddIndexColumn(#"Filtered Rows2", "Index", 0, 1),
#"Merged Queries" = Table.NestedJoin(Index2,{"Index"},Index1,{"Index"},"Index3",JoinKind.LeftOuter),
#"Expanded Index3" = Table.ExpandTableColumn(#"Merged Queries", "Index3", {"Value"}, {"Value.1"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Index3",{"Attribute", "Value - Copy"}),
#"Added Custom" = Table.AddColumn(#"Removed Columns", "Batch.1", each if [Index] = 0 then [Batch] else if #"Removed Columns"{[Index]-1}[Batch] = [Batch] then null else [Batch]),
#"Added Custom2" = Table.AddColumn(#"Added Custom", "Date.1", each if [Index] = 0 then [Date] else if #"Removed Columns"{[Index]-1}[Batch] = [Batch] then null else [Date]),
#"Removed Columns1" = Table.RemoveColumns(#"Added Custom2",{"Batch", "Date", "Index"}),
#"Reordered Columns" = Table.ReorderColumns(#"Removed Columns1",{"Batch.1", "Date.1", "Value.1", "Value"})
in #"Reordered Columns"
First of all a big thank you to #horseyride. I learned a lot from your suggested code. Sadly when I tried to add the date-column to the unpivot area as well I found a little flaw in the code. But thanks to the lessons I learned from it I was able to produce a slightly more generic version which basically follows the same algorithm.
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Date", type date}}),
#"Unpivot Columns" = Table.UnpivotOtherColumns(#"Changed Type", {"Batch"}, "Attribut", "Wert"),
Filter1 = Table.SelectRows(#"Unpivot Columns", each ([Attribut] <> "Amount1" and [Attribut] <> "Amount2" and [Attribut] <> "Amount3")),
ModFilter1 = Table.AddColumn(Filter1, "Benutzerdefiniert", each if [Attribut] = "Date" then [Attribut] else [Wert], type text),
Filter2 = Table.SelectRows(#"Unpivot Columns", each ([Attribut] <> "RawMaterial1" and [Attribut] <> "RawMaterial2" and [Attribut] <> "RawMaterial3")),
#"IndexFilter1" = Table.AddIndexColumn(ModFilter1, "Index", 0, 1),
#"IndexFilter2" = Table.AddIndexColumn(Filter2, "Index", 0, 1),
#"Join Filtered Indexes" = Table.NestedJoin(IndexFilter1,{"Index"},IndexFilter2,{"Index"},"IndexFilter2",JoinKind.LeftOuter),
#"Expand Joined Column" = Table.ExpandTableColumn(#"Join Filtered Indexes", "IndexFilter2", {"Wert"}, {"IndexFilter2.Wert"}),
#"Remove Columns" = Table.RemoveColumns(#"Expand Joined Column",{"Index", "Attribut", "Wert"}),
#"Rename Columns" = Table.RenameColumns(#"Remove Columns",{{"Benutzerdefiniert", "Attribut"}, {"IndexFilter2.Wert", "Wert"}})
in
#"Rename Columns"
I keep horseyrides answer checked as the right answer as he solves my initial question as it was.

Resources