Related
I receive data that looks like this:
Name
01/01/2023
01/02/2023
Revenue
Revenue
Chris
1
3
£100
£300
Colin
5
8
£500
£800
Pete
2
5
£200
£500
Where name is self-explanatory, the next two columns are dates (in UK format) with the number of days worked in the period shown below, and the final two columns are revenue.
I want to modify this data in Power Query so it looks like this:
Name
Date
Work Days
Revenue
Chris
01/01/2023
1
£100
Chris
01/02/2023
3
£300
Colin
01/01/2023
5
£500
Colin
01/02/2023
8
£800
Pete
01/01/2023
2
£200
Pete
01/02/2023
5
£500
I thought this would be some kind of a pivot operation but I can't figure it out.
Any assistance will be gratefully received.
Thanks,
Chris
One simple way
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
Set0=List.FirstN(Table.ColumnNames(Source),1),
Set1= List.Combine({Set0,List.Alternate(Table.ColumnNames(Source),1,1)}),
Set2=List.Combine({Set0,List.Alternate(List.RemoveFirstN(Table.ColumnNames(Source),1),1,1)}),
Part1 = Table.SelectColumns(Source,Set1),
Part2 = Table.SelectColumns(Source,Set2),
Date1 = Table.AddColumn(Part1,"Date" , each Table.ColumnNames(Part1){1}),
Date2 = Table.AddColumn(Part2,"Date" , each Table.ColumnNames(Part2){1}),
Rename1 = Table.RenameColumns(Date1,{{Table.ColumnNames(Part1){2}, "Revenue"}, {Table.ColumnNames(Part1){1}, "Work Days"}}),
Rename2 = Table.RenameColumns(Date2,{{Table.ColumnNames(Part2){2}, "Revenue"}, {Table.ColumnNames(Part2){1}, "Work Days"}}),
combined = Rename1 & Rename2
in combined
or
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added Index" = Table.AddIndexColumn(Source, "Index", 0, 1, Int64.Type),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Added Index", {"Index", "Name"}, "Attribute", "Value"),
#"Added Custom" = Table.AddColumn(#"Unpivoted Other Columns", "Date", each if Text.Start([Attribute],3)="Rev" then null else [Attribute]),
#"Added Custom2" = Table.AddColumn(#"Added Custom", "count", each if Text.Start([Attribute],3)="Rev" then null else [Value]),
#"Added Index1" = Table.AddIndexColumn(#"Added Custom2", "Index.1", 0, 1, Int64.Type),
#"Inserted Modulo" = Table.AddColumn(#"Added Index1", "Modulo", each Number.Mod([Index.1], 2), type number),
#"Sorted Rows" = Table.Sort(#"Inserted Modulo",{{"Index", Order.Ascending}, {"Modulo", Order.Ascending}, {"Attribute", Order.Ascending}}),
#"Filled Down" = Table.FillDown(#"Sorted Rows",{"Date", "count"}),
x=Table.AlternateRows(#"Filled Down",0,1,1),
#"Removed Other Columns" = Table.SelectColumns(x,{"Name", "Value", "Date", "count"})
in #"Removed Other Columns"
Here's another way:
Using List.Generate, create a List of Tables using each Date/Revenue Pair.
For each of the tables, ensure the Revenue Column is named Revenue (and not Revenue2, Revenu3, etc) and then Unpivot the table.
Then expand the column that has the list of tables
The rest is "housekeeping"
Code Edited to provide for varying numbers of "First Columns" to be retained before the sets of Date and Revenue columns
*Change #"Retained Column Count" to reflect that number of Columns
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
colNames = List.Buffer(Table.ColumnNames(Source)),
//How many columns at the beginning are non data pairs
#"Retained Column Count" = 4,
#"First Columns" = List.FirstN(colNames,#"Retained Column Count"),
#"Date Columns" = List.Range(colNames,#"Retained Column Count",(List.Count(colNames)-#"Retained Column Count")/2),
#"Revenue Columns" = List.LastN(colNames,List.Count(#"Date Columns")),
//set data types
types = List.Transform(#"First Columns", each {_, type text}) &
List.Transform(#"Date Columns", each {_, Int64.Type}) &
List.Transform(#"Revenue Columns", each {_, Currency.Type}),
#"Changed Type" = Table.TransformColumnTypes(Source, types, "en-GB"),
//create a list of tables consisting of each date/revenue pair
// then unpivot each table
// ensure Revenue column has the same name throughout
#"Data Pairs" = List.Generate(
()=>[t=Table.SelectColumns(#"Changed Type",#"First Columns" & {#"Date Columns"{0}} & {#"Revenue Columns"{0}}), idx=0],
each [idx] < List.Count(#"Date Columns"),
each [t=Table.SelectColumns(#"Changed Type",#"First Columns" & {#"Date Columns"{[idx]+1}} & {#"Revenue Columns"{[idx]+1}}), idx=[idx]+1],
each Table.Unpivot(
Table.RenameColumns([t], {Table.ColumnNames([t]){#"Retained Column Count"+1},"Revenue"}),
{#"Date Columns"{[idx]}},"Date","Work Days")),
//to a table
// then combine the tables with column names in desired order
#"Converted to Table" = Table.FromList(#"Data Pairs", Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandTableColumn(#"Converted to Table", "Column1", #"First Columns" & {"Date","Work Days","Revenue"}),
#"Changed Type with Locale" = Table.TransformColumnTypes(#"Expanded Column1",
List.Transform(#"First Columns", each {_, type text}) &
{{"Date", type date},
{"Work Days", Int64.Type},
{"Revenue", Currency.Type}}, "en-GB"),
#"Sorted Rows" = Table.Sort(#"Changed Type with Locale",{{"Name", Order.Ascending}, {"Date", Order.Ascending}})
in
#"Sorted Rows"
I have two tables that I'd like to merge, but I can't quite get the merge to work the way I'd like it to. The first is a list of days and some data, the second is a ranking of items sold that day along with the number sold, etc. I'd like to get the top 'x' items added as columns to my daily data. I got the merge to work, and so now I have a column that says 'table.' However if I expand the table, each item turns into its own row. Instead I'd like table item 2 to repeat all of its columns with a postfix.
date
total_sales
items
1/1/21
$50000
table
1/2/21
$40000
table
date
total_sales
items_1
units_1
items_2
units_2
1/1/21
$50000
pens
15
pencils
10
1/2/21
$40000
erasers
35
pens
5
etc.
I can do this with visual basic but I don't think that's the best way to go about it. Thanks for your help! Also, is there a specific term for this operation that I could have searched for?
Let's suppose your second table looks like this:
First, let's group by date where for each group we sort and index the subtable. (This is the hardest step.)
Once we have the index for each group, expand that column back so we're back to the start except with the new index column.
From here, you can filter the index to get only the top N and then unpivot the [items] and [units] columns.
Merge the [index] and [Attribute]
and then pivot on [Merged]
Ta da! Now your data is shaped so that you can easily merge it with the first table.
Here's the full code for this example. (You can paste this into the Advanced Editor and look through Applied Steps.)
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMtQ31DcyMDJU0lEqSM0rBlKGpkqxOugSyZk5YDkDqJwRTC61KLE4tQgkZ2yKLgc1EMM8hB5sWqBWGSnFxgIA", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [date = _t, items = _t, units = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"date", type date}, {"items", type text}, {"units", Int64.Type}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"date"}, {{"indexed", each Table.AddIndexColumn(Table.Buffer(Table.Sort(_, {{"units", Order.Descending}})), "Index", 1, 1, Int64.Type), type table}}),
#"Expanded indexed" = Table.ExpandTableColumn(#"Grouped Rows", "indexed", {"items", "units", "Index"}, {"items", "units", "Index"}),
#"Filtered Rows" = Table.SelectRows(#"Expanded indexed", each [Index] < 3),
#"Unpivoted Columns1" = Table.UnpivotOtherColumns(#"Filtered Rows", {"date", "Index"}, "Attribute", "Value"),
#"Merged Columns" = Table.CombineColumns(Table.TransformColumnTypes(#"Unpivoted Columns1", {{"Index", type text}}, "en-US"),{"Attribute", "Index"},Combiner.CombineTextByDelimiter("_", QuoteStyle.None),"Merged"),
#"Pivoted Column1" = Table.Pivot(#"Merged Columns", List.Distinct(#"Merged Columns"[Merged]), "Merged", "Value"),
#"Changed Type1" = Table.TransformColumnTypes(#"Pivoted Column1",{{"items_1", type text}, {"units_1", Int64.Type}, {"items_2", type text}, {"units_2", Int64.Type}})
in
#"Changed Type1"
The only place I'm doing special M language magic (not just clicking stuff in the GUI) is this step which I created by applying a couple of steps to one of the sub-tables and pasting the GUI-generated code back into the group by step (plus Table.Buffer wrapper to make sure the sorting "sticks").
#"Grouped Rows" =
Table.Group(
#"Changed Type",
{"date"},
{
{"indexed", each
Table.AddIndexColumn(
Table.Buffer(
Table.Sort(_, {{"units", Order.Descending}})
),
"Index", 1, 1, Int64.Type
), type table
}
}
),
I'm using this in a table on Excel, but it takes too long to calculate every time - no doubt.
=IF(COUNTIFS(T:T;T9371;R:R;">="&EOMONTH('Sheet1'!$B$2;-1)+1;R:R;"<="&EOMONTH('Sheet1'!$B$3;0))>0;1;0)
The idea is to find occurrences of that customer in the current month and mark it as 1, if there's any and 0 if not. Also find the ones who have bought in the last month...
How would we have it set in Power Query, instead?
Here's a sample of the data:
CustomerID
Customer
Date
Item
ThisMonth
LastMonth
T45678
ABC
09/15/21
Product A
1
0
T45678
ABC
09/15/21
Product B
1
0
T123645
BGT
08/10/21
Product A
0
1
Thank you!
An example of how I would do this in PQ.
Please read the comments in the code and explore the Applied Steps to understand the algorithm.
M Code
let
Source = Excel.CurrentWorkbook(){[Name="Table4"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{
{"CustomerID", type text}, {"Customer", type text}, {
"Date", type date}, {"Item", type text}}),
//Normalize all dates to start of month for grouping
//date type => datetime so can be used in grouping aggregation below
#"Added Custom" = Table.AddColumn(#"Changed Type", "BOM",
each Date.StartOfMonth([Date]), type datetime),
//group by Customer and BOM
// then return 1 or 0 if in current or previous month
#"Grouped Rows" = Table.Group(#"Added Custom", {"CustomerID", "BOM"}, {
{"all", each _, type table
[CustomerID=nullable text,
Customer=nullable text,
Date=nullable date,
Item=nullable text,
BOM=date]},
{"This Month", each if Date.IsInCurrentMonth([BOM]{0}) then 1 else 0, Int64.Type},
{"Last Month", each if Date.IsInPreviousMonth([BOM]{0}) then 1 else 0, Int64.Type}
}),
//Expand the table to restore ungrouped listings
#"Expanded all" = Table.ExpandTableColumn(#"Grouped Rows", "all",
{"Customer", "Date", "Item"}, {"Customer", "Date", "Item"}),
//Don't need BOM column anymore
#"Removed Columns1" = Table.RemoveColumns(#"Expanded all",{"BOM"})
in
#"Removed Columns1"
Original Data
Results
I have an Excel table with data organized such that each row is a sample and each column has a different property of that sample. However, I need to reorganize it so that it works with GraphPad Prism.
Currently the data is organized like this:
Sample ID
Exposure Level
Drug
Score 1
…
Score 22
101
1
A
0.675815
0.17351
102
1
B
0.276413
0.677079
103
2
A
0.914725
0.387529
104
3
A
0.504221
0.135295
105
3
B
0.963684
0.710081
106
2
B
0.964099
0.146872
And I want to make a box and whisker plot showing the score of each exposure level, like this:
I need to do this including all the samples and then again for just drug A and just drug B.
However, in order to do that in Prism, to my knowledge, each combination of variables you want needs to have in own column, like this:
Score 1 Exposure 1
Score 1 Exposure 2
Score 1 Exposure 3
Score 1 Exposure 1 (Just Drug A)
Score 1 Exposure 2 (Just Drug A)
Score 1 Exposure 3 (Just Drug A)
etc.
0.675815
0.914725
0.504221
0.675815
0.914725
0.504221
0.276413
0.964099
0.963684
This would be easy enough to do manually if there were just one score column, but there are twenty-two, so I'd rather not. Is there some automated way I can reorganize the data table like this?
To create a Box & Whiskers graph similar to what you show,
merely use the Exposure Level for the x-axis and the Score 1 column for the y-axis
To create a table similar to the results you show, you can use Power Query.
I created it as a single table, with each row representing a drug. You can then filter it by drug for your drug specific results.
The MCode is commented so by reading the comments, and also looking at the Applied Steps window, I hope I was clear in what was going on.
Most of the MCode is generated from the UI, but, especially, the colNames and ExpandTableColumns steps near the end are manually entered. Otherwise the number of columns in the expansion would not be flexible.
MCode
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
//Won't need ID column so get rid of it
#"Removed Columns2" = Table.RemoveColumns(Source,{"Sample ID"}),
//Unpivot the Score columns to put them in a single column
#"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Removed Columns2", {"Exposure Level", "Drug"}, "Attribute", "Value"),
//sort by Score, Attribute, Drug so the results will be properly ordered
#"Sorted Rows" = Table.Sort(#"Unpivoted Columns",{{"Attribute", Order.Ascending}, {"Exposure Level", Order.Ascending}, {"Drug", Order.Ascending}}),
//Create what will become a two line header column
// and remove the originals
#"Added Custom" = Table.AddColumn(#"Sorted Rows", "Headers", each "Exposure " & Text.From([Exposure Level]) & "#(lf)" & [Attribute]),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Exposure Level", "Attribute"}),
//Move headers to first column
#"Reordered Columns" = Table.ReorderColumns(#"Removed Columns",{"Headers", "Drug", "Value"}),
//Group by Drug
#"Grouped Rows" = Table.Group(#"Reordered Columns", {"Drug"}, {{"Grouped", each _, type table [Headers=text, Drug=text, Value=number]}}),
//Add an Index column
#"Added Index" = Table.AddIndexColumn(#"Grouped Rows", "Index", 0, 1, Int64.Type),
/*From each grouped table, remove Drug Column
and remove Header column EXCEPT fromk the first table
then Transpose each grouped table*/
#"Added Custom1" = Table.AddColumn(#"Added Index", "Custom", each
Table.Transpose(
if [Index] = 0 then
Table.RemoveColumns([Grouped],"Drug")
else
Table.RemoveColumns([Grouped],{"Headers","Drug"}))),
//Remove no longer needed Grouped and Index columns
#"Removed Columns1" = Table.RemoveColumns(#"Added Custom1",{"Grouped", "Index"}),
//Expand the table columns, promote headers, and rename the drug column to get final results
colNames = Table.ColumnNames(#"Removed Columns1"[Custom]{0}),
#"Expanded Custom" = Table.ExpandTableColumn(#"Removed Columns1", "Custom", colNames),
#"Promoted Headers" = Table.PromoteHeaders(#"Expanded Custom", [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"A", type text}, {"Exposure 1#(lf)Score 1", type number}, {"Exposure 2#(lf)Score 1", type number}, {"Exposure 3#(lf)Score 1", type number}, {"Exposure 1#(lf)Score 22", type number}, {"Exposure 2#(lf)Score 22", type number}, {"Exposure 3#(lf)Score 22", type number}}),
#"Renamed Columns" = Table.RenameColumns(#"Changed Type",{{"A", "Drug"}})
in
#"Renamed Columns"
EDIT
#user3316549 commented below that he might have multiple entries for the same drug for the same Score/Exposure and wanted the results for each shown separately.
A Pivot table would be useful here, except a classic pivot table will only have a single entry for each intersection of Drug with Score/Exposure.
This problem is solved with a custom function for the pivot that adds an extra row when needed. The credits for that function are included and you can examine the link for a detailed explanation of the algorithm used for that part of the code.
The custom function is added as a blank query. You can name it what you choose and call it that way in your main code.
M Code
Main Query
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
//Unpivot the Score columns to put them in a single column
#"Unpivoted Columns" = Table.UnpivotOtherColumns(Source, {"Sample ID","Exposure Level", "Drug"}, "Attribute", "Value"),
//sort by multiple columns so the results will be properly ordered to our liking
#"Sorted Rows" = Table.Sort(#"Unpivoted Columns",{{"Attribute", Order.Ascending}, {"Exposure Level", Order.Ascending}, {"Drug", Order.Ascending},{"Sample ID", Order.Ascending}}),
//Create what will become a two line header column
// and remove the originals
#"Added Custom" = Table.AddColumn(#"Sorted Rows", "Headers", each [Attribute] & "#(lf)" & "Exposure " & Text.From([Exposure Level])),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Sample ID","Exposure Level", "Attribute"}),
//custom pivot function for non-aggregation
pivotAll = fnPivotAll(#"Removed Columns","Headers","Value")
in
pivotAll
M Code
Custom Function named fnPivotAll
//credit: Cam Wallace https://www.dingbatdata.com/2018/03/08/non-aggregate-pivot-with-multiple-rows-in-powerquery/
(Source as table,
ColToPivot as text,
ColForValues as text)=>
let
PivotColNames = List.Buffer(List.Distinct(Table.Column(Source,ColToPivot))),
#"Pivoted Column" = Table.Pivot(Source, PivotColNames, ColToPivot, ColForValues, each _),
TableFromRecordOfLists = (rec as record, fieldnames as list) =>
let
PartialRecord = Record.SelectFields(rec,fieldnames),
RecordToList = Record.ToList(PartialRecord),
Table = Table.FromColumns(RecordToList,fieldnames)
in
Table,
#"Added Custom" = Table.AddColumn(#"Pivoted Column", "Values", each TableFromRecordOfLists(_,PivotColNames)),
#"Removed Other Columns" = Table.RemoveColumns(#"Added Custom",PivotColNames),
#"Expanded Values" = Table.ExpandTableColumn(#"Removed Other Columns", "Values", PivotColNames)
in
#"Expanded Values"
I have the following table on Power Query that contains a quantity of a product bought and his price:
I want to create another third column with the current average purchase price, however, I need to consider the history and the average should be weighted:
Can anyone help me?
To calculate a running weighted average, you merely have to generate a running total cost column (Price x Quantity) and a running total quanity column, then divide one by the other:
There are various ways to do this, and you don't need to do this by adding columns to the table, but I did it so you can examine the steps involved.
M-Code
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Month", type text}, {"Quantity", Int64.Type}, {"Price", type number}}),
#"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 1, 1),
#"Added Custom" = Table.AddColumn(#"Added Index", "Cost", each [Quantity]*[Price]),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "runTotCost", each List.Sum(List.Range(#"Added Custom"[Cost],0,[Index]))),
#"Added Custom2" = Table.AddColumn(#"Added Custom1", "runTotQuant", each List.Sum(List.Range(#"Added Custom"[Quantity],0,[Index]))),
#"Added Custom3" = Table.AddColumn(#"Added Custom2", "Weighted Average", each [runTotCost] / [runTotQuant]),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom3",{"Index", "Cost", "runTotCost", "runTotQuant"}),
#"Changed Type1" = Table.TransformColumnTypes(#"Removed Columns",{{"Weighted Average", type number}})
in
#"Changed Type1"
For better understanding, I suggest you do an internet search for methods of generating running totals. There are other methods that are more complicated, but more efficient.