Related
Is there any way in Excel Pivot or Power BI to do the rolling sum of the given data (let say monthly)?
Let say I have a list of cases, each row represent case count and amount. The project start date and end date varied as follows.
For, simplicity, if I demonstrate the data graphically, would be as follows.
What I'm try to do is to aggregate how much case counts and amounts in total for each chunk of month.
My goal is to produce below list using Pivot (if Pivot is not possible, then by Power Query) directly.
I could produce monthly aggregates using Filter function and Sum, then pivot that data to produce above result.
If there is a direct way of producing that aggregates in one step, that would be better. Please suggest it for me.
Please see sample data in below link
https://docs.google.com/spreadsheets/d/1vAKElb2-V_If-MMlPwHk_VGhYr8pkOg_gQfRYRrkbtc/edit?usp=share_link
Excel file in Zip
https://drive.google.com/file/d/1QqgNUrJlBuvin7iecsxsvexrGZXFIt-g/view?usp=share_link
Thank you in advance
LuZ
You can load the data into powerquery and transform from left to data table on right
code for that is
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added Custom1" = Table.AddColumn(Source, "Date", each List.Generate(()=>[x=[Start Date],i=0], each [i]<12, each [i=[i]+1,x=Date.AddMonths([x],1)], each [x])),
#"Expanded Custom" = Table.ExpandListColumn(#"Added Custom1", "Date"),
#"Added Custom" = Table.AddColumn(#"Expanded Custom", "Year", each Date.Year([Date])),
#"Added Custom2" = Table.AddColumn(#"Added Custom", "Month", each Date.Month([Date])),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom2",{"Start Date", "End Date", "Date"})
in #"Removed Columns"
Afterwards, load the powerquery back into excel as pivot report and generate your table
Alternatively, just use use
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added Custom1" = Table.AddColumn(Source, "Date", each List.Generate(()=>[x=[Start Date],i=0], each [i]<12, each [i=[i]+1,x=Date.AddMonths([x],1)], each [x])),
#"Expanded Custom" = Table.ExpandListColumn(#"Added Custom1", "Date"),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Custom",{"Start Date", "End Date"}),
#"Grouped Rows" = Table.Group(#"Removed Columns", {"Date"}, {{"Amount", each List.Sum([Amount]), type number}, {"Case Count", each List.Sum([Case Count]), type number}}),
#"Changed Type" = Table.TransformColumnTypes(#"Grouped Rows",{{"Date", type date}, {"Amount", type number}, {"Case Count", type number}})
in #"Changed Type"
to generate this table, then graph it
I have close to 400 excel files that contain client pricing information. The piece of work is to transform these into multiple database tables to load into Azure SQL Database. I'm tasked with the transformation.
The existing data follows a level of logic. 1 workbook per client. A single client can have multiple ICP's (shown in the example). What I'm looking for is some assistance to use Power Query to transform the data from the source to target formats. Once in Target, it'll be loaded in bulk to SQL server to perform additional transformation into required SQL tables.
Rows that need to be selected
Customer Name
Supply Address
ICP
Any subsequent row within the selected range (between ICPs) that contain data within the "Old Rates" or "New Rates" columns. Even if a single value is stored in one or the either, both columns for that row require extraction.
Example of source and target
Source and Target
Having to post as picture as I keep getting an error message about code not being formatted correctly, yet there is no code. Only two tables
first index the your table
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
rowscount = List.Count(List.Distinct(Source[type])),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"old rate", Percentage.Type}, {"new rate", Percentage.Type}}),
#"Replaced Value" = Table.ReplaceValue(#"Changed Type",null,0,Replacer.ReplaceValue,{"type", "detail", "old rate", "new rate"}),
#"Added Index" = Table.AddIndexColumn(#"Replaced Value", "Index", 0, 1, Int64.Type),
#"Integer-Divided Column" = Table.TransformColumns(#"Added Index", {{"Index", each Number.IntegerDivide(_, rowscount), Int64.Type}})
enter code here
in
#"Integer-Divided Column"
then you can pivot and merge the other data
let
Source = Table1,
#"Removed Columns" = Table.RemoveColumns(Source,{"old rate", "new rate"}),
#"Filtered Rows" = Table.SelectRows(#"Removed Columns", each ([type] = "customer name" or [type] = "ICP" or [type] = "supply address")),
#"Pivoted Column" = Table.Pivot(#"Filtered Rows", List.Distinct(#"Filtered Rows"[#"type"]), "type", "detail"),
#"Merged Queries" = Table.NestedJoin(#"Pivoted Column", {"Index"}, Table1, {"Index"}, "Table1", JoinKind.LeftOuter),
#"Expanded Table1" = Table.ExpandTableColumn(#"Merged Queries", "Table1", {"type", "old rate", "new rate"}, {"type", "old rate", "new rate"}),
#"Filtered Rows1" = Table.SelectRows(#"Expanded Table1", each ([type] = "Anytime" or [type] = "Day" or [type] = "EA Levy")),
#"Removed Columns1" = Table.RemoveColumns(#"Filtered Rows1",{"Index"})
in
#"Removed Columns1"
see also sample file
sample file
I need to transform data that is in multiple rows and multiple columns into unique rows, but there are specific rules around what i need. An example of the current data format is below:
The split should be based on the style, colour and unique upc but i need to copy some of the fields to each unique upc for the style and colour. I also need to show a parent child relationship.
The example below is how I want the data to be shown.
I've tried doing this in power query...but totally stuck!
Thanks in advance for any advice.
This seems to work in powerquery pasted into home...advanced editor.. assuming your initial table is range Table1 and similar to the sample structure you provided
Unpivot, remove numbers from the the attributes, group and add index for the later pivot, pivot. The rest is just custom columns and filling
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"style", "colour"}, "Attribute", "Value"),
#"removed numbers" = Table.TransformColumns(#"Unpivoted Other Columns",{{"Attribute", each Text.Remove(_, List.Transform({48..57}, each Character.FromNumber(_))), type text}}),
#"Grouped Rows" = Table.Group(#"removed numbers", {"style", "colour", "Attribute"}, {{"data", each Table.AddIndexColumn(_, "Index", 2, 1), type table}}),
#"Expanded data" = Table.ExpandTableColumn(#"Grouped Rows", "data", {"Value", "Index"}, {"Value", "Index"}),
#"Added Custom" = Table.AddColumn(#"Expanded data", "Custom", each if [Attribute]="description" then 1 else [Index]),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Index"}),
#"Pivoted Column" = Table.Pivot(#"Removed Columns", List.Distinct(#"Removed Columns"[Attribute]), "Attribute", "Value"),
#"Added Custom1" = Table.AddColumn(#"Pivoted Column", "parent", each if [description]=null then "child" else "parent"),
#"Filled Down" = Table.FillDown(#"Added Custom1",{"description"}),
#"Removed Columns1" = Table.RemoveColumns(#"Filled Down",{"Custom"})
in #"Removed Columns1"
Alternate version that uses index and transform on that column prior to pivoting
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"style", "color", "desc"}, "Attribute", "Value"),
#"removed numbers" = Table.TransformColumns(#"Unpivoted Other Columns",{{"Attribute", each Text.Remove(_, List.Transform({48..57}, each Character.FromNumber(_))), type text}}),
#"Added Index" = Table.AddIndexColumn(#"removed numbers", "Index", 1, .5),
#"Rounded Down" = Table.TransformColumns(#"Added Index",{{"Index", Number.RoundDown, Int64.Type}}),
#"Pivoted Column" = Table.Pivot(#"Rounded Down", List.Distinct(#"Rounded Down"[Attribute]), "Attribute", "Value"),
#"Added Custom1" = Table.AddColumn(#"Pivoted Column", "parent", each "child"),
#"Add parent" = Table.Combine({Table.AddColumn(Table.Distinct(Table.SelectColumns(#"Added Custom1",{"style", "color", "desc"})), "parent", each "parent"), #"Added Custom1"}),
#"Removed Columns" = Table.RemoveColumns(#"Add parent",{"Index"}),
#"Reordered Columns" = Table.ReorderColumns(#"Removed Columns",{"style", "color", "upc", "size", "desc", "parent"}),
#"Sorted Rows" = Table.Sort(#"Reordered Columns",{{"style", Order.Ascending}, {"color", Order.Ascending}, {"upc", Order.Ascending}})
in #"Sorted Rows"
Here is another Power Query method that uses a custom function to enable creation of a Pivot Table with no aggregation where there are multiple items.
Examine the comments in the M-Code and the Applied steps, and also the reference in the custom function, to understand how it works:
To enter the custom function, select to
New Query => Blank Query.
Rename the Query from (probably) Query1 to fnPivotAll
M Code
//Rename Table3 to your actual table name
let Source = Excel.CurrentWorkbook(){[Name="Table3"]}[Content],
//Unpivot all except the style and color columns
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"style", "colour"}, "Attribute", "Value"),
//remove digits from the UPC and SIZE attributes
remDigits = Table.TransformColumns(#"Unpivoted Other Columns",{
{"Attribute", each Text.Remove(_, List.Transform({48..57}, each Character.FromNumber(_))), type text}}),
//Pivot on Attribute Column
//Custom function to use when there are multiple values for the column
pivot = fnPivotAll(remDigits,"Attribute","Value"),
//Fill in the blank descriptions
#"Filled Down" = Table.FillDown(pivot,{"description"}),
//Group (by style, colour and description) to add a description row to each grouped table
#"Grouped Rows" = Table.Group(#"Filled Down", {"style", "colour", "description"}, {
{"All", each _, type table [style=text, colour=text, upc=number, size=any, description=text]},
{"addRow", each Table.InsertRows(_, 0, {[style=[style]{0}, colour=[colour]{0}, upc=null, size=null, description=[description]{0}]})}
}),
#"Removed Columns" = Table.RemoveColumns(#"Grouped Rows",{"style", "colour", "description", "All"}),
//expand the grouped table
#"Expanded addRow" = Table.ExpandTableColumn(#"Removed Columns", "addRow", {"style", "colour", "upc", "size", "description"}, {"style", "colour", "upc", "size", "description"}),
//Add column for Parent or child
#"Added Custom" = Table.AddColumn(#"Expanded addRow", "Parent", each if [upc] = null then "Parent" else "Child")
in
#"Added Custom"
Custom Function
named fnPivotAll -- Rename the Query
//credit: Cam Wallace https://www.dingbatdata.com/2018/03/08/non-aggregate-pivot-with-multiple-rows-in-powerquery/
(Source as table,
ColToPivot as text,
ColForValues as text)=>
let
PivotColNames = List.Buffer(List.Distinct(Table.Column(Source,ColToPivot))),
#"Pivoted Column" = Table.Pivot(Source, PivotColNames, ColToPivot, ColForValues, each _),
TableFromRecordOfLists = (rec as record, fieldnames as list) =>
let
PartialRecord = Record.SelectFields(rec,fieldnames),
RecordToList = Record.ToList(PartialRecord),
Table = Table.FromColumns(RecordToList,fieldnames)
in
Table,
#"Added Custom" = Table.AddColumn(#"Pivoted Column", "Values", each TableFromRecordOfLists(_,PivotColNames)),
#"Removed Other Columns" = Table.RemoveColumns(#"Added Custom",PivotColNames),
#"Expanded Values" = Table.ExpandTableColumn(#"Removed Other Columns", "Values", PivotColNames)
in
#"Expanded Values"
I've searched far and wide and haven't found an answer to this specific case, and wasn't able to adapt some of these solutions.
First of all, my data is a long list of attributes and their values for every product, structured like this:
Structured Initial Data
Note that some products have a single value per attributes, but (and here's my problem) some products have different values for the same attribute.
When I pivot the table in PowerQuery, i get errors where the products have multiple instances of the same attributes.
The resulting table that i'm looking for would be structured like this:
Structured Final Data
Thank you for your help!
See if this works for you
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Sorted Rows" = Table.Sort(Source,{{"Products", Order.Ascending}, {"Attributes", Order.Ascending}}),
#"Grouped Rows" = Table.Group(#"Sorted Rows", {"Products"}, {{"data", each _, type table}}),
#"Added Index1" = Table.AddIndexColumn(#"Grouped Rows", "Index", 0, 1),
#"Expanded data" = Table.ExpandTableColumn(#"Added Index1", "data", {"Attributes", "Values"}, {"Attributes", "Values"}),
mGroup = Table.Group(#"Expanded data" , {"Attributes","Products"}, {{"GRP", each Table.AddIndexColumn(_, "Index2", 1, 1), type table}}),
#"Expanded GRP" = Table.ExpandTableColumn(mGroup, "GRP", {"Values", "Index", "Index2"}, {"Values", "Index", "Index2"}),
#"Added Custom" = Table.AddColumn(#"Expanded GRP", "Row#", each [Index]+[Index2]),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Index", "Index2"}),
#"Pivoted Column" = Table.Pivot(#"Removed Columns", List.Distinct(#"Removed Columns"[Attributes]), "Attributes", "Values"),
#"Removed Columns1" = Table.RemoveColumns(#"Pivoted Column",{"Row#"}),
#"Reordered Columns" = Table.ReorderColumns(#"Removed Columns1",{"Products", "Each", "Pack"})
in #"Reordered Columns"
It groups on product and adds an index. Then it groups on product and Attribute and adds another index. The sum of those two are a unique row number you can use for pivoting
Source Data Table.
Desired Output
We want to Extract all the Unique values from Each Type Columns and Pivot the unique values as Column headers.
Somewhat similar to this but we have more then one columns to look up unique values.
Power Query - Transpose unique values and get matching values in rows
Number of Type columns in the Source table can increase or decrease over time.
The code below is created via standard menu options. This video takes you through the results of each step.
let
Source = SourceData,
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"Key"}, "Attribute", "Value"),
#"Added Custom" = Table.AddColumn(#"Unpivoted Other Columns", "Type", each if Text.Start([Attribute],4) = "Type" then [Value] else null),
#"Filled Down" = Table.FillDown(#"Added Custom",{"Type"}),
#"Filtered Rows" = Table.SelectRows(#"Filled Down", each not Text.StartsWith([Attribute], "Type")),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Attribute"}),
#"Pivoted Column" = Table.Pivot(#"Removed Columns", List.Distinct(#"Removed Columns"[Type]), "Type", "Value")
in
#"Pivoted Column"