After a week of research and many attempts, I kindly ask for help, please.
I have got the code works so close yet it's causing missed calculations.
I am trying to perform grouped running total with an if statement that evaluates if the next sum will result in a value greater than the value calculated with the previous total, and if does, add those calculations and then continue evaluating the next steps...
Here is my Code
let
Source = Excel.CurrentWorkbook(){[Name="SourceTable"]}[Content],
#"Added Index" = Table.AddIndexColumn(Source, "Index", 0, 1,
Int64.Type),
CorrectTypes = Table.TransformColumnTypes(#"Added Index",{{"Index",
Int64.Type}, {"Filter", type text}, {"Volume", type number}}),
each List.Sum(Table.SelectRows(CorrectTypes, (Q) => Q[Filter] =
[Filter] and Q[Index] <= [Index])[Volume]), type number),
Runing = Table.Group(CorrectTypes,"Filter",{"A", each let
A = Table.AddIndexColumn(_,"i")
in Table.AddColumn(A,"R", each
List.Accumulate(Table.SelectRows(A, (a)=> a[i]<=[i])[Volume],
[Running=0, Verifier = 1],
// Here the challenge begins
(s,l)=> [Running = if s[Running]+l >
(Number.RoundUp(s[Running]+[Volume]/[Cube])*[Cube]) then
((Number.RoundUp(s[Running]/[Cube])*[Cube])-s[Running])+s[Running]+
[Volume] else s[Running]+l , Verifier =Number.From(s[Running]+l
<=Number.RoundUp(s[Running]/[Cube])*[Cube] )] ))
}),
ExpandedR = Table.ExpandRecordColumn(
Table.ExpandTableColumn(Runing, "A", {"Volume","Cube","i", "R"}),
"R", {"Running", "Verifier"})
in
ExpandedR
To Explain the question best here is the Excel File demonstrating desired calculation.
Query Results And Desire Outcome through Excel Formulas
What is a super easy step in excel formulas, So far very challenging in the M language?
I have used many great examples from the web yet none answer the complexity of my query.
The data self contains only 3 columns:
Filter -to do the grouping,
Volume - to sum up,
cube - to use for exception calculations,
Here are the links to resources I have used through my tryouts
Conditional running total in Power Query
[POWER QUERY] Grouped Running Totals with a maximum condition and a verifier
Running Total Power Query with Treshold
Power Query All Over Running Totals
Power Query Running Total with Grouping
Link to Solussion
Here are Two Codes which resolve my problem thanks to chaps on PowerQueryForum... link below
The Function
fxCalc
(A)=>
let
RunningTotal = Table.AddColumn(
A,
"Running Total",
each
if [Volume] <
(Number.RoundUp(List.Sum(List.InsertRange(List.FirstN(A[Volume],
[Index]),0,{0})) / [Cube], 0) * [Cube])
-
List.Sum(List.InsertRange(List.FirstN(A[Volume],[Index]),0,{0}))
then #RunningTotal[Running Total]
{[Index]-1} + [Volume]
else List.Sum(List.FirstN(A[Volume],
[Index]+1))
+
((Number.RoundUp(List.Sum(List.InsertRange(List.FirstN(A[Volume],
[Index]),0,{0})) / [Cube], 0) * [Cube])
-
List.Sum(List.InsertRange(List.FirstN(A[Volume],[Index]),0,
{0})))
),
CubeFill = Table.AddColumn(
RunningTotal,
"Cube Fill",
each [Running Total]/[Cube]
),
PositionCount = Table.AddColumn(
CubeFill,
"Position Count",
each Number.RoundUp([Cube Fill],0)
),
RemainingSpace = Table.AddColumn(
PositionCount,
"Remaining Space",
each [Position Count] * [Cube] - [Running
Total]
)
in
RemainingSpace
Main Query
let
Source =
Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText
("i45Wc
lTSUTLQMzACU4amSrE6cDFTLGLmRKqzRBZzwmKHExa9TljscMYpZmKKRR3C3lgA",
BinaryEncoding.Base64)
, Compression.Deflate)), let _t = ((type
nullable text) meta [Serialized.Text = true]) in type table
[Filter = _t, Volume = _t, Cube = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Filter",
type text}, {"Volume", type number}, {"Cube", type number}}),
#"Grouped Rows" = Table.Group(#"Changed Type", {"Filter"},
{{"Group", each fxCalc(Table.AddIndexColumn(_,"Index",0,1))}}),
Combine = Table.Combine(#"Grouped Rows"[Group])
in
Combine
Option 2
let
Source =
Table.FromRows(Json.Document(Binary.Decompress
(Binary.FromText("i45WclTSUTLQMzACU4amQMpQKVYHLm6KEDdCFjdHiBvjUG+
CLG6JEDcFizth2GuGLI5kjjmyOJK9FmBxZwxxSyRxEyRzDA2QNSA5yBDo41gA",
BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type
nullable text) meta [Serialized.Text = true]) in type table [Filter
= _t, Volume = _t, Cube = _t, Index = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Filter",
type text}, {"Volume", type number}, {"Cube", type number},
{"Index", type text}}),
Aggregate = List.Accumulate(Table.ToRecords(#"Changed Type"), {},
(a,n)=> a & {
Record.AddField(Record.AddField(n, "CubeFill", n[Volume] + (if
List.IsEmpty(a) or List.Last(a)[CubeFill] + n[Volume] > n[Cube] or
List.Last(a)[Filter] <> n[Filter] then 0 else List.Last(a)
[CubeFill])),
"PositionCount", if List.IsEmpty(a) or List.Last(a)[Filter] <>
n[Filter] then 1 else if List.Last(a)[CubeFill] + n[Volume] >
n[Cube] then List.Last(a)
[PositionCount]+Number.RoundUp(n[Volume]/n[Cube], 0) else
List.Last(a)[PositionCount])}),
Output = Table.FromRecords(Aggregate)
in
Output
Related
source link
Hi, so I have 2 tables (need and supply) like this:
I am trying to add a custom column on the need table where for each item I want to retrieve the appropriate supply date based on the following condition (pseudocode) :
if
supply(Qty) >= need(Qty) and (supply(Supply date) <> null and |supply(Supply date) - need(Date)| < 31 days)
then supply(Supply date)
else
if
supply(Supply date) = null
then "NO"
else
"NON2"
Here's what I started doing:
x = Table.Column(source, Table.SelectRows(supply, each supply([Qty]) >= need([Qty]) and (supply[Supply date] <> null and ((supply([Supply date]) =< Date.AddMonths(need([Date]),1) or (supply([Supply date]) >= Date.AddMonths(need([Date]),-1)) )),Supply([date]),
if x <> null then x else "NO2"
Obviously I don't get what I want, that's why I come here asking for your help. Thx
Supply is a table. How can I apply Supply(Supply date) = null then "NO" to an entire table?
That said, see if this helps at all
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Item", type any}, {"Date", type datetime}, {"Qty", Int64.Type}}),
#"Merged Queries" = Table.NestedJoin(#"Changed Type", {"Item"}, supply, {"item"}, "supply", JoinKind.LeftOuter),
#"Added Custom" = Table.AddColumn(#"Merged Queries", "Custom", each
let ThisDate=[Date], ThisQty=[Qty] in try Table.SelectRows( [supply], each [Qty]>ThisQty and [Supply date]<>null
and Number.From([Supply date]) - Number.From(ThisDate)<31)[Supply date]{0} otherwise "NON")
in #"Added Custom"
Using PowerQuery I import data from a CSV that looks like this:
Report Title,,,,,
,Date,Type,USER_ID,PICKED_QTY,No of Hours
,31/10/2021,Type A,User_1,300,3
,31/10/2021,Type A,User_3,250,8
,01/11/2021,Type B,User_1,167,5
,01/11/2021,Type C,User_2,988,2
,02/11/2021,Type A,User_1,1113,4
Date,Type,USER_ID,PICKED_QTY,No of Hours,
03/11/2021,Type C,User_1,1500,5,
04/11/2021,Type A,User_1,200,8,
sometimes it looks like this (no second page) - which is where the problem is:
Report Title,,,,,
,Date,Type,USER_ID,PICKED_QTY,No of Hours
,31/10/2021,Type A,User_1,300,3
,31/10/2021,Type A,User_3,250,8
,01/11/2021,Type B,User_1,167,5
,01/11/2021,Type C,User_2,988,2
,02/11/2021,Type A,User_1,1113,4
I get the data into a readable format using this PQ (this source would be different, but references a table here for simplicity):
DataSource:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
SplitData = Table.SplitColumn(Source,"Column1", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv))
in
SplitData
I then use two queries to line the data up, so the Date columns are both in the same column, etc.
Query1:
let
Source = DataSource,
RemoveTopRows = Table.Skip(Source,1),
PromoteHeaders = Table.PromoteHeaders(RemoveTopRows, [PromoteAllScalars=true]),
FilterRows = Table.SelectRows(PromoteHeaders, each ([#""] = "")),
RemoveOtherColumns = Table.SelectColumns(FilterRows,{"Date", "Type", "USER_ID", "PICKED_QTY", "No of Hours"}),
ChangeType = Table.TransformColumnTypes(RemoveOtherColumns,{{"Date", type date}, {"Type", type text},
{"USER_ID", type text}, {"PICKED_QTY", Int64.Type},
{"No of Hours", Int64.Type}})
in
ChangeType
Query2:
let
Source = DataSource,
RemoveTopRows = Table.Skip(Source,1),
FilterRows = Table.SelectRows(RemoveTopRows, each ([Column1.1] <> "")),
PromoteHeaders = Table.PromoteHeaders(FilterRows, [PromoteAllScalars=true]),
RemoveOtherColumns = Table.SelectColumns(PromoteHeaders,{"Date", "Type", "USER_ID", "PICKED_QTY", "No of Hours"}),
FilterRows2 = Table.SelectRows(RemoveOtherColumns, each ([Date] <> "Date")),
ChangeType = Table.TransformColumnTypes(FilterRows2,{{"Date", type date}, {"Type", type text}, {"USER_ID", type text},
{"PICKED_QTY", Int64.Type}, {"No of Hours", Int64.Type}})
in
ChangeType
Finally, I join the previous two queries together and group to get my final table.
Query3:
let
Source = Query1,
AppendQueries = Table.Combine({Source, Query2}),
SortRows = Table.Sort(AppendQueries,{{"Date", Order.Ascending}}),
GroupRows = Table.Group(SortRows, {"Date", "Type"}, {{"Picked Qty", each List.Sum([PICKED_QTY]), type nullable number},
{"Total Hours", each List.Sum([No of Hours]), type nullable number}}),
AddDivision = Table.AddColumn(GroupRows, "Rate", each [Picked Qty] / [Total Hours], type number)
in
AddDivision
Question
Sometimes my raw data doesn't include a second page of data, so there's no need for Query2.
When this happens, if I don't manually add the headers for the second page I get an error: [Expression Error] The column 'Date' of the table wasn't found.
How do I avoid this? The error appears in Query2 with RemoveOtherColumns - without column headers it can't find the correct column, and in Query3 as it can't append a query that's returning an error.
Without re-writing all of it, you could just change the last line of Query2 to be
in try ChangeType otherwise Table.FromRecords({[Date = null, Type = null, USER_ID=null, PICKED_QTY=null, No of Hours = null]})
or
in try ChangeType otherwise Table.Skip(Table.FromRecords({[Date = null, Type = null, USER_ID=null, PICKED_QTY=null, No of Hours = null]}),1)
creating:
or just do the whole thing in one query
let Source = Csv.Document(File.Contents("C:\temp2\data.csv"),[Delimiter=",", Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Removed Top Rows" = Table.Skip(Source,1),
#"Filtered Rows" = Table.PromoteHeaders(Table.SelectRows(#"Removed Top Rows", each [Column1] = ""), [PromoteAllScalars=true]),
#"Filtered Rows2" = Table.PromoteHeaders(Table.SelectRows(#"Removed Top Rows", each [Column1] <> ""), [PromoteAllScalars=true]),
AppendQueries = Table.Combine({#"Filtered Rows",#"Filtered Rows2"}),
SortRows = Table.Sort(AppendQueries,{{"Date", Order.Ascending}}),
#"Changed Type1" = Table.TransformColumnTypes(SortRows,{{"PICKED_QTY", type number}, {"No of Hours", type number}}),
GroupRows = Table.Group(#"Changed Type1", {"Date", "Type"}, {{"Picked Qty", each List.Sum([PICKED_QTY]), type number}, {"Total Hours", each List.Sum([No of Hours]), type number}}),
AddDivision = Table.AddColumn(GroupRows, "Rate", each [Picked Qty] / [Total Hours], type number)
in AddDivision
If it possible to somehow refer to the previous value in the same column? I know there is option to use it in next column.
But it not really fit for me because I need some logic like this. If (calculation value>check value; yes value; previous value). I got error " A cyclic reference was encountered during evaluation " When I am trying refer back.
IF I understand what you want to do correctly, you can accomplish that with the List.Generate function. You generate a list according to your rules; then combine it with the original table.
M Code
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUTJVitWJBpI6SsZgljmQZQJmGQFZlmCWMZBlBGaZAVlmSrGxAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", Int64.Type}, {"Column2", Int64.Type}}),
//create conditional adding column
c1 = #"Changed Type"[Column1],
c2 = #"Changed Type"[Column2],
conditionalAdd = List.Generate(
()=>[res= if c1{0} + c2{0} > 10 then c1{0}+c2{0} else 0, idx=0],
each [idx] < List.Count(c1),
each [res=if c1{[idx]+1} + c2{[idx]+1} > 10 then c1{[idx]+1} + c2{[idx]+1} else [res],idx=[idx]+1],
each [res]),
//combine with original table
newTable =
Table.FromColumns(
Table.ToColumns(#"Changed Type") & {conditionalAdd},
Table.ColumnNames(#"Changed Type") & {"Conditional Add"}
)
in
newTable
Source
newTable
bear with me, this is my first attempt using the Power Query Formula Language. I need some advice on how to solve a particular problem sorting and filtering source data.
I now got this current source data, structured like this:
Using this power query:
let
Source = Excel.CurrentWorkbook(){[Name="EmployeeOrganization"]}[Content],
ListEmployees = Table.Group(Source, {"Organization"}, {{"Employee", each Text.Combine([Employee],","), type text}}),
CountEmployees = Table.AddColumn(ListEmployees, "Count", each List.Count(Text.Split([Employee],","))),
SplitEmployees = Table.SplitColumn(ListEmployees, "Employee", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv),List.Max(CountEmployees[Count])),
Transpose = Table.Transpose(SplitEmployees),
PromoteHeaders = Table.PromoteHeaders(Transpose, [PromoteAllScalars=true])
in
PromoteHeaders
I am able to produce the following result:
To avoid having to add the organization name to every single employee in the source, I would like the organization name to act as an parent-group, with the employees as children. I would also like the result to only fetch the organizations (+ employees) that has status Active = Yes.
The desired source should look similar to this:
So that the desired result should look similar to this: (Apple is gone due to Active = NO)
I am stuck at this point and need some advice on how can I modify my Power Query Formula to:
Only fetch Organizations that are Active (Does not matter if they have employees or not)
Somehow link the children Employees to the correct Organizations. (Without having to write the org name in every adjacent employee column)
(Excel file can be found her)
In PQ, you'll need to fill in the blank rows, then Pivot with no aggregation.
See the comments in the code, and follow the Applied Steps to understand the algorithm
Source
Custom Function
Rename: fnPivotAll
//credit: Cam Wallace https://www.dingbatdata.com/2018/03/08/non-aggregate-pivot-with-multiple-rows-in-powerquery/
(Source as table,
ColToPivot as text,
ColForValues as text)=>
let
PivotColNames = List.Buffer(List.Distinct(Table.Column(Source,ColToPivot))),
#"Pivoted Column" = Table.Pivot(Source, PivotColNames, ColToPivot, ColForValues, each _),
TableFromRecordOfLists = (rec as record, fieldnames as list) =>
let
PartialRecord = Record.SelectFields(rec,fieldnames),
RecordToList = Record.ToList(PartialRecord),
Table = Table.FromColumns(RecordToList,fieldnames)
in
Table,
#"Added Custom" = Table.AddColumn(#"Pivoted Column", "Values", each TableFromRecordOfLists(_,PivotColNames)),
#"Removed Other Columns" = Table.RemoveColumns(#"Added Custom",PivotColNames),
#"Expanded Values" = Table.ExpandTableColumn(#"Removed Other Columns", "Values", PivotColNames)
in
#"Expanded Values"
Basic Query
let
//Read in data and set data types
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45W8k12yc9LzEkpVtJRAqLI1GKlWJ1oEDMgtSS1CCQK5XvlpyLzEvPgXMeCgpxUiH6/fJgC38SiSiT1jjmZyXAN7vn56TAdyDYmluYgaXHKTwLzYgE=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Organization = _t, Employee = _t, Active = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Organization", type text}, {"Employee", type text}, {"Active", type text}}),
//replace blanks with null if not already there
#"Replaced Value" = Table.ReplaceValue(#"Changed Type","",null,Replacer.ReplaceValue,{"Organization", "Employee", "Active"}),
//fill down the Company and active columns
#"Filled Down" = Table.FillDown(#"Replaced Value",{"Organization", "Active"}),
//Filter to show only Active="Yes and Employee not null
#"Filtered Rows" = Table.SelectRows(#"Filled Down", each ([Employee] <> null) and ([Active] = "Yes")),
//Pivot with no aggregation
//could do this with grouping, but easier (and maybe faster, with a custom function
pivotAll = fnPivotAll(#"Filtered Rows","Organization","Employee"),
//remove unneeded Active column and set data types
#"Removed Columns" = Table.RemoveColumns(pivotAll,{"Active"}),
typed = Table.TransformColumnTypes(#"Removed Columns",
List.Transform(Table.ColumnNames(#"Removed Columns"),each {_, Text.Type}))
in
typed
typed Results
I am trying to write a query that takes a table and multiplies every number in the table by 100. I've gotten close, but I am having trouble applying it correctly to every column. Below is the code I have so far. The line starting with ReplaceTable is the line I have working for one column, and the line below was my attempt at getting it to work for other columns. I am dealing with a small subset currently, but the real data will potentially have ~100 columns, so I do not want to do this by hand. If there's a better way to do this task, please let me know. I am new to Power Query, so if able please explain my error/the solution so I can learn. Thanks!
let
Source = Excel.CurrentWorkbook(){[Name="Data"]}[Content],
//Organization will always be of type text. The others will be should be numbers, unless user error
#"Changed Type" = Table.TransformColumnTypes(Source, {{"Organization", type text}, {"A", Int64.Type}, {"B", Int64.Type}, {"C", Int64.Type}}),
//function to replace all values in all columns with multiplied values
MultiplyReplace = (DataTable as table, DataTableColumns as list) =>
let
Counter = Table.ColumnCount(DataTable),
ReplaceCol = (DataTableTemp, i) =>
let
colName = {DataTableColumns{i}},
col = Table.Column(DataTableTemp, colName),
//LINE THAT WORKS- want this functionality for ALL columns
ReplaceTable = Table.ReplaceValue(DataTableTemp, each[A], each if [A] is number then [A]*100 else [A], Replacer.ReplaceValue, colName)
//ReplaceTable = Table.ReplaceValue(DataTableTemp, each col, each if col is number then col*100 else col, Replace.ReplaceValue, colName)
in
if i = Counter-1 then ReplaceTable else #ReplaceCol(ReplaceTable, i+1)
in
ReplaceCol(DataTable, 0),
allColumns = Table.ColumnNames(#"Changed Type"),
#"Multiplied Numerics" = MultiplyReplace(#"Changed Type", allColumns)
//#"Restored Type" = Value.ReplaceTypes(#"Multiplied Numerics", #"Changed Type")
in
#"Multiplied Numerics"
The issue involves the scope of the functions and the variables.
With a hard-coded column name (such as [A]), the code is understanding the shorthand to actually mean _[A]. Within a Table.ReplaceValue function, that _ is referencing the current Record or row. However, the col variable is referencing the entire table column. When used in the replacer function, it causes an error. (Unfortunately(?), errors in replacer functions are just ignored with no error message, so issues can be hard to trace.)
In the corrected code, I got rid of the col variable, since it's being determined at the wrong scope level. I changed colName to being text instead of a list, and then used the Record.Field function with _ (the current record within the Table.ReplaceValue function) and the text value colName to extract the desired record for the calculations with the Table.ReplaceValue function itself.
Corrected Code
let
Source = Excel.CurrentWorkbook(){[Name="Data"]}[Content],
//Organization will always be of type text. The others will be should be numbers, unless user error
#"Changed Type" = Table.TransformColumnTypes(Source, {{"Organization", type text}, {"A", Int64.Type}, {"B", Int64.Type}, {"C", Int64.Type}}),
//function to replace all values in all columns with multiplied values
MultiplyReplace = (DataTable as table, DataTableColumns as list) =>
let
Counter = Table.ColumnCount(DataTable),
ReplaceCol = (DataTableTemp, i) =>
let
colName = DataTableColumns{i},
//LINE THAT WORKS- want this functionality for ALL columns
ReplaceTable = Table.ReplaceValue(DataTableTemp,each Record.Field(_, colName), each if Record.Field(_, colName) is number then Record.Field(_, colName)*100 else Record.Field(_, colName),Replacer.ReplaceValue,{colName})
//ReplaceTable = Table.ReplaceValue(DataTableTemp, each col, each if col is number then col*100 else col, Replace.ReplaceValue, colName)
in
if i = Counter-1 then ReplaceTable else #ReplaceCol(ReplaceTable, i+1)
in
ReplaceCol(DataTable, 0),
allColumns = Table.ColumnNames(#"Changed Type"),
#"Multiplied Numerics" = MultiplyReplace(#"Changed Type", allColumns)
//#"Restored Type" = Value.ReplaceTypes(#"Multiplied Numerics", #"Changed Type")
in
#"Multiplied Numerics"