Apply Power Query to all columns - excel

I have a Power query that finds and replaces values listed in a table that I work through from here Bulk Find And Replace In Power Query
But I need to apply it to All columns.
How to do this without listing all the columns as they are dynamic and keep changing
Thanks
What I have so far
let
Source = Excel.CurrentWorkbook(){[Name="MyData"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Job Title", type text}}),
BulkReplaceStep = fBulkReplace(#"Changed Type", MyFindReplace, {"Job Title","Job Title2"})
in
BulkReplaceStep
The find/replace data table
let
Source = Excel.CurrentWorkbook(){[Name="MyFindReplace"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Find", type text}, {"Replace", type text}})
in
#"Changed Type
Bulkreplace
let BulkReplace = (DataTable as table, FindReplaceTable as table, DataTableColumn as list) =>
let
//Convert the FindReplaceTable to a list using the Table.ToRows function
//so we can reference the list with an index number
FindReplaceList = Table.ToRows(FindReplaceTable),
//Count number of rows in the FindReplaceTable to determine
//how many iterations are needed
Counter = Table.RowCount(FindReplaceTable),
//Define a function to iterate over our list
//with the Table.ReplaceValue function
BulkReplaceValues = (DataTableTemp, n) =>
let
//Replace values using nth item in FindReplaceList
ReplaceTable = Table.ReplaceValue(
DataTableTemp,
//replace null with empty string
if FindReplaceList{n}{0} = null then "" else FindReplaceList{n}{0},
if FindReplaceList{n}{1} = null then "" else FindReplaceList{n}{1},
Replacer.ReplaceText,
DataTableColumn
)
in
//if we are not at the end of the FindReplaceList
//then iterate through Table.ReplaceValue again
if n = Counter - 1
then ReplaceTable
else #BulkReplaceValues(ReplaceTable, n + 1),
//Evaluate the sub-function at the first row
Output = BulkReplaceValues(DataTable, 0)
in
Output
in
BulkReplace

This works
Change this:
BulkReplaceStep = fBulkReplace(#"Changed Type", MyFindReplace, {"Job Title","Job Title2"})
To This:
BulkReplaceStep = fBulkReplace(#"Changed Type", MyFindReplace, Table.ColumnNames(#"Changed Type"))

Related

Restrict transformation to Header row or Row 1 Power Query

I need to find and replace the headers of my Source Table in Power query
I am able to do this with BulkReplace
But this searches the entire table, is there a way to restrict BulkReplace to only the headers, or if not then I can demote the headers and run BulkReplace on just Row 1 of the Source Table
Thank you
sumAppHeads (Find Replace Table)
In my Power Query, I have
BulkReplaceStepHeaders = fBulkReplaceStep(#"Demoted Headers", sumAppHeaders, Table.ColumnNames(#"Demoted Headers")),
let BulkReplace = (DataTable as table, FindReplaceTable as table, DataTableColumn as list) =>
let
//Convert the FindReplaceTable to a list using the Table.ToRows function
//so we can reference the list with an index number
FindReplaceList = Table.ToRows(FindReplaceTable),
//Count number of rows in the FindReplaceTable to determine
//how many iterations are needed
Counter = Table.RowCount(FindReplaceTable),
//Define a function to iterate over our list
//with the Table.ReplaceValue function
BulkReplaceValues = (DataTableTemp, n) =>
let
//Replace values using nth item in FindReplaceList
ReplaceTable = Table.ReplaceValue(
DataTableTemp,
//replace null with empty string in nth item
if FindReplaceList{n}{0} = null then "" else FindReplaceList{n}{0},
if FindReplaceList{n}{1} = null then "" else FindReplaceList{n}{1},
Replacer.ReplaceValue,
DataTableColumn
)
in
//if we are not at the end of the FindReplaceList
//then iterate through Table.ReplaceValue again
if n = Counter - 1
then ReplaceTable
else #BulkReplaceValues(ReplaceTable, n + 1),
//Evaluate the sub-function at the first row
Output = BulkReplaceValues(DataTable, 0)
in
Output
in
BulkReplace
Demote the headers
Transpose the table
Replace the old column names that are now all in Column1
Transpose the table back
Promote the headers
Try this
BulkReplaceStepHeaders = fBulkReplaceStep(Table.FirstN(#"Demoted Headers", 1),sumAppHeaders,Table.ColumnNames(#"Demoted Headers")) & Table.Skip(#"Demoted Headers", 1),
This grabs the column names, merges against the replace table to find new names, then does a rename to use the new names
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Merged Queries" = Table.NestedJoin(Table.FromList(Table.ColumnNames(Source)), {"Column1"}, ReplaceTable, {"Find"}, "Table2", JoinKind.LeftOuter),
#"Expanded Table2" = Table.ExpandTableColumn(#"Merged Queries", "Table2", {"Replace"}, {"Replace"}),
#"NewNames" = Table.AddColumn(#"Expanded Table2", "Custom", each if [Replace]=null then [Column1] else [Replace])[Custom],
#"Rename"=Table.RenameColumns( Source, List.Zip( { Table.ColumnNames( Source ), #"NewNames" } ) )
in #"Rename"

Previous value in the same column Power Bi

If it possible to somehow refer to the previous value in the same column? I know there is option to use it in next column.
But it not really fit for me because I need some logic like this. If (calculation value>check value; yes value; previous value). I got error " A cyclic reference was encountered during evaluation " When I am trying refer back.
IF I understand what you want to do correctly, you can accomplish that with the List.Generate function. You generate a list according to your rules; then combine it with the original table.
M Code
let
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45WMlTSUTJVitWJBpI6SsZgljmQZQJmGQFZlmCWMZBlBGaZAVlmSrGxAA==", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Column1 = _t, Column2 = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", Int64.Type}, {"Column2", Int64.Type}}),
//create conditional adding column
c1 = #"Changed Type"[Column1],
c2 = #"Changed Type"[Column2],
conditionalAdd = List.Generate(
()=>[res= if c1{0} + c2{0} > 10 then c1{0}+c2{0} else 0, idx=0],
each [idx] < List.Count(c1),
each [res=if c1{[idx]+1} + c2{[idx]+1} > 10 then c1{[idx]+1} + c2{[idx]+1} else [res],idx=[idx]+1],
each [res]),
//combine with original table
newTable =
Table.FromColumns(
Table.ToColumns(#"Changed Type") & {conditionalAdd},
Table.ColumnNames(#"Changed Type") & {"Conditional Add"}
)
in
newTable
Source
newTable

Power Query Applying a Function Across Every Column

I am trying to write a query that takes a table and multiplies every number in the table by 100. I've gotten close, but I am having trouble applying it correctly to every column. Below is the code I have so far. The line starting with ReplaceTable is the line I have working for one column, and the line below was my attempt at getting it to work for other columns. I am dealing with a small subset currently, but the real data will potentially have ~100 columns, so I do not want to do this by hand. If there's a better way to do this task, please let me know. I am new to Power Query, so if able please explain my error/the solution so I can learn. Thanks!
let
Source = Excel.CurrentWorkbook(){[Name="Data"]}[Content],
//Organization will always be of type text. The others will be should be numbers, unless user error
#"Changed Type" = Table.TransformColumnTypes(Source, {{"Organization", type text}, {"A", Int64.Type}, {"B", Int64.Type}, {"C", Int64.Type}}),
//function to replace all values in all columns with multiplied values
MultiplyReplace = (DataTable as table, DataTableColumns as list) =>
let
Counter = Table.ColumnCount(DataTable),
ReplaceCol = (DataTableTemp, i) =>
let
colName = {DataTableColumns{i}},
col = Table.Column(DataTableTemp, colName),
//LINE THAT WORKS- want this functionality for ALL columns
ReplaceTable = Table.ReplaceValue(DataTableTemp, each[A], each if [A] is number then [A]*100 else [A], Replacer.ReplaceValue, colName)
//ReplaceTable = Table.ReplaceValue(DataTableTemp, each col, each if col is number then col*100 else col, Replace.ReplaceValue, colName)
in
if i = Counter-1 then ReplaceTable else #ReplaceCol(ReplaceTable, i+1)
in
ReplaceCol(DataTable, 0),
allColumns = Table.ColumnNames(#"Changed Type"),
#"Multiplied Numerics" = MultiplyReplace(#"Changed Type", allColumns)
//#"Restored Type" = Value.ReplaceTypes(#"Multiplied Numerics", #"Changed Type")
in
#"Multiplied Numerics"
The issue involves the scope of the functions and the variables.
With a hard-coded column name (such as [A]), the code is understanding the shorthand to actually mean _[A]. Within a Table.ReplaceValue function, that _ is referencing the current Record or row. However, the col variable is referencing the entire table column. When used in the replacer function, it causes an error. (Unfortunately(?), errors in replacer functions are just ignored with no error message, so issues can be hard to trace.)
In the corrected code, I got rid of the col variable, since it's being determined at the wrong scope level. I changed colName to being text instead of a list, and then used the Record.Field function with _ (the current record within the Table.ReplaceValue function) and the text value colName to extract the desired record for the calculations with the Table.ReplaceValue function itself.
Corrected Code
let
Source = Excel.CurrentWorkbook(){[Name="Data"]}[Content],
//Organization will always be of type text. The others will be should be numbers, unless user error
#"Changed Type" = Table.TransformColumnTypes(Source, {{"Organization", type text}, {"A", Int64.Type}, {"B", Int64.Type}, {"C", Int64.Type}}),
//function to replace all values in all columns with multiplied values
MultiplyReplace = (DataTable as table, DataTableColumns as list) =>
let
Counter = Table.ColumnCount(DataTable),
ReplaceCol = (DataTableTemp, i) =>
let
colName = DataTableColumns{i},
//LINE THAT WORKS- want this functionality for ALL columns
ReplaceTable = Table.ReplaceValue(DataTableTemp,each Record.Field(_, colName), each if Record.Field(_, colName) is number then Record.Field(_, colName)*100 else Record.Field(_, colName),Replacer.ReplaceValue,{colName})
//ReplaceTable = Table.ReplaceValue(DataTableTemp, each col, each if col is number then col*100 else col, Replace.ReplaceValue, colName)
in
if i = Counter-1 then ReplaceTable else #ReplaceCol(ReplaceTable, i+1)
in
ReplaceCol(DataTable, 0),
allColumns = Table.ColumnNames(#"Changed Type"),
#"Multiplied Numerics" = MultiplyReplace(#"Changed Type", allColumns)
//#"Restored Type" = Value.ReplaceTypes(#"Multiplied Numerics", #"Changed Type")
in
#"Multiplied Numerics"

Split a table into multiple smaller tables based on a column's value-Power Query

I have a table like this that I got using "combine & edit" option in power query that has information from multiple sheets from multiple .xlsx files. Sheetnames never change and they'll stay the same, excel files can change.
Now, I want many tables splitting by column1's value firstkey. So, I can get multiple tables like this,
I have been Googling to find an answer, still no success. There are threads like this, that requires you to duplicate the original table and filter each value.
However, in my case, I want to automate in a way if I have new .xlsx files. So, if I get a value Brooklyn Park instead of Bursville, it should be filtered based on Column1's value.
How can I do this Power Query?
EDIT
As requested, original excel sheet for one file,
M code:
let
Source = Excel_Export,
#"Trimmed Text" = Table.TransformColumns(Source,{{"Column1", Text.Trim, type text}}),
#"Cleaned Text" = Table.TransformColumns(#"Trimmed Text",{{"Column1", Text.Clean, type text}}),
#"Filtered Rows" = Table.SelectRows(#"Cleaned Text", each ([Source.Name] = "Burnsville.xlsx")),
#"Transposed Table" = Table.Transpose(#"Filtered Rows"),
#"Removed Top Rows" = Table.Skip(#"Transposed Table",1),
#"Promoted Headers" = Table.PromoteHeaders(#"Removed Top Rows", [PromoteAllScalars=true]),
#"Renamed Columns" = Table.RenameColumns(#"Promoted Headers",{{"Address", "Address Number"}, {"Column3", "StreetName"}, {"Column4", "City"}})
in
#"Renamed Columns"
I used this code to create a function to automate for each file.
The M code you've posted indicates there being at least 3 columns, but your first image shows only two columns. It also appears to refer to another query (Excel_Export). I was expecting it to show how you achieved the table in the first image, so am not too sure what's going on.
Regarding the insertion of blank rows, you can try the function below.
Code:
fxInsertBlankRows = (tableToTransform as table) =>
let
blankRowToInsert =
let
headers = Table.ColumnNames(tableToTransform),
emptyTable = Table.FromColumns(List.Transform(headers, each {""}), headers),
toListOfRecords = Table.ToRecords(emptyTable)
in
toListOfRecords,
insertionIndexes =
let
isolateColumn = Table.SelectColumns(tableToTransform, {"Column1"}),
indexes = Table.PositionOf(isolateColumn, [Column1="firstKey"], Occurrence.All)
in
indexes,
insertBlankRows = List.Accumulate(insertionIndexes, tableToTransform, (tableState, currentIndex) =>
Table.InsertRows(tableState, currentIndex, blankRowToInsert)
)
in
insertBlankRows,
Say you want to use the above function on the #"Renamed Columns" step in the M code you posted (assuming #"Renamed Columns" is a table, which I'm fairly sure it is). You would change the way the code ends as per below:
#"Renamed Columns" = Table.RenameColumns(#"Promoted Headers",{{"Address", "Address Number"}, {"Column3", "StreetName"}, {"Column4", "City"}})
fxInsertBlankRows = (tableToTransform as table) =>
let
blankRowToInsert =
let
headers = Table.ColumnNames(tableToTransform),
emptyTable = Table.FromColumns(List.Transform(headers, each {""}), headers),
toListOfRecords = Table.ToRecords(emptyTable)
in
toListOfRecords,
insertionIndexes =
let
isolateColumn = Table.SelectColumns(tableToTransform, {"Column1"}),
indexes = Table.PositionOf(isolateColumn, [Column1="firstKey"], Occurrence.All)
in
indexes,
insertBlankRows = List.Accumulate(insertionIndexes, tableToTransform, (tableState, currentIndex) =>
Table.InsertRows(tableState, currentIndex, blankRowToInsert)
)
in
insertBlankRows,
invokeFunction = fxInsertBlankRows(#"Renamed Columns")
in
invokeFunction
Seemed like a fun challenge. Here's a standalone example, I tried to make it succinct:
let
SourceTable = Table.FromRecords({
[Cities = "City1", Info = "Info1"],[Cities = "City1", Info = "Info2"],
[Cities = "City1", Info = "Info3"],[Cities = "City2", Info = "Info1"],
[Cities = "City2", Info = "Info2"],[Cities = "City3", Info = "Info1"],
[Cities = "City3", Info = "Info2"],[Cities = "City3", Info = "Info3"],
[Cities = "City3", Info = "Info4"],[Cities = "City3", Info = "Info5"]
}),
SortedTable = Table.Sort(SourceTable,{{"Cities", Order.Ascending},{"Info", Order.Ascending}}),
DistinctCities = List.Distinct(SortedTable[Cities]),
DistinctCitiesAfterFirst = if List.Count(DistinctCities) > 1 then List.RemoveRange(DistinctCities,0) else {},
CityOffsets = List.Transform(DistinctCitiesAfterFirst, each (List.PositionOf(SortedTable[Cities],_) + List.PositionOf(DistinctCitiesAfterFirst,_) - 1)),
SortedTableWithBlankRows = List.Accumulate(
CityOffsets,
SortedTable,
((tableState, currentOffset) =>
Table.InsertRows(
tableState,
currentOffset,
{
Record.FromList(List.Repeat({""},Table.ColumnCount(SortedTable)),Table.ColumnNames(SortedTable))
}
)
)
)
in
SortedTableWithBlankRows

Convert column to cell string Power Query

I need to fit all the values of a column in Power Query into a 1-cell string separated by commas, as the example below:
To do this, I have the following piece of code:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Transposed Table" = Table.Transpose(Source),
#"Merged Columns" = Table.CombineColumns(#"Transposed Table",{"Column1", "Column2", "Column3"},Combiner.CombineTextByDelimiter(",", QuoteStyle.None),"Merged"),
#"KeepString" = #"Merged Columns"[Merged]{0}
in
#"KeepString"
The problem with this code is that it assumes there will always be 3 columns, which is not always the case. How can I merge all columns (regardless of how many there are) into one?
You can do this with List.Accumulate:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
KeepString = List.Accumulate(Source[User], "", (state, current) => if state = "" then current else state & "," & current)
in
KeepString
You can also use Table.ColumnNames to get the list of all the column names. You can pass this into Table.CombineColumns, so your modified solution would be:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Transposed Table" = Table.Transpose(Source),
#"Merged Columns" = Table.CombineColumns(#"Transposed Table", Table.ColumnNames(#"Transposed Table"),Combiner.CombineTextByDelimiter(",", QuoteStyle.None),"Merged"),
#"KeepString" = #"Merged Columns"[Merged]{0}
in
#"KeepString"
You can also use a shorter code, like this:
let
Source=Excel.CurrentWorkbook( {[Name="Table1"]}[Content],
Result = Text.Combine(Source[User], ",")
in
Result

Resources