I need to find and replace the headers of my Source Table in Power query
I am able to do this with BulkReplace
But this searches the entire table, is there a way to restrict BulkReplace to only the headers, or if not then I can demote the headers and run BulkReplace on just Row 1 of the Source Table
Thank you
sumAppHeads (Find Replace Table)
In my Power Query, I have
BulkReplaceStepHeaders = fBulkReplaceStep(#"Demoted Headers", sumAppHeaders, Table.ColumnNames(#"Demoted Headers")),
let BulkReplace = (DataTable as table, FindReplaceTable as table, DataTableColumn as list) =>
let
//Convert the FindReplaceTable to a list using the Table.ToRows function
//so we can reference the list with an index number
FindReplaceList = Table.ToRows(FindReplaceTable),
//Count number of rows in the FindReplaceTable to determine
//how many iterations are needed
Counter = Table.RowCount(FindReplaceTable),
//Define a function to iterate over our list
//with the Table.ReplaceValue function
BulkReplaceValues = (DataTableTemp, n) =>
let
//Replace values using nth item in FindReplaceList
ReplaceTable = Table.ReplaceValue(
DataTableTemp,
//replace null with empty string in nth item
if FindReplaceList{n}{0} = null then "" else FindReplaceList{n}{0},
if FindReplaceList{n}{1} = null then "" else FindReplaceList{n}{1},
Replacer.ReplaceValue,
DataTableColumn
)
in
//if we are not at the end of the FindReplaceList
//then iterate through Table.ReplaceValue again
if n = Counter - 1
then ReplaceTable
else #BulkReplaceValues(ReplaceTable, n + 1),
//Evaluate the sub-function at the first row
Output = BulkReplaceValues(DataTable, 0)
in
Output
in
BulkReplace
Demote the headers
Transpose the table
Replace the old column names that are now all in Column1
Transpose the table back
Promote the headers
Try this
BulkReplaceStepHeaders = fBulkReplaceStep(Table.FirstN(#"Demoted Headers", 1),sumAppHeaders,Table.ColumnNames(#"Demoted Headers")) & Table.Skip(#"Demoted Headers", 1),
This grabs the column names, merges against the replace table to find new names, then does a rename to use the new names
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Merged Queries" = Table.NestedJoin(Table.FromList(Table.ColumnNames(Source)), {"Column1"}, ReplaceTable, {"Find"}, "Table2", JoinKind.LeftOuter),
#"Expanded Table2" = Table.ExpandTableColumn(#"Merged Queries", "Table2", {"Replace"}, {"Replace"}),
#"NewNames" = Table.AddColumn(#"Expanded Table2", "Custom", each if [Replace]=null then [Column1] else [Replace])[Custom],
#"Rename"=Table.RenameColumns( Source, List.Zip( { Table.ColumnNames( Source ), #"NewNames" } ) )
in #"Rename"
Related
I have a Power query that finds and replaces values listed in a table that I work through from here Bulk Find And Replace In Power Query
But I need to apply it to All columns.
How to do this without listing all the columns as they are dynamic and keep changing
Thanks
What I have so far
let
Source = Excel.CurrentWorkbook(){[Name="MyData"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Job Title", type text}}),
BulkReplaceStep = fBulkReplace(#"Changed Type", MyFindReplace, {"Job Title","Job Title2"})
in
BulkReplaceStep
The find/replace data table
let
Source = Excel.CurrentWorkbook(){[Name="MyFindReplace"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Find", type text}, {"Replace", type text}})
in
#"Changed Type
Bulkreplace
let BulkReplace = (DataTable as table, FindReplaceTable as table, DataTableColumn as list) =>
let
//Convert the FindReplaceTable to a list using the Table.ToRows function
//so we can reference the list with an index number
FindReplaceList = Table.ToRows(FindReplaceTable),
//Count number of rows in the FindReplaceTable to determine
//how many iterations are needed
Counter = Table.RowCount(FindReplaceTable),
//Define a function to iterate over our list
//with the Table.ReplaceValue function
BulkReplaceValues = (DataTableTemp, n) =>
let
//Replace values using nth item in FindReplaceList
ReplaceTable = Table.ReplaceValue(
DataTableTemp,
//replace null with empty string
if FindReplaceList{n}{0} = null then "" else FindReplaceList{n}{0},
if FindReplaceList{n}{1} = null then "" else FindReplaceList{n}{1},
Replacer.ReplaceText,
DataTableColumn
)
in
//if we are not at the end of the FindReplaceList
//then iterate through Table.ReplaceValue again
if n = Counter - 1
then ReplaceTable
else #BulkReplaceValues(ReplaceTable, n + 1),
//Evaluate the sub-function at the first row
Output = BulkReplaceValues(DataTable, 0)
in
Output
in
BulkReplace
This works
Change this:
BulkReplaceStep = fBulkReplace(#"Changed Type", MyFindReplace, {"Job Title","Job Title2"})
To This:
BulkReplaceStep = fBulkReplace(#"Changed Type", MyFindReplace, Table.ColumnNames(#"Changed Type"))
I have a column in my table that has some text values (input) which I would like to convert to numbers (output) for each unique text value, so that I can do some regression analysis:
Input
Output
AOP
1
AOS
2
AOS
2
AOS
2
AOP
1
null
0 or null
AOP
1
I initially tried to do this do this with several Transform: Replace Values steps, however I don't know how to:
make this flexible to different numbers of unique values (not hardcode 3 replacements but handle n where n is the number of unique values in input)
repeat this for many columns of my table
avoid looping as far as possible
What's a better approach?
One way is add custom column with below formula, and do that for each column you care to apply it to, using the value of each text character to generate a unique number
= try
List.Accumulate(Text.ToList([Input]), "", (state, current)=>
state&Number.ToText(Character.ToNumber(current), "0000")) otherwise null
this would transform all column's text into unique numbers, replacing the original data:
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
Function = (x) => try List.Accumulate(Text.ToList(x), "", (state, current)=> state&Number.ToText(Character.ToNumber(current), "0000")) otherwise null,
TransformList = List.Transform(Table.ColumnNames(Source), each {_ , Function}),
Output = Table.TransformColumns(Source, TransformList)
in Output
this would transform all column's text into unique numbers, appending the new columns to existing columns:
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
Function = (x) => try List.Accumulate(Text.ToList(x), "", (state, current)=> state&Number.ToText(Character.ToNumber(current), "0000")) otherwise null,
TransformList = List.Transform(Table.ColumnNames(Source), each {_ , Function}),
Output = Table.TransformColumns(Source, TransformList),
Numericals=Table.RenameColumns( Output, List.Zip( { Table.ColumnNames( Output), List.Transform(Table.ColumnNames(Output), each _ &"number") } ) ),
#"Merged Queries" = Table.NestedJoin(Table.AddIndexColumn(Source, "Index", 0, 1),{"Index"},Table.AddIndexColumn(Numericals, "Index2", 0, 1),{"Index2"},"Tabl2",JoinKind.LeftOuter),
#"Expanded Tabl2" = Table.ExpandTableColumn(#"Merged Queries", "Tabl2", Table.ColumnNames( Numericals),Table.ColumnNames( Numericals)),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Tabl2",{"Index"})
in #"Removed Columns"
I am trying to write a query that takes a table and multiplies every number in the table by 100. I've gotten close, but I am having trouble applying it correctly to every column. Below is the code I have so far. The line starting with ReplaceTable is the line I have working for one column, and the line below was my attempt at getting it to work for other columns. I am dealing with a small subset currently, but the real data will potentially have ~100 columns, so I do not want to do this by hand. If there's a better way to do this task, please let me know. I am new to Power Query, so if able please explain my error/the solution so I can learn. Thanks!
let
Source = Excel.CurrentWorkbook(){[Name="Data"]}[Content],
//Organization will always be of type text. The others will be should be numbers, unless user error
#"Changed Type" = Table.TransformColumnTypes(Source, {{"Organization", type text}, {"A", Int64.Type}, {"B", Int64.Type}, {"C", Int64.Type}}),
//function to replace all values in all columns with multiplied values
MultiplyReplace = (DataTable as table, DataTableColumns as list) =>
let
Counter = Table.ColumnCount(DataTable),
ReplaceCol = (DataTableTemp, i) =>
let
colName = {DataTableColumns{i}},
col = Table.Column(DataTableTemp, colName),
//LINE THAT WORKS- want this functionality for ALL columns
ReplaceTable = Table.ReplaceValue(DataTableTemp, each[A], each if [A] is number then [A]*100 else [A], Replacer.ReplaceValue, colName)
//ReplaceTable = Table.ReplaceValue(DataTableTemp, each col, each if col is number then col*100 else col, Replace.ReplaceValue, colName)
in
if i = Counter-1 then ReplaceTable else #ReplaceCol(ReplaceTable, i+1)
in
ReplaceCol(DataTable, 0),
allColumns = Table.ColumnNames(#"Changed Type"),
#"Multiplied Numerics" = MultiplyReplace(#"Changed Type", allColumns)
//#"Restored Type" = Value.ReplaceTypes(#"Multiplied Numerics", #"Changed Type")
in
#"Multiplied Numerics"
The issue involves the scope of the functions and the variables.
With a hard-coded column name (such as [A]), the code is understanding the shorthand to actually mean _[A]. Within a Table.ReplaceValue function, that _ is referencing the current Record or row. However, the col variable is referencing the entire table column. When used in the replacer function, it causes an error. (Unfortunately(?), errors in replacer functions are just ignored with no error message, so issues can be hard to trace.)
In the corrected code, I got rid of the col variable, since it's being determined at the wrong scope level. I changed colName to being text instead of a list, and then used the Record.Field function with _ (the current record within the Table.ReplaceValue function) and the text value colName to extract the desired record for the calculations with the Table.ReplaceValue function itself.
Corrected Code
let
Source = Excel.CurrentWorkbook(){[Name="Data"]}[Content],
//Organization will always be of type text. The others will be should be numbers, unless user error
#"Changed Type" = Table.TransformColumnTypes(Source, {{"Organization", type text}, {"A", Int64.Type}, {"B", Int64.Type}, {"C", Int64.Type}}),
//function to replace all values in all columns with multiplied values
MultiplyReplace = (DataTable as table, DataTableColumns as list) =>
let
Counter = Table.ColumnCount(DataTable),
ReplaceCol = (DataTableTemp, i) =>
let
colName = DataTableColumns{i},
//LINE THAT WORKS- want this functionality for ALL columns
ReplaceTable = Table.ReplaceValue(DataTableTemp,each Record.Field(_, colName), each if Record.Field(_, colName) is number then Record.Field(_, colName)*100 else Record.Field(_, colName),Replacer.ReplaceValue,{colName})
//ReplaceTable = Table.ReplaceValue(DataTableTemp, each col, each if col is number then col*100 else col, Replace.ReplaceValue, colName)
in
if i = Counter-1 then ReplaceTable else #ReplaceCol(ReplaceTable, i+1)
in
ReplaceCol(DataTable, 0),
allColumns = Table.ColumnNames(#"Changed Type"),
#"Multiplied Numerics" = MultiplyReplace(#"Changed Type", allColumns)
//#"Restored Type" = Value.ReplaceTypes(#"Multiplied Numerics", #"Changed Type")
in
#"Multiplied Numerics"
I have a table like this that I got using "combine & edit" option in power query that has information from multiple sheets from multiple .xlsx files. Sheetnames never change and they'll stay the same, excel files can change.
Now, I want many tables splitting by column1's value firstkey. So, I can get multiple tables like this,
I have been Googling to find an answer, still no success. There are threads like this, that requires you to duplicate the original table and filter each value.
However, in my case, I want to automate in a way if I have new .xlsx files. So, if I get a value Brooklyn Park instead of Bursville, it should be filtered based on Column1's value.
How can I do this Power Query?
EDIT
As requested, original excel sheet for one file,
M code:
let
Source = Excel_Export,
#"Trimmed Text" = Table.TransformColumns(Source,{{"Column1", Text.Trim, type text}}),
#"Cleaned Text" = Table.TransformColumns(#"Trimmed Text",{{"Column1", Text.Clean, type text}}),
#"Filtered Rows" = Table.SelectRows(#"Cleaned Text", each ([Source.Name] = "Burnsville.xlsx")),
#"Transposed Table" = Table.Transpose(#"Filtered Rows"),
#"Removed Top Rows" = Table.Skip(#"Transposed Table",1),
#"Promoted Headers" = Table.PromoteHeaders(#"Removed Top Rows", [PromoteAllScalars=true]),
#"Renamed Columns" = Table.RenameColumns(#"Promoted Headers",{{"Address", "Address Number"}, {"Column3", "StreetName"}, {"Column4", "City"}})
in
#"Renamed Columns"
I used this code to create a function to automate for each file.
The M code you've posted indicates there being at least 3 columns, but your first image shows only two columns. It also appears to refer to another query (Excel_Export). I was expecting it to show how you achieved the table in the first image, so am not too sure what's going on.
Regarding the insertion of blank rows, you can try the function below.
Code:
fxInsertBlankRows = (tableToTransform as table) =>
let
blankRowToInsert =
let
headers = Table.ColumnNames(tableToTransform),
emptyTable = Table.FromColumns(List.Transform(headers, each {""}), headers),
toListOfRecords = Table.ToRecords(emptyTable)
in
toListOfRecords,
insertionIndexes =
let
isolateColumn = Table.SelectColumns(tableToTransform, {"Column1"}),
indexes = Table.PositionOf(isolateColumn, [Column1="firstKey"], Occurrence.All)
in
indexes,
insertBlankRows = List.Accumulate(insertionIndexes, tableToTransform, (tableState, currentIndex) =>
Table.InsertRows(tableState, currentIndex, blankRowToInsert)
)
in
insertBlankRows,
Say you want to use the above function on the #"Renamed Columns" step in the M code you posted (assuming #"Renamed Columns" is a table, which I'm fairly sure it is). You would change the way the code ends as per below:
#"Renamed Columns" = Table.RenameColumns(#"Promoted Headers",{{"Address", "Address Number"}, {"Column3", "StreetName"}, {"Column4", "City"}})
fxInsertBlankRows = (tableToTransform as table) =>
let
blankRowToInsert =
let
headers = Table.ColumnNames(tableToTransform),
emptyTable = Table.FromColumns(List.Transform(headers, each {""}), headers),
toListOfRecords = Table.ToRecords(emptyTable)
in
toListOfRecords,
insertionIndexes =
let
isolateColumn = Table.SelectColumns(tableToTransform, {"Column1"}),
indexes = Table.PositionOf(isolateColumn, [Column1="firstKey"], Occurrence.All)
in
indexes,
insertBlankRows = List.Accumulate(insertionIndexes, tableToTransform, (tableState, currentIndex) =>
Table.InsertRows(tableState, currentIndex, blankRowToInsert)
)
in
insertBlankRows,
invokeFunction = fxInsertBlankRows(#"Renamed Columns")
in
invokeFunction
Seemed like a fun challenge. Here's a standalone example, I tried to make it succinct:
let
SourceTable = Table.FromRecords({
[Cities = "City1", Info = "Info1"],[Cities = "City1", Info = "Info2"],
[Cities = "City1", Info = "Info3"],[Cities = "City2", Info = "Info1"],
[Cities = "City2", Info = "Info2"],[Cities = "City3", Info = "Info1"],
[Cities = "City3", Info = "Info2"],[Cities = "City3", Info = "Info3"],
[Cities = "City3", Info = "Info4"],[Cities = "City3", Info = "Info5"]
}),
SortedTable = Table.Sort(SourceTable,{{"Cities", Order.Ascending},{"Info", Order.Ascending}}),
DistinctCities = List.Distinct(SortedTable[Cities]),
DistinctCitiesAfterFirst = if List.Count(DistinctCities) > 1 then List.RemoveRange(DistinctCities,0) else {},
CityOffsets = List.Transform(DistinctCitiesAfterFirst, each (List.PositionOf(SortedTable[Cities],_) + List.PositionOf(DistinctCitiesAfterFirst,_) - 1)),
SortedTableWithBlankRows = List.Accumulate(
CityOffsets,
SortedTable,
((tableState, currentOffset) =>
Table.InsertRows(
tableState,
currentOffset,
{
Record.FromList(List.Repeat({""},Table.ColumnCount(SortedTable)),Table.ColumnNames(SortedTable))
}
)
)
)
in
SortedTableWithBlankRows
I need to fit all the values of a column in Power Query into a 1-cell string separated by commas, as the example below:
To do this, I have the following piece of code:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Transposed Table" = Table.Transpose(Source),
#"Merged Columns" = Table.CombineColumns(#"Transposed Table",{"Column1", "Column2", "Column3"},Combiner.CombineTextByDelimiter(",", QuoteStyle.None),"Merged"),
#"KeepString" = #"Merged Columns"[Merged]{0}
in
#"KeepString"
The problem with this code is that it assumes there will always be 3 columns, which is not always the case. How can I merge all columns (regardless of how many there are) into one?
You can do this with List.Accumulate:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
KeepString = List.Accumulate(Source[User], "", (state, current) => if state = "" then current else state & "," & current)
in
KeepString
You can also use Table.ColumnNames to get the list of all the column names. You can pass this into Table.CombineColumns, so your modified solution would be:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Transposed Table" = Table.Transpose(Source),
#"Merged Columns" = Table.CombineColumns(#"Transposed Table", Table.ColumnNames(#"Transposed Table"),Combiner.CombineTextByDelimiter(",", QuoteStyle.None),"Merged"),
#"KeepString" = #"Merged Columns"[Merged]{0}
in
#"KeepString"
You can also use a shorter code, like this:
let
Source=Excel.CurrentWorkbook( {[Name="Table1"]}[Content],
Result = Text.Combine(Source[User], ",")
in
Result