Power Query - duplicate multiple columns Mcode - excel

I need to duplicate multiple columns with PQ, and yes I can do it manually having either 10 steps or 1 entangled step with 10 duplicate column commands.
while I had similar problem with adding columns I used:
#"Added Custom" =
Table.AddColumn(#"Changed Type", "record", each [
Custom = "DTO",
#"Custom.1" = Date.ToText([Accounting Date],"YYYYMMDD"),
#"Custom.2" = "SA",
#"Custom.3" = "DTO " & Date.ToText([Accounting Date], "yyyy.MM"),
#"Custom.4" = null,
#"Custom.5" = null,
#"Custom.6" = null,
#"Custom.7" = null,
#"Custom.8" = null,
#"Custom.9" = if Text.StartsWith([General Ledger Code],"5") then [Cost Center] else null,
#"Custom.10" = null,
#"Custom.11" = null,
#"Custom.12" = null,
#"Custom.13" = if Text.StartsWith([General Ledger Code], "5") then "IF" else null
]),
#"Expanded record" = Table.ExpandRecordColumn(#"Added Custom", "record", {"Custom", "Custom.1", "Custom.2", "Custom.3", "Custom.4", "Custom.5", "Custom.6", "Custom.7", "Custom.8", "Custom.9", "Custom.10", "Custom.11", "Custom.12", "Custom.13"}, {"Custom", "Custom.1", "Custom.2", "Custom.3", "Custom.4", "Custom.5", "Custom.6", "Custom.7", "Custom.8", "Custom.9", "Custom.10", "Custom.11", "Custom.12", "Custom.13"}),
and it worked for adding columns
however I failed while trying to duplicate this with:
#"Duplicated test" = Table.AddColumn(#"Expanded record", "duplicated", each [
#"Custom.14" = Table.DuplicateColumn(#"Expanded record", "Custom.1", "Custom.1 - Copy"),
#"Custom.15" = Table.DuplicateColumn(#"Expanded record", "Custom.3", "Custom.3 - Copy")
]),
#"Expanded duplicated" = Table.ExpandRecordColumn(#"Duplicated test", "duplicated", {"Custom.14", "Custom.15"}, {"Custom.14", "Custom.15"}),
After expanding columns i received my whole table instead of just 2 additional columns.
Is there a way to simplify duplicating columns like adding columns?

Yes, you can simplify duplicating the columns by just using the Table.AddColumn function as you did in the first step, no need for the Table.DuplicateColumn function:
#"Added Custom1" = Table.AddColumn(#"Expanded record", "duplicated", each [
Custom.14 = [Custom.1],
Custom.15 = [Custom.3]
]),
#"Expanded duplicated" = Table.ExpandRecordColumn(#"Added Custom1", "duplicated", {"Custom.14", "Custom.15"}, {"Custom.14", "Custom.15"})

Alternatively, you can use List.Accumulate to make it like you are passing a simple list of column names to the native Table.DuplicateColumn function like this:
table_source = Table.FromRecords({
[CustomerID = 1, Name = "Bob", Phone = "123-4567"],
[CustomerID = 2, Name = "Jim", Phone = "987-6543"],
[CustomerID = 3, Name = "Paul", Phone = "543-7890"],
[CustomerID = 4, Name = "Ringo", Phone = "232-1550"]
}),
table_duplicated = List.Accumulate({"Name", "Phone"}, table_source, (state as table, current as text) =>
Table.DuplicateColumn(state, current, current & " - Copy")
)
Here's what the output looks like:

Related

Split decimal from text as batch

Attempting to split decimal numbers in batch using a prevo=ious formula provided on here however the result is an error stating that null or "" or "x" (where is a number) cant be converted to the type list.
The formula:
=try Text.Remove([Column1],Text.ToList(Text.Remove([Column1],{"0".."9","."}))) otherwise null works when applied to a single column however when trying to create a create a table from these columns I get the followings errors:
Desired Output:
M Code:
let
Source = Excel.CurrentWorkbook(){[Name="Table19"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each Table.FromColumns({
(try Text.Remove([Column1],Text.ToList(Text.Remove([Column1],{"0".."9","."}))) otherwise null),
(try Text.Remove([Column2],Text.ToList(Text.Remove([Column2],{"0".."9","."}))) otherwise null)
}))
in
#"Added Custom"
I would like to be able to generate a Table.FromColumns, for n columns which I can then expand. This is just an example and in reality, the number of columns can vary quite a lot.
Update
To better visualise what I am trying to do in power query I wish to create this scenario:
Such that this table can be expanded to:
Probably something obvious but any help appreciated.
I would just
split the columns based on character transition, including the decimal in the character list.
Then Trim the resultant columns to remove any leading/following spaces
Note: Code edited to allow for any number of columns to be split in two. Column names can be dynamic also
let
Source = Excel.CurrentWorkbook(){[Name="Table21"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", type text}}),
//Generate new table from all the columns
//create List of columns
colList = Table.ToColumns(#"Changed Type"),
colNames = Table.ColumnNames(#"Changed Type"),
//convert each column
splitCols = List.Generate(
()=>[colPair=
List.Transform(colList{0},(li)=>
Splitter.SplitTextByCharacterTransition(
{"0".."9","."}, (c) => not List.Contains({"0".."9","."}, c))
(li)),
cn = colNames{0},
idx=0],
each [idx] < List.Count(colList),
each [colPair=
List.Transform(colList{[idx]+1},(li)=>
Splitter.SplitTextByCharacterTransition(
{"0".."9","."}, (c) => not List.Contains({"0".."9","."}, c))
(li)),
cn=colNames{[idx]+1},
idx=[idx]+1],
each List.Zip([colPair]) & {List.Transform({1..2}, (n)=> [cn] & "." & Text.From(n))}),
newCols = List.Combine(List.Transform(splitCols, each List.RemoveLastN(_,1))),
newColNames = List.Combine(List.Transform(splitCols, each List.Last(_))),
newTable = Table.FromColumns(newCols,newColNames),
//trim the excess spaces
trimOps = List.Transform(Table.ColumnNames(newTable), each {_, Text.Trim}),
trimAll = Table.TransformColumns(newTable, trimOps)
in
trimAll
Example with three columns
Again, if you want to retain the original columns in your result table, you need to change three lines in the code:
...
newCols = Table.ToColumns(#"Changed Type") & List.Combine(List.Transform(splitCols, each List.RemoveLastN(_,1))),
newColNames = Table.ColumnNames(#"Changed Type") & List.Combine(List.Transform(splitCols, each List.Last(_))),
newTable = Table.FromColumns(newCols,newColNames),
...
Edited to be usable for multiple columns
let Source =Excel.CurrentWorkbook(){[Name="Table3"]}[Content],
#"Added Index" = Table.AddIndexColumn(Source, "Index", 0, 1, Int64.Type),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Added Index", {"Index"}, "Attribute", "Value"),
#"Split Column by Delimiter" = Table.SplitColumn(#"Unpivoted Other Columns", "Value", Splitter.SplitTextByEachDelimiter({" "}, QuoteStyle.Csv, false), {"Value.1", "Value.2"}),
#"Removed Columns" = Table.RemoveColumns(#"Split Column by Delimiter",{"Value.2"}),
#"rename1" = Table.TransformColumns(#"Removed Columns",{{"Attribute", each _&"a", type text}}),
#"Pivoted Column" = Table.RemoveColumns(Table.Pivot(#"rename1", List.Distinct(#"Lowercased Text"[Attribute]), "Attribute", "Value.1"),{"Index"}),
#"Removed Columns2" = Table.RemoveColumns(#"Split Column by Delimiter",{"Value.1"}),
rename = Table.TransformColumns(#"Removed Columns2",{{"Attribute", each _ & "b", type text}}),
#"Pivoted Column1" = Table.RemoveColumns(Table.Pivot(rename, List.Distinct(rename[Attribute]), "Attribute", "Value.2"),{"Index"}),
TFC = Table.FromColumns(Table.ToColumns(Source)&Table.ToColumns(#"Pivoted Column")&Table.ToColumns(#"Pivoted Column1"),Table.ColumnNames(Source)&Table.ColumnNames(#"Pivoted Column")&Table.ColumnNames(#"Pivoted Column1"))
in TFC
I would just duplicate the two original columns (Add Column > Duplicate column) and then split the resulting columns on the left most " " delimiter. No M code needed.

How to specify array element based on a value in Power Query for JSON

I have a JSON source of nested arrays. Each object has a "Key" element. I want to select the element where Key="canada". How would I specify this?
What works:
let
Source = Json.Document(Web.Contents("https://kustom.radio-canada.ca/covid-19")),
Source1 = Source{22},
But it's hardcoded as the 22nd value.
What doesn't work:
Source1 = Source{[Key="canada"]}
This returns:
Expression.Error: We cannot convert a value of type Record to type Number.
Details:
Value=
Key=canada
Type=[Type]
So, how would I specify this?
Try this
let
Source = Json.Document(Web.Contents("https://kustom.radio-canada.ca/covid-19")),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Key", "Api", "Name", "NameFr", "Country", "CountryFr", "State", "StateFr", "Lat", "Long", "Confirmed", "Deaths", "Recovered", "Population", "History", "Regions", "DateUpdate"}, {"Key", "Api", "Name", "NameFr", "Country", "CountryFr", "State", "StateFr", "Lat", "Long", "Confirmed", "Deaths", "Recovered", "Population", "History", "Regions", "DateUpdate"}),
#"Filtered Rows" = Table.SelectRows(#"Expanded Column1", each ([Key] = "canada"))
in
#"Filtered Rows"

How to do a grouped running total with Power Query in Excel?

My question or rather problem, doesn't fall to far from this post.
I've checked these sources to get to where I am with the code below;
link 1
link 2
let
Source = Table.Combine({CITIBank_USDChecking, QNBFinansbank_TRYMevduat}),
#"Sort Rows" = Table.Sort(Source,{{"Date", Order.Ascending}}),
#"Add Custom ""Num"" Col" = Table.AddColumn(#"Sort Rows", "Num", each null),
#"Add Custom ""Payee"" Col" = Table.AddColumn(#"Add Custom ""Num"" Col", "Payee", each null),
#"Add Custom ""Tag"" Col" = Table.AddColumn(#"Add Custom ""Payee"" Col", "Tag", each null),
#"Add Custom ""Category"" Col" = Table.AddColumn(#"Add Custom ""Tag"" Col", "Category", each null),
#"Reorder Cols" = Table.ReorderColumns(#"Add Custom ""Category"" Col",{"Account", "Currency", "Date", "Num", "Payee", "Description", "Tag", "Category", "Clr", "Debit", "Credit"}),
BufferValues = List.Buffer(#"Reorder Cols"[Debit]),
BufferGroup = List.Buffer(#"Reorder Cols"[Account]),
runTotal = Table.FromList (
fxGroupedRunningTotal (BufferValues , BufferGroup) , Splitter.SplitByNothing() , {"runningDebit"}
),
combineColumns = List.Combine (
{ Table.ToColumns (#"Reorder Cols") , Table.ToColumns(runTotal) }
),
#"Convert to Table" = Table.FromColumns (
combineColumns , List.Combine ( { Table.ColumnNames(#"Reorder Cols") , {"runningDebit"} } )
),
#"Removed Columns" = Table.RemoveColumns(#"Convert to Table",{"runningDebit"})
in
#"Removed Columns"
The issue with doing grouped running totals -in my case- is the resetting that occurs once the list hits a value of 0. I don't know how to get around this issue at this point. Basically, I'm trying to get a running total of debits by account, where the debit is a list of decimal numbers and 0 at times. Maybe #alexis-olson can chime in...
This would be the running group sum of Debits based on the sample starting image
Basically, you add an index
Compare the Account to the Account in the row below. If they dont match, copy over the index
Fill upwards. You now have unique groupings for Account
Add a formula for running sum based on that grouping
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added Index" = Table.AddIndexColumn(Source, "Index", 0, 1),
#"Added Custom" = Table.AddColumn(#"Added Index", "Custom", each try if [Account] = #"Added Index"{[Index]+1}[Account] then null else [Index] otherwise [Index]),
#"Filled Up" = Table.FillUp(#"Added Custom",{"Custom"}),
#"Added Custom2" = Table.AddColumn(#"Filled Up","runningDebit",(i)=>List.Sum(Table.SelectRows(#"Filled Up", each [Custom] = i[Custom] and [Index]<=i[Index]) [Debit]), type number)
in #"Added Custom2"

Powerquery to split delimited columns into unique rows

I have a spreadsheet that looks like this:
I would like it to look like this at the end:
I know that I can use PowerQuery to delimit and expand into rows, but that would only work on the first column. The second time doing that would introduce lots of duplicates.
Any help?
I'd just do them separately and combine
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Removed Columns" = Table.RemoveColumns(Source,{"People"}),
#"Filtered Rows" = Table.SelectRows(#"Removed Columns", each ([Emails] <> null)),
SCBD = Table.ExpandListColumn(Table.TransformColumns(#"Filtered Rows", {{"Emails", Splitter.SplitTextByDelimiter(";", QuoteStyle.None)}}), "Emails"),
#"Removed Columns1" = Table.RemoveColumns(Source,{"Emails"}),
#"Filtered Rows1" = Table.SelectRows(#"Removed Columns1", each ([People] <> null)),
SCBD1 = Table.ExpandListColumn(Table.TransformColumns(#"Filtered Rows1", {{"People", Splitter.SplitTextByDelimiter(";", QuoteStyle.None)}}), "People"),
Combined = SCBD & SCBD1,
#"Sorted Rows" = Table.Sort(Combined,{{"Name", Order.Ascending}, {"Emails", Order.Descending}})
in #"Sorted Rows"
You lose original order with this approach (you could attempt to re-sort at the end if you wanted).
If at some point you get more columns that need to be similarly split, you can just add to the columnsToSplit list.
let
dataFromSheet = Table.FromColumns({{"Cereal Killers", "Acme Products", "Arkham Asylum"}, {"123 Sugar Way", "345 Whoville Place", "Gotham City"}, {"abc#def.com; xyz#abc.com; jkl#mno.com", null, "the.scarecrow#nolan.com; jokesonyou#joker.com"}, {"Tony Tiger; Toucan Sam; Lucky Leprauchan", "W. Coyote; R. Runner; R. Rabbit", null}}, type table [Name=text, Address=text, Emails=nullable text, People=nullable text]),
columnsToSplit = {"Emails","People"},
loopOverColumnsToSplit = List.Accumulate(columnsToSplit, #table({}, {}), (tableState, currentColumn) =>
let
reduceColumns = Table.SelectColumns(dataFromSheet, {"Name", "Address"} & {currentColumn}),
dropNullRows = Table.SelectRows(reduceColumns, each Record.Field(_, currentColumn) <> null),
splitIntoList = Table.TransformColumns(dropNullRows, {{currentColumn, each Text.Split(_, "; "), type list}}),
expandList = Table.ExpandListColumn(splitIntoList, currentColumn),
appendToAccumulatedTable = tableState & expandList
in appendToAccumulatedTable
)
in
loopOverColumnsToSplit
If preserving order is important, then maybe try approach below (which might take a bit longer as it has a few extra steps).
let
dataFromSheet = Table.FromColumns({{"Cereal Killers", "Acme Products", "Arkham Asylum"}, {"123 Sugar Way", "345 Whoville Place", "Gotham City"}, {"abc#def.com; xyz#abc.com; jkl#mno.com", null, "the.scarecrow#nolan.com; jokesonyou#joker.com"}, {"Tony Tiger; Toucan Sam; Lucky Leprauchan", "W. Coyote; R. Runner; R. Rabbit", null}}, type table [Name=text, Address=text, Emails=nullable text, People=nullable text]),
columnsToSplit = {"Emails","People"},
numberOfColumnsToSplit = List.Count(columnsToSplit),
loopOverColumnsToSplit = List.Accumulate(List.Positions(columnsToSplit), #table({}, {}), (tableState, currentIndex) =>
let
currentColumn = columnsToSplit{currentIndex},
reduceColumns = Table.SelectColumns(dataFromSheet, {"Name", "Address"} & {currentColumn}),
dropNullRows = Table.SelectRows(reduceColumns, each Record.Field(_, currentColumn) <> null),
addIndex = Table.AddIndexColumn(dropNullRows, "toSortBy", currentIndex, numberOfColumnsToSplit),
splitIntoList = Table.TransformColumns(addIndex, {{currentColumn, each Text.Split(_, "; "), type list}}),
expandList = Table.ExpandListColumn(splitIntoList, currentColumn),
appendToAccumulatedTable = tableState & expandList
in appendToAccumulatedTable
),
sorted = Table.Sort(loopOverColumnsToSplit, {"toSortBy", Order.Ascending}),
dropHelperColumn = Table.RemoveColumns(sorted, {"toSortBy"})
in
dropHelperColumn
Just to clarify, if you have a row where the values in Emails and People columns are both null, then that row will not be present in the output table.

Split a table into multiple smaller tables based on a column's value-Power Query

I have a table like this that I got using "combine & edit" option in power query that has information from multiple sheets from multiple .xlsx files. Sheetnames never change and they'll stay the same, excel files can change.
Now, I want many tables splitting by column1's value firstkey. So, I can get multiple tables like this,
I have been Googling to find an answer, still no success. There are threads like this, that requires you to duplicate the original table and filter each value.
However, in my case, I want to automate in a way if I have new .xlsx files. So, if I get a value Brooklyn Park instead of Bursville, it should be filtered based on Column1's value.
How can I do this Power Query?
EDIT
As requested, original excel sheet for one file,
M code:
let
Source = Excel_Export,
#"Trimmed Text" = Table.TransformColumns(Source,{{"Column1", Text.Trim, type text}}),
#"Cleaned Text" = Table.TransformColumns(#"Trimmed Text",{{"Column1", Text.Clean, type text}}),
#"Filtered Rows" = Table.SelectRows(#"Cleaned Text", each ([Source.Name] = "Burnsville.xlsx")),
#"Transposed Table" = Table.Transpose(#"Filtered Rows"),
#"Removed Top Rows" = Table.Skip(#"Transposed Table",1),
#"Promoted Headers" = Table.PromoteHeaders(#"Removed Top Rows", [PromoteAllScalars=true]),
#"Renamed Columns" = Table.RenameColumns(#"Promoted Headers",{{"Address", "Address Number"}, {"Column3", "StreetName"}, {"Column4", "City"}})
in
#"Renamed Columns"
I used this code to create a function to automate for each file.
The M code you've posted indicates there being at least 3 columns, but your first image shows only two columns. It also appears to refer to another query (Excel_Export). I was expecting it to show how you achieved the table in the first image, so am not too sure what's going on.
Regarding the insertion of blank rows, you can try the function below.
Code:
fxInsertBlankRows = (tableToTransform as table) =>
let
blankRowToInsert =
let
headers = Table.ColumnNames(tableToTransform),
emptyTable = Table.FromColumns(List.Transform(headers, each {""}), headers),
toListOfRecords = Table.ToRecords(emptyTable)
in
toListOfRecords,
insertionIndexes =
let
isolateColumn = Table.SelectColumns(tableToTransform, {"Column1"}),
indexes = Table.PositionOf(isolateColumn, [Column1="firstKey"], Occurrence.All)
in
indexes,
insertBlankRows = List.Accumulate(insertionIndexes, tableToTransform, (tableState, currentIndex) =>
Table.InsertRows(tableState, currentIndex, blankRowToInsert)
)
in
insertBlankRows,
Say you want to use the above function on the #"Renamed Columns" step in the M code you posted (assuming #"Renamed Columns" is a table, which I'm fairly sure it is). You would change the way the code ends as per below:
#"Renamed Columns" = Table.RenameColumns(#"Promoted Headers",{{"Address", "Address Number"}, {"Column3", "StreetName"}, {"Column4", "City"}})
fxInsertBlankRows = (tableToTransform as table) =>
let
blankRowToInsert =
let
headers = Table.ColumnNames(tableToTransform),
emptyTable = Table.FromColumns(List.Transform(headers, each {""}), headers),
toListOfRecords = Table.ToRecords(emptyTable)
in
toListOfRecords,
insertionIndexes =
let
isolateColumn = Table.SelectColumns(tableToTransform, {"Column1"}),
indexes = Table.PositionOf(isolateColumn, [Column1="firstKey"], Occurrence.All)
in
indexes,
insertBlankRows = List.Accumulate(insertionIndexes, tableToTransform, (tableState, currentIndex) =>
Table.InsertRows(tableState, currentIndex, blankRowToInsert)
)
in
insertBlankRows,
invokeFunction = fxInsertBlankRows(#"Renamed Columns")
in
invokeFunction
Seemed like a fun challenge. Here's a standalone example, I tried to make it succinct:
let
SourceTable = Table.FromRecords({
[Cities = "City1", Info = "Info1"],[Cities = "City1", Info = "Info2"],
[Cities = "City1", Info = "Info3"],[Cities = "City2", Info = "Info1"],
[Cities = "City2", Info = "Info2"],[Cities = "City3", Info = "Info1"],
[Cities = "City3", Info = "Info2"],[Cities = "City3", Info = "Info3"],
[Cities = "City3", Info = "Info4"],[Cities = "City3", Info = "Info5"]
}),
SortedTable = Table.Sort(SourceTable,{{"Cities", Order.Ascending},{"Info", Order.Ascending}}),
DistinctCities = List.Distinct(SortedTable[Cities]),
DistinctCitiesAfterFirst = if List.Count(DistinctCities) > 1 then List.RemoveRange(DistinctCities,0) else {},
CityOffsets = List.Transform(DistinctCitiesAfterFirst, each (List.PositionOf(SortedTable[Cities],_) + List.PositionOf(DistinctCitiesAfterFirst,_) - 1)),
SortedTableWithBlankRows = List.Accumulate(
CityOffsets,
SortedTable,
((tableState, currentOffset) =>
Table.InsertRows(
tableState,
currentOffset,
{
Record.FromList(List.Repeat({""},Table.ColumnCount(SortedTable)),Table.ColumnNames(SortedTable))
}
)
)
)
in
SortedTableWithBlankRows

Resources