Related
I have the following data with duplicates which I wish to identify. I do not wish to remove these so unique value only won't work. I want to be able to identify them but just saying null.
I have attempted to self-reference the code but end up just duplicating the original result.
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}}),
#"Removed Duplicates" = Table.Distinct(#"Changed Type"),
#"Merged Queries" = Table.NestedJoin(Source, {"Column1"}, #"Removed Duplicates", {"Column1"}, "Removed Duplicates", JoinKind.LeftOuter)
in
#"Merged Queries"
You can use List.Generate to generate a list with your requirements. And then you can either replace the first column or add the list as a second column.
This needs to be done in the Advanced Editor.
Please note there is a difference between the text string "null" and the power query null value. Based on your example screenshot, I assumed you wanted the "null" text string. If you prefer the null value, remove the surrounding quotes in the code
M Code
let
//Change next line to reflect your actual data source
Source = Excel.CurrentWorkbook(){[Name="Table13"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}}),
//change 2nd and later duplicates to null
dupsNull = List.Generate(
()=>[v=#"Changed Type"[Column1]{0}, idx=0],
each [idx]<Table.RowCount(#"Changed Type"),
each [v=if List.PositionOf(#"Changed Type"[Column1],#"Changed Type"[Column1]{[idx]+1},Occurrence.First) = [idx]+1
then #"Changed Type"[Column1]{[idx]+1} else "null", idx=[idx]+1],
each [v]),
//either add as a column or replace the first column
#"add de-duped" = Table.FromColumns(
Table.ToColumns(#"Changed Type") & {dupsNull},
type table[Column1=text, Column2=text])
in
#"add de-duped"
Here's another way. First, add an index column. Then add another column using List.PositionOf to get the row of the first occurrence of each value in the column. Then add one last column to compare the index and List.PositionOf, to determine which row entries should be null.
Let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}}),
#"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 0, 1, Int64.Type),
#"Added Custom" = Table.AddColumn(#"Added Index", "Custom", each List.PositionOf(#"Added Index"[Column1],[Column1])),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "Custom.1", each if [Index] = [Custom] then [Column1] else null)
in
#"Added Custom1"
Here a solution that doesn't require to add a new column. It returns the same column just with duplicated values replaced with "null":
let
Source = Excel.CurrentWorkbook(){[Name="TB_INPUT"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}}),
removeDups = (lst as list) =>
List.Accumulate(lst, {}, (x, y) => x & {if List.Contains(x, y) then "null" else y}),
replacedValues = removeDups(Table.Column(#"Changed Type", "Column1")),
#"replaced Values" = Table.FromList(replacedValues, null, type table[Column1 = Text.Type ])
in
#"replaced Values"
it uses a List.Accumulate function to simplify the process to generate the corresponding list with the specified requirements.
The output will be the following in Power Query:
and in Excel:
If you want an empty cell instead of "null" token, then in the function removeDups replace "null" with null.
If you want to consider a more general case, lets say you have more than one column in the input Excel Table and you want to replace duplicated values in more than one column at the same time.
Let's say we have the following input in Excel:
The following code can be used to replace duplicates in Column1 and Column2:
let
Source = Excel.CurrentWorkbook(){[Name="TB_GralCase"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", Int64.Type}}),
listOfColumns = {"Column1", "Column2"},
remainingColumns = List.Difference(Table.ColumnNames(#"Changed Type"), listOfColumns),
removeDups = (lst as list) =>
List.Accumulate(lst, {}, (x, y) => x & {if List.Contains(x, y) then "null" else y}),
replacedValues = List.Transform(listOfColumns, each removeDups(Table.Column( #"Changed Type", _))),
#"replaced values" = Table.FromColumns(
replacedValues & Table.ToColumns(Table.SelectColumns( #"Changed Type", remainingColumns)),
listOfColumns & remainingColumns
)
in
#"replaced values"
In listOfColumns variable, you define the list of columns you want to replace duplicates.
The the output in Power Query will be:
Attempting to split decimal numbers in batch using a prevo=ious formula provided on here however the result is an error stating that null or "" or "x" (where is a number) cant be converted to the type list.
The formula:
=try Text.Remove([Column1],Text.ToList(Text.Remove([Column1],{"0".."9","."}))) otherwise null works when applied to a single column however when trying to create a create a table from these columns I get the followings errors:
Desired Output:
M Code:
let
Source = Excel.CurrentWorkbook(){[Name="Table19"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom", each Table.FromColumns({
(try Text.Remove([Column1],Text.ToList(Text.Remove([Column1],{"0".."9","."}))) otherwise null),
(try Text.Remove([Column2],Text.ToList(Text.Remove([Column2],{"0".."9","."}))) otherwise null)
}))
in
#"Added Custom"
I would like to be able to generate a Table.FromColumns, for n columns which I can then expand. This is just an example and in reality, the number of columns can vary quite a lot.
Update
To better visualise what I am trying to do in power query I wish to create this scenario:
Such that this table can be expanded to:
Probably something obvious but any help appreciated.
I would just
split the columns based on character transition, including the decimal in the character list.
Then Trim the resultant columns to remove any leading/following spaces
Note: Code edited to allow for any number of columns to be split in two. Column names can be dynamic also
let
Source = Excel.CurrentWorkbook(){[Name="Table21"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}, {"Column2", type text}}),
//Generate new table from all the columns
//create List of columns
colList = Table.ToColumns(#"Changed Type"),
colNames = Table.ColumnNames(#"Changed Type"),
//convert each column
splitCols = List.Generate(
()=>[colPair=
List.Transform(colList{0},(li)=>
Splitter.SplitTextByCharacterTransition(
{"0".."9","."}, (c) => not List.Contains({"0".."9","."}, c))
(li)),
cn = colNames{0},
idx=0],
each [idx] < List.Count(colList),
each [colPair=
List.Transform(colList{[idx]+1},(li)=>
Splitter.SplitTextByCharacterTransition(
{"0".."9","."}, (c) => not List.Contains({"0".."9","."}, c))
(li)),
cn=colNames{[idx]+1},
idx=[idx]+1],
each List.Zip([colPair]) & {List.Transform({1..2}, (n)=> [cn] & "." & Text.From(n))}),
newCols = List.Combine(List.Transform(splitCols, each List.RemoveLastN(_,1))),
newColNames = List.Combine(List.Transform(splitCols, each List.Last(_))),
newTable = Table.FromColumns(newCols,newColNames),
//trim the excess spaces
trimOps = List.Transform(Table.ColumnNames(newTable), each {_, Text.Trim}),
trimAll = Table.TransformColumns(newTable, trimOps)
in
trimAll
Example with three columns
Again, if you want to retain the original columns in your result table, you need to change three lines in the code:
...
newCols = Table.ToColumns(#"Changed Type") & List.Combine(List.Transform(splitCols, each List.RemoveLastN(_,1))),
newColNames = Table.ColumnNames(#"Changed Type") & List.Combine(List.Transform(splitCols, each List.Last(_))),
newTable = Table.FromColumns(newCols,newColNames),
...
Edited to be usable for multiple columns
let Source =Excel.CurrentWorkbook(){[Name="Table3"]}[Content],
#"Added Index" = Table.AddIndexColumn(Source, "Index", 0, 1, Int64.Type),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Added Index", {"Index"}, "Attribute", "Value"),
#"Split Column by Delimiter" = Table.SplitColumn(#"Unpivoted Other Columns", "Value", Splitter.SplitTextByEachDelimiter({" "}, QuoteStyle.Csv, false), {"Value.1", "Value.2"}),
#"Removed Columns" = Table.RemoveColumns(#"Split Column by Delimiter",{"Value.2"}),
#"rename1" = Table.TransformColumns(#"Removed Columns",{{"Attribute", each _&"a", type text}}),
#"Pivoted Column" = Table.RemoveColumns(Table.Pivot(#"rename1", List.Distinct(#"Lowercased Text"[Attribute]), "Attribute", "Value.1"),{"Index"}),
#"Removed Columns2" = Table.RemoveColumns(#"Split Column by Delimiter",{"Value.1"}),
rename = Table.TransformColumns(#"Removed Columns2",{{"Attribute", each _ & "b", type text}}),
#"Pivoted Column1" = Table.RemoveColumns(Table.Pivot(rename, List.Distinct(rename[Attribute]), "Attribute", "Value.2"),{"Index"}),
TFC = Table.FromColumns(Table.ToColumns(Source)&Table.ToColumns(#"Pivoted Column")&Table.ToColumns(#"Pivoted Column1"),Table.ColumnNames(Source)&Table.ColumnNames(#"Pivoted Column")&Table.ColumnNames(#"Pivoted Column1"))
in TFC
I would just duplicate the two original columns (Add Column > Duplicate column) and then split the resulting columns on the left most " " delimiter. No M code needed.
bear with me, this is my first attempt using the Power Query Formula Language. I need some advice on how to solve a particular problem sorting and filtering source data.
I now got this current source data, structured like this:
Using this power query:
let
Source = Excel.CurrentWorkbook(){[Name="EmployeeOrganization"]}[Content],
ListEmployees = Table.Group(Source, {"Organization"}, {{"Employee", each Text.Combine([Employee],","), type text}}),
CountEmployees = Table.AddColumn(ListEmployees, "Count", each List.Count(Text.Split([Employee],","))),
SplitEmployees = Table.SplitColumn(ListEmployees, "Employee", Splitter.SplitTextByDelimiter(",", QuoteStyle.Csv),List.Max(CountEmployees[Count])),
Transpose = Table.Transpose(SplitEmployees),
PromoteHeaders = Table.PromoteHeaders(Transpose, [PromoteAllScalars=true])
in
PromoteHeaders
I am able to produce the following result:
To avoid having to add the organization name to every single employee in the source, I would like the organization name to act as an parent-group, with the employees as children. I would also like the result to only fetch the organizations (+ employees) that has status Active = Yes.
The desired source should look similar to this:
So that the desired result should look similar to this: (Apple is gone due to Active = NO)
I am stuck at this point and need some advice on how can I modify my Power Query Formula to:
Only fetch Organizations that are Active (Does not matter if they have employees or not)
Somehow link the children Employees to the correct Organizations. (Without having to write the org name in every adjacent employee column)
(Excel file can be found her)
In PQ, you'll need to fill in the blank rows, then Pivot with no aggregation.
See the comments in the code, and follow the Applied Steps to understand the algorithm
Source
Custom Function
Rename: fnPivotAll
//credit: Cam Wallace https://www.dingbatdata.com/2018/03/08/non-aggregate-pivot-with-multiple-rows-in-powerquery/
(Source as table,
ColToPivot as text,
ColForValues as text)=>
let
PivotColNames = List.Buffer(List.Distinct(Table.Column(Source,ColToPivot))),
#"Pivoted Column" = Table.Pivot(Source, PivotColNames, ColToPivot, ColForValues, each _),
TableFromRecordOfLists = (rec as record, fieldnames as list) =>
let
PartialRecord = Record.SelectFields(rec,fieldnames),
RecordToList = Record.ToList(PartialRecord),
Table = Table.FromColumns(RecordToList,fieldnames)
in
Table,
#"Added Custom" = Table.AddColumn(#"Pivoted Column", "Values", each TableFromRecordOfLists(_,PivotColNames)),
#"Removed Other Columns" = Table.RemoveColumns(#"Added Custom",PivotColNames),
#"Expanded Values" = Table.ExpandTableColumn(#"Removed Other Columns", "Values", PivotColNames)
in
#"Expanded Values"
Basic Query
let
//Read in data and set data types
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45W8k12yc9LzEkpVtJRAqLI1GKlWJ1oEDMgtSS1CCQK5XvlpyLzEvPgXMeCgpxUiH6/fJgC38SiSiT1jjmZyXAN7vn56TAdyDYmluYgaXHKTwLzYgE=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Organization = _t, Employee = _t, Active = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Organization", type text}, {"Employee", type text}, {"Active", type text}}),
//replace blanks with null if not already there
#"Replaced Value" = Table.ReplaceValue(#"Changed Type","",null,Replacer.ReplaceValue,{"Organization", "Employee", "Active"}),
//fill down the Company and active columns
#"Filled Down" = Table.FillDown(#"Replaced Value",{"Organization", "Active"}),
//Filter to show only Active="Yes and Employee not null
#"Filtered Rows" = Table.SelectRows(#"Filled Down", each ([Employee] <> null) and ([Active] = "Yes")),
//Pivot with no aggregation
//could do this with grouping, but easier (and maybe faster, with a custom function
pivotAll = fnPivotAll(#"Filtered Rows","Organization","Employee"),
//remove unneeded Active column and set data types
#"Removed Columns" = Table.RemoveColumns(pivotAll,{"Active"}),
typed = Table.TransformColumnTypes(#"Removed Columns",
List.Transform(Table.ColumnNames(#"Removed Columns"),each {_, Text.Type}))
in
typed
typed Results
I have a spreadsheet that looks like this:
I would like it to look like this at the end:
I know that I can use PowerQuery to delimit and expand into rows, but that would only work on the first column. The second time doing that would introduce lots of duplicates.
Any help?
I'd just do them separately and combine
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Removed Columns" = Table.RemoveColumns(Source,{"People"}),
#"Filtered Rows" = Table.SelectRows(#"Removed Columns", each ([Emails] <> null)),
SCBD = Table.ExpandListColumn(Table.TransformColumns(#"Filtered Rows", {{"Emails", Splitter.SplitTextByDelimiter(";", QuoteStyle.None)}}), "Emails"),
#"Removed Columns1" = Table.RemoveColumns(Source,{"Emails"}),
#"Filtered Rows1" = Table.SelectRows(#"Removed Columns1", each ([People] <> null)),
SCBD1 = Table.ExpandListColumn(Table.TransformColumns(#"Filtered Rows1", {{"People", Splitter.SplitTextByDelimiter(";", QuoteStyle.None)}}), "People"),
Combined = SCBD & SCBD1,
#"Sorted Rows" = Table.Sort(Combined,{{"Name", Order.Ascending}, {"Emails", Order.Descending}})
in #"Sorted Rows"
You lose original order with this approach (you could attempt to re-sort at the end if you wanted).
If at some point you get more columns that need to be similarly split, you can just add to the columnsToSplit list.
let
dataFromSheet = Table.FromColumns({{"Cereal Killers", "Acme Products", "Arkham Asylum"}, {"123 Sugar Way", "345 Whoville Place", "Gotham City"}, {"abc#def.com; xyz#abc.com; jkl#mno.com", null, "the.scarecrow#nolan.com; jokesonyou#joker.com"}, {"Tony Tiger; Toucan Sam; Lucky Leprauchan", "W. Coyote; R. Runner; R. Rabbit", null}}, type table [Name=text, Address=text, Emails=nullable text, People=nullable text]),
columnsToSplit = {"Emails","People"},
loopOverColumnsToSplit = List.Accumulate(columnsToSplit, #table({}, {}), (tableState, currentColumn) =>
let
reduceColumns = Table.SelectColumns(dataFromSheet, {"Name", "Address"} & {currentColumn}),
dropNullRows = Table.SelectRows(reduceColumns, each Record.Field(_, currentColumn) <> null),
splitIntoList = Table.TransformColumns(dropNullRows, {{currentColumn, each Text.Split(_, "; "), type list}}),
expandList = Table.ExpandListColumn(splitIntoList, currentColumn),
appendToAccumulatedTable = tableState & expandList
in appendToAccumulatedTable
)
in
loopOverColumnsToSplit
If preserving order is important, then maybe try approach below (which might take a bit longer as it has a few extra steps).
let
dataFromSheet = Table.FromColumns({{"Cereal Killers", "Acme Products", "Arkham Asylum"}, {"123 Sugar Way", "345 Whoville Place", "Gotham City"}, {"abc#def.com; xyz#abc.com; jkl#mno.com", null, "the.scarecrow#nolan.com; jokesonyou#joker.com"}, {"Tony Tiger; Toucan Sam; Lucky Leprauchan", "W. Coyote; R. Runner; R. Rabbit", null}}, type table [Name=text, Address=text, Emails=nullable text, People=nullable text]),
columnsToSplit = {"Emails","People"},
numberOfColumnsToSplit = List.Count(columnsToSplit),
loopOverColumnsToSplit = List.Accumulate(List.Positions(columnsToSplit), #table({}, {}), (tableState, currentIndex) =>
let
currentColumn = columnsToSplit{currentIndex},
reduceColumns = Table.SelectColumns(dataFromSheet, {"Name", "Address"} & {currentColumn}),
dropNullRows = Table.SelectRows(reduceColumns, each Record.Field(_, currentColumn) <> null),
addIndex = Table.AddIndexColumn(dropNullRows, "toSortBy", currentIndex, numberOfColumnsToSplit),
splitIntoList = Table.TransformColumns(addIndex, {{currentColumn, each Text.Split(_, "; "), type list}}),
expandList = Table.ExpandListColumn(splitIntoList, currentColumn),
appendToAccumulatedTable = tableState & expandList
in appendToAccumulatedTable
),
sorted = Table.Sort(loopOverColumnsToSplit, {"toSortBy", Order.Ascending}),
dropHelperColumn = Table.RemoveColumns(sorted, {"toSortBy"})
in
dropHelperColumn
Just to clarify, if you have a row where the values in Emails and People columns are both null, then that row will not be present in the output table.
I have a table like this that I got using "combine & edit" option in power query that has information from multiple sheets from multiple .xlsx files. Sheetnames never change and they'll stay the same, excel files can change.
Now, I want many tables splitting by column1's value firstkey. So, I can get multiple tables like this,
I have been Googling to find an answer, still no success. There are threads like this, that requires you to duplicate the original table and filter each value.
However, in my case, I want to automate in a way if I have new .xlsx files. So, if I get a value Brooklyn Park instead of Bursville, it should be filtered based on Column1's value.
How can I do this Power Query?
EDIT
As requested, original excel sheet for one file,
M code:
let
Source = Excel_Export,
#"Trimmed Text" = Table.TransformColumns(Source,{{"Column1", Text.Trim, type text}}),
#"Cleaned Text" = Table.TransformColumns(#"Trimmed Text",{{"Column1", Text.Clean, type text}}),
#"Filtered Rows" = Table.SelectRows(#"Cleaned Text", each ([Source.Name] = "Burnsville.xlsx")),
#"Transposed Table" = Table.Transpose(#"Filtered Rows"),
#"Removed Top Rows" = Table.Skip(#"Transposed Table",1),
#"Promoted Headers" = Table.PromoteHeaders(#"Removed Top Rows", [PromoteAllScalars=true]),
#"Renamed Columns" = Table.RenameColumns(#"Promoted Headers",{{"Address", "Address Number"}, {"Column3", "StreetName"}, {"Column4", "City"}})
in
#"Renamed Columns"
I used this code to create a function to automate for each file.
The M code you've posted indicates there being at least 3 columns, but your first image shows only two columns. It also appears to refer to another query (Excel_Export). I was expecting it to show how you achieved the table in the first image, so am not too sure what's going on.
Regarding the insertion of blank rows, you can try the function below.
Code:
fxInsertBlankRows = (tableToTransform as table) =>
let
blankRowToInsert =
let
headers = Table.ColumnNames(tableToTransform),
emptyTable = Table.FromColumns(List.Transform(headers, each {""}), headers),
toListOfRecords = Table.ToRecords(emptyTable)
in
toListOfRecords,
insertionIndexes =
let
isolateColumn = Table.SelectColumns(tableToTransform, {"Column1"}),
indexes = Table.PositionOf(isolateColumn, [Column1="firstKey"], Occurrence.All)
in
indexes,
insertBlankRows = List.Accumulate(insertionIndexes, tableToTransform, (tableState, currentIndex) =>
Table.InsertRows(tableState, currentIndex, blankRowToInsert)
)
in
insertBlankRows,
Say you want to use the above function on the #"Renamed Columns" step in the M code you posted (assuming #"Renamed Columns" is a table, which I'm fairly sure it is). You would change the way the code ends as per below:
#"Renamed Columns" = Table.RenameColumns(#"Promoted Headers",{{"Address", "Address Number"}, {"Column3", "StreetName"}, {"Column4", "City"}})
fxInsertBlankRows = (tableToTransform as table) =>
let
blankRowToInsert =
let
headers = Table.ColumnNames(tableToTransform),
emptyTable = Table.FromColumns(List.Transform(headers, each {""}), headers),
toListOfRecords = Table.ToRecords(emptyTable)
in
toListOfRecords,
insertionIndexes =
let
isolateColumn = Table.SelectColumns(tableToTransform, {"Column1"}),
indexes = Table.PositionOf(isolateColumn, [Column1="firstKey"], Occurrence.All)
in
indexes,
insertBlankRows = List.Accumulate(insertionIndexes, tableToTransform, (tableState, currentIndex) =>
Table.InsertRows(tableState, currentIndex, blankRowToInsert)
)
in
insertBlankRows,
invokeFunction = fxInsertBlankRows(#"Renamed Columns")
in
invokeFunction
Seemed like a fun challenge. Here's a standalone example, I tried to make it succinct:
let
SourceTable = Table.FromRecords({
[Cities = "City1", Info = "Info1"],[Cities = "City1", Info = "Info2"],
[Cities = "City1", Info = "Info3"],[Cities = "City2", Info = "Info1"],
[Cities = "City2", Info = "Info2"],[Cities = "City3", Info = "Info1"],
[Cities = "City3", Info = "Info2"],[Cities = "City3", Info = "Info3"],
[Cities = "City3", Info = "Info4"],[Cities = "City3", Info = "Info5"]
}),
SortedTable = Table.Sort(SourceTable,{{"Cities", Order.Ascending},{"Info", Order.Ascending}}),
DistinctCities = List.Distinct(SortedTable[Cities]),
DistinctCitiesAfterFirst = if List.Count(DistinctCities) > 1 then List.RemoveRange(DistinctCities,0) else {},
CityOffsets = List.Transform(DistinctCitiesAfterFirst, each (List.PositionOf(SortedTable[Cities],_) + List.PositionOf(DistinctCitiesAfterFirst,_) - 1)),
SortedTableWithBlankRows = List.Accumulate(
CityOffsets,
SortedTable,
((tableState, currentOffset) =>
Table.InsertRows(
tableState,
currentOffset,
{
Record.FromList(List.Repeat({""},Table.ColumnCount(SortedTable)),Table.ColumnNames(SortedTable))
}
)
)
)
in
SortedTableWithBlankRows