Related
I need to get these project description rows merged into a single row so that there will be consistency in the number of a rows per record so that I can transpose them into proper columns through Power Query. (see image) I understand how to execute a transpose w/ Power Query if the number of rows are consistent across records but I cannot figure out how to do this if the number of rows differ. The data comes from a PDF which is horribly formatted and breaks the Project Description information in to separate rows. < THAT IS THE KEY PROBLEM. Apart from that the rest is cake. See snippet to see what I mean.
Each transposed record will have seven columns:
Director Analysis
Address
Project
Area
Notice Date
Project Description
Appeal
I can get everything I need including the headers. I just can't figure out how to merge the rows under Project Description so that I can proceed w/ the transpose.
here is the link to view a screenshot of my sheet
This is a kludge but seems to work. Assumes the column we want to operate on is named column a in powerquery
It looks for anything between the rows that contain Project Description and Appeals must be
Create a shifted row, so we can see what is on the row above
Add index
Use custom columns to determine which rows need filtering out, and which rows are the start and end rows to combine based on the first column and the shifted first column
Merge text together based on that info, merge that back into original table, then remove the extra rows
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
// create shifted row
shiftedList = {null} & List.RemoveLastN(Source[a],1),
custom3 = Table.ToColumns(Source) & {shiftedList},
custom4 = Table.FromColumns(custom3,Table.ColumnNames(Source) & {"Next Row Header"}),
#"Added Index" = Table.AddIndexColumn(custom4, "Index", 0, 1, Int64.Type),
#"Added Custom" = Table.AddColumn(#"Added Index", "Custom", each try if Text.Contains([Next Row Header],"Project Description" ) then [Index] else if Text.Contains([a],"Appeals must be") then [Index] else null otherwise 0),
#"Filled Down" = Table.FillDown(#"Added Custom",{"Custom"}),
#"Added Custom1" = Table.AddColumn(#"Filled Down", "Custom.1", each try if Text.Contains([Next Row Header],"Project Desc") then "remove" else if Text.Contains([a],"Appeals must be") then "keep" else null otherwise "keep"),
#"Filled Down1" = Table.FillDown(#"Added Custom1",{"Custom.1"}),
#"Filtered Rows1" = Table.SelectRows(#"Filled Down1", each ([Custom.1] = "remove")),
#"Grouped Rows1" = Table.Group(#"Filtered Rows1", {"Custom"}, {{"Count", each Text.Combine(List.Transform([a], Text.From), ","), type text}}),
#"Merged Queries" = Table.NestedJoin(#"Filled Down1", {"Index"}, #"Grouped Rows1", {"Custom"}, "Table2", JoinKind.LeftOuter),
#"Expanded Table2" = Table.ExpandTableColumn(#"Merged Queries", "Table2", {"Count"}, {"Count"}),
#"SwapValue"= Table.ReplaceValue( #"Expanded Table2", each [Custom.1], each if [Count] = null then [Custom.1] else "keep", Replacer.ReplaceValue,{"Custom.1"}),
#"Final Swap"=Table.ReplaceValue(#"SwapValue",each [a], each if [Count]=null then [a] else [Count] , Replacer.ReplaceValue,{"a"}),
#"Filtered Rows" = Table.SelectRows(#"Final Swap", each ([Custom.1] = "keep")),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Next Row Header", "Index", "Custom", "Custom.1", "Count"})
in #"Removed Columns"
I have a column in excel containing a long list similar to the following:
alfa.zulu#test.com
9v46by8
9016767312
TX961779
1DM90F4
bravo.zulu#test.com
B935536
24086942
9486388284
UAUG350583
0P47MB2
asd65f4
813asdg
357yvjy
jxvn97
iopu634
charlie.zulu#test.com
1DM90F4
0P47MB2
delta.zulu#test.com
9016767312
asd65f4
357yvjy
iopu634
echo.zulu#test.com
9v46by8
TX961779
B935536
I need to transpose the list, BUT every time I have an email address, I need to jump on down to the next row and start all over, such as the following:
alfa.zulu#test.com 9v46by8 9016767312 TX961779 1DM90F4
bravo.zulu#test.com B935536 24086942 9486388284 UAUG350583 0P47MB2 asd65f4 813asdg 357yvjy
charlie.zulu#test.com 1DM90F4 0P47MB2
delta.zulu#test.com 9016767312 asd65f4 357yvjy iopu634
echo.zulu#test.com 9v46by8 TX961779 B935536
Is there any way to achieve this without using vba?
Thanks in advance!
This can be done by combining the INDEX, AGGREGATE and SEARCH functions.
But there are some prerequisites:
The SEARCH function will search for cells with the # symbol - so it should be only in email addresses
At the end of the list, the # symbol must be entered in the first blank cell
Formula:
=IFERROR(INDEX(INDEX($A$1:$A$30,AGGREGATE(15,6,(1/ISNUMBER(SEARCH("#",$A$1:$A$30)))*ROW($A$1:$A$30),ROW())):INDEX($A$1:$A$30,AGGREGATE(15,6,(1/ISNUMBER(SEARCH("#",$A$1:$A$30)))*(ROW($A$1:$A$30)-1),ROW()+1)),COLUMN()-2),"")
If the list is very long, it may be better to follow Ron's advice.
With Power Query:
Make the column data type = text
Test if an entry is email -- using the # but could be more sophisticated
Add an Index column
Add another column which contains a unique number each time there is an email in column 1
Fill down with the unique numbers so each "group" will have the same number
Group the rows on the unique numbers column
Extract the data from each row into a delimited list
Add some logic to enable variations in the numbers of potential columns, else power query will not adapt.
Split the list of data into new columns based on the delimiter
Along the way, we delete extraneous columns
Paste the code below into the Power Query Editor
Change the Table in Line 2 to reflect the real table name in your worksheet.
Double click on the statements in the Applied Steps window to explore what is being done at each step
A refresh is all that should be required if your data table changes.
M Code
let
Source = Excel.CurrentWorkbook(){[Name="Table3"]}[Content],
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Column1", type text}}),
#"Added Custom" = Table.AddColumn(#"Changed Type", "isEmail", each Text.Contains([Column1],"#")),
#"Added Index" = Table.AddIndexColumn(#"Added Custom", "Index", 0, 1, Int64.Type),
#"Added Custom1" = Table.AddColumn(#"Added Index", "Grouper", each if [isEmail] then [Index] else null),
#"Filled Down" = Table.FillDown(#"Added Custom1",{"Grouper"}),
#"Removed Columns" = Table.RemoveColumns(#"Filled Down",{"isEmail", "Index"}),
#"Grouped Rows" = Table.Group(#"Removed Columns", {"Grouper"}, {{"Grouped", each _, type table [Column1=nullable text, Grouper=number]}}),
#"Added Custom2" = Table.AddColumn(#"Grouped Rows", "Value", each Table.Column([Grouped],"Column1")),
#"Removed Columns2" = Table.RemoveColumns(#"Added Custom2",{"Grouper", "Grouped"}),
#"Added Custom3" = Table.AddColumn(#"Removed Columns2", "numSplits", each List.Count([Value])),
//Make column splitting dynamic for each refresh, in case maximum number of columns changes
splits = List.Max(Table.Column(#"Added Custom3","numSplits")),
newColList = List.Zip({List.Repeat({"Value"},splits),List.Generate(() => 1, each _ <= splits, each _ +1)}),
#"Converted to Table" = Table.FromList(newColList, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
newColNamesTbl = Table.TransformColumns(#"Converted to Table", {"Column1", each Text.Combine(List.Transform(_, Text.From)), type text}),
newColNamesList = Table.Column(newColNamesTbl,"Column1"),
#"Extracted Values" = Table.TransformColumns(#"Added Custom3", {"Value", each Text.Combine(List.Transform(_, Text.From), ";"), type text}),
#"Removed Columns1" = Table.RemoveColumns(#"Extracted Values",{"numSplits"}),
#"Split Column by Delimiter" = Table.SplitColumn(#"Removed Columns1", "Value", Splitter.SplitTextByDelimiter(";", QuoteStyle.Csv), newColNamesList)
in
#"Split Column by Delimiter"
Source Data
Results
I'm importing a bunch of columns to do some analysis on in Excel power query. Some of the analysis columns need to be inserted after a certain column, but every option for adding a column only lets me append the column to the very end. I want to insert the new columns after the one named "Total" for readability.
Bellow a function than outputs the list of re-arranged column names.
ReorderList:
(tableName as table, toBeMovedColumnName as any, optional afterColumnName as text) as list=>
//tableName - the name of the table we want to reorder.
//toBeMovedColumnName - the name of the column you want to change the position. Can be a list of column names.
//columnName - the name of the column you want the toBeMovedColumnName to be positioned after. If omited toBeMovedColumnName will be placed as the first column.
let
columnNames = Table.ColumnNames(tableName),
positionOf = if afterColumnName is null or afterColumnName = "" then 0 else List.PositionOf(columnNames, afterColumnName) + 1,
toBeMovedList = if Value.Is(toBeMovedColumnName, type list) = true then toBeMovedColumnName else {toBeMovedColumnName},
intermediaryList = List.Combine({List.FirstN(columnNames,positionOf),toBeMovedList}),
intermediaryList2 = List.RemoveItems(columnNames,intermediaryList),
reorderList = List.Combine({intermediaryList,intermediaryList2})
in
reorderList
Usage like this:
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Added Custom" = Table.AddColumn(Source, "Custom1", each 4),
#"Reordered Columns" = Table.ReorderColumns(#"Added Custom", ReorderList(#"Added Custom","Custom1","Total"))
in
#"Reordered Columns"
Sample below.
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
// get baseline column names. Use this before inserting new analysis columns
Names = Table.ColumnNames(Source),
TotalSpot = List.PositionOf(Names,"Total"),
// add any code or steps here ; this is random sample. don't use
#"Added Custom" = Table.AddColumn(Source, "Custom1", each 4),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "Custom2", each 5),
#"Added Custom2" = Table.AddColumn(#"Added Custom1", "Custom3", each 6),
// insert this after all your new columns are added
// it moves all new columns to the right of the Total column
// replace #"Added Custom2" in step below with previous step name
#"Reordered Columns" = Table.ReorderColumns(#"Added Custom2",List.Combine ({List.FirstN(Names,TotalSpot+1),List.RemoveItems(Table.ColumnNames(#"Added Custom2"),Names),List.RemoveFirstN (Names,TotalSpot+1)}))
in #"Reordered Columns"
Maybe this is a very simple question, but I'm trying to figure out how to do this, as I have hundreds of columns and the idea of doing it by hand, splitting them into separate queries and then append them doesn't seem to be very practical.
I've been working on a query and it returns me values in the following format:
Date | Time | Value | Time | Value...
A | B | C | D | E...
But I need to transform it to look like:
Date | Time | Value
A | B | C
A | D | E
Thanks for the help!
Using no custom code:
Load data into powerquery using Data ... From Table/Range...
Right-click Date column, choose unpivot other columns
Add column... index column... use default column name Index
Add column...Custom Column... with formula =Number.Mod([Index],2) and default name Custom
This converts the index column into alternating 0/1s
(Assuming your 2nd column is named Value.1) Add column...Custom Column... with formula =#"Added Custom"{[Index]+1}[Value.1] and default name Custom.1
That will place the value from the row below the current one into current row
Remove alternating row by clicking arrow next to Custom column and removing [x] next the the 1
Click-Select the Attribute, Index and Custom columns, right-click Remove Columns
Load and Close
Assuming your data is loaded as range Table1 you could use this code, pasted into Home...Advanced...
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"Date"}, "Attribute", "Value.1"),
#"Added Index" = Table.AddIndexColumn(#"Unpivoted Other Columns", "Index", 0, 1),
#"Added Custom" = Table.AddColumn(#"Added Index", "Custom", each Number.Mod([Index],2)),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "Custom.1", each #"Added Custom"{[Index]+1}[Value.1]),
#"Filtered Rows" = Table.SelectRows(#"Added Custom1", each ([Custom] = 0)),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Attribute", "Index", "Custom"})
in #"Removed Columns"
If you are willing to use some custom code, this creates two tables, one table with odd columns and one with even columns, unpivots each of them, adds an index to both, then merges them back on that index. Works for any number of columns, might be faster than above for larger data sets.
Assuming your data is loaded as range Table1 you could use this code, pasted into Home...Advanced...
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
OddUnpivot= Table.AddIndexColumn(Table.UnpivotOtherColumns(Table.RemoveColumns(Source,List.RemoveFirstN(List.Alternate(Table.ColumnNames(Source),1,1,1),1)), {"date"}, "Attribute", "Value"), "Index", 0, 1),
EvenUnpivot= Table.AddIndexColumn(Table.UnpivotOtherColumns(Table.RemoveColumns(Source,List.Alternate(Table.ColumnNames(Source),1,1)), {"date"}, "Attribute", "Value"), "Index", 0, 1),
#"Merged Queries" = Table.NestedJoin(OddUnpivot,{"Index"},EvenUnpivot,{"Index"},"Table2",JoinKind.LeftOuter),
#"Expanded Table" = Table.ExpandTableColumn(#"Merged Queries", "Table2", {"Value"}, {"Value.1"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded Table",{"Attribute", "Index"})
in #"Removed Columns"
LATER UPDATE:
More generically, I've decided I like this method better
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
// 1 base columns, then groups of 2 columns, stack them
Combo = List.Transform(List.Split(List.Skip(Table.ColumnNames(Source),1),2), each List.FirstN(Table.ColumnNames(Source),1) & _),
#"Added Custom" =List.Accumulate(
Combo,
#table({"Column1"}, {}),
(state,current)=> state & Table.Skip(Table.DemoteHeaders(Table.SelectColumns(Source, current)),1)
)
in #"Added Custom"
Source Data Table.
Desired Output
We want to Extract all the Unique values from Each Type Columns and Pivot the unique values as Column headers.
Somewhat similar to this but we have more then one columns to look up unique values.
Power Query - Transpose unique values and get matching values in rows
Number of Type columns in the Source table can increase or decrease over time.
The code below is created via standard menu options. This video takes you through the results of each step.
let
Source = SourceData,
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(Source, {"Key"}, "Attribute", "Value"),
#"Added Custom" = Table.AddColumn(#"Unpivoted Other Columns", "Type", each if Text.Start([Attribute],4) = "Type" then [Value] else null),
#"Filled Down" = Table.FillDown(#"Added Custom",{"Type"}),
#"Filtered Rows" = Table.SelectRows(#"Filled Down", each not Text.StartsWith([Attribute], "Type")),
#"Removed Columns" = Table.RemoveColumns(#"Filtered Rows",{"Attribute"}),
#"Pivoted Column" = Table.Pivot(#"Removed Columns", List.Distinct(#"Removed Columns"[Type]), "Type", "Value")
in
#"Pivoted Column"