From this original table,
I made a second table (using power query).
This second table is to be used for data validation purposes, and I need it to depend on the first table so that any changes will follow through. The problem I'm running into is that my second table is not quite how I want it, I would like to remove any duplicates from each individual column. When I try to remove duplicates in power query, it removes whole rows (which makes sense, I agree), is there a way to remove duplicates from single columns?
Here's the M code I'm using to get from table1 to table2
let
Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
#"Removed Columns" = Table.RemoveColumns(Source,{"Grade", "fb", "fv", "fc", "fcp", "ft", "E", "E05"}),
#"Grouped Rows" = Table.FromColumns(Table.Group(#"Removed Columns", {"Catégorie"}, {{"Count", each List.InsertRange([Essence],0,List.Distinct([Catégorie]))}})[Count]),
#"Promoted Headers" = Table.PromoteHeaders(#"Grouped Rows", [PromoteAllScalars=true])
in
#"Promoted Headers"
If you have a Source table with columns A, B, and C and want to return a table of each column with duplicates removed, then you can write M code like this:
= Table.FromColumns({
List.Distinct(Source[A]),
List.Distinct(Source[B]),
List.Distinct(Source[C])},
{"A","B","C"})
More generically (without using explicit column names), you can do it in a few steps like this:
ToColumns = Table.FromList(Table.ToColumns(Source), Splitter.SplitByNothing(), null, null, ExtraValues.Error),
RemoveDuplicates = Table.TransformColumns(ToColumns, {{"Column1", each List.Distinct(_)}}),
FromColumns = Table.FromColumns(RemoveDuplicates[Column1], Table.ColumnNames(Source))
if you are using a new version of excel go to data menu from the top menu then > highlight the column you want and press remove duplicates.
it remove only values from the selected column
first:
then:
Related
I have a table of data which is consisted of 18 columns and 2.017 rows. I can get the row that has the highest (MAX) value in a cell but I need the row that has the most cells with higher values and have them in DESC order. I haven't managed yet to find a relevant post to this.
Here follows an example:
Using numbers up to 10 for illustration, the following shows the logic behind. (The actual numbers are those shown in Exhibit1)
Thank you
EDIT:
I am adding the below in order to try to clarify further. I am not sure if it is the correct path to go but I hope it makes sense.
In Exhibit2 I am indexing each column Desc (Based on Exhibit1) and then =SUM in the end of the row. Following this logic, the name having the lowest total is the one with the most high values (not the highest) in its row.
The result table is the following
Although possible with formulas and helper tables/columns, this can also be accomplished using Power Query, available in Windows Excel 2010+ and Excel 365 (Windows or Mac)
To use Power Query
Select some cell in your Data Table
Data => Get&Transform => from Table/Range or from within sheet
When the PQ Editor opens: Home => Advanced Editor
Make note of the Table Name in Line 2
Paste the M Code below in place of what you see
Change the Table name in line 2 back to what was generated originally.
Read the comments and explore the Applied Steps to understand the algorithm
As we discussed in our Chat, I transform each column into a list of Ranked Entries; then sum the ranks for each row and sort as you have laid out.
M Code
let
Source = Excel.CurrentWorkbook(){[Name="Table5"]}[Content],
//type all the columns
data = Table.TransformColumnTypes(Source,{
{"Order", Int64.Type},
{"Name", type text}} &
List.Transform(List.RemoveFirstN(Table.ColumnNames(Source),2), each {_, type number})
),
//Replace with ranks
//generate list of transforms to dynamically include all columns
cols = List.RemoveFirstN(Table.ColumnNames(data),2),
xForms = List.Transform(cols, (c)=> {c, each List.PositionOf(List.Sort(Table.Column(data,c),Order.Descending),_)}),
ranks = Table.TransformColumns(data,xForms),
//add Index column to enable row-wise sums
// then add the sumRank column and delete the Index column
#"Added Index" = Table.AddIndexColumn(ranks, "Index", 0, 1, Int64.Type),
#"Added Custom" = Table.AddColumn(#"Added Index", "sumRank", each
List.Sum(
Record.ToList(
Record.RemoveFields(#"Added Index"{[Index]},{"Order","Name","Index"})
)
)),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Index"}),
//join back with the original data table
//extract the sumRank column
join = Table.NestedJoin(data,{"Order","Name"}, #"Removed Columns",{"Order","Name"}, "joined",JoinKind.FullOuter),
#"Expanded joined" = Table.ExpandTableColumn(join, "joined", {"sumRank"}, {"sumRank"}),
//sort by the sumRank column, then remove it
#"Sorted Rows" = Table.Sort(#"Expanded joined",{{"sumRank", Order.Ascending}}),
#"Removed Columns1" = Table.RemoveColumns(#"Sorted Rows",{"sumRank"})
in
#"Removed Columns1"
This set-up is volatile, so I would only adopt it if non-volatile alternatives are not forthcoming.
An additional column in your table with the following formula:
=SUM(COUNTIF(OFFSET([Column1],,TRANSPOSE(ROW(INDIRECT("1:"&COLUMNS(Table1[#[Column1]:[Column4]])))-1)),">="&Table1[#[Column1]:[Column4]]))
which you can then use to sort your table.
Note that this formula will most likely require committing with CTRL+SHIFT+ENTER for your version of Excel.
Amend the table and column names as required, noting that the part
Table1[#[Column1]:[Column4]]
as well as including the table name, should comprise the leftmost and rightmost of the contiguous columns to be interrogated.
I wanted to know if Power Query in Excel can handle matching something from another worksheet and keeping only the matching row and the row above it all the while not sorting the list.
Above is the report I get sent daily. It contains orders going out. But we only give our customers their orders if they paid, which our system also catches as an "order". Our database is created that links these two orders together but it does it in a single column with the order in above the order out.
The above is the flat text file from the database that shows the OUT orders and the IN orders (i.e. payments). They are sorted by IN and linked OUT order. The numbers are randomly made by the system.
Can Power Query be used to import this flat text file from the database, match those OUT orders from "Today's OUTS" sheet and the OrdersINs which is always the single row above?
I want to just end up with a sheet that contains Today's OUTS and their linked Order INs.
Thank you.
Yes, it can.
Read in the two tables
Add an Index column to the "Links" table to be able to restore original order
Do Table.Join with JoinKind.FullOuter (all rows from both)
Sort according to the Index column
At this point one could either
add a custom column to reference the previous row if there is something in the OUTS column or,
my preference as it will often be faster: offset the Links column by one; then filter out the nulls
Please read the comments in the code and explore the Applied Steps to better understand the algorithm:
M Code
let
Source = Excel.CurrentWorkbook(){[Name="Outs"]}[Content],
Outs = Table.TransformColumnTypes(Source,{{"Today's OUTS", type text}}),
Source2 = Excel.CurrentWorkbook(){[Name="Links"]}[Content],
Links = Table.TransformColumnTypes(Source2,{{"Order Links", type text}}),
//Add index column to links to restore order after join
#"Added Index" = Table.AddIndexColumn(Links, "Index", 0, 1, Int64.Type),
Joined = Table.Join(Outs,"Today's OUTS", #"Added Index", "Order Links", JoinKind.FullOuter),
#"Sorted Rows" = Table.Sort(Joined,{{"Index", Order.Ascending}}),
#"Removed Columns" = Table.RemoveColumns(#"Sorted Rows",{"Index"}),
//offset Links by one row (usually faster than using Index to reference previous row
prevRow = let
ShiftedList = {null} & List.RemoveLastN(Table.Column(#"Removed Columns", "Order Links"),1),
Custom1 = Table.ToColumns(#"Removed Columns") & {ShiftedList},
Custom2 = Table.FromColumns(Custom1, Table.ColumnNames(#"Removed Columns") & {"Order IN"})
in
Custom2,
#"Removed Columns1" = Table.RemoveColumns(prevRow,{"Order Links"}),
//Filter out the nulls
#"Filtered Rows" = Table.SelectRows(#"Removed Columns1", each ([#"Today's OUTS"] <> null))
in
#"Filtered Rows"
Edit: Outs without Links will show up in the Outs column with a blank in the In column. Not sure how you might want to handle this
As an example heres:
and here's
and I want it to add rows not found in table 2, to the bottom of table 1, like this
I can do a full join in R and it works, but cant figure out how to do it in power query
Append one table to the other (Home ... Append queries...)
Highlight the relevant columns upon which to match
Right click. Remove duplicates.
#"Appended Query" = Table.Combine({#"PriorStepName", OtherTable}),
#"Removed Duplicates" = Table.Distinct(#"Appended Query")
Another way
Start in Table2.
Merge in Table1, matching all columns, using Left
Anti-join as merge type
That just leaves items in Table2 not found in Table1
Remove extra column
Append Table1
So in Table2... assuming two columns to match named Column1 and Column2
#"Merged Queries" = Table.NestedJoin(#"PriorStepName",{"Column1", "Column2"},Table1,{"Column1", "Column2"},"Table1",JoinKind.LeftAnti),
#"Removed Columns" = Table.RemoveColumns(#"Merged Queries",{"Table1"}),
#"Appended Query" = Table.Combine({#"Removed Columns", Table1})
I am trying to combine worksheets from two different workbooks with Power Query and I have trouble doing that.
I do not want to merge the two workbooks.
I do not want to create relationships or "joints".
However, I want to get very specific information for one workbook which has only one column. The "ID" column.
The ID column has rows with letter tags : AB or BE.
Following these letters, sepcific numeric ranges are associated.
For both AB and BE, number ranges first from 0000 to 3000 and from 3000 to 6000.
I thus have the following possibilities:
From AB0000 to AB3000
From AB3001 to AB6000
From BE0000 to BE3000
From BE3001 to AB6000
Each category match to the a specific item in my column geography, from the other workbook:
From AB0000 to AB3000, it is ItalyZ
From AB3001 to AB6000, it is ItalyB
From BE0000 to BE3000, it is UKY
From BE3001 to AB6000, it is UKM
I am thus trying to find the highest number associated to the first AB category, the second AB category, the first BE category, and the second.
I then want to "bring" this number in the other query and increment it each time that matching country is found in the other workbook.
For example :
AB356 is the highest number in the first workbook.
Once the first "ItalyB" is found, the column besides writes "AB357".
Once the second is "ItalyB" is found, the column besides write "AB358".
Here is the one columned worksheet:
Here is the other worksheet with the various countries in geography:
Here is an example of results:
have one column (geography) with
I think that this is something which I should work towards:
I added the index column, with a start as one, because each row (even row zero) should increment either of the four matching code.
In order to keep moving forward I have also been trying to create some sort of mapping in third excel sheet, that I imported in Power BI, but I am not sure that this is a good way forward:
I have the following result when I create a blank query:
After a correction, I still get this result when creating the blank query:
This is not an easy answer as there are many steps to get to your result. I have choosen for m-query because of the complexity.
In PBi click on Transform data, now you are in m-query.
The table with the ID's (I called it "HighestID") needs expansion
because we need to be able to map on prefix
You need a mapping table
("GeoMapping"), else there is no relation between the Prefixes and
the geolocation.
We need the newID on the Geo-table (which I called "Geo").
Expand the HighestID table.
Click on the table and open the Advanced Editor, look at your code and compare it to the one below, the last 2 steps are essential, there I add two columns (Prefix and Number) which we need later.
let
Source = Csv.Document(File.Contents("...\HighestID.csv"),[Delimiter=";", Columns=1, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
#"Changed Type1" = Table.TransformColumnTypes(#"Promoted Headers",{{"ID", type text}}),
#"Added Custom" = Table.AddColumn(#"Changed Type1", "Prefix", each Text.Middle([ID],0,2), type text),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "Number", each Number.FromText(Text.Middle([ID],2,5)))
in
#"Added Custom1"
Result:
Create mapping table
Click right button under your last table and click Blank Query:
Paste the source below, ensure the name of the merg table equals the name of your table. As I mentioned, I called it HighestID.
let
Source = #table({"Prefix", "Seq_Start", "Seq_End","GeoLocation"},{{"AB",0,2999,"ItalyZ"},{"AB",3000,6000,"ItalyB"},{"BC",0,299,"UKY"},{"BC",3000,6000,"UKM"}}),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Seq_Start", Int64.Type}, {"Seq_End", Int64.Type}}),
#"Merged Queries" = Table.NestedJoin(#"Changed Type", {"Prefix"}, HighestID, {"Prefix"}, "HighestID", JoinKind.LeftOuter),
#"Expanded HighestID" = Table.ExpandTableColumn(#"Merged Queries", "HighestID", {"Number"}, {"Number"}),
#"Filtered Rows" = Table.SelectRows(#"Expanded HighestID", each [Number] >= [Seq_Start] and [Number] <= [Seq_End]),
#"Grouped Rows" = Table.Group(#"Filtered Rows", {"Prefix", "Seq_Start", "Seq_End", "GeoLocation"}, {{"NextSeq", each List.Max([Number]) + 1, type number}})
in
#"Grouped Rows"
Result:
Adding the NextSeq Column
This is the hard bit because when I would only give you teh code, I am afraid it will not work so I give you the steps you need to do.
1.Select the table, right click on Geography and click Group by. select as below:
Merge with table Geomapping as below:
Expand the GeoMapping with NextSeq
Add a custom column:
Remove columns not needed so only custom is left created in step 4.
Expand the column (all select). End result all your columns you had earlier plus an Index column.
In Excel, what VBA code will help me explode/enrich data in table A by applying the % shares in table B to produce the desired output in table C? Not all companies need to be enriched.
screenshot of relevant tables in Excel
I envisage some loop to match on company name and then to enrich Table B by inserting the necessary rows to show the resulting shared $ by Team.
I'd use Power Query for that, not VBA. Load both table A and Table B into Power Query. Then create a query that merges the two tables on the company column, using a full outer join. Then extract the team and share columns. Create a new column for the calculation of the $ value, delete the columns not required and remove null values from the Team column. The result looks like this:
M Code generated by clicking the buttons in Power Query Editor and entering one IF statement manually looks like this:
let
Source = Table.NestedJoin(TableA,{"company"},TableB,{"company"},"NewColumn",JoinKind.FullOuter),
#"Expanded NewColumn" = Table.ExpandTableColumn(Source, "NewColumn", {"Team", "PercShare"}, {"Team", "PercShare"}),
#"Added Custom" = Table.AddColumn(#"Expanded NewColumn", "Dollars", each if [Team] = null then [dollar] else [dollar] *([PercShare]/100)),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"PercShare"}),
#"Replaced Value" = Table.ReplaceValue(#"Removed Columns",null,"",Replacer.ReplaceValue,{"Team"})
in
#"Replaced Value"