I wanted to know if Power Query in Excel can handle matching something from another worksheet and keeping only the matching row and the row above it all the while not sorting the list.
Above is the report I get sent daily. It contains orders going out. But we only give our customers their orders if they paid, which our system also catches as an "order". Our database is created that links these two orders together but it does it in a single column with the order in above the order out.
The above is the flat text file from the database that shows the OUT orders and the IN orders (i.e. payments). They are sorted by IN and linked OUT order. The numbers are randomly made by the system.
Can Power Query be used to import this flat text file from the database, match those OUT orders from "Today's OUTS" sheet and the OrdersINs which is always the single row above?
I want to just end up with a sheet that contains Today's OUTS and their linked Order INs.
Thank you.
Yes, it can.
Read in the two tables
Add an Index column to the "Links" table to be able to restore original order
Do Table.Join with JoinKind.FullOuter (all rows from both)
Sort according to the Index column
At this point one could either
add a custom column to reference the previous row if there is something in the OUTS column or,
my preference as it will often be faster: offset the Links column by one; then filter out the nulls
Please read the comments in the code and explore the Applied Steps to better understand the algorithm:
M Code
let
Source = Excel.CurrentWorkbook(){[Name="Outs"]}[Content],
Outs = Table.TransformColumnTypes(Source,{{"Today's OUTS", type text}}),
Source2 = Excel.CurrentWorkbook(){[Name="Links"]}[Content],
Links = Table.TransformColumnTypes(Source2,{{"Order Links", type text}}),
//Add index column to links to restore order after join
#"Added Index" = Table.AddIndexColumn(Links, "Index", 0, 1, Int64.Type),
Joined = Table.Join(Outs,"Today's OUTS", #"Added Index", "Order Links", JoinKind.FullOuter),
#"Sorted Rows" = Table.Sort(Joined,{{"Index", Order.Ascending}}),
#"Removed Columns" = Table.RemoveColumns(#"Sorted Rows",{"Index"}),
//offset Links by one row (usually faster than using Index to reference previous row
prevRow = let
ShiftedList = {null} & List.RemoveLastN(Table.Column(#"Removed Columns", "Order Links"),1),
Custom1 = Table.ToColumns(#"Removed Columns") & {ShiftedList},
Custom2 = Table.FromColumns(Custom1, Table.ColumnNames(#"Removed Columns") & {"Order IN"})
in
Custom2,
#"Removed Columns1" = Table.RemoveColumns(prevRow,{"Order Links"}),
//Filter out the nulls
#"Filtered Rows" = Table.SelectRows(#"Removed Columns1", each ([#"Today's OUTS"] <> null))
in
#"Filtered Rows"
Edit: Outs without Links will show up in the Outs column with a blank in the In column. Not sure how you might want to handle this
Related
I have a table of data which is consisted of 18 columns and 2.017 rows. I can get the row that has the highest (MAX) value in a cell but I need the row that has the most cells with higher values and have them in DESC order. I haven't managed yet to find a relevant post to this.
Here follows an example:
Using numbers up to 10 for illustration, the following shows the logic behind. (The actual numbers are those shown in Exhibit1)
Thank you
EDIT:
I am adding the below in order to try to clarify further. I am not sure if it is the correct path to go but I hope it makes sense.
In Exhibit2 I am indexing each column Desc (Based on Exhibit1) and then =SUM in the end of the row. Following this logic, the name having the lowest total is the one with the most high values (not the highest) in its row.
The result table is the following
Although possible with formulas and helper tables/columns, this can also be accomplished using Power Query, available in Windows Excel 2010+ and Excel 365 (Windows or Mac)
To use Power Query
Select some cell in your Data Table
Data => Get&Transform => from Table/Range or from within sheet
When the PQ Editor opens: Home => Advanced Editor
Make note of the Table Name in Line 2
Paste the M Code below in place of what you see
Change the Table name in line 2 back to what was generated originally.
Read the comments and explore the Applied Steps to understand the algorithm
As we discussed in our Chat, I transform each column into a list of Ranked Entries; then sum the ranks for each row and sort as you have laid out.
M Code
let
Source = Excel.CurrentWorkbook(){[Name="Table5"]}[Content],
//type all the columns
data = Table.TransformColumnTypes(Source,{
{"Order", Int64.Type},
{"Name", type text}} &
List.Transform(List.RemoveFirstN(Table.ColumnNames(Source),2), each {_, type number})
),
//Replace with ranks
//generate list of transforms to dynamically include all columns
cols = List.RemoveFirstN(Table.ColumnNames(data),2),
xForms = List.Transform(cols, (c)=> {c, each List.PositionOf(List.Sort(Table.Column(data,c),Order.Descending),_)}),
ranks = Table.TransformColumns(data,xForms),
//add Index column to enable row-wise sums
// then add the sumRank column and delete the Index column
#"Added Index" = Table.AddIndexColumn(ranks, "Index", 0, 1, Int64.Type),
#"Added Custom" = Table.AddColumn(#"Added Index", "sumRank", each
List.Sum(
Record.ToList(
Record.RemoveFields(#"Added Index"{[Index]},{"Order","Name","Index"})
)
)),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Index"}),
//join back with the original data table
//extract the sumRank column
join = Table.NestedJoin(data,{"Order","Name"}, #"Removed Columns",{"Order","Name"}, "joined",JoinKind.FullOuter),
#"Expanded joined" = Table.ExpandTableColumn(join, "joined", {"sumRank"}, {"sumRank"}),
//sort by the sumRank column, then remove it
#"Sorted Rows" = Table.Sort(#"Expanded joined",{{"sumRank", Order.Ascending}}),
#"Removed Columns1" = Table.RemoveColumns(#"Sorted Rows",{"sumRank"})
in
#"Removed Columns1"
This set-up is volatile, so I would only adopt it if non-volatile alternatives are not forthcoming.
An additional column in your table with the following formula:
=SUM(COUNTIF(OFFSET([Column1],,TRANSPOSE(ROW(INDIRECT("1:"&COLUMNS(Table1[#[Column1]:[Column4]])))-1)),">="&Table1[#[Column1]:[Column4]]))
which you can then use to sort your table.
Note that this formula will most likely require committing with CTRL+SHIFT+ENTER for your version of Excel.
Amend the table and column names as required, noting that the part
Table1[#[Column1]:[Column4]]
as well as including the table name, should comprise the leftmost and rightmost of the contiguous columns to be interrogated.
I am moving my first steps in PowerQuery, so here's my problem. I have a raw data table which list countries and certain products. For each product there is the "market" value followed by a MyValue (meaning my own sales of that product in that country). An example here:
raw table
What I was trying to obtain with PowerQuery is a table that unpivots the products category and leaves two columns, one for Market and one for MyValue.
I tried in many ways and the closest to the result I could get was splitting the original table in two, one for the Market and one for MyValues. Then unpivot each one of them in PowerQuery so that I could get them in this way:
Market
And
MyValue
I tried then to merge the two tables but can't work it out. Of course I could do that manually but I'm sure there a way to do it with PowerQuery, either splitting into 2 tables, unpivoting and then merging or - even better - with a single query.
The result I'm aiming at is like
Desired Result
You are close.
After you unpivot, you need to create a custom column that you can pivot on, and also modify the names in the resultant "attribute" column.
Read the comments in the code and explore the Applied Steps window to understand the algorithm
M Code
let
Source = Excel.CurrentWorkbook(){[Name="rawTable"]}[Content],
//generalized "typer" in case you add other Items
#"Changed Type" = Table.TransformColumnTypes(Source,{
{"Country", type text}, {"Date", type date}} &
List.Transform(List.RemoveFirstN(Table.ColumnNames(Source),2),each {_, Int64.Type})),
//Unpivot all except Country|Date
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Changed Type", {"Country", "Date"}, "Item", "Value"),
//Add Custom Column to create Pivot column for "Market" and "MyValue
#"Added Custom" = Table.AddColumn(#"Unpivoted Other Columns", "Custom", each
if Text.StartsWith([Item],"My")
then "Market"
else "MyValue"),
//Replace "My" so Item Labels will be consistent
#"Replaced Value" = Table.ReplaceValue(#"Added Custom","My","",Replacer.ReplaceText,{"Item"}),
//Pivot with no aggregation (unless you want to)
#"Pivoted Column" = Table.Pivot(#"Replaced Value", List.Distinct(#"Replaced Value"[Custom]), "Custom", "Value"),
//Sort "Items" to original Column Order
itemSortOrder = List.Distinct(#"Replaced Value"[Item]),
sorted = Table.Sort(#"Pivoted Column",
{{"Country", Order.Ascending},
each List.PositionOf(itemSortOrder,[Item])
})
in
sorted
Hopefully, this is what you want for a result
thank you so much for having spent your time to help me.
I think I solved my problem using the List.Zip function. Solution was not mine but I took if from THIS video. With this trick, I don't even have to split the original source data into two tables (market & MyShare).
It perfectly does what I needed to with little if no effort for data-cleaning...
I hope I'm going to explain myself well enough. I'm importing spreadsheets from a folder into Power Query, but I want to filter out (drop) any groups of data (by a name column) where that group has at least one instance of a given value.
Example, I have a theoretical table with two columns. One has repeating names and the other has scattered values of A or B or C. I want the query to look for a group of names, then see if any of the records in that group has either and A, B or C in the second column. If found, it drops that entire group and if none, then it allows that entire group through.
I'm not sure if this is too complex for Power Query or whether I need to do this outside Excel. If the latter, what would the equivalent SQL statement be (for MS Access, as I'm trying to keep this as basic as possible)?
Theres about 500 ways to do this. One easy way is to filter Column2 for ABC, remove that column, then remove duplicates from Column1. That give you a unique list of groups to remove, and just merge that against the original data and remove those rows
sample:
let Source = Excel.CurrentWorkbook(){[Name="Table1"]}[Content],
// all unique groups from Col1 that have a/b/c in Col2
#"Filtered Rows" = Table.SelectRows(Source, each ([Column2] = "a" or [Column2] = "b" or [Column2] = "c")),
#"Removed Columns"= Table.SelectColumns(#"Filtered Rows",{"Column1"})
#"Removed Duplicates" = Table.Distinct(#"Removed Columns"),
// merge that into original table and filter
#"Merged Queries" = Table.NestedJoin(Source,{"Column1"},#"Removed Duplicates",{"Column1"},"Table2",JoinKind.LeftOuter),
#"Expanded Table2" = Table.ExpandTableColumn(#"Merged Queries", "Table2", {"Column1"}, {"Column1.1"}),
#"Filtered Rows1" = Table.SelectRows(#"Expanded Table2", each ([Column1.1] <> null)),
#"Removed Columns1" = Table.RemoveColumns(#"Filtered Rows1",{"Column1.1"})
in #"Removed Columns1"
I am trying to combine worksheets from two different workbooks with Power Query and I have trouble doing that.
I do not want to merge the two workbooks.
I do not want to create relationships or "joints".
However, I want to get very specific information for one workbook which has only one column. The "ID" column.
The ID column has rows with letter tags : AB or BE.
Following these letters, sepcific numeric ranges are associated.
For both AB and BE, number ranges first from 0000 to 3000 and from 3000 to 6000.
I thus have the following possibilities:
From AB0000 to AB3000
From AB3001 to AB6000
From BE0000 to BE3000
From BE3001 to AB6000
Each category match to the a specific item in my column geography, from the other workbook:
From AB0000 to AB3000, it is ItalyZ
From AB3001 to AB6000, it is ItalyB
From BE0000 to BE3000, it is UKY
From BE3001 to AB6000, it is UKM
I am thus trying to find the highest number associated to the first AB category, the second AB category, the first BE category, and the second.
I then want to "bring" this number in the other query and increment it each time that matching country is found in the other workbook.
For example :
AB356 is the highest number in the first workbook.
Once the first "ItalyB" is found, the column besides writes "AB357".
Once the second is "ItalyB" is found, the column besides write "AB358".
Here is the one columned worksheet:
Here is the other worksheet with the various countries in geography:
Here is an example of results:
have one column (geography) with
I think that this is something which I should work towards:
I added the index column, with a start as one, because each row (even row zero) should increment either of the four matching code.
In order to keep moving forward I have also been trying to create some sort of mapping in third excel sheet, that I imported in Power BI, but I am not sure that this is a good way forward:
I have the following result when I create a blank query:
After a correction, I still get this result when creating the blank query:
This is not an easy answer as there are many steps to get to your result. I have choosen for m-query because of the complexity.
In PBi click on Transform data, now you are in m-query.
The table with the ID's (I called it "HighestID") needs expansion
because we need to be able to map on prefix
You need a mapping table
("GeoMapping"), else there is no relation between the Prefixes and
the geolocation.
We need the newID on the Geo-table (which I called "Geo").
Expand the HighestID table.
Click on the table and open the Advanced Editor, look at your code and compare it to the one below, the last 2 steps are essential, there I add two columns (Prefix and Number) which we need later.
let
Source = Csv.Document(File.Contents("...\HighestID.csv"),[Delimiter=";", Columns=1, Encoding=1252, QuoteStyle=QuoteStyle.None]),
#"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
#"Changed Type1" = Table.TransformColumnTypes(#"Promoted Headers",{{"ID", type text}}),
#"Added Custom" = Table.AddColumn(#"Changed Type1", "Prefix", each Text.Middle([ID],0,2), type text),
#"Added Custom1" = Table.AddColumn(#"Added Custom", "Number", each Number.FromText(Text.Middle([ID],2,5)))
in
#"Added Custom1"
Result:
Create mapping table
Click right button under your last table and click Blank Query:
Paste the source below, ensure the name of the merg table equals the name of your table. As I mentioned, I called it HighestID.
let
Source = #table({"Prefix", "Seq_Start", "Seq_End","GeoLocation"},{{"AB",0,2999,"ItalyZ"},{"AB",3000,6000,"ItalyB"},{"BC",0,299,"UKY"},{"BC",3000,6000,"UKM"}}),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Seq_Start", Int64.Type}, {"Seq_End", Int64.Type}}),
#"Merged Queries" = Table.NestedJoin(#"Changed Type", {"Prefix"}, HighestID, {"Prefix"}, "HighestID", JoinKind.LeftOuter),
#"Expanded HighestID" = Table.ExpandTableColumn(#"Merged Queries", "HighestID", {"Number"}, {"Number"}),
#"Filtered Rows" = Table.SelectRows(#"Expanded HighestID", each [Number] >= [Seq_Start] and [Number] <= [Seq_End]),
#"Grouped Rows" = Table.Group(#"Filtered Rows", {"Prefix", "Seq_Start", "Seq_End", "GeoLocation"}, {{"NextSeq", each List.Max([Number]) + 1, type number}})
in
#"Grouped Rows"
Result:
Adding the NextSeq Column
This is the hard bit because when I would only give you teh code, I am afraid it will not work so I give you the steps you need to do.
1.Select the table, right click on Geography and click Group by. select as below:
Merge with table Geomapping as below:
Expand the GeoMapping with NextSeq
Add a custom column:
Remove columns not needed so only custom is left created in step 4.
Expand the column (all select). End result all your columns you had earlier plus an Index column.
Currently I am making a schedule of class times where the secretary adds names to the list for however many seats are available for that room, and it shows whether or not they have passed the test already. The managers would like a count of how many times the physician may have no showed. Column A is the seat number (plays no real role), column B is the name slot, which pulls a searchable list from a master list, with the "=Cell("contents")" trick because there are too many for a straight drop down. Column C is at VLOOKUP to check their current test status to help not double book. And finally, Column D is a checkbox if they no show.
I have a separate sheet that is keeping track of these no shows, it records the name, a count of 1, and the date they skipped.
Question 1, is there a way to not have to make each checkbox individually and link each individually? There's 8 weeks of class with 60+ seats.
Question 2, is there a way to make it add rows to this sheet only if checked off so there isn't 900 blank rows for a pivot table?
Code used on "NoShow" sheet:
=IF(Schedule!D5=TRUE,Schedule!B5,"")
=IF(A2<>"","1","")
=IF(Schedule!D5=TRUE,TODAY(),"")
This can be done easier with Power Query. In this example, I have:
One table on each worksheet, for each training date. No shows are indicated with "Yes".
Each table is named t_ and the table name.
Then Power Query consolidates all of the tables into one and produces one table showing all of the consolidated records, that is summarized with a pivot table, and another with unique names, that can be used for your drop-down menu.
When you have a new date, just add a new worksheet with a table for that date, fill in the info and Refresh the calculations.
Here is the table of consolidated data...
Here is the pivot that counts the no shows...
To get the summary table...
After you set up your tables, insert a blank query by going to Data > Get and Transform Data > Get Data > From Other Sources > Blank Query.
Then click Advanced Editor, delete any existing text and paste this:
let
Source = Excel.CurrentWorkbook(),
#"Filtered Rows" = Table.SelectRows(Source, each ([Name] <> "Summary")),
#"Expanded Content1" = Table.ExpandTableColumn(#"Filtered Rows", "Content", {"Seat Number", "Name of Physician", "No Show?"}, {"Seat Number", "Name of Physician", "No Show?"}),
#"Duplicated Column" = Table.DuplicateColumn(#"Expanded Content1", "Name", "Name - Copy"),
#"Removed Columns" = Table.RemoveColumns(#"Duplicated Column",{"Seat Number"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"Name - Copy", "Date"}}),
#"Extracted Text After Delimiter" = Table.TransformColumns(#"Renamed Columns", {{"Date", each Text.AfterDelimiter(_, "_"), type text}}),
#"Changed Type" = Table.TransformColumnTypes(#"Extracted Text After Delimiter",{{"Date", type date}}),
#"Reordered Columns" = Table.ReorderColumns(#"Changed Type",{"Name", "Date", "Name of Physician", "No Show?"}),
#"Renamed Columns1" = Table.RenameColumns(#"Reordered Columns",{{"Name", "Table Name"}})
in
#"Renamed Columns1"
Then click Close and Load To > New Worksheet.
To get the unique names table....
Follow the same steps above, but in a new blank query, paste this text...
let
Source = Excel.CurrentWorkbook(),
#"Filtered Rows" = Table.SelectRows(Source, each ([Name] <> "Summary")),
#"Expanded Content1" = Table.ExpandTableColumn(#"Filtered Rows", "Content", {"Seat Number", "Name of Physician", "No Show?"}, {"Seat Number", "Name of Physician", "No Show?"}),
#"Duplicated Column" = Table.DuplicateColumn(#"Expanded Content1", "Name", "Name - Copy"),
#"Removed Columns" = Table.RemoveColumns(#"Duplicated Column",{"Seat Number"}),
#"Renamed Columns" = Table.RenameColumns(#"Removed Columns",{{"Name - Copy", "Date"}}),
#"Extracted Text After Delimiter" = Table.TransformColumns(#"Renamed Columns", {{"Date", each Text.AfterDelimiter(_, "_"), type text}}),
#"Changed Type" = Table.TransformColumnTypes(#"Extracted Text After Delimiter",{{"Date", type date}}),
#"Reordered Columns" = Table.ReorderColumns(#"Changed Type",{"Name", "Date", "Name of Physician", "No Show?"}),
#"Renamed Columns1" = Table.RenameColumns(#"Reordered Columns",{{"Name", "Table Name"}}),
#"Removed Columns1" = Table.RemoveColumns(#"Renamed Columns1",{"No Show?", "Date", "Table Name"}),
#"Removed Duplicates" = Table.Distinct(#"Removed Columns1")
in
#"Removed Duplicates"
Then Close and Load To > New Worksheet.
Then you can select the data in summary table and Insert Pivot Table. Add the names to the Rows section and the No Shows to the Values section. In the Row Labels column header, click Value Filters > Greater Than 0 (to remove the blanks). With the pivot table, you can double-click on the number of no shows and a new worksheet will be created, showing you where that calculation came from, so there's not need for the hyperlink.
Question 1, is there a way to not have to make each checkbox individually and link each individually? There's 8 weeks of class with 60+ seats.
Yes! See proposed system solution below.
Question 2, is there a way to make it add rows to this sheet only if checked off so there isn't 900 blank rows for a pivot table?
Yes, format the range you are using as a "Table" and the table adds rows automatically. Use in conjunction with proposed solution.
Proposed Solution:
You should be able to find a solution with PowerPivot using (i) a simple data model comprised of one table (fact table) tracks the class dates and people who miss the class and a master list of potential attendees (lookup/dimension table), where you relate the class tracking table to the potential attendee table via the name and (ii) a pivot table that easily summarizes who has missed. From the Pivot Table, you can create cool charts or slides as needed
Your secretary merely has to update changes in the fact table or dimension table. If the seat number plays no real purpose, it makes sense to leave it out.
Fact Table Columns: Date, Name, Missed (assign 1)
Dimension/LookUpTable Columns: Name, Test Taken, (other relevant info)
Note, you are using the entire name(first name and last name) in one cell. If you want to be rigorous, you may want to assign each name it's own unique ID and use that as the primary key for the LookUp table in the case there are two identical names.
Hopes this helps!