Calculate Overlap in Excel Dataset - excel

Question

I have an Excel file with ~500,000+ rows of data, each of which has (amongst other things) an ID and a certain value I'd like to manipulate. I'll use an example consisting of names and foods. The data looks something like this:
Name Food
Alex Melon
Alex Burger
Bruce Apple
Charlie Water
Alice Apple
Bruce Melon
Bruce Apple
Bruce Plum
I'd like to find the overlap in foods between any pair of names, giving me a result that would tell me (for example) that for the pairing of Bruce vs Alex, 2/3 of Bruce's data is unique and 1/3 is the same is Alex's list, whilst for Alex 1/2 his data in unique and 1/2 is the same as Bruce.
There is no consistency in the amount of foods a person can have listed alongside their name. And its entirely possible for some people to have foods not found alongside anyone else.
I thought to present it through something like this, where each percentage sign is the overlap for that pairing (read by row, so C2 would be the proportion of Alex's data also found in Alice's, whilst B3 would be the proportion of Alice's data also found in Alex's):
Alex Alice Bruce Charlie
Alex - % % %
Alice % - % %
Bruce % % - %
Charlie % % % -
I've been struggling to think and find a formulae or VBA script that would achieve this and calculate the overlap. I've considered creating (i) a helper column that concatenates the name and food, (ii) a new de-duplicated unique list of foods and maps this against the helper column. However, as far as I can tell, whilst that will help me summarise which foods go with which person, it won't help me find out the overlap between each person's list of foods.
Any help would be greatly appreciated!

Answer 1

This was fun!
You can use Power Query for this.
Highlight your data and insert a table, make sure you say the first row is columnm headers.
Go to the Table ribbon and change the table's name to "PersonTag"
Go to the Data ribbon and in the 'Get & Transform' section, click "From Table"
This opens up Power Query, with a new query called "PersonTag"
Highlight both columns, then on the "Home" ribbon, choose "Remove Rows - Remove Duplicates"
In the "Home" ribbon, click "Manage - Reference"
You've just created a new query that refers to the PersonTag query! Rename it to PersonCount.
Highlight the "Name" column, and in the "Transform" ribbon, click "Group By" and group by name, creating a new column called "PersonCount" that is the count of the rows.
Go back to editing the PersonTag query
Create a new query (or copy an existing one, it doesn't matter how), name it "PersonTagPersonTag", and then go to the "Home" ribbon, click "Advanced Editor" and replace whatever's there with the following.
let
Source = PersonTag,
// Recursion! Now each row contains it's own "PersonTag" table!
#"Added Custom" = Table.AddColumn(Source, "2nd", each PersonTag),
#"Expanded 2nd" = Table.ExpandTableColumn(#"Added Custom", "2nd", {"Name", "Food"}, {"2nd.Name", "2nd.Food"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded 2nd",{{"2nd.Name", type text}, {"2nd.Food", type text}}),
// We only want rows where the foods match and the names don't
#"Filtered Rows" = Table.SelectRows(#"Changed Type", each ([Name] <> [2nd.Name] and [Food] = [2nd.Food]))
in
#"Filtered Rows"
Now we're going to group by Name and 2nd.Name to get the row count of matches in a "PersonPersonCount" columns, bring in the PersonCount query we created earlier to get the total foods each Name, then create a PercentMatch column by dividing PersonPersonCount by PersonCount. Then we can get rid of the Count columns because we don't need them, and pivot by 2nd.Name! Create another query (I named mine "PersonvPerson").
let
Source = PersonTagPersonTag,
#"Grouped Rows" = Table.Group(Source, {"Name", "2nd.Name"}, {{"PersonPersonCount", each Table.RowCount(_), type number}}),
// Bring in PersonCount query
#"Merged Queries" = Table.NestedJoin(#"Grouped Rows",{"Name"},PersonCount,{"Name"},"PersonCount",JoinKind.LeftOuter),
// If you click the column type icon in the column title in the previous step, you get the dialog box you can fill out that does this step for you
#"Expanded PersonCount" = Table.ExpandTableColumn(#"Merged Queries", "PersonCount", {"PersonCount"}, {"PersonCount"}),
#"Added PercentMatch" = Table.AddColumn(#"Expanded PersonCount", "PercentMatch", each [PersonPersonCount] / [PersonCount]),
#"Changed Type" = Table.TransformColumnTypes(#"Added PercentMatch",{{"PercentMatch", Percentage.Type}}),
#"Removed Other Columns" = Table.SelectColumns(#"Changed Type",{"Name", "2nd.Name", "PercentMatch"}),
#"Sorted Rows" = Table.Sort(#"Removed Other Columns",{{"2nd.Name", Order.Ascending}}),
// I did this by highlighting the "2nd.Name" column, going to the "Transform" ribbon, and clicking "Pivot Column"
#"Pivoted Column" = Table.Pivot(#"Sorted Rows", List.Distinct(#"Sorted Rows"[#"2nd.Name"]), "2nd.Name", "PercentMatch", List.Sum)
in
#"Pivoted Column"
Exit the Power Query window and keep your changes. By default, creating new queries automatically create new tabs on the worksheet that contain the data. Delete the tabs you don't want to keep.
If you're anything like I was a few months ago, your jaw is dropping at what you can do with power queries. I gave you the code because I was too lazy to tell you every little click to create the code, but don't be overwhelmed!!! I just clicked around to create each next step and it created the code for me! They made it easy to click around and try/undo things.