Excel: Remove Duplicates based on time condition - excel

Question

I'm looking to remove duplicates from a 250,000 row excel sheet based on a 3 month rolling time condition.
We have a lot of usersIDs and the dates which they visited but a lot of these visits are very far apart (sometimes over a year) and a lot of them are within the same day/couple day period.
The best way to explain what I want to do is with an example. So if they first visited on 1st Jan, 1st Jan, 3rd Jan, 8th Feb, 4th June, 5th June, 1st Dec, 1st Dec, 2nd Dec, I would want to grab that first date of 1st Jan, 4th June and 1st Dec.
If they visited 1st Jan, 1st Jan, 3rd Jan, 8th Feb, 9th Apr then 1st August, 1st Sept, I would want 1st Jan and 8th August.
So we want to grab the first date, then see how often they visit within 3 months of each visit and if they leave for more than a 3 month period, grab the first date that they return. Sometimes they come back 4 or 5 times after 3 months and the data can span several years.
Is there a way for me to achieve this? It would be great to get some help as this is driving me mad.
Cheers

Answer 1

If the UserID is in column A and the VisitDate is in B with the headings in row 1 and then a blank row in 2 and the data starting in row 3 then try this (explanation below):
Array Formula version:
sort the rows ascending by VisitDate
in B2 put 1/1/1900 so it won't match anything (but it has to be a date)
in C3 put this array formula (press control-shift-enter instead of just enter):
=SUM((B$2:B2<DATE(YEAR(B3),MONTH(B3)-3,DAY(B3)))*(A$2:A2=A3))=SUM((A$2:A2=A3)*1)
Copy the formula in C3 down to every row of data
Filter on Unique = TRUE
if you want to resort you will need to copy and paste back column C by values
New non-array formula version:
sort the rows ascending by VisitDate
in B2 put 1/1/1900 so it won't match anything (but it has to be a date)
in C3 put this normal formula (just press enter):
=COUNTIFS(B$2:B2,"<"&DATE(YEAR(B3),MONTH(B3)-3,DAY(B3)),A$2:A2,A3)=COUNTIF(A$2:A2,A3)
Copy the formula in C3 down to every row of data
Filter on Unique = TRUE
if you want to resort you will need to copy and paste back column C by values
This produces the following with my sample data (array formulas may take a very long time to calculate for lots of rows):
| A | B | C
---+--------+------------+--------
1 | UserID | VisitDate | Unique
2 | | 1/01/1900 |
3 | a | 1/01/2017 | TRUE
4 | a | 1/01/2017 | FALSE
5 | b | 2/01/2017 | TRUE
6 | b | 2/01/2017 | FALSE
7 | a | 3/01/2017 | FALSE
8 | c | 3/01/2017 | TRUE
9 | c | 3/01/2017 | FALSE
10 | b | 4/01/2017 | FALSE
11 | c | 5/01/2017 | FALSE
12 | a | 8/02/2017 | FALSE
13 | b | 9/02/2017 | FALSE
14 | c | 10/02/2017 | FALSE
15 | a | 4/06/2017 | TRUE
16 | a | 5/06/2017 | FALSE
17 | b | 5/06/2017 | TRUE
18 | b | 6/06/2017 | FALSE
19 | c | 6/06/2017 | TRUE
20 | c | 7/06/2017 | FALSE
21 | a | 1/12/2017 | TRUE
22 | a | 1/12/2017 | FALSE
23 | a | 2/12/2017 | FALSE
24 | b | 2/12/2017 | TRUE
25 | b | 2/12/2017 | FALSE
26 | b | 3/12/2017 | FALSE
27 | c | 3/12/2017 | TRUE
28 | c | 3/12/2017 | FALSE
29 | c | 4/12/2017 | FALSE
Because the formula compares the current row with all the rows above looking for rows with dates in the past the data needs to be sorted with the oldest dates first.
How the array formula works:
=SUM((B$2:B2<DATE(YEAR(B3),MONTH(B3)-3,DAY(B3)))*(A$2:A2=A3))=SUM((A$2:A2=A3)*1)
DATE(YEAR(B3),MONTH(B3)-3,DAY(B3)) is 3 months ago (even if it is 92 days)
(B$2:B2<DATE(YEAR(B3),MONTH(B3)-3,DAY(B3))) is an array of TRUE/FALSE values which has a TRUE for every row above that is older than 3 months ago
(A$2:A2=A3) is an array of TRUE/FALSE values which has a TRUE for every row above that matches the user ID
(B$2:B2<DATE(YEAR(B3),MONTH(B3)-3,DAY(B3)))*(A$2:A2=A3) does an AND of the arrays so 1 is returned (TRUE*TRUE=1) for each row above that has the same name and a date that is older than 3 months ago
SUM((B$2:B2<DATE(YEAR(B3),MONTH(B3)-3,DAY(B3)))*(A$2:A2=A3)) adds all the TRUE rows above that have the same name and a date that is older than 3 months ago
SUM((A$2:A2=A3)*1) adds the number of rows above that have the same name (TRUE*1=1)
=SUM((B$2:B2<DATE(YEAR(B3),MONTH(B3)-3,DAY(B3)))*(A$2:A2=A3))=SUM((A$2:A2=A3)*1) compares the two sums and returns TRUE if all the rows above that have the same name are all older than 3 months ago
Methodology:
I originally just played with a column of dates - no userID. I wanted to find a way to know if the date on a particular was more than 3 months after all the dates before it (I implicitly assumed that the dates were sorted). I reasoned that if a count of the dates before the current row matched a count of the dates before the current row that were older than 3 months in the past then I would have the answer I wanted. So I originally put this formula in C3 and copied it down:
=COUNTIF(B$2:B2,"<"&(B3-90))=COUNTA(B$2:B2)
Then change it to 3 months instead of 90 days:
=COUNTIF(B$2:B2,"<"&DATE(YEAR(B3),MONTH(B3)-3,DAY(B3)))=COUNTA(B$2:B2)
And then to add the userID we need a way to compare multiple criteria - this is where COUNTIFS comes in (if you have Excel 2007 or better):
=COUNTIFS(B$2:B2,"<"&DATE(YEAR(B3),MONTH(B3)-3,DAY(B3)),A$2:A2,A3)=COUNTIF(A$2:A2,A3)
And then I converted it to this array formula:
=SUM((B$2:B2<DATE(YEAR(B3),MONTH(B3)-3,DAY(B3)))*(A$2:A2=A3))=SUM((A$2:A2=A3)*1)
In retrospect I don't know if giving the array formula was a good idea or not: I don't know whether the array formula would be better/faster than COUNTIFS or not. So use whichever you prefer.