Which formulae in MS Excel can we use for -
equi-depth binning
equi-width binning
Here's what I used. The data I was binning was in A2:A2001.
Equi-width:
I calculated the width in a separate cell (U2), using this formula:
=(MAX($A$2:$A$2001) - MIN($A$2:$A$2001) + 0.00000001)/10
10 is the number of bins. The + 0.00000000001 is there because without it, values equal to the maximum were getting put into their own bin.
Then, for the actual binning, I used this:
=ROUNDDOWN(($A2-MIN($A$2:$A$2001))/$U$2, 0)
This function is finding how many bin-widths above the minimum your value is, by dividing (value - minimum) by the bin width. We only care about how many full bin-widths fit into the value, not fractional ones, so we use ROUNDDOWN to chop off all the fractional bin-widths (that is, show 0 decimal places).
Equi-depth
This one is simpler.
=ROUNDDOWN(PERCENTRANK($A$2:$A$2001, $A2)*10, 0)
First, get the percentile rank of the current cell ($A2) out of all the cells being binned ($A$2:$A$2001). This will be a value between 0 and 1, so to convert it into bins, just multiply by the total number of bins you want (I used 10). Then, chop off the decimals the same way as before.
For either of these, if you want your bins to start at 1 rather than 0, just add a +1 to the end of the formula.
Best approach is to use the built-in method:
http://support.microsoft.com/kb/214269
I think the VBA version of the addin (step 3 with most versions) will also give you the code.
Put this formula in B1:
=MAX( ROUNDUP( PERCENTRANK($A$1:$A$8, A1) *4, 0),1)
Fill down the formula all across B column and you are done. The formula divides the range into 4 equal buckets and it returns the bucket number which the cell A1 falls into. The first bucket contains the lowest 25% of values.
General pattern is:
=MAX( ROUNDUP ( PERCENTRANK ([Range], [TestCell]) * [NumberOfBuckets], 0), 1)
You may have to build the matrix to graph.
For the bin bracket you could use =PERCENTILE() for equi-depth and a proportion of the difference =Max(Data) - Min(Data) for equi-width.
You could obtain the frequency with =COUNTIF(). The bin's Mean could be obtained using =SUMPRODUCT((Data>LOWER_BRACKET)*(Data<UPPER_BRACKET)*Data)/frequency
More complex statistics could be reached hacking around with SUMPRODUCT and/or Array formulas (which I do not recommend since are very hard to comprehend for a non-programmer)
Related
I am currently trying to check if a number in a comma-separated string is within a number interval. What I am trying to do is to check if an area code (from the comma-separated string) is within the interval of an area.
The data:
AREAS
Area interval
Name
Number of locations
1000-1499
Area 1
?
1500-1799
Area 2
?
1800-1999
Area 3
?
GEOLOCATIONS
Name
Areas List
Location A
1200, 1400
Location B
1020, 1720
Location C
1700, 1920
Location D
1940, 1950, 1730
The result I want here is the number of unique locations in the "Areas list" within the area interval. So Location D should only count ONCE in the 1800-1999 "area", and the Location A the same in the 1000-1499 location. But location B should count as one in both 1000-1499 and one in 1500-1799 (because a number from each interval is in the comma-separated string in "Areas list"):
Area interval
Name
Number of locations
1000-1499
Area 1
2
1500-1799
Area 2
3
1800-1999
Area 3
2
How is this possible?
I have tried with a COUNTIFS, but it doesnt seem to do the job.
Here is one option using FILTERXML():
Formula in C2:
=SUM(FILTERXML("<x><t>"&TEXTJOIN("</s></t><t>",,"1<s>"&SUBSTITUTE(B$7:B$10,", ","</s><s>"))&"</s></t></x>","//t[count(.//*[.>="&SUBSTITUTE(A2,"-","][.<=")&"])>0]"))
Where:
"<x><t>"&TEXTJOIN("</s></t><t>",,"1<s>"&SUBSTITUTE(B$7:B$10,", ","</s><s>"))&"</s></t></x>" - Is the part where we construct a valid piece of XML. The theory here is that we use three axes here. Each t-node will be named a literal 1 to make sure that once we return them with xpath we can sum the result. The outer x-nodes are there to make sure Excel will handle the inner axes correctly. If you are curious to know how this xml-syntax looks at the end, it's best to step through using the 'Evaluate Formula' function on the Data-tab;
//t[count(.//*[.>="&SUBSTITUTE(A2,"-","][.<=")&"])>0]")) - Basically means that we collect all t-nodes where the count of child s-nodes that are >= to the leftmost number and <= to the rightmost number is larger than zero. For A2 the xpath would look like //t[count(.//*[.>=1000][.<=1499])>0]")) after substitution. In short: //t - Select t-nodes, where count(.//* select all child-nodes where count of nodes that fullfill both requirements [.>=1000][.<=1499] is larger than zero;
Since all t-nodes equal the number 1, the SUM() of these t-nodes equals the amount of unique locations that have at least one area in its Areas List;
Important to note that FILTERXML() will result into an error if no t-nodes could be found. That would mean we need to wrap the FILTERXML() in an IFERROR(...., 0) to counter that and make the SUM() still work correctly.
Or, wrap the above in BYROW():
Formula in C2:
=BYROW(A2:A4,LAMBDA(a,SUM(FILTERXML("<x><t>"&TEXTJOIN("</s></t><t>",,"1<s>"&SUBSTITUTE(B$7:B$10,", ","</s><s>"))&"</s></t></x>","//t[count(.//*[.>="&SUBSTITUTE(a,"-","][.<=")&"])>0]"))))
Using MMULT and TEXTSPLIT:
=LET(rng,TEXTSPLIT(D2,"-"),
tarr,IFERROR(--TRIM(TEXTSPLIT(TEXTJOIN(";",,$B$2:$B$5),",",";")),0),
SUM(--(MMULT((tarr>=--TAKE(rng,,1))*(tarr<=--TAKE(rng,,-1)),SEQUENCE(COLUMNS(tarr),,1,0))>0)))
I am in very distinguished company but will add my version anyway as byrow probably is a slightly different approach
=LET(range,B$2:B$5,
lowerLimit,--#TEXTSPLIT(E2,"-"),
upperLimit,--INDEX(TEXTSPLIT(E2,"-"),2),
counts,BYROW(range,LAMBDA(r,SUM((--TEXTSPLIT(r,",")>=lowerLimit)*(--TEXTSPLIT(r,",")<=upperLimit)))),
SUM(--(counts>0))
)
Here the ugly way to do it, with A LOT of helper columns. But not so complicated 🙂
F4= =TRANSPOSE(FILTERXML("<m><r>"&SUBSTITUTE(B4;",";"</r><r>")&"</r></m>";"//r"))
F11= =TRANSPOSE(FILTERXML("<m><r>"&SUBSTITUTE(A11;"-";"</r><r>")&"</r></m>";"//r"))
F16= =SUM(F18:F21)
F18= =IF(SUM(($F4:$O4>=$F$11)*($F4:$O4<=$G$11))>0;1;"")
G18= =IF(SUM(($F4:$O4>=$F$12)*($F4:$O4<=$G$12))>0;1;"")
H18= =IF(SUM(($F4:$O4>=$F$13)*($F4:$O4<=$G$13))>0;1;"")
I would like to know how does Excel think to calculate the values on the function PERCENTILE.INC. I'm making some studies on Percentile and Quartile, I got the below results:
How does Excel think to calculate the values on column F?
Here's the formulas I'm using:
=PERCENTILE.INC(B2:B21; 0,75) ==> F2
=PERCENTILE.INC(B2:B21; 0,50) ==> F3
=PERCENTILE.INC(B2:B21; 0,25) ==> F4
=PERCENTILE.INC(B2:B21; 0,00) ==> F5
Short answer - the position of a given percentile when the data is sorted in ascending order, using percentile.inc, is given by
(N-1)p+1
where p is the required percentile as a fraction from 0 to 1 and N is the number of points.
If this expression gives a whole number, you take the value at this position (e.g. percentile zero gives 1, so its value is exactly 22). If it's not a whole number, you interpolate between the value at the position given by the whole number part (e.g. for p=0.25 it's 5 and the value at this position is 52) and the value at the position one higher (in this case position 6 so the number is 55), then multiply the difference of the two values (3) by the fraction part (0.75) giving you 2.25 and finally add this to the lower of the two values giving you 54.25. A shorter way of saying this is that you go a quarter of the way between the two nearest values. So you have:
If you wished to show the logic as an Excel formula, you could implement the expression shown here on the right (where h, in the second column of the table, is the position calculated from the formula above and x is the value at that position)
like this:
=LET(p,J3,
range,I$2:I$21,
N,COUNT(range),
position,(N-1)*p+1,
lower,FLOOR(position,1),
fraction,MOD(position,1),
upper,CEILING(position,1),
lowerValue,INDEX(range,lower),
upperValue,INDEX(range,upper),
difference,upperValue-lowerValue,
lowerValue+fraction*difference)
I need to create a running product from a column of numbers (I could use a row, but a column is easier to demonstrate here.) The input might be any arbitrary array. In fact, in the application where I would deploy this, it will not be a range, but rather another dynamic array within a LAMBDA formula. Here is an example of the Input column of numbers and the desired Output from the formula:
Inputs
Expected Dynamic Array Output
10
10
8
80
3
240
4
960
5
4800
The formula would spill the results.
There are lots of solutions for a running total, but I've found no solution for a running product. I have tried a few different approaches, including SUBTOTAL and AGGREGATE with no success. I have also built a number of approaches that get the result, but are hard-coded to a fixed number of rows. I need the formula to adapt to any arbitrarily sized number of rows. The following formula is the closest I have gotten so far.
This LET formula delivers the result, but, as you can see is fixed to 5 rows:
=LET( a, {10;8;3;4;5},
v, SEQUENCE( ROWS(a) ), h, TRANSPOSE( v ),
stagr, (v - h + 1) * (v >= h),
m, IFERROR(INDEX( a, IF(stagr>0,stagr,-1), ), 1),
almost, INDEX(m,v,h) * INDEX(m,v,h+1) * INDEX(m,v,h+2) * INDEX(m,v,h+3) * INDEX(m,v,h+4),
result, INDEX( almost, , 1 ),
result )
The arbitrary array of numbers input is placed in the variable a.
The next step is to create some indexes that will be used to address these numbers: v is a sequence of vertical rows for each number in a and h is a the same sequence, but transposed into columns. stagr is an index matrix that is created from v and h that will later be used to address each item in a to form it into a multiplication matrix. If you replace the last result with stagr, you can see the shape of stagr. It just shifts a column down by one row until they are shifted all the way down.
Now we create the mulitplication matrix m using stagr by simply using INDEX, like this: INDEX(a,stagr). But this is not exactly what is needed because it takes the first row value (10) and replicates it because an INDEX of 0 is treated the same as 1. To get what we want, I forced an error by using and internal IF statement like this: INDEX( a, IF(stagr>0,stagr,-1) ) to replace the 0 results with -1. i.e. it will produce this:
Now, replace the errors with 1's by using IFERROR, so this explains how m is created and why. The result is a matrix like this:
and by multiplying m row-wise, we get the output we want, but this is where I fail.
For illustration, I created a variable almost that shows how I am trying to do a row-wise multiplication.
almost, INDEX(m,v,h) * INDEX(m,v,h+1) * INDEX(m,v,h+2) * INDEX(m,v,h+3) * INDEX(m,v,h+4)
You can see that I crudely multiplied one column times the next and the next... and using h + offset to get there. This produces the almost matrix and result just delivers the first column of that matrix, which contains the answer.
While an answer might be a good replacement for almost that would be dynamically sized, that is not my real question. I want a running product and I suspect that there is a wholly different approach than simply replacing my almost.
Just to be clear, the result must be a dynamic array that spills with no helper cells or CSE drag-down.
oh... and no VBA. (#stackoverflow - please add a no-VBA tag)
The only way I can find is to use DPRODUCT with OFFSET, but that requires a title row. It does not matter what is in the title row(it can even be empty), just that it is included.
=DPRODUCT(OFFSET(A1,0,0,SEQUENCE(COUNT(A:A),,2)),1,$ZZ1:$ZZ2)
The $ZZ1:$ZZ2 can be any empty cell reference.
If the values in A are dynamic then we can do:
=DPRODUCT(OFFSET(A1,0,0,SEQUENCE(ROWS(A2#),,2)),1,$ZZ:$ZZ)
There are plenty of interesting answers here. But, if summation is easy why not take logarithms of the number you want to multiply, sum those logarithms and then calculate the exponent of your sum to return to the product of the original numbers.
i.e. exploit the fact that ln(a * b) = ln(a) + ln(b)
Whilst not available to everybody (yet) we can use SCAN()
Formula in A1:
=SCAN(1,{10,8,3,4,5},LAMBDA(a,b,a*b))
The 1st parameter is our starting value, meaning the 1st calculation in the nested LAMBDA() is '1*10'.
The 2nd parameter can both take a 1D- & 2D-array (written or range-reference).
The 3rd parameter is a nested LAMBDA() where the result of our recursive function will then be used for the 2nd calculation; '10*8'. And the 3rd...etc. etc.
In the above sample a vertical array is spilled but when horizontal input is used this will obviously result in an horizontal spilled output. When a 2D-array is used this will spill a 2D-array as result.
Excel
Need to find nearest float in a table, for each integer 0..99
https://www.excel-easy.com/examples/closest-match.html explains a great technique for finding the CLOSEST number from an array to a constant cell.
I need to perform this for many values (specifically, find nearest to a vertical list of integers 0..99 from within a list of floats).
Array formulas don't allow the compare-to value (integers) to change as we move down the list of integers, it treats it like a constant location.
I tried Tables, referring to the integers (works) but the formula from the above web site requires an Array operation (F2, control shift Enter), which are not permitted in Tables. Correction: You can enter the formula, control-enter the array function for one cell, copy the formulas, then insert table. Don't change the search cell reference!
Update:
I can still use array operations, but I manually have to copy the desired function into each 100 target cells. No biggie.
Fixed typo in formula. See end of question for details about "perfection".
Example code:
AI4=some integer
AJ4=MATCH(MIN(ABS(Table[float_column]-AI4)), ABS(Table[float_column]-AI4), 0)
repeat for subsequent integers in AI5...AI103
Example data:
0.1 <= matches 0
0.5
0.95 <= matches 1
1.51 <= matches 2
2.89
Consider the case where target=5, and 4.5, 5.5 exist in the list. One gives -0.5 and the other +0.5. Searching for ABS(-.5) will give the first one. Either one is decent, unless your data is non-monotonic.
This still needs a better solution.
Thanks in advance!
I had another problem, which pushed to a better solution.
Specifically, since the Y values for the X that I am interested in can be at varying distances in X, I will interpolate X between the X point before and after. Ie search for less than or equal, also greater than or equal, interpolate the desired X, then interpolate the Y values.
I could go a step further and interpolate N - 1 to N + 1, which will give cleaner results for noisy data.
I have some delicate formula comparing numbers that are between 1 and 0, but Excel insists in round them up inside the formula, which was detected whilst evaluating it.
I have
=IF(C65=(LARGE(BA$65:BA$88;1));1;0)
where C65 = 0.91
and the largest number in the BA65:BA88 range is 1.
it detects incorrectly that C65 and 1 are equal and outputs the wrong result.
On evaluating i saw that it was rouding up 0.91 to 1.
i've tried using ROUND(C65,2) but no success
how can I force the formula to have a decimal vaule on the comparison step?
EDIT:
the 0.91 comes from a Cell with another formula
=IFERROR(IF(LARGE(C$65:C$88;$AA73)=LARGE(C$65:C$88;$AA72);LARGE(C$65:C$88;$AA73)-$AA73*0,01;LARGE(C$65:C$88;$AA73));0)
basically it is being used to get itens ranked from a list (index() and rank() werent helpful at all at achieving what i want so i decided to pick the numbers and make a decimal subtraction so i can find precise rank location with MATCH() only )
the results from BA65 to BA88 at the rows are
4,00 (from 4)
3,00 (from 3)
2,00 (from 2)
1,96 (from repeated 2)
1,00 (from 1)
0,94 (from repeated 1)
0,93 (from second repeated 1)
0,92 (for third repeated 1)
0,91 (forth)
0,90 (fifth)
0,89 (sixth)
0 is fallback value / not ranked
for some reason, after i changed the C65 formula to
=NOT(IFERROR(LARGE(C$65:C$88;$AA65));
IF(LARGE(C$65:C$88;$AA65)=LARGE(C$65:C$88;$AA64);
LARGE(C$65:C$88;$AA65)-$AA65*0,01;
LARGE(C$65:C$88;$AA65))
)
it got it right.
may be it was an issue with the IFERROR() output