I'm performing array calculations that are taking a long time to complete. I'd like to optimize my formulas some more. All of the formulas are of the same nature - they perform some high-level function (Average, Slope, Min, Max) across a column of values. However, not all cells in a column are included in the array. I use multiple IF criteria to choose which cells get included. All comparisons are made to the current row. Here's an example of the data:
A B C D E
1 Company Generation Date Value ToCalculate
2 Abc 1 1/1/2010 5.6
3 ... ... ... ... ...
E would look something like this
{=Average(If(A2=A2:A1000, If(B2=B2:B1000, If(C2 > C2:C1000, D2:D1000))))}
So once E2 is calculated then I have to autofill down column E. Column F, G, H, ... Uses the same approach, either selects different values to operate on or a different function to perform. My dataset is quite large, and with only a few of these the spreadsheet is taking an hour plus to compute. Every so often I'll add a fourth criteria, all other criteria being the same.
Is there an efficiency? Some thoughts:
Can I use a single array per column instead of thousands per column?
Can I condense the first three criteria so that the output is row numbers? Perhaps then subsequent formulas won't have to search for multiple criteria but can just perform the function?
or somehow build the crtieria up? So a new column returns all rows where the company is the same. another column returns all rows from the first column where generation is the same...and so on...
For the Average you can do without arrays:
=AVERAGEIFS(D2:D$1000,A2:A$1000,A2,B2:B$1000,B2,C2:C$1000,"<="&C2)
As there is also a COUNTIFS and a SUMIFS, I think your slopes could be calculated the same way.
For the rest of the functions (max, min, etc), we should analyze case by case.
I did a slight performance test, and this is apparently better, but of course my datasets are just mocked.
HTH!
Note: Excel 2007 and up only!
Edit - Answering your comment.
Without knowing the dimensions of the problem is difficult to give advice, but I'll risk one anyway:
You could write a VBA function that:
1) Generates a new sheet for each company-generation pair
2) Sorts the data in those sheets by date
3) Adds the formulas to those sheets (no conditionals needed in this context)
4) Recalculates and Gets the results from those formulas and populates the original sheet
5) Deletes the auxiliary sheets
To capture the rows and re-use try this approach:
Sort the data by Company & Generation.
Make a unique list of Companies & generations (use Advanced Filter, Unique Only, Copy)
For each Company generation pair in the list build 2 columns of formulae. First column gives the count of rows in the data for this pair (use COUNTIFS), second column gives the first row in the data for this pair (=first row for previous pair+count of rows for previous pair). Then you can use a function like OFFSET to return only the rows of data for the Company-Generation pair and embed this inside the final function/array formula (AVERAGEIFS etc) You could extend this sort and count approach to include dates if you wanted. There is a drawback that if the list of cities and generations change you have to change the list of uniques and associated formulas. There are examples of this approach on my website athttp://www.decisionmodels.com/optspeedk.htmhttp://www.decisionmodels.com/optspeedj.htm
Related
I'm having trouble filtering an excel table. M, it is a set of two rows from two tables, where it is necessary to find duplicates.
2 rows with duplicates
Some idents are repeated, they are present both in the current and previous months. In the example below, with the help with this function =IFERROR(MATCH(A2;B:B;0); "NO"), I obtained information about which data from last month is repeated in the current month and exactly in which row it is located. The code for determining whether it is repeated is as follows =COUNTIFS($A$2:$B$13;A2)>1
duplicates and if repeated
I would like to retrieve only duplicates from the list, I tried the code =IFERROR(INDEX(A:A;SMALL(IF(NOT(D$2:D$104=TRUE);ROW(B2)-ROW(INDEX(B2;1;1))+1);ROW(G:G)));" ERROR")to get the ones that are repeat and skip those ones that arent, but the result is not as desired. In line G, you can see an example of how Excel gives me data regarding the entered function. In cell H, it is shown how I would like a new row to be created with only duplicates.
Current vs. desired display
In this example, the columns are a bit small, but in reality there could be at least a thousand rows, so I would need help filtering those.
You implied these columns were present in two different tables. So I used Tables with structured references. You can convert to normal addressing if you require that instead.
If you have Windows Excel 2021 or later, you can use:
=FILTERXML("<t><s>" &TEXTJOIN("</s><s>",,UNIQUE(LastMonth[Last month marks],FALSE,TRUE),UNIQUE(CurrentMonth[Current Month],FALSE,TRUE))& "</s></t>","//s[following::*=.]")
Create a list of distinct items for each row
Create an XML by concatenating the items into an array using Textjoin
Extract only those items that are followed by an identical item
With your earlier version of Excel, again, I would still use Tables and structured references but I would also use a Helper Column
D2: =IFERROR(MATCH(lastMonth[#[last month]],currentMonth[current month],0),"NO") *and fill down*
E2: =IFERROR(INDEX(currentMonth[current month], AGGREGATE(15,6,[Duplicates in Which Row],ROWS($1:1))),"")
So, I've searched for an answer to this, but I can't find anything. Hopefully some Excel guru out there has an easy answer.
CONTEXT
I have a sheet that has two columns; a list of airport codes (Col A) and a list of fuel gallons (Col B). Column A has a bunch of duplicate entries, column B is always different. It's basically a giant list of fill-up events for aircraft over time at different airports. The airports can be the same, because it's one row per fill-up event.
PROBLEM
What I want to do is have a formula that takes the enter data set, finds all identical entries in Col A, sums the Col B values for the matches, and spits out the result on a separate sheet with one entry for every set/match.
OTHER STUFF
I do not have a reference list for Column A and I would rather not create one since there are thousands of entries. I would like to just write a formula that does all this at once, using the data itself as the reference.
All the answers I find are "create a reference list on a separate sheet", and it's driving me crazy!
Thanks in advance for any help!
-rt
Sounds that you need a formula version of remove duplicated for column A, and a simple sumif for column B.
Column A
=IFERROR(INDEX(Data!A$1:A$1000,SMALL(IF(
MATCH(Data!A$1:A$1000,Data!A$1:A$1000,0)=ROW(Data!A$1:A$1000),ROW(Data!A$1:A$1000)),ROW())),"")
Array Formula so please press Ctrl + Shift + Enter to complete it. After that you might see a {} outside the formula.
Column B
=SUMIF(Data!A$1:A$1000,A2,Data!B$1:B$1000)
Just change the range for your data.
Reminders: The formula in columnA should starts from Row#1, or you have to add some offset constant for adjustments.
Since the returning value of MATCH() represents the position of the key in the given array. If we wanted it to be equal to its row number, we have to add some constant if the array is not started from ROW#1. So the adjustment of data in Range(B3:B1000) is below.
=IFERROR(INDEX('Event Data'!B$3:B$1000,SMALL(IF(
MATCH('Event Data'!B$3:B$1000,
'Event Data'!B$3:B$1000,0)+2=ROW('Event Data'!B$3:B$1000),
ROW('Event Data'!B$3:B$1000)),ROW())-2),"")
Further more, the range should exactly the same as the data range. If you need it larger than the data range for future expandability, an IFERROR() should added into the formula.
=IFERROR(INDEX('Event Data'!B$3:B$1000,SMALL(IFERROR(IF(MATCH(
'Event Data'!B$3:B$1000,'Event Data'!B$3:B$1000,0)+2
=ROW('Event Data'!B$3:B$1000),
ROW('Event Data'!B$3:B$1000)),FALSE),ROW())-2),"")
Lastly, I truly recommended that you should use the Remove Duplicated built in excel since the array formula is about O(n^2) of time complexity and memory usage also. And every time you entered any data in even other cells, it will automatically re-calculate all formulas once the calculation option in your excel is automatic. This will pull down the performance.
I'm looking for a way to count the number of times two cells appear side by side in Excel - like intersections. Sometimes in my data (of about 550 records) A Road will appear next to B Road, which is a count of 1. If it occurs again, later in the data the count would be 2. But if B Road appears in the first column and A Road appears in the second column, I can't find a way to make that number 3 in the count.
I've tried concatenating the data, but I need to be able to write this formula without inserting specific criteria (like searching for A Road) because it would be easier in that case to do this manually. Does anyone know if there's a formula for find the occurrence of the same two variables between two columns without specific criteria?
If the order of values in the two columns were important (i.e. A Road followed by B Road is different than B Road followed by A Road), then a simple pivot table would provide the counts you need. You would just put Col M on the rows, Col N on the columns and the count of any field as the value.
But the OP has said that A Road followed by B Road should be counted in the same total as B Road followed by A Road. Let's modify the concatenation in Col O to become =IF(M2<N2,CONCAT(M2," & ",N2),CONCAT(N2," & ",M2)). That provides a canonical form of each combination regardless of order. Having done that, then it is again an easy matter to create a pivot table that shows all the required counts -- just put the concatenated value on the rows and the count as the value.
If I correctly understand your intent, then try this array formula : =SUM(IF(EXACT(B$2:B3&C$2:C3,B3&C3),1,""))+SUM(IF(EXACT(B$2:B3&C$2:C3,C3&B3),1,"")).The second sum formula accounts for any reverse order of adjacent street occurrences.Then copy down the column as needed. Enter with Ctrl+Shift+Enter.
I am trying to assemble a index/match combination and am having trouble figuring out how to make it work. I have experience with a lot of the formula types in excel, but unfortunately I am pretty ignorant when it comes to these functions.
I will explain what I am trying to do first, but I have attached 3 images at the end that will probably make things more clear.
In order to identify the specific values I want, I am having to use helper cells. These helper cells are denoted with the (helper) tag in the pictures. These cells go through and grab the adjusted closing price of the stock (column A) at the beginning (column C) and the end (Column D) of a dynamically calculated period.
I would like to consolidate these values into numerical order in columns F and G. The thought is that the first non zero number in C/D is belongs to the first predefined period and should go into columns F/G beside the #1 (column E). This gets carried on through all of the periods (ex: 2nd non zero goes beside the number 2, third nonzero number goes beside the number 3 etc.)
This is just an example of one stock. I need the function or formula to be dynamic enough to work on a wide variety of distributions. Sometimes there are up to 100 dynamically calculated periods within the stock analysis.
Below are the images that should provide more clarity
Image 1 is an example of what the data looks like
Image 2 is a crudely drawn example of how I would like the data to move
Image 3 is the desired result
Image 1
Image 2
Image 3
Updated image for Scott Craner showing out of order results
Please let me know if I can clarify any confusion.
If you just need to return the first value of each period (column C) and the last value of each period (column D), you could use index match and lookup to do this without even using helper columns.
Try this in cell F2
=INDEX(A2:A50,MATCH(E2,B2:B50,0))
And this in cell G2
=LOOKUP(E2,B2:B50,A2:A50)
Depending on much variance is in your overall number of rows, you could use indirect references in the formulas to dynamically update the ranges.
Example:
=INDEX(A2:INDIRECT("A"&COUNTA(A:A)),MATCH(E2,B2:INDIRECT("B"&COUNTA(A:A)),0))
You will need to open macro. Then do the following in recorded macro.
+ Filter only non-null value in C/D
+ Select whole column in C/D then copy the whole column
+ Turn off Filter
+ Paste the whole C/D in F/G
+ Stop macro
Gook Luck
Put this formula if F2:
=INDEX(INDEX(C:C,MATCH($E2,$B:$B,0)):INDEX(C:C,MATCH($E2,$B:$B,0)+COUNTIF($B:$B,$E2)-1),MATCH(1E+99,INDEX(C:C,MATCH($E2,$B:$B,0)):INDEX(C:C,MATCH($E2,$B:$B,0)+COUNTIF($B:$B,$E2)-1)))
Copy over one column and down the list.
I have column A for Job Type.
I have column B for Experience Range.
I have column C for salary.
These are in a sheet named 'Data'.
I have created a pivot table with Job Type as rows and Experience Range as columns. (in another sheet)
I want to find median for each cell in the Pivot Table.
How do I do that?
e.g. how do I find the median of Salary for all 'Stage Manager' (cell A2) with 'less than 2 years' (cell B1) experience?
Please help. Thank you!
As you've no doubt discovered, the median isn't listed on the grouping options (sum, min, max, count, etc).
There doesn't appear to be a direct way around the problem; however, you may be able to hack your way around it by adding custom a formula inside the table. Then use the median() function in the formula.
Another approach is abandon pivot tables and use array formulas instead.
You can use * for and and + for or. FALSE is coded as 0 and TRUE as 1.
0*1 = 1, so if only one condition is met it will be FALSE(AND)
1+0 or 1+1 is >0, so if only one condition is met, it will still be >0 (OR)
You can use this fact to do a test as to whether your condition is >0 and then transform your array, so that if the condition is true, the array is the array but otherwise it equal . This will basically eliminate all rows where the conditions are not met (making them count as blank in any array formula)
You can use this tick for a large number of AND and OR conditions.
If you use name manager to name your arrays, then you can very easily vary conditions very fast and create a simulated pivot table.
You can then use the aggregate function to ignore hidden rows (e.g. aggregate can calculate median, percentiles and other stuff), and then the filter function can effectively work as a pivot filter.
I usually make my first column a test column that tests if a row is hidden or not and returns 1 if visible and 0 otherwise. Then I build that condition into my condition stack to act as an additional AND switch. This means suddenly your formulas react to the filter to only consider values that are filtered in.
This stuff is really great to do multi-conditional calculations on a large amount of data without the need to manually calculate after each filter - the calculations happen on the fly, as you filter!
Calculations can take 3-5 seconds depending on how many conditions you have and the size of your raw data, so not quite as fast as a pivot table, but very convenient.