I am working at moving my data over from Excel to Matlab. I have some data that I want to average based on multiple criteria. I can accomplish this by looping, but want to do it with matrix operations only, if possible.
Thus far, I have managed to do so with a single criterion, using accumarray as follows:
data=[
1 3
1 3
1 3
2 3
2 6
2 9];
accumarray(data(:,1),data(:,2))./accumarray(data(:,1),1);
Which returns:
3
6
Corresponding to the averages of items 1 and 2, respectively. I have at least three other columns that I need to include in this averaging but don't know how I can add that in. Any help is much appreciated.
For your single column, you don't need to call accumarray twice, you can provide a function handle to mean as the fourth input
mu = accumarray(data(:,1), data(:,2), [], #mean);
For multiple columns, you can use the row indices as the second input to accumarray and then use those from within the anonymous function to access the rows of your data to operate on.
data = [1 3 5
1 3 10
1 3 8
2 3 7
2 6 9
2 9 12];
tmp = accumarray(data(:,1), 1:size(data, 1), [], #(rows){mean(data(rows,2:end), 1)});
means = cat(1, tmp{:});
% 3.0000 7.6667
% 6.0000 9.3333
Related
I have the following table in Excel.
A B C D E
1 i x y mu sigma
2 0 1 2 =average(b$1:b1,b3:b$12) =stdev.s(b$1:b1,b3:b$12)
3 1 3 4 =average(b$1:b2,b4:b$12) =stdev.s(b$1:b2,b4:b$12)
4 2 2 1 =average(b$1:b3,b5:b$12) =stdev.s(b$1:b3,b5:b$12)
5 3 1 2 ... ...
6 4 2 5
7 5 4 7
8 6 8 1
9 7 2 3
10 8 5 9
11 9 1 3
The ith mu calculates the average without the ith observation—the leave-one-out average. I can also calculate the leave-one-out standard deviations, but how can I do this for the leave-one-out correlations then? correl requires two arrays and I cannot feed two or more distant cells as one array using commas. Can I input non-consecutive cells as one array? For example, I tried =correl((b$1:b1,b3:b$12),(c$1:c1,c3:c$12)), but failed. Thanks for your reading.
Because CORREL takes arrays we can use an array formula:
=CORREL(INDEX(B:B,N(IF({1},MODE.MULT(IF(ROW($B$2:$B$11)<>ROW(),ROW($B$2:$B$11)*{1,1}))))),INDEX(C:C,N(IF({1},MODE.MULT(IF(ROW($C$2:$C$11)<>ROW(),ROW($C$2:$C$11)*{1,1}))))))
Use this easier array formula as per the OP(I overthought this):
=CORREL(IF(ROW($B$2:$B$11)=ROW(),"",$B$2:$B$11),IF(ROW($C$2:$C$11)=ROW(),"",$C$2:$C$11))
Being an array formula one needs to use Ctrl-shift-Enter instead of Enter when exiting edit mode.
This will pass two arrays of only the numbers not on the row where the formula is placed.
Now, eventually when microsoft releases the dynamic array formula into Office 365 this can be simplified using FILTER()
=CORREL(FILTER($B$2:$B$11,ROW($B$2:$B$11)<>ROW()),FILTER($C$2:$C$11,ROW($C$2:$C$11)<>ROW()))
I have a rather big pandas data frame (more than 1 million rows) with columns containing either strings or numbers. Now I would like to split the strings in one column before the expression "is applied".
An example to explain what I mean:
What I have:
a b description
2 4 method A is applied
10 5 titration is applied
3 1 computation is applied
What I am looking for:
a b description
2 4 method A
10 5 titration
3 1 computation
I tried the following,
df.description = df.description.str.split('is applied')[0]
But this didn't bring the desired result.
Any ideas how to do it? :-)
You are close, need str[0]:
df.description = df.description.str.split(' is applied').str[0]
Alternative solution:
df.description = df.description.str.extract('(.*)\s+is applied')
print (df)
a b description
0 2 4 method A
1 10 5 titration
2 3 1 computation
But for better performance use list comprehension:
df.description = [x.split(' is applied')[0] for x in df.description]
you can use replace
df.description = df.description.str.replace(' is applied','')
df
a b description
0 2 4 method A
1 10 5 titration
2 3 1 computation
I'm facing an issue with the use of Redim in VBA of MS Excel in a 3D array. How can I append a new array with different dimension than the first one to the first array and then another one and then ...; and then be able to read a specific group of data from this newly created matrix (array) with different number of columns in each row, for example, the 2nd group of data. Here is an example to clarify the matter.
Arr1 is in 3x3
Arr2 is in 5x5
Arr3 is in 7x7
Arr4 is in 9x9
Arr5 is in 11x11 dimensions.
Now if I append them together the resulting matrix should be in this format - for example:
1 2 3
4 5 6
7 8 9
1 2 3 4 5
6 7 8 9 1
2 3 4 5 6
7 8 9 1 2
3 4 5 6 7
...
...
1 2 3 ... 11
1 2 3 ... 11
(another 8 rows)
1 2 3 ... 11
(total of 11 rows)
Next I want to be able to read a group of data (sub-matrix) from the resulting matrix, for example the 2nd group of data, which is in 5x5.
If I use Redim in between each loop, while appending those new group of arrays and so to adjust the dimensions of the first matrix and continuing to do so, the last dimension will be in 11x11, which does not give me the needed dimension; in this case 5x5.
I was wondering, if there is any recommendation you could kindly provide. Thank you.
I'm looking for a way to compare multiple rows with data to each other, trying to find the best possible match. Each number in every column must be an approximately match the other numbers in the same column.
Example:
Customer #1: 1 5 10 9 7 7 8 2 3
Customer #2: 10 5 9 3 5 7 4 3 2
Customer #3: 1 4 10 9 8 7 6 2 2
Customer #4: 9 5 6 7 2 1 10 5 6
In this example customer #1 and #3 is quite similar, and I need to find a way to highlight or sort the rows so I can easily find the best match.
I've tried using conditional formatting to highlight the numbers that are the similar, but that is quite confusing, because the amount of data is quite big.
Any ideas of how I could solve this?
Thanks!
The following formula entered in (say) L1 and pulled down gives the best match with the current row based on the sum of the absolute differences between corresponding cells:-
=MIN(IF(ROW($C$1:$K$4)<>ROW(),(MMULT(ABS($C1:$K1-$C$1:$K$4),TRANSPOSE(COLUMN($C$1:$K$4))^0))))
It is an array formula and must be entered with CtrlShiftEnter.
You can then sort on column L to bring the customers with lowest similarity scores to the top or use conditional formatting to highlight rows with a certain similarity value.
EDIT
If you wanted to penalise large differences in individual columns more heavily than small differences to try and avoid pairs of customers which are fairly similar except for having some columns very different, you could try something like the square of the differences:-
=MIN(IF(ROW($C$1:$K$4)<>ROW(),(MMULT(($C1:$K1-$C$1:$K$4)^2,TRANSPOSE(COLUMN($C$1:$K$4))^0))))
then the scores for your test data would come out as 7,127,7,127.
I'm assuming you want to compare customers 2-4 with customer 1 and that you are comparing only within each column. In this case, you could implement a 'scoring system' using multiple IFs. For example,:
A B C D E
1 Customer 1 1 1 2
2 Customer 2 1 2 2
3 Customer 3 0 1 0
you could use in E2
=if(B2=$B$1,1,0)+if(C2=$C$1,1,0)+if(D2=$D$1,1,0)
This will return a 'score' of 1 when you have a match and a 'score' of 0 when you don't. It then adds up the scores and your highest value will be your best match. Copying down would then give
A B C D E
1 Customer 1 1 1 2
2 Customer 2 1 2 2 2
3 Customer 3 0 1 0 1
so customer 2 is the best match.
I'm not sure how to ask this question without illustrating it, so here goes:
I have results from a test which has tested peoples attitudes (scores 1-5) to different voices on a 16 different scales. The data set looks like this (where P1,P2,P3 are participants, A, B, C are voices)
Aformal Apleasant Acool Bformal etc
P1 2 3 1 4
P2 5 4 2 4
P3 1 2 4 3
However, I want to rearrange my data to look like this:
formal pleasant cool
P1A 3 3 5
P1B 2 1 6
P1C etc
P1D
This would mean a lot more rows (multiple rows per participant), and a lot fewer columns. Is it doable without having to manually reenter all the scores in a new excel file?
Sure, no problem. I just hacked this solution:
L M N O P Q
person# voice# formal pleasant cool
1 1 P1A 2 3 1
1 2 P1B 4 5 2
1 3 P1C 9 9 9
2 1 P2A 5 4 2
2 2 P2B 4 4 1
2 3 P2C 9 9 9
3 1 P3A 1 2 4
3 2 P3B 3 3 2
3 3 P3C 9 9 9
Basically, in columns L and M, I made two columns with index numbers. Voice numbers go from 1 to 3 and repeat every 3 rows because there are nv=3 voices (increase this if you have voices F, G, H...). Person numbers are also repeated for 3 rows each, because there are nv=3 voices.
To make the row headers (P1A, etc.), I used this formula: ="P" & L2 & CHAR(64+M2) at P1A and copied down.
To make the new table of values, I used this formula: =OFFSET(B$2,$L2-1,($M2-1)*3) at P1A-formal, and copied down and across. Note that B$2 corresponds to the cell address for P1-Aformal in the original table of values (your example).
I've used this indexing trick I don't know how many times to quickly rearrange tables of data inherited from other people.
EDIT: Note that the index columns are also made (semi)automatically using a formula. The first nv=3 rows are made manually, and then subsequent rows refer to them. For example, the formula in L5 is =L2+1 and the formula in M5 is =M2. Both are copied down to the end of the table.