Suppose an Excel file has vectorized 2-by-2 symmetric matrices line by line as follows:
line
A: element11
B: element22
C: element12=element21
D: DETERMINANT
1
2
3
0
6
2
5
7
0
35
3
11
13
0
143
…
…
…
…
…
Thus, the first value of DETERMINANT column can be computed from a matrix {2 0,0 3}, the second value can be from a matrix {5 0,0 7}, and so on. Instead of reshaping the A1:C1 values, the D1 value can be directly computed via =MDETERM(IF(A1:B1=TRANSPOSE(A1:B1),A1:B1,C1)) because IF can actually reshape the A1:C1 values to the diagonal matrix of interest.
The point is that, in Excel, matrix functions such as MDETERM directly require matrices, while the matrices in the data are stored row by row after vectorization. Unfortunately, the IF solution above can only reshape 2-by-2 symmetric matrices and handle neither asymmetric matrices nor matrices with more than two off-diagonal elements. Do we have any other solutions more general than the illustrated IF approach to unvectorize matrices?
In addition, I am also interested in vectorizing matrices in Excel as well; for example, the matrix in A1:C1 can be reshaped, inverted, and then restored in the D1:G1 area. Thanks for your help!
I think you might have meant
=IF(COLUMN(A1:B1)=TRANSPOSE(COLUMN(A1:B1)),A1:B1,C1)
But you can certainly expand on this idea. Suppose you have the diagonal elements of a symmetric 4X4 matrix in A1:D1 and the off-diagonal elements in E1:J1.
Then you could use
=IF(ROW(A1:D4)=COLUMN(A1:D4),INDEX(A1:J1,ROW(A1:D4)),
IF(ROW(A1:D4)<COLUMN(A1:D4),INDEX(E1:J1,(ROW(A1:D4)-1)*4+COLUMN(A1:D4)-ROW(A1:D4)*(ROW(A1:D4)+1)/2),
INDEX(E1:J1,(COLUMN(A1:D4)-1)*4+ROW(A1:D4)-COLUMN(A1:D4)*(COLUMN(A1:D4)+1)/2)))
A bit more general with Excel 365 Let function:
=LET(N,4,mat,A1:INDEX(A1:Z26,N,N),
IF(ROW(mat)=COLUMN(mat),
INDEX(1:1,ROW(mat)),
IF(ROW(mat)<COLUMN(mat),
INDEX(1:1,(ROW(mat)-1)*N+COLUMN(mat)-ROW(mat)*(ROW(mat)+1)/2+N),
INDEX(1:1,(COLUMN(mat)-1)*N+ROW(mat)-COLUMN(mat)*(COLUMN(mat)+1)/2+N))))
Related
Number of days before vaccination (x) bacteria count (1000 pieces) (y)
1 112
2 148
3 241
4 363
5 585
I Need to find 2 things
first calculate with growth function third day count and I have been counted.
=GROWTH(I3:I4;H3:H4;H5)
But I need to calculate parameters of growth function( 𝑌=𝑎.𝑏^𝑋)
So how to calculate a and b? I tried to use excel solver but i didn't solve
Seems like LOGEST is designed for what you want:
the LOGEST function calculates an exponential curve that fits your
data and returns an array of values that describes the curve. Because
this function returns an array of values, it must be entered as an
array formula.
Note that there is a difference in how the equation is expressed on an x-y chart with an exponential trendline, and by the function. On the chart, m is expressed as a power of e, so to convert the value returned by the formula to be the same as what is seen on the chart, you would do something like:
=LN(INDEX(LOGEST(known_y,known_x),1))
You are dealing with an exponentional growth, you want to describe. The basic way to handle this, is to take the logarythm of the whole thing, and apply linear regression on that, using the Linest() function.
I am confused by NumPy concepts of array and vector, let's say we have a 1-D array as below. From the 'shape' method, I can see the dimension. (10,) means 1 dimension with 10 elements.
a = np.arange(10)
print(a)
a.shape
[0 1 2 3 4 5 6 7 8 9]
(10,)
Now I got to know a method called np.newaxis, to convert the array in to a row vector. So I am wondering in NumPy do we assume a row or column vector always has 2-dimension? (I think in linear algebra, vector can live in any dimension, or there is a conceptual difference in NumPy and linear algebra when we are referring to 'dimension'?). Since the vector was converted by add 1 dimension by calling np.newaxis.
print(a[np.newaxis:])
print(a[np.newaxis,:].shape)
[0 1 2 3 4 5 6 7 8 9]
(1, 10)
Please note that DIMENSION word may take different sense in different context. For example in linear algebra (1, 1) is a vector in the 2D space and (1, 1, 1) is the vector in the 3D space and both of them are 1D arrays in programming langages. The collection of 3D vectors is matrix in linear algebra and 2D array in programming languages.
In terms of linear algebra shape (10,) is 1 10-dimensional vector or 10 scalar values. Shape (10,2) is 2 10-dimensional vectors or 10 2-dimensional vectors.
Let's consider linear algebra matrix multiplication formula:
AB(i,j) = sum(A[i,k] * B[k,j])
This formula remains valid for vectors if we assume that the row vector is a matrix of dimension (1, N), and the column vector is a matrix of dimension (N, 1).
NumPy uses the same approach. But NumPy allows not only 2D arrays, but also 1D, 3D and so on. Such arrays are useful for other computational models. If you are interested in this, you can read more about tensors.
You can rearrange elements of ndarray with .reshape(...) method. When changing the shape of the array, all elements remain in place, but the addressing of the elements changes.
If we assume that the row vector and column vector are special objects, we will be forced to complicate the calculation rules, which is very impractical.
Simply run the following:
a.reshape(-1,1)
Given a numpy 1d array, which we might conceptually call a vector since it is a one-dimensional sequence of values, it is useful to be able to create what could conceptually be called a column vector or a row vector with the same values as the original vector.
In numpy, the conceptual column vector corresponds to a 2d array with 1 column, and the conceptual row vector corresponds to a 2d array with 1 row.
These can be created easily as follows, using either the numpy constant [np.newaxis][1] or simply None (since np.newaxis is defined as an alias to None):
import numpy as np
vec = np.array([0, 1, 2])
row_vec = vec[None]
col_vec = vec[:, None]
print(vec, '\n')
print(row_vec, '\n')
print(col_vec, '\n')
Output:
[0 1 2]
[[0 1 2]]
[[0]
[1]
[2]]
if you want to reshape the array you should use:
new_array = np.reshape(old_array, (-1, 2))
where -1 is the size of the new array from the size of the given array.
I have 3 questions:
1)
The confusion matrix for sklearn is as follows:
TN | FP
FN | TP
While when I'm looking at online resources, I find it like this:
TP | FP
FN | TN
Which one should I consider?
2)
Since the above confusion matrix for scikit learn is different than the one I find in other rescources, in a multiclass confusion matrix, what's the structure will be? I'm looking at this post here:
Scikit-learn: How to obtain True Positive, True Negative, False Positive and False Negative
In that post, #lucidv01d had posted a graph to understand the categories for multiclass. is that category the same in scikit learn?
3)
How do you calculate the accuracy of a multiclass? for example, I have this confusion matrix:
[[27 6 0 16]
[ 5 18 0 21]
[ 1 3 6 9]
[ 0 0 0 48]]
In that same post I referred to in question 2, he has written this equation:
Overall accuracy
ACC = (TP+TN)/(TP+FP+FN+TN)
but isn't that just for binary? I mean, for what class do I replace TP with?
The reason why sklearn has show their confusion matrix like
TN | FP
FN | TP
like this is because in their code, they have considered 0 to be the negative class and one to be positive class. sklearn always considers the smaller number to be negative and large number to positive. By number, I mean the class value (0 or 1). The order depends on your dataset and class.
The accuracy will be the sum of diagonal elements divided by the sum of all the elements.p The diagonal elements are the number of correct predictions.
As the sklearn guide says: "(Wikipedia and other references may use a different convention for axes)"
What does it mean? When building the confusion matrix, the first step is to decide where to put predictions and real values (true labels). There are two possibilities:
put predictions to the columns, and true labes to rows
put predictions to the rows, and true labes to columns
It is totally subjective to decide which way you want to go. From this picture, explained in here, it is clear that scikit-learn's convention is to put predictions to columns, and true labels to rows.
Thus, according to scikit-learns convention, it means:
the first column contains, negative predictions (TN and FN)
the second column contains, positive predictions (TP and FP)
the first row contains negative labels (TN and FP)
the second row contains positive labels (TP and FN)
the diagonal contains the number of correctly predicted labels.
Based on this information I think you will be able to solve part 1 and part 2 of your questions.
For part 3, you just sum the values in the diagonal and divide by the sum of all elements, which will be
(27 + 18 + 6 + 48) / (27 + 18 + 6 + 48 + 6 + 16 + 5 + 21 + 1 + 3 + 9)
or you can just use score() function.
The scikit-learn convention is to place predictions in columns and real values in rows
The scikit-learn convention is to put 0 by default for a negative class (top) and 1 for a positive class (bottom). the order can be changed using labels = [1,0].
You can calculate the overall accuracy in this way
M = np.array([[27, 6, 0, 16], [5, 18,0,21],[1,3,6,9],[0,0,0,48]])
M
sum of diagonal
w = M.diagonal()
w.sum()
99
sum of matrices
M.sum()
160
ACC = w.sum()/M.sum()
ACC
0.61875
That's my first question here ever, though I am reading questions here since a few year.
I am looking for a way to do the following with excel formula
count how many are line matching a criteria. Sounds maybe easy, but so far I didn't manage it, probably because I didn't do it the right way.
I have a table of this kind (here pets, but also work with any "object" array, like worker and their efficiency)
01.10.2018 02.10.2018 03.10.2018
Menu Wg Sz Menu Wg Sz Menu Wg Sz
Lassie Dry food 23 65 Dry food 22 65 Dry food 23 65
Fusel Meat fodder 12 49 Dry food 14 49 Fish fodder 13 49
Bobo Fish fodder 33 86 Meat fodder 32 86 Meat fodder 34 86
I am asking myself the questions like this:
How many pets ate Fish fodder?
How many pets are under 50cm?
I can do easily this on a row level and then add a sum cell (let's say in A column):
COUNTIF(3:3,"Fish fodder")
COUNTIF(4:4,"Fish fodder")
COUNTIF(5:5,"Fish fodder")
COUNTIF(A:A,">0")
But I am looking for a way to do this in a formula for single cell.
I was thinking to use the crtl+shif+enter way, but then i also need to do it on each row an extra cell to be able cumulate the results.
I hope someone can help.
Thank you.
According the COUNTIF formula you gave, I guess this is something you need.
B9 =SUMPRODUCT(--(MMULT(--($B$3:$J$5=$A9),TRANSPOSE(COLUMN($B$3:$J$3)))>0))
B10 =SUMPRODUCT(--(MMULT(--($B$3:$J$5=$A10),TRANSPOSE(COLUMN($B$3:$J$3)))>0))
B11 =SUMPRODUCT(--(MMULT(--(($B$3:$J$5<50)*(($B$2:$J$2)="Sz")),TRANSPOSE(COLUMN($B$3:$J$3)))>0))
All formulas here are Array Formula so please press Ctrl + Shift + Enter to complete them.
The trick is, in matrix [n x m]*[m x 1] = [n x 1]. However in excel, matrix * matrix directly is not a matrix multiplication [#1]. Array * array returns an array with a11*b11, a12*b12, a13*b13 and so on. We have to use a formula called MMULT for matrix multiplying.
Therefore we built up a [3 x 9] matrix first, and we compare it with the criteria "Dry food" then. We get a [3 x 9] matrix full of True or False, so we add double minus sign before the matrix, forcing them become 1 and 0.
The TRANSPOSE is for generating a [9 x 1] matrix, the value is actually not so important once they are greater than 0. Actually we can use a ROW(1:9) and the effect will be the same. However not everyone knows how to adjust the reference in ROW(). A benefit of TRANSPOSE(COLUMN()) is that the reference inside is just the same as the origin data area.
After executing MMULT, the result become a [3 x 1] matrix. And if it is matched with the criteria, the value is greater than 0, others will be 0. So the next part is checking every elements in side the matrix is >0 or not. And then we add a double minus sign again for converting the boolean to 0 and 1. The last part here is simply sum them up by SUMPRODUCT.
[#1] More about matrix multiplication here: https://en.wikipedia.org/wiki/Matrix_multiplication
You could do this with a simple array formulas, they would need to be configured specific to each question. Here are some examples:
Specific Name, Date, and Food:
Food Type By Date:
These are using array multiplication, basically you end up with an array of 1s and 0s and you just sum them.
These are array formulas and must be confirmed with Ctrl+Shift+Enter
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 1 year ago.
Improve this question
I am doing my graduation project in the field of computer vision, and i have only taken one course in statistics that discussed very basic concepts, and now i am facing more difficulty in rather advanced topics, so i need help (book, tutorial, course, ..etc) to grasp and review the basic ideas and concepts in statistics and then dive into the details (statistical details) used in computer vision.
You can calculate False Positives/False negatives, etc with this Confusion Matrix PyTorch example:
import torch
def confusion(prediction, truth):
""" Returns the confusion matrix for the values in the `prediction` and `truth`
tensors, i.e. the amount of positions where the values of `prediction`
and `truth` are
- 1 and 1 (True Positive)
- 1 and 0 (False Positive)
- 0 and 0 (True Negative)
- 0 and 1 (False Negative)
"""
confusion_vector = prediction / truth
# Element-wise division of the 2 tensors returns a new tensor which holds a
# unique value for each case:
# 1 where prediction and truth are 1 (True Positive)
# inf where prediction is 1 and truth is 0 (False Positive)
# nan where prediction and truth are 0 (True Negative)
# 0 where prediction is 0 and truth is 1 (False Negative)
true_positives = torch.sum(confusion_vector == 1).item()
false_positives = torch.sum(confusion_vector == float('inf')).item()
true_negatives = torch.sum(torch.isnan(confusion_vector)).item()
false_negatives = torch.sum(confusion_vector == 0).item()
return true_positives, false_positives, true_negatives, false_negatives
You could use nn.BCEWithLogitsLoss (remove the sigmoid therefore) and set the pos_weight > 1 to increase the recall. Or further optimize it with using Dice Coefficients to penalize the model for false positives, with something like:
def Dice(y_true, y_pred):
"""Returns Dice Similarity Coefficient for ground truth and predicted masks."""
#print(y_true.dtype)
#print(y_pred.dtype)
y_true = np.squeeze(y_true)/255
y_pred = np.squeeze(y_pred)/255
y_true.astype('bool')
y_pred.astype('bool')
intersection = np.logical_and(y_true, y_pred).sum()
return ((2. * intersection.sum()) + 1.) / (y_true.sum() + y_pred.sum() + 1.)
IOU Calculations Explained
Count true positives (TP)
Count false positives (FP)
Count false negatives (FN)
Intersection = TP
Union = TP + FP + FN
IOU = Intersection/Union
The left side is our ground truth, while the right side contains our predictions. The highlighted cells on the left side note which class we are looking at for statistics on the right side. The highlights on the right side note true positives in a cream color, false positives in orange, and false negatives in yellow (note that all others are true negatives — they are predicted as this individual class, and should not be based on the ground truth).
For Class 0, only the top row of the 4x4 matrix should be predicted as zeros. This is a rather simplified version of a real ground truth. In reality, the zeros could be anywhere in the matrix. On the right side, we see 1,0,0,0, meaning the first is a false negative, but the other three are true positives (aka 3 for Intersection as well). From there, we need to find anywhere else where zero was falsely predicted, and we note that happens once on the second row, and twice on the fourth row, for a total of three false positives.
To get the union, we add up TP (3), FP (3) and FN (1) to get seven. The IOU for this class, therefore, is 3/7.
If we do this for all the classes and average the IOUs, we get:
Mean IOU = [(3/7) + (2/6) + (3/4) + (1/6)] / 4 = 0.420
You will also want to learn how to pull the statistics for mAP (Mean Avg Precision):
https://www.youtube.com/watch?v=pM6DJ0ZZee0
https://towardsdatascience.com/breaking-down-mean-average-precision-map-ae462f623a52#1a59
https://medium.com/#hfdtsinghua/calculate-mean-average-precision-map-for-multi-label-classification-b082679d31be
Compute Covariance Matrixes
The variance of a variable describes how much the values are spread. The covariance is a measure that tells the amount of dependency between two variables.
A positive covariance means that the values of the first variable are large when values of the second variables are also large. A negative covariance means the opposite: large values from one variable are associated with small values of the other.
The covariance value depends on the scale of the variable so it is hard to analyze it. It is possible to use the correlation coefficient that is easier to interpret. The correlation coefficient is just the normalized covariance.
A positive covariance means that large values of one variable are associated with big values from the other (left). A negative covariance means that large values of one variable are associated with small values of the other one (right).
The covariance matrix is a matrix that summarises the variances and covariances of a set of vectors and it can tell a lot of things about your variables. The diagonal corresponds to the variance of each vector:
A matrix A and its matrix of covariance. The diagonal corresponds to the variance of each column vector. Let’s check with the formula of the variance:
With n the length of the vector, and x̄ the mean of the vector. For instance, the variance of the first column vector of A is:
This is the first cell of our covariance matrix. The second element on the diagonal corresponds of the variance of the second column vector from A and so on.
Note: the vectors extracted from the matrix A correspond to the columns of A.
The other cells correspond to the covariance between two column vectors from A. For instance, the covariance between the first and the third column is located in the covariance matrix as the column 1 and the row 3 (or the column 3 and the row 1):
The position in the covariance matrix. Column corresponds to the first variable and row to the second (or the opposite). The covariance between the first and the third column vector of A is the element in column 1 and row 3 (or the opposite = same value).
Let’s check that the covariance between the first and the third column vector of A is equal to -2.67. The formula of the covariance between two variables Xand Y is:
The variables X and Y are the first and the third column vectors in the last example. Let’s split this formula to be sure that it is crystal clear:
The sum symbol (Σ) means that we will iterate on the elements of the vectors. We will start with the first element (i=1) and calculate the first element of X minus the mean of the vector X:
Multiply the result with the first element of Y minus the mean of the vector Y:
Reiterate the process for each element of the vectors and calculate the sum of all results:
Divide by the number of elements in the vector.
EXAMPLE - Let’s start with the matrix A:
We will calculate the covariance between the first and the third column vectors:
and
Which is x̄=3, ȳ=4, and n=3 so we have:
Code example -
Using NumPy, the covariance matrix can be calculated with the function np.cov.
It is worth noting that if you want NumPy to use the columns as vectors, the parameter rowvar=False has to be used. Also, bias=True divides by n and not by n-1.
Let’s create the array first:
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
A = np.array([[1, 3, 5], [5, 4, 1], [3, 8, 6]])
Now we will calculate the covariance with the NumPy function:
np.cov(A, rowvar=False, bias=True)
Finding the covariance matrix with the dot product
There is another way to compute the covariance matrix of A. You can center A around 0. The mean of the vector is subtracted from each element of the vector to have a vector with mean equal to 0. It is multiplied with its own transpose, and divided by the number of observations.
Let’s start with an implementation and then we’ll try to understand the link with the previous equation:
def calculateCovariance(X):
meanX = np.mean(X, axis = 0)
lenX = X.shape[0]
X = X - meanX
covariance = X.T.dot(X)/lenX
return covariance
print(calculateCovariance(A))
Output:
array([[ 2.66666667, 0.66666667, -2.66666667],
[ 0.66666667, 4.66666667, 2.33333333],
[-2.66666667, 2.33333333, 4.66666667]])
The dot product between two vectors can be expressed:
It is the sum of the products of each element of the vectors:
If we have a matrix A, the dot product between A and its transpose will give you a new matrix:
Visualize data and covariance matrices
In order to get more insights about the covariance matrix and how it can be useful, we will create a function to visualize it along with 2D data. You will be able to see the link between the covariance matrix and the data.
This function will calculate the covariance matrix as we have seen above. It will create two subplots — one for the covariance matrix and one for the data. The heatmap() function from Seaborn is used to create gradients of colour — small values will be coloured in light green and large values in dark blue. We chose one of our palette colours, but you may prefer other colours. The data is represented as a scatterplot.
def plotDataAndCov(data):
ACov = np.cov(data, rowvar=False, bias=True)
print 'Covariance matrix:\n', ACov
fig, ax = plt.subplots(nrows=1, ncols=2)
fig.set_size_inches(10, 10)
ax0 = plt.subplot(2, 2, 1)
# Choosing the colors
cmap = sns.color_palette("GnBu", 10)
sns.heatmap(ACov, cmap=cmap, vmin=0)
ax1 = plt.subplot(2, 2, 2)
# data can include the colors
if data.shape[1]==3:
c=data[:,2]
else:
c="#0A98BE"
ax1.scatter(data[:,0], data[:,1], c=c, s=40)
# Remove the top and right axes from the data plot
ax1.spines['right'].set_visible(False)
ax1.spines['top'].set_visible(False)
Uncorrelated data
Now that we have the plot function, we will generate some random data to visualize what the covariance matrix can tell us. We will start with some data drawn from a normal distribution with the NumPy function np.random.normal().
This function needs the mean, the standard deviation and the number of observations of the distribution as input. We will create two random variables of 300 observations with a standard deviation of 1. The first will have a mean of 1 and the second a mean of 2. If we randomly draw two sets of 300 observations from a normal distribution, both vectors will be uncorrelated.
np.random.seed(1234)
a1 = np.random.normal(2, 1, 300)
a2 = np.random.normal(1, 1, 300)
A = np.array([a1, a2]).T
A.shape
Note 1: We transpose the data with .T because the original shape is (2, 300) and we want the number of observations as rows (so with shape (300, 2)).
Note 2: We use np.random.seed function for reproducibility. The same random number will be used the next time we run the cell. Let’s check how the data looks like:
A[:10,:]
array([[ 2.47143516, 1.52704645],
[ 0.80902431, 1.7111124 ],
[ 3.43270697, 0.78245452],
[ 1.6873481 , 3.63779121],
[ 1.27941127, -0.74213763],
[ 2.88716294, 0.90556519],
[ 2.85958841, 2.43118375],
[ 1.3634765 , 1.59275845],
[ 2.01569637, 1.1702969 ],
[-0.24268495, -0.75170595]])
Nice, we have two column vectors; Now, we can check that the distributions are normal:
sns.distplot(A[:,0], color="#53BB04")
sns.distplot(A[:,1], color="#0A98BE")
plt.show()
plt.close()
We can see that the distributions have equivalent standard deviations but different means (1 and 2). So that’s exactly what we have asked for.
Now we can plot our dataset and its covariance matrix with our function:
plotDataAndCov(A)
plt.show()
plt.close()
Covariance matrix:
[[ 0.95171641 -0.0447816 ]
[-0.0447816 0.87959853]]
We can see on the scatterplot that the two dimensions are uncorrelated. Note that we have one dimension with a mean of 1 (y-axis) and the other with the mean of 2 (x-axis).
Also, the covariance matrix shows that the variance of each variable is very large (around 1) and the covariance of columns 1 and 2 is very small (around 0). Since we ensured that the two vectors are independent this is coherent. The opposite is not necessarily true: a covariance of 0 doesn’t guarantee independence.
Correlated data
Now, let’s construct dependent data by specifying one column from the other one.
np.random.seed(1234)
b1 = np.random.normal(3, 1, 300)
b2 = b1 + np.random.normal(7, 1, 300)/2.
B = np.array([b1, b2]).T
plotDataAndCov(B)
plt.show()
plt.close()
Covariance matrix:
[[ 0.95171641 0.92932561]
[ 0.92932561 1.12683445]]
The correlation between the two dimensions is visible on the scatter plot. We can see that a line could be drawn and used to predict y from x and vice versa. The covariance matrix is not diagonal (there are non-zero cells outside of the diagonal). That means that the covariance between dimensions is non-zero.
From this point with Covariance Matrcies, you can research further on the following:
Mean normalization
Standardization or normalization
Whitening
Zero-centering
Decorrelate
Rescaling