My problem is the following. I have a dataset with 10 variables and 8 samples. Each sample has been analysed for triplicate, therefore I have a dataset of 24 rows. However, some analyses (variable) were not performed in triplicate. In the case where the analysis was only done once, I have to introduce NA in order to fill the blanks. In the cases where the analysis was performed more than three times, I have to introduce new rows that add NA to the analysis which were in fact done three times.
My ulterior goal is to apply ANOVA to this dataset.I have thought about repeating the value in the case where I only have 1 analysis, and randomly eliminating values in the cases where I have more than 3 analysis, but I have the feeling this is not the most orthodox way to proceed.
I hope it is clear enough.
Thanks in advance!
Related
I am helping with a retrospective study and the data isn't very well organized. Also, I am new to statistics, so I took a stab at analyzing the data myself. We will be getting the help of a statistician later on, but not sure when yet.
We are looking at about 100 patients and each patient was followed up with for a variable amount of time. Throughout each patient's follow-up, there were a variable amount of observations made at various timepoints. The observations included a set of lab values, anthropometric data, and demographic data. So to conduct the analysis, we split up the observations into time bins (eg. 6 months follow-up, 1 year follow up, etc). Then for each time point, we categorized each patient in one of 3 groups based on the outcome of interest. Also, for each time point, we selected one observation to represent one patient during that timepoint (since there could be many within the same time bin). For the analysis, we did the following:
1 . ANOVA within each timepoint to compare the 3 groups of outcomes . Looking at select independent variables of interest.
2 . For the same variables of interest above, do a repeated measures ANOVA to see if it's changing over time.
3 . Test for correlations between the variables of interest mentioned above and other independent variables.
4 . Test each independent variable in a univariate binomial logistic regression to see if it predicts outcome. There were 3 groups, so we did pairwise regressions (eg. (outcome 1 + 2) vs (outcome 3), and (outcome 1) vs (outcome 2 + 3)).
5 . Do a multivariate binomial logistic regression with forward elimination using only the significant independent variables retained from step 4.
6 . If any independent variables of interest are retained in the MV regression, run it again testing for potential interactions with any variables it was correlated with from step 3. We tried to do this by making a new variable that is the product of the two variables and putting it into the regression.
What I'm trying to show with this analysis is that one key independent variable explains the difference in outcomes among the patients. So far my analysis seems to be doing this, as it seems to be one of the few variables retained at step 6 and with a good significance value. So sorry if this is very confusing to read.
I have the following 3 cases of a numeric metric on a time series(t,t1,t2 etc denotes different hourly comparisons across periods)
If you notice the 3 graphs t(period of interest) clearly has a drop off for image 1 but not so much for image 2 and image 3. Assume this is some sort of numeric metric(raw metric or derived) and I want to create a system/algo which specifically catches case 1 but not case 2 or 3 with t being the point of interest. While visually this makes sense and is very intuitive I am trying to design a way to this in python using the dataframes shown in the picture.
Generally the problem is how do I detect when the time series is behaving very differently from any of the prior weeks.
Edit: When I say different what I really mean is, my metric trends together across periods in t1 to t4 but if they dont and try to separate out of the envelope, that to me is an anomaly. If you notice chart 1 you can see t tries to split out from rest of the tn this is an anomaly for me. in other cases t is within the bounds of other time periods. Hope this helps.
With small data the best is if you can come up with a good transformation into a simpler representation.
In this case I would try the following:
Distance to the median along the time-axis. Then a summary of that, could be median, Mean-Squared-Error etc
Median of the cross-correlation of the signals
I am a user of microsoft excels solver, and am pretty sure it is not possible to solve to maximize for two values. I was wondering if anyone might have another clever way to do this.
Basically I have a column of numbers between 1 and 30 that I need to look over about and pull out 9 to 10 values (out of 200) based on a couple other constraints. I would also like to not just maximize this value, but also a probability value (range from 0 to 1) that I would also like to maximize.
Adding them up won't work as that would grossly undervalue the probability value and multiplying may do the opposite by overvaluing the probability. Any Strategies to handle this problem would be greatly appreciated.
This is an example of multi-objective optimization, which has an extensive literature. As the Wikipedia article shows, this can lead to some pretty deep waters.
By far the easiest approach is that of linear scalarization. This refers to replacing a vector of 2 (or more) objective functions by a single (hence scalar) objective function which is a linear combination of the objective function. What you can do with the solver is to create 2 cells to hold the relative weights to assign to the two objectives. These will be 2 numbers in the range 0 and 1 which sum to 1. Then create a new objective function which is the SUMPRODUCT (linear combination) of these weights and the objectives. Then -- jut use the solver to optimize this objective function. If you aren't happy with the results -- adjust the weights. There is no one right answer. One of the advantages of this approach is that it allows a decision maker to clarify the relevant importance of the objectives.
I have been playing around a Kaplan Meier Survival analysis. I have 3 conditions mutually exclusive. Let says:
condition 1 is 'I am not able to work (and not working)'
condition 2 is 'I am able to work and I am working'
condition 3 is 'I am able to work but I am not working'
I am trying to have an overall likelihood to be either way in C1, C2 or C3.
I have done 3 separates Survival analysis (one for each condition) and add the cumulative proportion for the same time, but to the total is slightly superior to 1 (between 1.02 to 1.06 to be exact). I was wondering how to explain this over estimation. Is it something in my logic or the way the censored data are estimated? (or else).
Thanks,
I have 2 columns and multiple rows of data in excel. Each column represents an algorithm and the values in rows are the results of these algorithms with different parameters. I want to make statistical significance test of these two algorithms with excel. Can anyone suggest a function?
As a result, it will be nice to state something like "Algorithm A performs 8% better than Algorithm B with .9 probability (or 95% confidence interval)"
The wikipedia article explains accurately what I need:
http://en.wikipedia.org/wiki/Statistical_significance
It seems like a very easy task but I failed to find a scientific measurement function.
Any advice over a built-in function of excel or function snippets are appreciated.
Thanks..
Edit:
After tharkun's comments, I realized I should clarify some points:
The results are merely real numbers between 1-100 (they are percentage values). As each row represents a different parameter, values in a row represents an algorithm's result for this parameter. The results do not depend on each other.
When I take average of all values for Algorithm A and Algorithm B, I see that the mean of all results that Algorithm A produced are 10% higher than Algorithm B's. But I don't know if this is statistically significant or not. In other words, maybe for one parameter Algorithm A scored 100 percent higher than Algorithm B and for the rest Algorithm B has higher scores but just because of this one result, the difference in average is 10%.
And I want to do this calculation using just excel.
Thanks for the clarification. In that case you want to do an independent sample T-Test. Meaning you want to compare the means of two independent data sets.
Excel has a function TTEST, that's what you need.
For your example you should probably use two tails and type 2.
The formula will output a probability value known as probability of alpha error. This is the error which you would make if you assumed the two datasets are different but they aren't. The lower the alpha error probability the higher the chance your sets are different.
You should only accept the difference of the two datasets if the value is lower than 0.01 (1%) or for critical outcomes even 0.001 or lower. You should also know that in the t-test needs at least around 30 values per dataset to be reliable enough and that the type 2 test assumes equal variances of the two datasets. If equal variances are not given, you should use the type 3 test.
http://depts.alverno.edu/nsmt/stats.htm