I am new using this Solver function. Is there is way to use Solver for more than 200 sets of data. Maybe using VBA? Really hope someone could help me
No, you need to buy their commercial product (from Frontline Systems) or build one yourself. For Linear/Non-linear optimization there are a lot of open source libraries available in C++ so you may compile a DLL and invoke it from a VBA wrapper.
Also, it's not difficult to implement it completely in VBA. The original guys who developed the solver have given a very very detailed explanation of its workings here. These are the guys who actually made it so you can easily replicate it - especially for the linear case.
A bit dated, but will give you an idea on how to proceed.
https://www.utexas.edu/courses/lasdon/design.htm
Especially useful are the sections where they describe how they build the Jacobian matrix numerically (for Generalized Gradient Descent in NLopt) and how they use a small perturbation to determine the linear coefficients for the Simplex Method. Thus for a linear problem, the only trick is to generate the Coefficient Matrix and then the solver can proceed completely in-memory and generate
the optimal solution
Shadow prices, reduced costs, etc.
Related
I've created a specific model on excel which uses historical returns of firms and aims to find the optimal point between the return and a stop-loss point. Basically I need the optimal stop-loss %. I've been using excel's solver, however, it only finds the local optimum. Can solver somehow find the global optimal point or even a few optimal points? If not, are there any programs that can be recommended to use?
P.S. I am not extremely good at mathematics, and transporting all of the information from excel to an equation or something is probably not an option.
Please help, thanks.
You can either upgrade for the commercial solver which has better support to find a global optimum. Alternatively, you can write a macro that will initialize the various variables to random values and repeatedly launch the optimizer and checking within each loop if a better solution has been found and store the value of the variables before continuing the search.
See https://www.solver.com/upgrade-excel-solver
Note: I am not affiliated with this company. I would rather have you use open source solutions. This is possible with some training.
I have many equations with many unknowns (my data is in Excel) and currently I am using matrix method to solve them.
I use inbuilt MMULT (matrix multiply) and MINVERSE (matrix inverse) in following form :- Result = MMULT (MINVERSE(matrix1),matrix2)
Here lies the problem, my matrices are of the order of 2000 x 2000 or more, and Excel takes lot of time in doing inverse (matrix multiplication is very fast).
What is the best way for me to speed up the process? I don't mind exporting data to any external 3rd party program (compatible with Excel) and then importing the inversed matrix back to Excel.
I don't know much C / C++, but I feel if we compile the Matrix Inverse Function into a DLL and then use the same from excel VBA, maybe the speed will improve. Please help.
Please see following link :
Why is MATLAB so fast in matrix multiplication?
It is found that MATLAB has highest speed for matrix calculations. Can we use any such library in Excel / VBA ?
I have found certain libraries such as LAPACK, ARMADILO etc which are in C / C++ / C# or .NET. How can I use compiled versions of these libraries in my Excel VBA?
I am the author of the blog linked by John Coleman. There are quite a few options available, depending on your familiarity with compiling and linking to different languages. I'd say the main options are (in order of ease of implementation):
Use the Python Numpy and Scipy libraries with ExcelPython or xlwings (which now includes ExcelPython). The most recent post on the subject at my blog (including free downloads) is:
https://newtonexcelbach.wordpress.com/2016/01/04/xlscipy-python-scipy-for-excel-update-with-new-functions/
Advantages:
Free
Easy to implement (especially using Anaconda Python, which now includes xlwings).
No compilation required.
The Scipy code is fast, and includes packages for sparse matrices, which where applicable makes a huge difference to solving time, and the maximum size of matrix that can be handled.
Free and open-source spreadsheet implementation available on my blog.
Disadvantages:
The overhead in transferring large data sets from VBA to Python can be significant, in fact for small to medium sized matrices it is often greater than the solution time.
Use the Python version of the Alglib library. Search my blog for Alglib for recent examples using Python, and older examples using C++ and C#.
Advantages and disadvantages:
As for Numpy and Scipy, except there is a commercial version available as well, which may offer performance advantages (I haven't tried it).
Use Numpy/Scipy with Pyxll.
https://newtonexcelbach.wordpress.com/2013/09/10/python-matrix-functions-using-pyxll/ for sample spreadsheet.
Advantages:
The data transfer overhead should be much reduced.
More mature and documentation is better than the current xlwings.
Disadvantages:
Commercial package (but free for evaluation and non-commercial use)
Not open source.
Use open source C++ or Fortran packages with Visual Studio or other compiler, to produce xll based functions.
Advantages:
Potentially the best all-round performance.
Disadvantages:
More coding required.
Implementation more difficult, especially if you want to distribute to others.
32 bit/ 64 bit issues likely to be much harder to resolve.
As mentioned in the comments, matrix inversion is much slower than other matrix solution methods, but even allowing for that, the built in Excel Matrix functions are very slow compared with alternative compiled routines, and the advantages of installing one of the alternatives listed above is well worth the effort.
I am looking for a vba library for solving nonlinear least-squares problems. Usually I use Matlab functions lsqcurvefit(fun,x0,xdata,ydata) and lsqnonlin(fun,x0) which have the same nls algorithm but different problem specification. Does anyone know vba library (function) with problem specification as in lsqnonlin(fun,x0)? Of course it is possible to adjust ALGLIB library but it requires time...
Thanks in advance.
This question will probably get a better response on SuperUser, but here's a possible solution:
In Excel, if you create a Scatter chart of your data, you can right-click on the line in the chart, and choose 'Add Trendline'. This will give you several options for the type ofcurve-fitting you want to do.
However, all the built-in methods are for 'linear' data and will most likely not be optimal for non-linear data.
Microsoft has augmented the existing Simplex (Linear) and Gradient (Non-linear) solver engines of the standard Solver Add-In by an Evolutionary solver engine aiming at non-smooth discontinuous problems where global optimal solutions are generally hard (or most of the time even impossible) to find with the other engines. In fact, it is one of the solvers that was previously only available through Frontline's Premium Solver product line, so I think it can be considered a generous addition to the standard solver that ships with Excel.
I haven't heard a lot about people using this new engine and guess that most solver users haven't noticed this recent addition by Microsoft. I become aware of it here: http://office.microsoft.com/en-us/excel-help/what-s-new-in-excel-2010-HA010369709.aspx
I would therefore like to hear about your opinions and experiences with it, also with respect to reasonable settings as it seems to take a lot more time to converge than the other methods.
I've used Evolutionary solver engine developing a MPS (master production schedule) trying to involve most aspects and get an optimal solution, I've found this:
Sometimes it gives the optimal solution, but sometimes you have to give it some kind of hints, such as moving some variables, as will, and see what happens, therefore, I wouldn't recommend it for final decisions, but give it a chance!
I'm writing an app to help facilitate some research, and part of this involves doing some statistical calculations. Right now, the researchers are using a program called SPSS. Part of the output that they care about looks like this:
They're really only concerned about the F and Sig. values. My problem is that I have no background in statistics, and I can't figure out what the tests are called, or how to calculate them.
I thought the F value might be the result of the F-test, but after following the steps given on Wikipedia, I got a result that was different from what SPSS gives.
This website might help you out a bit more. Also this one.
I'm working from a fairly rusty memory of a statistics course, but here goes nothing:
When you're doing analysis of variance (ANOVA), you actually calculate the F statistic as the ratio from the mean-square variances "between the groups" and the mean-square variances "within the groups". The second link above seems pretty good for this calculation.
This makes the F statistic measure exactly how powerful your model is, because the "between the groups" variance is explanatory power, and "within the groups" variance is random error. High F implies a highly significant model.
As in many statistical operations, you back-determine Sig. using the F statistic. Here's where your Wikipedia information comes in slightly handy. What you want to do is - using the degrees of freedom given to you by SPSS - find the proper P value at which an F table will give you the F statistic you calculated. The P value where this happens [F(table) = F(calculated)] is the significance.
Conceptually, a lower significance value shows a very strong ability to reject the null hypothesis (which for these purposes means to determine your model has explanatory power).
Sorry to any math folks if any of this is wrong. I'll be checking back to make edits!!!
Good luck to you. Stats is fun, just maybe not this part. =)
I assume from your question that your research colleagues want to automate the process by which certain statistical analyses are performed (i.e., they want to batch process data sets). You have two options:
1) SPSS is now scriptable through python (as of version 15) - go to spss.com and search for python. You can write python scripts to automate data analyses and extract key values from pivot tables, and then process the answers any way you like. This has the virtue of allowing an exact comparison between the results from your python script and the hand-calculated efforts in SPSS of your collaborators. Thus you won't have to really know any statistics to do this work (which is a key advantage)
2) You could do this in R, a free statistics environment, which could probably be scripted. This has the disadvantage that you will have to learn statistics to ensure that you are doing it correctly.
Statistics is hard :-). After a year of reading and re-reading books and papers and can only say with confidence that I understand the very basics of it.
You might wish to investigate ready-made libraries for whichever programming language you are using, because they are many gotcha's in math in general and statistics in particular (rounding errors being an obvious example).
As an example you could take a look at the R project, which is both an interactive environment and a library which you can use from your C++ code, distributed under the GPL (ie if you are using it only internally and publishing only the results, you don't need to open your code).
In short: don't do this by hand, link/use existing software. And sain_grocen's answer is incorrect. :(
These are all tests for significance of parameter estimates that are typically used in Multivariate response Multiple Regressions. These would not be simple things to do outside of a statistical programming environment. I would suggest either getting the output from a pre-existing statistical program, or using one that you can link to and use that code.
I'm afraid that the first answer (sain_grocen's) will lead you down the wrong path. His explanation is likely of a special case of what you are actually dealing with. The anova explained in his links is for a single variate response, in a balanced design. These aren't the F statistics you are seeing. The names in your output (Pillai's Trace, Hotelling's Trace,...) are some of the available multivariate versions. They have F distributions under certain assumptions. I can't explain a text books worth of material here, I would advise you to start by looking at
"Applied Multivariate Statistical Analysis" by Johnson and Wichern
Can you explain more why SPSS itself isn't a fine solution to the problem? Is it that it generates pivot tables as output that are hard to manipulate? Is it the cost of the program?
F-statistics can arise from any number of particular tests. The F is just a distribution (loosely: a description of the "frequencies" of groups of values), like a Normal (Gaussian), or Uniform. In general they arise from ratios of variances. Opinion: many statisticians (myself included), find F-based tests to be unstable (jargon: non-robust).
The particular output statistics (Pillai's trace, etc.) suggest that the original analysis is a MANOVA example, which as other posters describe is a complicated, and hard to get right procedure.
I'm guess also that, based on the MANOVA, and the use of SPSS, this is a psychology or sociology project... if not please enlighten. It might be that other, simpler models might actually be easier to understand and more repeatable. Consult your local university statistical consulting group, if you have one.
Good luck!
Here's an explanation of MANOVA ouptput, from a very good site on statistics and on SPSS:
Output with explanation:
http://faculty.chass.ncsu.edu/garson/PA765/manospss.htm
How and why to do MANOVA or multivariate GLM:
(same path as above, but terminating in '/manova.htm')
Writing software from scratch to calculate these outputs would be both lengthy and difficult;
there's lots of numerical problems and matrix inversions to do.
As Henry said, use Python scripts, or R. I'd suggest working with somebody who knows SPSS if scripting.
In addition, SPSS itself is capable of exporting the output tables to files using something called OMS.
A script within SPSS can do this.
Find out who in your research group knows SPSS and work with them.