I have many equations with many unknowns (my data is in Excel) and currently I am using matrix method to solve them.
I use inbuilt MMULT (matrix multiply) and MINVERSE (matrix inverse) in following form :- Result = MMULT (MINVERSE(matrix1),matrix2)
Here lies the problem, my matrices are of the order of 2000 x 2000 or more, and Excel takes lot of time in doing inverse (matrix multiplication is very fast).
What is the best way for me to speed up the process? I don't mind exporting data to any external 3rd party program (compatible with Excel) and then importing the inversed matrix back to Excel.
I don't know much C / C++, but I feel if we compile the Matrix Inverse Function into a DLL and then use the same from excel VBA, maybe the speed will improve. Please help.
Please see following link :
Why is MATLAB so fast in matrix multiplication?
It is found that MATLAB has highest speed for matrix calculations. Can we use any such library in Excel / VBA ?
I have found certain libraries such as LAPACK, ARMADILO etc which are in C / C++ / C# or .NET. How can I use compiled versions of these libraries in my Excel VBA?
I am the author of the blog linked by John Coleman. There are quite a few options available, depending on your familiarity with compiling and linking to different languages. I'd say the main options are (in order of ease of implementation):
Use the Python Numpy and Scipy libraries with ExcelPython or xlwings (which now includes ExcelPython). The most recent post on the subject at my blog (including free downloads) is:
https://newtonexcelbach.wordpress.com/2016/01/04/xlscipy-python-scipy-for-excel-update-with-new-functions/
Advantages:
Free
Easy to implement (especially using Anaconda Python, which now includes xlwings).
No compilation required.
The Scipy code is fast, and includes packages for sparse matrices, which where applicable makes a huge difference to solving time, and the maximum size of matrix that can be handled.
Free and open-source spreadsheet implementation available on my blog.
Disadvantages:
The overhead in transferring large data sets from VBA to Python can be significant, in fact for small to medium sized matrices it is often greater than the solution time.
Use the Python version of the Alglib library. Search my blog for Alglib for recent examples using Python, and older examples using C++ and C#.
Advantages and disadvantages:
As for Numpy and Scipy, except there is a commercial version available as well, which may offer performance advantages (I haven't tried it).
Use Numpy/Scipy with Pyxll.
https://newtonexcelbach.wordpress.com/2013/09/10/python-matrix-functions-using-pyxll/ for sample spreadsheet.
Advantages:
The data transfer overhead should be much reduced.
More mature and documentation is better than the current xlwings.
Disadvantages:
Commercial package (but free for evaluation and non-commercial use)
Not open source.
Use open source C++ or Fortran packages with Visual Studio or other compiler, to produce xll based functions.
Advantages:
Potentially the best all-round performance.
Disadvantages:
More coding required.
Implementation more difficult, especially if you want to distribute to others.
32 bit/ 64 bit issues likely to be much harder to resolve.
As mentioned in the comments, matrix inversion is much slower than other matrix solution methods, but even allowing for that, the built in Excel Matrix functions are very slow compared with alternative compiled routines, and the advantages of installing one of the alternatives listed above is well worth the effort.
Related
I have a Gurobi licence and I am after a good MILP/LP modelling language, which should be
free/open source
intuitive, i.e. something that looks like (taken from MiniZinc)
var int: x;
constraint x >= 0.5;
solve minimize x;
fast: the time to build the model and send it to Gurobi should be of similar order to the best ones (AMPL GAMS etc.)
flexible/powerful (ability to deal with 3D+ arrays, activate/deactivate constraints easily, provide initial solutions to the solver, etc.)
Of course, and correct me if I'm wrong, AMPL GAMS fail at 1), Python and R fail at 2) (and perhaps at 3)?).
How about GLPK, Minizinc, ZIMPL etc.? They satisfy 1) and 2) but what about 3) and 4)? Are they as good as AMPL in this regard? If not, is there a modelling language satisfying 1-4?
I've used AMPL with Gurobi for mid-sized MIPs (~ 100k-1m variables?) and MiniZinc, mostly with Gecode, for smaller combinatorial problems. I've seen some Gurobi work done with R and Python, but haven't used it that way myself.
I'm less familiar with the other options. My understanding is that GAMS is quite similar to AMPL and much of what I have to say about AMPL may also be valid for GAMS, but I can't vouch for it.
Of course, and correct me if I'm wrong, AMPL GAMS fail at 1),
Yes, generally. There is an exception which probably isn't helpful for your specific requirements but might be useful to others: you can get free use of AMPL, Gurobi, and many other optimisation products, by using the NEOS web service. This is restricted to academic non-commercial purposes and you have to grant NEOS certain rights in relation to the problems you send them; definitely read those terms of service before using it. It also requires waiting for an available server, so if speed is a high priority this probably isn't the solution for you.
Python and R fail at 2) (and perhaps at 3)?).
In my limited experience, yes for (2). AMPL, GAMS, and MiniZinc are designed specifically for defining optimisation problems, so it's unsurprising that their syntax is more user-friendly for that purpose than languages like Python and R.
The flip-side to this is that if you want to do just about anything other than defining an optimisation problem with these languages, Python/R/etc. will probably be better for that purpose.
On speed: for the problems I usually work with, AMPL takes maybe a couple of seconds to build and presolve a MIP model which takes Gurobi a couple of minutes to solve. Obviously this is going to vary somewhat with hardware and details of the problem, but in general I would expect build time to be small compared to solve time for any of the solutions under discussion. Even with a good solver like Gurobi, big MIPs are hard. Many of the serious optimisation programmers I've met do use Python, so I presume the performance side is good enough.
However, that doesn't mean the choice of language/platform is irrelevant to speed. One of the nice features of AMPL (and also GAMS) is presolve, which attempts to reduce the problem size before sending it to the solver. My standard problems have a lot of redundant variables and constraints; AMPL identifies and eliminates many of these, reducing the problem size by about 80% and giving a noticeable improvement in solver time (as compared to runs where I switch off presolve, which I sometimes do for debugging-related reasons). This might be a consideration if you expect a lot of redundancy.
flexible/powerful (ability to deal with 3D+ arrays, activate/deactivate constraints easily, provide initial solutions to the solver, etc.)
MiniZinc handles up to 6D arrays, which may or may not be enough depending on your applications.
It's more flexible than AMPL in some areas and less so in others. AMPL has a lot of set-based functionality that I find useful (e.g. I can define a variable whose index set is something like "pairs of non-identical cities separated by no more than 500 km") and MiniZinc doesn't have this. OTOH, MiniZinc seems to be better than AMPL for solver-hopping, e.g. if I write a MZ model with a combinatorial constraint like "alldifferent" but then try to run it on a solver that doesn't recognise such constraints, MZ will translate it into something the solver can deal with.
I haven't tried deactivating constraints in MZ other than by commenting them out, so I can't help there, and similarly on providing initial solutions.
Overall, MiniZinc is a good choice to consider. Some pluses and minuses relative to AMPL ("free" being a big plus!) but it fills a similar niche.
IMHO, there is no such system if you consider the Python interfaces/modeling environments to SCIP or Gurobi too complicated:
x = model.addVar()
y = model.addVar(vtype="INTEGER")
model.setObjective(x + y)
model.addCons(2*x - y*y >= 0)
model.optimize()
To me this looks quite natural and straight forward. The immense benefit of using an actual programming language instead of modeling language is that you can do anything in there, while there will always be boundaries in the latter.
If you are a looking for a modeling GUI, you should check out LITIC. It can be used almost entirely with drag-and-drop operations: https://litic.com/showcase.html
I've used a lot of the options mentioned, and some not yet mentioned
GAMS
GAMS' Python API
GAMS' MATLAB API
AMPL
FICO Xpress Mosel
FICO Xpress Model's Python API
IBM ILOG OPL
Gurobi's Python API
PuLP (Python)
Pyomo (Python)
Python-MIP
JuMP (Julia)
MATLAB Optimization Toolbox
Google OR-Tools
Based on your requirements, I'd suggest trying Python-MIP, PuLP or JuMP. They are free and have easy syntax with no limit on array dimensionality.
Take a look at Google or-tools. I’m not sure if getting initial solution to the solver is available in all of its interfaces, but if you use it in python, it should probably satisfy all 1-4.
I am new to the subject "modeling of physical systems". I read some basic literature and did some tutorials in Modelica and Simulink/Simscape. I wanted to ask you, if I understand the following content correctly:
Symbolic manipulation is the process of transforming a differential-algebraic system of equation (physical model: DAE) into a system of differential equations (ODE) that can be solved by standard solvers (Runge, Kutta, BDF, ...)
There are also solver that can solve DAE's directly. But Modelica (openModelica, Dymola) and Simscape transfer the System into an ODE (why are this methods better compared to direct DAE solvers?)
A "flat Modelica code" is the result ( = ODE) of the transformation.
Thank you very much for your answers.
Symbolic processing for Modelica includes:
remove object oriented structure and obtain an hybrid DAE (flat Modelica)
perform matching, index reduction, casualization to get an ODE
perform optimization (tearing, common subexpression elimination, etc)
generate code for a particular solver
OpenModelica can also solve the system in DAE mode without transforming it to ODE and I guess other Modelica tools can also do that.
A "flat Modelica code" is Modelica code where the object orientation is removed, connect equations are expanded to normal equations. The result is a hybrid DAE.
See Modelica Spec 3.3 for more info about all this (for example Appendix C):
https://modelica.org/documents/ModelicaSpec33Revision1.pdf
So I think your understanding of the terminology is very good too.
Due to the declarative way (opposed to imperative) of programming in modelica, we get immediately very high numbers of algebraic equations. Solving these (partly) symbolically has, above all, these essential advantages:
Speed. Without eliminating algebraic loops, modelica would not be practically usable for any real-world problem and even then only in simple cases no algebraic equations remain. It would be too slow and would force you to do transformations manually yourself in modelica too (as in imperative languages e.g. in C/C++ or Simulink). Even today modelica can still be slower than manually transformed and optimized solutions.
Moreover modelica applications often need simulations in real-time.
Correctness. Symbolic transformations are based on proofs and modelica applications often are in the area of safety critical or cyber-physical systems.
One additional consideration is that there are different forms of DAEs, and modeling often lead to high-index DAEs that are complicated to solve numerically (*). (Note "high" means index greater than 1, typically 2 - but sometimes even higher.)
Symbolic transformations can reduce high-index DAEs to semi-explicit index 1 DAEs, and then by (numerically) solving the systems of equations they are transformed into ODEs.
Thus even if a tool solves DAEs directly it is normally the semi-explicit index 1 DAEs that are solved, not the original high index DAE.
(I know this answer is late. The hybrid part for the symbolic transformations is more complicated, still working on that.)
For more information see https://en.wikipedia.org/wiki/Differential-algebraic_system_of_equations
(*): There are some solvers for high index DAEs (in particular index 2), but typically they rely on a specific structure of the model and finding that structure requires similar techniques as reducing the index to 1.
I am new using this Solver function. Is there is way to use Solver for more than 200 sets of data. Maybe using VBA? Really hope someone could help me
No, you need to buy their commercial product (from Frontline Systems) or build one yourself. For Linear/Non-linear optimization there are a lot of open source libraries available in C++ so you may compile a DLL and invoke it from a VBA wrapper.
Also, it's not difficult to implement it completely in VBA. The original guys who developed the solver have given a very very detailed explanation of its workings here. These are the guys who actually made it so you can easily replicate it - especially for the linear case.
A bit dated, but will give you an idea on how to proceed.
https://www.utexas.edu/courses/lasdon/design.htm
Especially useful are the sections where they describe how they build the Jacobian matrix numerically (for Generalized Gradient Descent in NLopt) and how they use a small perturbation to determine the linear coefficients for the Simplex Method. Thus for a linear problem, the only trick is to generate the Coefficient Matrix and then the solver can proceed completely in-memory and generate
the optimal solution
Shadow prices, reduced costs, etc.
I am trying to implement an algorithm in C# that has some math function inside like :
Minimize a ||xu( i ) - ∑j aj( i ) bj ||2 + β || a( i ) ||1
but I'm not sure if there is any direct method in C# for optimizing and I couldn't find any thing in forums , or maybe you can offer any suggestion to solve this optimization problem in C# !?
There are several solvers with C# bindings that could be suitable for your problem, for example:
MS Solver Foundation, Microsoft's platform for mathematical optimization, that provides solvers for a large range of problems, including unconstrained and constrained least squares. Some effort to dig into the API, but in return derivatives are automatically computed etc.
ALGLIB provides a large range of different optimizers and is available in C#.
IPOPT was mentioned in Ali's comment above. I have developed the C# binding to IPOPT, and it is available for download here. I am of course biased, but personally I believe that the API is fairly straightforward. You need to manually provide the derivatives if you are using csipopt. (There is actually also a library that incorporates csipopt and provides automated derivative computation, Funclib.)
COBYLA2 (if you have nonlinear constraints) and BOBYQA (if you have bound or no constraints) are derivative-free direct search methods developed by Michael Powell. I have ported these solvers to C#, 100% managed code. The C# COBYLA2 project is available here, and the C# BOBYQA project is available here.
Microsoft has augmented the existing Simplex (Linear) and Gradient (Non-linear) solver engines of the standard Solver Add-In by an Evolutionary solver engine aiming at non-smooth discontinuous problems where global optimal solutions are generally hard (or most of the time even impossible) to find with the other engines. In fact, it is one of the solvers that was previously only available through Frontline's Premium Solver product line, so I think it can be considered a generous addition to the standard solver that ships with Excel.
I haven't heard a lot about people using this new engine and guess that most solver users haven't noticed this recent addition by Microsoft. I become aware of it here: http://office.microsoft.com/en-us/excel-help/what-s-new-in-excel-2010-HA010369709.aspx
I would therefore like to hear about your opinions and experiences with it, also with respect to reasonable settings as it seems to take a lot more time to converge than the other methods.
I've used Evolutionary solver engine developing a MPS (master production schedule) trying to involve most aspects and get an optimal solution, I've found this:
Sometimes it gives the optimal solution, but sometimes you have to give it some kind of hints, such as moving some variables, as will, and see what happens, therefore, I wouldn't recommend it for final decisions, but give it a chance!