Currently I have been using MATLAB to solve the problems that I have been dealing with. In my MATLAB code, I call CPlex or GUROBI solvers to solve large Linear Programming problems. In each call of my MATLAB code, CPlex or GUROBI is called more than 10^5 times. This causes a high computational load and it takes too much to solve larger problems. However, I want to switch to another programming language which is,
capable of calling one of these (CPlex or GUROBI ),
capable of calling one of these solvers (CPlex or GUROBI ) too many times (let's say 10^6 times) and in each call without significant reduction in performance and without any significant increase in run time.
capable of using dynamic arrays (namely, when I run my code, It will get some user parameters and it will define matrices of different sizes in each run.)
capable of defining multi-dimensional arrays, not array of arrays.
At this point I have found three options
Fortran: It seems quite OK but I have some concerns whether can it efficiently be able to call solvers too many times. It seems that I can only use Fortran with Cplex. I googled Fortran+GUROBI but results were not encouraging.
Visual Basic.NET : It seems that calling both Cplex and GUROBI is possible with Visual Basic but I am not sure about the performance.
Python: Both integrable with CPlex and GUROBI .
In conclusion, I request your suggestions to go forward. My preference is to start with a programming language in which it is possible to use both GUROBI and CPlex.
My recommendation is using Python with
Pyomo.
Pyomo (www.pyomo.org) "... is a Python-based open-source software package that supports a diverse set of optimization capabilities for formulating and analyzing optimization models." As such, you can use Python constructs to help build a model. After building the model, you can then run it on gurobi, cplex and many more. And since the solver is just an argument in the solve function, running it in both is just a one word difference. Below is a simple example.
# model.py
from __future__ import division
from pyomo.environ import *
model = ConcreteModel()
model.x = Var([1,2], domain=NonNegativeReals)
model.OBJ = Objective(expr = 2 * model.x[1] + 3 * model.x[2])
model.Constraint1 = Constraint(expr=3 * model.x[1] + 4 * model.x[2] >= 1)
From the command line you could then solve
pyomo solve model.py --solver=gurobi
or
pyomo solve model.py --solver=cplex
You could solve it using a script as well. This sky is the limit and the support group is great.
Related
I searched about the JAGS's manual and one post (here in 2012) about the ordinary differential equation (ode). I was thinking JAGS can because it's similar to WinBUGS (which does it through WBdiff interface). However, if I let JAGS read in my ode code, it cannot even recognize the D(y[...], t) expression.
Can JAGS deal with ode? Maybe I missed a plug-in in JAGS like WBdiff?
While WinBUGS/OpenBUGS/JAGS have almost equivalent syntax/feature sets, there are a few differences between them: one of these is that there is no ODE solver included as part of a standard JAGS installation.
However, JAGS is extensible using user-specified modules (like the plug-ins you mentioned), which provide new functions/distributions using C++ code that can then be used within JAGS when that module is loaded. It would certainly be possible to implement an ODE this way using e.g. the ODE solvers included in the boost C++ library. To do so you will need the following:
Familiarity with C++
Instructions for how to build a module for JAGS
I can't help you with the former, so this may be a dead-end for you if you have never used C++ before. But there is a tutorial available for how to build a JAGS module: https://pubmed.ncbi.nlm.nih.gov/23959766/ This article shows how to build a standalone module, but if you are happy to accept the limitation of using JAGS from R (as most people do) then it is MUCH easier to build a JAGS module within an R package - you could follow code in the runjags package as an example https://cran.r-project.org/package=runjags
If you are thinking of trying to do this yourself then I could potentially help with a few pointers along the way. Of course, it is also possible that someone else has already done this, but if so then I am not aware of it.
I need to solve a linear program (LP) with a mix of binary and continuous variables (MILP).
The data I use as constraints and objectives come from pandas data frame manipulations, so they are in matrix (or I should probably say numpy array) format, and the variables to solve for are sometimes 100's / 1000's.
I know how to use matrices and vectors to setup and solve MILP's in R, but as I coded part of the data handling in python, I wanted to include the LP part in the python script.
So I looked for MILP solvers for python, and I found this post:
Python Mixed Integer Linear Programming
and this one:
https://realpython.com/linear-programming-python/#what-is-mixed-integer-linear-programming
Apart from many links no longer working in the first post, my impression, after browsing through the docs of some of the suggested solvers, was that the LP functions in scipy, which take matrices and vectors as input, cannot solve MILP's (see also this post); on the other hand, interfaces like pulp etc., which do solve MILP's, seem to have their own very special input where one has to specify one by one the equations and name each variable. That would be very inconvenient, as I already have a simple numpy array where each column is a variable and each row is a vector of constraint coefficients.
Does anyone know of a python interface to a MILP solver that takes matrices and vectors as input, instead of requiring to write everything as explicit text equations?
Thanks!
PS I should probably mention that I am looking for interfaces to non-commercial solvers.
EDIT adding my current conclusion, after following the advice from users, for future record and in case it helps someone else
In short: as pulp does the job and comes with a (by all accounts) very good LP solver (CBC), I gave in and accepted to rewrite my problem as a set of equations and inequalities, instead of using the matrix format.
It seems to work, it's indeed fast, at least for the problems I tried so far.
My advice to anyone trying to use it on large problems with many variables would be: look up LpAffineExpression, it's very useful to put together complicated constraints. But first you need to define your variables using LpVariable. A variable can also be a list of variables (that's what stumped me initially).
Before trying pulp, I tried cvxopt, which is easy to install, but then it's true, like the users below said, it has its own bizarre format, and GLPK unfortunately is not a very fast solver.
When I still tried to stick to my matrix input format, I tried cvxpy. On paper this is very good, and in fact would allow me to write the problem as a sum of squares min, which is in fact what my original problem was (I turned it into a LP by replacing sum of squares with sum of abs values, and converting that to LP by known techniques).
I got all the way to writing my objective and constraints in matrix format, and then the solver failed... in fact even the example problem published in the cvxpy documentation failed with the same error. Big waste of time.
Anyway, lesson learned: no point fighting: sometimes you just have to do what python wants you to do ;)
There are some specifics missing and no real definition for how close to exactly what you have as input must be accepted, but Google's OR tools mention a method to define problems for the various solvers they wrap through arrays. This may be workable for you.
For 100s/1000s of variables, this is probably not an option, but I have added an answer to the scipy.optimize question with a toy implementation of branch-and-bound on top of linprog which takes it's args in arrays. If your problem is highly constrained, it might give some answers in a day or two :).
For future reference:
Make sure you have at least pip install scipy>=1.9.3 and use from scipy.optimize import milp as explained here using the HiGHS solvers via method='highs'
I have a Gurobi licence and I am after a good MILP/LP modelling language, which should be
free/open source
intuitive, i.e. something that looks like (taken from MiniZinc)
var int: x;
constraint x >= 0.5;
solve minimize x;
fast: the time to build the model and send it to Gurobi should be of similar order to the best ones (AMPL GAMS etc.)
flexible/powerful (ability to deal with 3D+ arrays, activate/deactivate constraints easily, provide initial solutions to the solver, etc.)
Of course, and correct me if I'm wrong, AMPL GAMS fail at 1), Python and R fail at 2) (and perhaps at 3)?).
How about GLPK, Minizinc, ZIMPL etc.? They satisfy 1) and 2) but what about 3) and 4)? Are they as good as AMPL in this regard? If not, is there a modelling language satisfying 1-4?
I've used AMPL with Gurobi for mid-sized MIPs (~ 100k-1m variables?) and MiniZinc, mostly with Gecode, for smaller combinatorial problems. I've seen some Gurobi work done with R and Python, but haven't used it that way myself.
I'm less familiar with the other options. My understanding is that GAMS is quite similar to AMPL and much of what I have to say about AMPL may also be valid for GAMS, but I can't vouch for it.
Of course, and correct me if I'm wrong, AMPL GAMS fail at 1),
Yes, generally. There is an exception which probably isn't helpful for your specific requirements but might be useful to others: you can get free use of AMPL, Gurobi, and many other optimisation products, by using the NEOS web service. This is restricted to academic non-commercial purposes and you have to grant NEOS certain rights in relation to the problems you send them; definitely read those terms of service before using it. It also requires waiting for an available server, so if speed is a high priority this probably isn't the solution for you.
Python and R fail at 2) (and perhaps at 3)?).
In my limited experience, yes for (2). AMPL, GAMS, and MiniZinc are designed specifically for defining optimisation problems, so it's unsurprising that their syntax is more user-friendly for that purpose than languages like Python and R.
The flip-side to this is that if you want to do just about anything other than defining an optimisation problem with these languages, Python/R/etc. will probably be better for that purpose.
On speed: for the problems I usually work with, AMPL takes maybe a couple of seconds to build and presolve a MIP model which takes Gurobi a couple of minutes to solve. Obviously this is going to vary somewhat with hardware and details of the problem, but in general I would expect build time to be small compared to solve time for any of the solutions under discussion. Even with a good solver like Gurobi, big MIPs are hard. Many of the serious optimisation programmers I've met do use Python, so I presume the performance side is good enough.
However, that doesn't mean the choice of language/platform is irrelevant to speed. One of the nice features of AMPL (and also GAMS) is presolve, which attempts to reduce the problem size before sending it to the solver. My standard problems have a lot of redundant variables and constraints; AMPL identifies and eliminates many of these, reducing the problem size by about 80% and giving a noticeable improvement in solver time (as compared to runs where I switch off presolve, which I sometimes do for debugging-related reasons). This might be a consideration if you expect a lot of redundancy.
flexible/powerful (ability to deal with 3D+ arrays, activate/deactivate constraints easily, provide initial solutions to the solver, etc.)
MiniZinc handles up to 6D arrays, which may or may not be enough depending on your applications.
It's more flexible than AMPL in some areas and less so in others. AMPL has a lot of set-based functionality that I find useful (e.g. I can define a variable whose index set is something like "pairs of non-identical cities separated by no more than 500 km") and MiniZinc doesn't have this. OTOH, MiniZinc seems to be better than AMPL for solver-hopping, e.g. if I write a MZ model with a combinatorial constraint like "alldifferent" but then try to run it on a solver that doesn't recognise such constraints, MZ will translate it into something the solver can deal with.
I haven't tried deactivating constraints in MZ other than by commenting them out, so I can't help there, and similarly on providing initial solutions.
Overall, MiniZinc is a good choice to consider. Some pluses and minuses relative to AMPL ("free" being a big plus!) but it fills a similar niche.
IMHO, there is no such system if you consider the Python interfaces/modeling environments to SCIP or Gurobi too complicated:
x = model.addVar()
y = model.addVar(vtype="INTEGER")
model.setObjective(x + y)
model.addCons(2*x - y*y >= 0)
model.optimize()
To me this looks quite natural and straight forward. The immense benefit of using an actual programming language instead of modeling language is that you can do anything in there, while there will always be boundaries in the latter.
If you are a looking for a modeling GUI, you should check out LITIC. It can be used almost entirely with drag-and-drop operations: https://litic.com/showcase.html
I've used a lot of the options mentioned, and some not yet mentioned
GAMS
GAMS' Python API
GAMS' MATLAB API
AMPL
FICO Xpress Mosel
FICO Xpress Model's Python API
IBM ILOG OPL
Gurobi's Python API
PuLP (Python)
Pyomo (Python)
Python-MIP
JuMP (Julia)
MATLAB Optimization Toolbox
Google OR-Tools
Based on your requirements, I'd suggest trying Python-MIP, PuLP or JuMP. They are free and have easy syntax with no limit on array dimensionality.
Take a look at Google or-tools. I’m not sure if getting initial solution to the solver is available in all of its interfaces, but if you use it in python, it should probably satisfy all 1-4.
While there are many questions on SE regarding sand-boxing CPython, most focus on providing a more or less complete Python environment.
In this case, I'm interested in using Python for basic math expressions, something of this complexity, eg:
(sin(a) * cos(b) / tan(c) ** sqrt(d)) - e
Now I could create my own expression evaluator in Python, however I don't want to sacrifice performance (or have to maintain it and have good Python compatibility for all the corner cases).
I looked into numba and numexpr, both very interesting projects, but neither have security/sand-boxing as a goal.
I considered writing a minimal version of ceval.c which runs a restricted set of CPython's bytecodes, but this is still quite some effort.
Instead I did a quick test that restricts name-space and checks the compiled expressions opcodes before executing, from my initial tests this works well, though I'm not totally confident its secure either.
While this isn't clear-cut, using simple expressions means I don't necessarily need:
multiple lines of code.
import statements.
defining functions & classes.
getattr or getitem access.
for & while loops.
So my question is:
Is there a reliably secure way to execute math expressions in CPython that can co-exist in the same process as complete CPython scripts?
Note, since security may be too vague a term, for the purpose of discussion.
Reading or removing files on the users system.
Running open or any functions in os or shutil.
I have many equations with many unknowns (my data is in Excel) and currently I am using matrix method to solve them.
I use inbuilt MMULT (matrix multiply) and MINVERSE (matrix inverse) in following form :- Result = MMULT (MINVERSE(matrix1),matrix2)
Here lies the problem, my matrices are of the order of 2000 x 2000 or more, and Excel takes lot of time in doing inverse (matrix multiplication is very fast).
What is the best way for me to speed up the process? I don't mind exporting data to any external 3rd party program (compatible with Excel) and then importing the inversed matrix back to Excel.
I don't know much C / C++, but I feel if we compile the Matrix Inverse Function into a DLL and then use the same from excel VBA, maybe the speed will improve. Please help.
Please see following link :
Why is MATLAB so fast in matrix multiplication?
It is found that MATLAB has highest speed for matrix calculations. Can we use any such library in Excel / VBA ?
I have found certain libraries such as LAPACK, ARMADILO etc which are in C / C++ / C# or .NET. How can I use compiled versions of these libraries in my Excel VBA?
I am the author of the blog linked by John Coleman. There are quite a few options available, depending on your familiarity with compiling and linking to different languages. I'd say the main options are (in order of ease of implementation):
Use the Python Numpy and Scipy libraries with ExcelPython or xlwings (which now includes ExcelPython). The most recent post on the subject at my blog (including free downloads) is:
https://newtonexcelbach.wordpress.com/2016/01/04/xlscipy-python-scipy-for-excel-update-with-new-functions/
Advantages:
Free
Easy to implement (especially using Anaconda Python, which now includes xlwings).
No compilation required.
The Scipy code is fast, and includes packages for sparse matrices, which where applicable makes a huge difference to solving time, and the maximum size of matrix that can be handled.
Free and open-source spreadsheet implementation available on my blog.
Disadvantages:
The overhead in transferring large data sets from VBA to Python can be significant, in fact for small to medium sized matrices it is often greater than the solution time.
Use the Python version of the Alglib library. Search my blog for Alglib for recent examples using Python, and older examples using C++ and C#.
Advantages and disadvantages:
As for Numpy and Scipy, except there is a commercial version available as well, which may offer performance advantages (I haven't tried it).
Use Numpy/Scipy with Pyxll.
https://newtonexcelbach.wordpress.com/2013/09/10/python-matrix-functions-using-pyxll/ for sample spreadsheet.
Advantages:
The data transfer overhead should be much reduced.
More mature and documentation is better than the current xlwings.
Disadvantages:
Commercial package (but free for evaluation and non-commercial use)
Not open source.
Use open source C++ or Fortran packages with Visual Studio or other compiler, to produce xll based functions.
Advantages:
Potentially the best all-round performance.
Disadvantages:
More coding required.
Implementation more difficult, especially if you want to distribute to others.
32 bit/ 64 bit issues likely to be much harder to resolve.
As mentioned in the comments, matrix inversion is much slower than other matrix solution methods, but even allowing for that, the built in Excel Matrix functions are very slow compared with alternative compiled routines, and the advantages of installing one of the alternatives listed above is well worth the effort.