What would be a good choice of programming language in which to implement a decision tree? The results of the implementation will be for personal use only, so no need to consider ability to publish etc.
I have heard that Octave is a good option, can anyone explain why a matrix based language is recommended for implementing decision trees?
I have used Standard ML both to implement decision trees and to write a compiler from a domain-specific language into decision trees. I've also compiled similar decision trees into C code.
It really depends what you want to do with decision trees. If you are trying to do something sophisticated or you are trying to make the decision trees especially easy to read and write, I would suggest either creating a domain-specific language or embedding domain-specific operators into Haskell or Standard ML. If you just want to get going, you could start with ML (easier than Haskell for a beginner) and that preserves some options for later.
In general, ML and Haskell are both very good at representing and manipulating trees of all kinds.
I can't explain why someone would recommend a matrix-based language for decision trees.
I am pretty sure that the first decision tree was written in LISP.
Still many such algorithm is still written in LISP.
You can find many documentation if you decide to choose LISP.
Scheme is also a good language for that purpose and it is simpler/smaller than LISP.
Also the learning curve is fast in both languages.
IMHO
Related
How is ML (and its variants e.g. SML) a metalanguage. What is the object language that ML describes? Is it just because functions are considered values, therefore code is treated in the same way as data?
It actually comes from its original use case.
ML was designed as a language to write theorem provers. In this case, ML is the programming language that you use to describe the theory. It's the language above the theory: the meta-language. Or, as Milner would put it in the original paper:
We also discuss extending these results to richer languages; a type-checking algorithm based on W is in fact already implemented and working,
for the metalanguage ML in the Edinburgh LCF system.
The name stuck, so now it's called like this, even though it doesn't describe an object language in the general sense.
ML originally was the meta language for the LCF theorem prover developed by Milner in the 70s. You could use it define and perform proofs in this system, e.g., by writing proof tactics in ML. See also the Wikipedia article
I would like to develop a decision support system for diagnosis disease. I am newbie for the programming. Can anyone suggest which programming language is most suitable?
That really depends on what you want to do exactly (do you want to use prebuilt libraries?).
I see you added weka as a tag. Java is a good option, since it's so versatile, powerful, fast and fairly easy to use, implementable as a web service, plus you can use the weka library for quickly building trees.
But really, any other programming language (C++, python, matlab) will have the power to build a tree.
You have to design a decision mathematical model. In order to do that, you need a back ground in statistic, data mining or even fuzzy logic.
And then, you code the decision making algorithm to get the final results. I suggest you should have a validation phase to ensure your program is useful and correct.
I'd like to use Haskell for stochastic simulation, but I don't know how. I've read Hutton's 'Programming in Haskell', and I'm comfortable writing deterministic functional programs. However, I don't know how to start writing stochastic simulations of the sort that are easy in imperative languages like R or python. Is there a tutorial or primer on this that I could read, or can anyone provide some tips on getting started?
There's a nice self-contained paper Erwig and Kollmansberger: Functional Pearls - Probabilistic Functional Programming in Haskell on this topic. I used this as a starting point for writing a natural language processor based on Hidden Markov Models in Haskell. There's a package that is based on this paper, which also seems to provide a basic interface to R plotting.
There's also an entry on the HaskellWiki with more links to hackage. In particular, the ProbabilityMonads package might be useful for you.
http://learnyouahaskell.com/a-fistful-of-monads#the-list-monad
This small section in Learn You a Haskell talks about using the List monad and functor functions to easily deal with non-determinism. May be a little simplistic depending on what your needs are, but make good use of the tools that are already in the standard library.
I'm hoping to use either Haskell or OCaml on a new project because R is too slow. I need to be able to use support vectory machines, ideally separating out each execution to run in parallel. I want to use a functional language and I have the feeling that these two are the best so far as performance and elegance are concerned (I like Clojure, but it wasn't as fast in a short test). I am leaning towards OCaml because there appears to be more support for integration with other languages so it could be a better fit in the long run (e.g. OCaml-R).
Does anyone know of a good tutorial for this kind of analysis, or a code example, in either Haskell or OCaml?
Hal Daume has written several major machine learning algorithms during his Ph.D. (now he is an assistant professor and rising star in machine learning community)
On his web page, there are a SVM, a simple decision tree and a logistic regression all in OCaml. By reading these code, you can have a feeling how machine learning models are implemented in OCaml.
Another good example of writing basic machine learning models is Owl library for scientific and numeric computations in OCaml.
I'd also like to mention F#, a new .Net language similar to OCaml. Here's a factor graph model written in F# analyzing Chess play data. This research also has a NIPS publication.
While FP is suitable for implementing machine learning and data mining models. But what you can get here most is NOT performance. It is right that FP supports parallel computing better than imperative languages, like C# or Java. But implementing a parallel SVM, or decision tree, has very little relation to do with the language! Parallel is parallel. The numerical optimizations behind machine learning and data mining are usually imperative, writing them pure-functionally is usually hard and less efficient. Making these sophisticated algorithms parallel is very hard task in the algorithm level, not in the language level. If you want to run 100 SVM in parallel, FP helps here. But I don't see the difficulty running 100 libsvm parallel in C++, not to consider that the single thread libsvm is more efficient than a not-well-tested haskell svm package.
Then what do FP languages, like F#, OCaml, Haskell, give?
Easy to test your code. FP languages usually have a top-level interpreter, you can test your functions on the fly.
Few mutable states. This means that passing the same parameter to a function, this function always gives the same result, thus debugging is easy in FPs.
Code is succinct. Type inference, pattern matching, closures, etc. You focus more on the domain logic, and less on the language part. So when you write the code, your mind is mainly thinking about the programming logic itself.
Writing code in FPs is fun.
The only problem I can see is that OCaml doesn't really support multicore parallelism, while GHC has excellent support and performance. If you're looking to use multiple threads of execution, on multiple calls, GHC Haskell will be a lot easier.
Secondly, the Haskell FFI is more powerful (that is, it does more with less code) than OCaml's, and more libraries are avaliable (via Hackage: http://hackage.haskell.org ) so I don't think foreign interfaces will be a deciding factor.
As far as multi-language integration goes, combining C and Haskell is remarkably easy, and I say this as someone who is (unlike dons) not really much of an expert on either. Any other language that integrates well with C shouldn't be much trickier; you can always fall back to a thin interface layer in C if nothing else. For better or worse, C is still the lingua franca of programming, so Haskell is more than acceptable for most cases.
...but. You say you're motivated by performance issues, and want to use "a functional language". From this I infer you're not previously familiar with the languages you ask about. Among Haskell's defining features are that it, by default, uses non-strict evaluation and immutable data structures--which are both incredibly useful in many ways, but it also means that optimizing Haskell for performance is often dramatically different from other languages, and well-honed instincts may lead you astray in baffling ways. You may want to browse performance-related topics on the Haskell wiki to get a feel for the issues.
Which isn't to say that you can't do what you want in Haskell--you certainly can. Both laziness and immutability can in fact be exploited for performance benefits (Chris Okasaki's thesis provides some nice examples). But be aware that there'll be a bit of a learning curve when it comes to dealing with performance.
Both Haskell and OCaml provide the lovely benefits of using an ML-family language, but for most programmers, OCaml is likely to offer a gentler learning curve and better immediate results.
It's hard to give a definitive answer on this. Haskell has the advantages that Don mentioned along with having a more powerful type system and cleaner syntax. OCaml will be easier to learn if you coming from almost any other language (this is because Haskell is as function as functional languages get), and working with mutable random access structures can be a little clunky in Haskell. You will also likely find the performance characteristics of your OCaml code more intuitive than Haskell because of Haskell's lazy evaluation.
Really, I would recommend you evaluate both if you have the time. Here are some relevant Haskell resources:
http://hackage.haskell.org/package/hslibsvm
http://hackage.haskell.org/package/HSvm
Real World Haskell: this is a great freely available book for Haskell
Learn You a Haskell: this tutorial is just plain fun to read
Oh, if you look further into Haskell be sure to sign up for the Haskell Beginners and Haskell Cafe lists. The community is friendly and eager to help out newcomers (is my bias showing?).
If speed is your prime concern then go for C. Haskell is pretty good performance wise but you are never going to get as fast as C. To my knowledge the only functional language that has bettered C in a benchmark is Stalin Scheme but that is very old and nobody really knows how it works.
I've written genetic programming libraries where performance was key and I wrote it in a functional style in C. The functional style allowed me to easily parallelise it using OMP and it scales linearly upto 8 cores within a single process. You certainly can't do that in OCaml although Haskell is improving all the time with regards to concurrency and parallelism.
The downside of using C was that it took me months to finally find all the bugs and stop the core dumps which was extremely challenging because of the concurrency. Haskell would probably have caught 90% of those bugs on the first compilation.
So speed at any cost ? Looking back I'd wish I'd used Haskell as I could stand it to be 2 - 3 times slower if I'd saved over a month in development time.
While dons is correct that multicore parallelism at the thread level is better supported in Haskell, it sounds like you could live with process level parallelism (from your phrase: ideally separating out each execution to run in parallel.) which is supported quite well in OCaml. Keith pointed out that Haskell has a more powerful type system, but it can also be said that OCaml has a more powerful module system than Haskell.
As others have pointed out, OCaml's learning curve will be lower than Haskell's; you'll likely be more productive more quickly in OCaml. That said, learning OCaml is a great stepping-stone towards learning Haskell because many of the underlying concepts are very similar, so you could always migrate to Haskell later and find a lot of things familiar there. And as you pointed out, there is an OCaml-R bridge.
As an examples of Haskell and Ocaml in machine learning see stuff at Hal Daume and Lloyd Allison homepages. IMO it's is much more straightforward to achieve C++-like performance in Ocaml, than in Haskell. Through, as already said, Haskell has much nicer community (packages, tools and support), syntax&features (i.e. FFI, probability monads via typeclasses) and parallel programming support.
Having revamped OCaml-R, I've got a few comments to make on integrating OCaml and R. It might be worthwile to use OCaml to call R code, it works, but is not yet exactly straightforward. So using it to pilot R is worthwile. Integrating R functionality much more thoroughly is still cumbersome as, for example, much remains to be done to export R's type system and data to OCaml in a seamless way (you will have work to do). Moreover, the interaction of R's GC and OCaml's GC is a delicate point: you free n values in O(n^2) time, which isn't nice (to solve this point, you either need a more flexible R API, as far as I understand it, or to implement a GC in the binding itself as a big R array for proper interaction between GCs).
In a nutshell, I'd go for the "pilot R from OCaml" approach.
Contributions on the GC interaction layer and on mapping R datatypes to OCaml are most welcome.
You may want to take a look at this : http://www.haskell.org/pipermail/haskell-cafe/2010-May/077243.html
Late answer but a machine learning library in Haskell is available here : https://github.com/mikeizbicki/HLearn
This library implements various ML algorithms who are designed to have a much faster cross-validation than the usual implementations. It is based on the following paper Algebraic classifiers: a generic approach to fast cross-validation,
online training, and parallel training. The authors claims a 400x speed-up compared to the same task in Weka.
for haskell, consider checking hasktorch (which I managed to use for my AI thesis). for ocaml there seem to be tensorflow bindings.
What's the advantage, for someone who is not a theoretical computer scientist, of learning a purely functional language like Haskell or Erlang instead of an "impure" language with a strong functional component, like Python or version 2 of the D programming language? My arguments are as follows:
No paradigm is perfect. Languages that are single-paradigm, regardless of what paradigm that is, are very restrictive.
Python or D will ease you into functional programming while you get real work done. Haskell or Erlang will force you to learn functional style all at once before you can actually do anything of interest in them.
Edit: By "impure" functional languages, what I really mean is strongly multiparadigm languages w/ a functional flavor, not strongly functional languages with some non-functional abilities.
It all depends on what are trying to achieve. If your goal to write production code in a functional language - a 'pure' functional language can make it more difficult.
But if you are trying to learn new concepts, 'pure' language gives you the benefit of guarding where you are sliding off the mark of functional concepts. Once you have clear understanding of differences you can go to a mixed environments - like F#, but before that it is all too easy to slip to the OOP way of doing things and because of this miss the advantages of functional approach.
In some other thread I offered an opinion that a better way of learning F# is to start with let us say Haskell (and was voted down for this), but if you learn F# to do OOP than what's the point - stay with C#
You almost answered your own question:
Haskell or Erlang will force you to learn functional style all at once before you can actually do anything of interest in them.
Actually, depending on what you mean by 'of interest', you can be productive in Haskell in a week.
The main reason for anyone to learn Haskell (language theorists already know it, and other kinds of theorists are too busy proving theorems to be bothered with programming) is that learning Haskell will change the way you think about programming. Especially the type system, list comprehensions (stolen for Python—the highest form of praise), and programming with pattern matching. You will be able to use many of your new thoughts in all the programming you do. And Haskell will force you to think new thoughts in a way that other langauges won't.
Erlang is an honorable language but has nothing comparable to Haskell's type system.
If you like Paul Graham you can read more about this line of reasoning in his essay Beating the Averages, especially the part about the "Blub Paradox". Just substitute Haskell for Lisp :-)
You learn very little if your "new" language is just a slight permutation of what you already know. It's like asking, "Why learn Chinese when I can just get a dialect coach to teach me to speak with a Scottish brogue?" I guess it's fine if you enjoy speaking with a brogue, but you're not really expanding your expertise very much.
Learning a functional language teaches you a new way of looking at things. Impure or mixed-paradigm languages like OCaml are good as well, but it can be tempting to use the impure elements as a crutch to avoid having to look at the problem in a new way. No, functional languages are not a magic bullet, but they do have a lot of interesting benefits, and you're robbing yourself of those benefits if you learn a language that has a "functional components" but doesn't really work like a real functional language.
For example, in a pure functional language like Haskell, state is very carefully isolated from the rest of your program. This makes all sorts of optimizations trivial that are very hard in other languages. For example, state is the enemy of parallel processing. In Haskell, you can just look at a function's type and be 100% confident that it won't create any side effects.
This way of thinking in functions that work with immutable data structures something that a pure functional language can teach you. I'm not saying pure functional languages are "the best," but they have their benefits. It's another tool in your belt. And you won't get that tool by sticking with what's familiar.
Is Erlang purely functional? I honestly thought it wasn't.
I'm not an expert on this, so take my words with a grain of salt.
Side-effects prevent (or make more complicated) lots of optimizations, including parallelism. So, in theory, going pure functional should mean better optimization for faster code, although I'm not sure this is true in practice. Even if it does not, it might someday be... it will be fun if a purely functional language comes along which makes using up all the cores easy peasy.
Additionally, in a way, programming without side-effects makes for easier to understand programs.
Because only pure is declarative at the operational semantics level. Pure functional programming (i.e. eliminating accidental dependencies) is required for parallelism.
Also, without the immutability restriction, you won't be forced to think about how to do algorithms just as fast as they can be done in imperative, or how to model interaction with the real world without state spaghetti, using functional reactive programming.
Thus you won't be maximizing your skills for writing code that is maximally composable and extensible.