Related
I want to write a video editing software, and the "logical" conclusion is that the language I must to use is C++... But I don't like it (sorry c++ fans)
I would like to write it with something cool, like Lisp or Haskell or Erlang... But I don't know if the open source implementation of those languages (I don't have money to buy licenses) let me made a competitive software (in the performance area)
What do you think? what do you recommend?
I can't speak to Lisp, but both Erlang and Haskell are capable of the performance necessary for video processing. Achieving that performance is likely to be more difficult than with C++ because there are fewer existing libraries in the domain, so you'll have to implement more yourself. Which means you'll have to be capable of writing high-performance code yourself. In Haskell I expect this would require a significant investment of time (6 months minimum) to become proficient.
Which language you choose should depend a great deal upon the goals of the project. If it's a hobby project, or you want to learn a lot about processing algorithms (and therefore don't mind having to do a lot of low-level coding yourself), there's nothing wrong with using an out-of-mainstream language. Haskell has bindings to a lot of things you would probably want to use eventually, such as a wrapper for GLSL.
As somebody working with audio processing (including real-time), I can say that Haskell's performance hasn't been a problem for me. For a recent project I did write some functions in C, but that was necessary to implement a custom vectorization scheme. Doing high-level work in Haskell and calling out to C when necessary is a perfectly valid approach, although thankfully it's less necessary now than in the past.
Of course, this presumes a few things about the nature of your project. If you want something you can use right away, Haskell, Lisp, and Erlang are probably not the languages for you because there are fewer resources. Have you considered Processing? It's Java, I don't know if you consider that better than C++ or worse.
I had motivations besides productivity for working in Haskell (and my productivity took a big hit for a while), without those other goals I wouldn't have persevered. If you want to write something to use it, stick with what's going to be most productive. If you have other motivations, tell us what they are and it's more likely people will make helpful suggestions.
For what it's worth, Wings3D is written in Erlang.
You could always try D, if you want something somewhat similar to C++ but not C++. Also, D could use some love.
For both Haskell and Erlang, the open source implementation is the standard, most efficient available implementation available. There's no reason that Haskell shouldn't be performant enough for your needs -- for video stuff I assume you'll be using matrices and such. There are quality bindings available for BLAS & co for Haskell. I don't know of a great deal of existing video editing work, but Alberto Ruiz (the author of HMatrix) has done work with Haskell and computer vision: http://dis.um.es/profesores/alberto/research.html
There's also a great deal of work on sound libraries and processing in Haskell.
I'd use the language that gives me the best coverage by third-party libraries for what I'm trying to do; for manipulating video data that's probably going to be a mainstream language like C++.
If this project is for fun/to learn a new language then by all means, take the road less traveled. But if this is something you need to ship in a reasonable amount of time, avoiding the best tools for the job because you don't like them is unsound strategy.
That depends at least on what's your goal with the project. If it's a hobby project and you want to learn a different language, then you should choose that language. In this case, however, I assume you're familiar with video processing. On the other hand, if you want to learn about video processing, I'd recommend using a language you are already comfortable with.
Now, if it's a professional project of a decent size (video processing software can be huge) you should probably consider using different languages for different things. The kind of systems I work with usually require writing some code in C (for efficiency reasons), but we always try to keep that to de indispensable minimum and use a higher level language for most of the system behaviour (we use erlang, but that applies to any other higher level language).
IMO, writing big systems in C or C++ is almost a suicide. There are projects that succeed, but I find that much harder than complementing the C part with higher level languages.
There is already some video streaming server written in Erlang http://erlyvideo.org/. You can look for some inspiration https://github.com/erlyvideo/erlyvideo.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
Have you actually "tried" (means programmed in, not just read an article on it) Erlang and decided against it for a project? If so, why? Also, if you have opted to go back to your old language, or to use another functional language like F#, Haskell, Clojure, Scala, or something else then this counts too, and state why.
I returned to Haskell for my personal projects from Erlang for the simple virtue of Haskell's amazing type system. Erlang gives you a ton of tools to handle when things go wrong. Haskell gives you tools to keep you from going wrong in the first place.
When working in a language with a strong type system you are effectively proving free theorems about your code every time you compile.
You also get a bunch of overloading sugar from Haskell's typeclass machinery, but that is largely secondary to me -- even if it does allow me to express a number of abstractions that would be terribly verbose or non-idiomatic and unusable in Erlang (e.g. Haskell's category-extras).
I love Erlang, I love its channels and its effortless scalability. I turn to it when these are the things I need. Haskell isn't a panacea. I give up a better operational understanding of space consumption. I give up the magical one pass garbage collector. I give up OTP patterns and all that effortless scalability.
But its hard for me to give up the security blanket that, as is commonly said, in Haskell, if it typechecks, it is probably correct.
We use Haskell, OCaml and (now) F# so for us it has nothing to do with lack of C-like syntax. Rather we skip Erlang because:
It's dynamically typed (we're fans of Haskell's type system)
Doesn't provide a 'real' string type (I understand why, but it's annoying that this hasn't been corrected at the language level yet)
Tends to have poor (incomplete or unmaintained) database drivers
It isn't batteries included and doesn't appear to have a community working on correcting this. If it does, it isn't highly visible. Haskell at least has Hackage, and I'd guess that's what has us choosing that language over any other. In Windows environments F# is about to have the ultimate advantage here.
There are probably other reasons I can't think of right now, but these are the major points.
The best reason to avoid Erlang is when you cannot commit to the functional way of programming.
I read an anti-Erlang blog rant a few weeks ago, and one of the author's criticisms of Erlang is that he couldn't figure out how to make a function return a different value each time he called it with the same arguments. What he really hadn't figured out is that Erlang is that way on purpose. That's how Erlang manages to run so well on multiple processors without explicit locking. Purely functional programming is side-effect-free programming. You can arm-twist Erlang into working like our ranting blogger wanted, adding side effects, but in doing so you throw away the value Erlang offers.
Pure functional programming is not the only right way to program. Not everything needs to be mathematically rigorous. If you determine your application would be best written in a language that misuses the term "function", better cross Erlang off your list.
I have used Erlang in a few project already. I often use it for restful services. Where I don't use it however is for complex front end web applications where tools like Ruby on Rails are far better. But for the powerbroker behind the scenes I know of no better tool than Erlang.
I also use a few applications written in Erlang. I use CouchDB and RabbitMQ a bit and I have set up a few EJabberd servers. These applications are the most powerful, easiest and flexible tools in their field.
Not wanting to use Erlang because it does not use JVM is in my mind pretty silly. JVM is not some magical tool that is the best in doing everything in the world. In my mind the ability to choose from an arsenal of different tools and not being stuck in a single language or framework is what separates experts from code monkeys.
PS: After reading my comment back in context I noticed it looked like I was calling oxbow_lakes a code monkey. I really wasn't and apologize if he took it like that. I was generalizing about types of programmers and I would never call an individual such a negative name based on one comment by him. He is probably a good programmer even though I encourage him to not make the JVM some sort of a deal breaker.
Whilst I haven't, others on the internet have, e.g.
We investigated the relative merits of
C++ and Erlang in the implementation
of a parallel acoustic ray tracing
algorithm for the U.S. Navy. We found
a much smaller learning curve and
better debugging environment for
parallel Erlang than for
pthreads-based C++ programming. Our
C++ implementation outperformed the
Erlang program by at least 12x.
Attempts to use Erlang on the IBM Cell
BE microprocessor were frustrated by
Erlang's memory footprint. (Source)
And something closer to my heart, which I remember reading back in the aftermath of the ICFP contest:
The coding was very straightforward,
translating pseudocode into C++. I
could have used Java or C#, but I'm at
the point where programming at a high
level in C++ is just as easy, and I
wanted to retain the option of quickly
dropping down into some low-level
bit-twiddling if it came down to it.
Erlang is my other favorite language
for hacking around in, but was worried
about running into some performance
problem that I couldn't extricate
myself from. (Source)
For me, the fact that Erlang is dynamically typed is something that makes me wary. Although I do use dynamically typed languages because some of them are just so very problem-oriented (take Python, I solve a lot of problems with it), I wish they were statically typed instead.
That said, I actually intended to give Erlang a try for some time, and I’ve just started downloading the source. So your “question” achieved something after all. ;-)
I know Erlang since university, but have never used it in my own projects so far. Mainly because I'm mostly developing desktop applications, and Erlang is not a good language for making nice GUIs. But I will soon implement a server application, and I will give Erlang a try, because that's what it's good for. But I'm worring that I need more librarys, so maybe I'll try with Java instead.
A number of reasons:
Because it looks alien from anyone used to the C family of languages
Because I wanted to be able to run on the Java Virtual Machine to take advantage of tools I knew and understood (like JConsole) and the years of effort which have gone into JIT and GC.
Because I didn't want to have to rewrite all the (Java) libraries I've built up over the years.
Because I have no idea about the Erlang "ecosystem" (database access, configuration, build etc).
Basically I am familiar with Java, its platform and ecosystem and I have invested much effort into building stuff which runs on the JVM. It was easier by far to move to scala
I Decided against using Erlang for my project that was going to be run with a lot of shared data on a single multi-processor system and went with Clojure becuase Clojure really gets shared-memory-concurrency. When I have worked on distributed data storage systems Erlang was a great fit because Erlang really shines at distributed message passing systems. I compare the project to the best feature in the language and choose accordingly
Used it for a message gateway for a proprietary, multi-layered, binary protocol. OTP patterns for servers and relationships between services as well as binary pattern matching made the development process very easy. For such a use case I'd probably favor Erlang over other languages again.
The JVM is not a tool, it is a platform. Although I am all in favour of choosing the best tool for the job the platform is mostly already determined. Unless I am developing something standalone, from scratch and without the desire to reuse any existing code/library (three aspects that are unlikely in isolation already) I may be free to choose the platform.
I do use multiple tools and languages but I mainly targetg the JVM platform. That precludes Erlang for most if not all of my projects, as interesting as some of it concepts are.
Silvio
While I liked many design aspects of the Erlang runtime and the OTP platform, I found it to be a pretty annoying program language to develop in. The commas and periods are totally lame, and often require re-writing the last character of many lines of code just to change one line. Also, some operations that are simple in Ruby or Clojure are tedious in Erlang, for example string handling.
For distributed systems relying on a shared database the Mnesia system is really powerful and probably a good option, but I program in a language to learn and to have fun, and Erlang's annoying factor started to outweigh the fun factor once I had gotten past the basic bank account tutorials and started writing plugins for an XMPP server.
I love Erlang from the concurrency standpoint. Erlang really did concurrency right. I didn't end up using erlang primarily because of syntax.
I'm not a functional programmer by trade. I generally use C++, so I'm covet my ability to switch between styles (OOP, imperative, meta, etc). It felt like Erlang was forcing me to worship the sacred cow of immutable-data.
I love it's approach to concurrency, simple, beautiful, scalable, powerful. But the whole time I was programming in Erlang I kept thinking, man I'd much prefer a subset of Java that disallowed data sharing between thread and used Erlangs concurrency model. I though Java would have the best bet of restricting the language the feature set compatible with Erlang's processes and channels.
Just recently I found that the D Programing language offers Erlang style concurrency with familiar c style syntax and multi-paradigm language. I haven't tried anything massively concurrent with D yet, so I can't say if it's a perfect translation.
So professionally I use C++ but do my best to model massively concurrent applications as I would in Erlang. At some point I'd like to give D's concurrency tools a real test drive.
I am not going to even look at Erlang.
Two blog posts nailed it for me:
Erlang machinery walks the whole list to figure out whether they have a message to process, and the only way to get message means walking the whole list (I suspect that filtering messages by pid also involves walking the whole message list)
http://www.lshift.net/blog/2010/02/28/memory-matters-even-in-erlang
There are no miracles, indeed, Erlang does not provide too many services to deal with unavoidable overloads - e.g. it is still left to the application programmer to deal checking for available space in the message queue (supposedly by walking the queue to figure out the current length and I suppose there are no built-in mechanisms to ensure some fairness between senders).
erlang - how to limit message queue or emulate it?
Both (1) and (2) are way below naive on my book, and I am sure there are more software "gems" of similar nature sitting inside Erlang machinery.
So, no Erlang for me.
It seems that once you have to deal with a large system that requires high performance under overload C++ + Boost is still the only game in town.
I am going to look at D next.
I wanted to use Erlang for a project, because of it's amazing scalability with number of CPU'S. (We use other languages and occasionally hit the wall, leaving us with having to tweak the app)
The problem was that we must deliver our application on several platforms: Linux, Solaris and AIX, and unfortunately there is no Erlang install for AIX at the moment.
Being a small operation precludes the effort in porting and maintaining an AIX version of Erlang, and asking our customers to use Linux for part of our application is a no go.
I am still hoping that an AIX Erlang will arrive so we can use it.
Why do people insist on using trivial mathematical problems like finding numbers in the Fibonacci sequence for language benchmarks? Don't these usually get optimized to relativistic speeds? Isn't the brunt of the bottlenecks usually in I/O, system API calls, operations on strings and structures, processing large quantities of data, abstract object-oriented stuff, etc?
It is a throwback to the old days, when compiler technology for what we would now call basic math was still evolving rapidly.
Now, compiler evolution is more focused on exploiting new instructions for niche operations, 64-bit math, and so on.
Micro-benchmarks such as the ones you mention were useful, though, when evaluating the efficiency of the hotspot compiler when Java was first launched, and in evaluating the efficiency of .NET versus C/C++.
Your suggestion that I/O and system calls are the likely bottlenecks is correct, at least for some space of problems. But I notice you suggested string operations. One person's irrelevant micro-benchmark is another person's critical performance metric.
EDIT: ps, I also remember using linpack and other micro-benchmarks to compare versions of the JVM, and to compare vendors of the JVM. From v4 to v5 there was a big jump in perf, I guess the JIT compiler got more effective. Also, IBM's JVM was ahead of Sun's at that time, on Windows-x86.
Because if you want to benchmark the language/compiler, these "math problems" are good indicators of the "bare speed" of the generated code. Either they use the iterative solution, which is a tight loop and indicates how well can the compiler push the instructions to the processor, or they use the recursive solution, which indicates how does it handle recursive calls of short functions (inlining, tail-recursion etc.) (although the Ackermann function is usually used for that too).
Usually, the benchmark suite for the language contain tests benchmarking other parts as well - eg. gzip compression, text searching, object creation, virtual function call, exception throw/catch benchmarks.
The other things you've noticed, syscalls and IO are usually not included because
syscalls are in fact not that slow - applications don't spend significant porion of the time in the kernel, except for test specifically targeted at them or when something is seriously wrong with the program
syscall and IO performance does not depend on the language, but rather on the OS & hardware
I'd think a simple, well-established algorithm would remove the possibility that the benchmark is biased (whether through ignorance or malice) to favor one language. It is very difficult to write a complex program in two different languages exactly the same. Testing something like the efficiency of a multithreaded application in c# vs java, for example, would require developers skilled in multithreaded development both languages, and there would still be questions as to whether the benchmark app properly represents the general case, or if it is misrepresenting a special case that only one language handles well.
Back when the sieve of eratosthanes was a popular benchmark for C compilers, I thought it would be funny if one of the compiler authors would recognize the sieve code and replace it with a pre-computed lookup.
we all writing code for single processor.
i wonder when we all are able to write code on multi processors?
what do we need (software tools, logic, algorithms) for this switching?
edit: in my view, as we do many task parallely, same way we need to convert those real life solutions(algorithms) to computer lang. just as OOPs coding did for procedural coding. OOPs is more real life coding style than procedural one. so i hope for that kind of solutions.
I think the most important requirement is a good language that has native constructs that support parallelism or one that can automatically generate parallel code. There are quite a few languages that fit that description, but none of them is popular enough to really be considered for mainstream use. That, in turn is caused by several things:
By their very nature, these languages are very different from today's imperative languages, and are therefor harder to learn (or at least seem that way).
They often lack good tools and libraries, making them unusable for any "real" project.
Of course, if it were more popular more people would be willing to learn it and there would be more support, so it's a kind of cycle that's pretty hard to break out of. I guess all we can do is hope. :)
An example of a language designed with heavy parallelization in mind is Erlang - and it's actually used in commercial projects.
What we need are natural abstractions for highly-concurrent algorithms. Actors (think: Erlang) go a long way in this direction, but they aren't a one-size-fits-all solution. Some more specific abstractions like fork/join or map/reduce can be even easier to apply to common problems.
The trick with all of these concurrency abstractions is they require functional-style programming. Concurrency doesn't mesh well with shared mutable state. As they say, "Locks considered harmful". Since most developers come from a strictly imperative background, switching to a shared-nothing continuation passing approach is often extremely challenging.
Incidentally, with respect to concurrency abstractions, Clojure has some very interesting features in this direction. Not only does it have sort-of actors, but it also defines a transactional memory model (think: databases) along with a global, atomic references mechanism. These two features allow concurrent operations to share "mutable" state without ever having to worry about locking or race conditions.
In the end, it comes down to education. Much of the needed theoretical work into concurrency abstractions has already been done, we just need to accept it. Unfortunately, as Erlang and Haskell prove, sometimes the best ideas remain relegated to an extremely fringe demographic. Hopefully efforts like Scala and Clojure will succeed in bringing the more advanced abstractions into the mainstream by sneaking them onto an existing, well-supported platform (the JVM).
Unfortunately for massive concurrent programming - unless there is a breakthrough in compilers to help, we will be throwing out a lot of what we know about algorithms (I think Don Knuth even said that). Read about Erlang for a glimpse of this possible future.
There are several tools/languages that are popular or are gaining popularity. If you use FORTRAN, C, or C++, you can use OpenMP (not too hard to implement) or the Message Passing Interface (MPI) libraries (powerful and greatest speedup potential, but also complex and difficult). OpenMP uses preprocessor directives to mark areas that can be parallelized, especially loops. MPI uses messages that pass data back and forth between processes, and the greatest difficulty is keeping everything synchronized without hitting bottlenecks and keeping processes waiting. I would say MPI is definitely on the way out, however. It's become clear in the scientific/high-performance computing communities that the speedup is rarely worth the additional development time.
As for up and coming languages, check out Fortress. It's still being designed, but the goal is to create a language even easier for scientific computing than FORTRAN. Programs will be specified in a very high level mathematical syntax. Additionally, parallelism will be implicit; the programmer will have to work to do things in serial. Plus, it's being championed by Sun and is based on java, so it will be portable.
There is no simple answer, and in many ways even the complex answers are currently inadequate or incomplete. You'll get a better answer if you are more specific about the replies you want: pointers to dev libraries and tools, instructional materials, pointers to current research projects and issues in this area, or something else?
The most important requirement is to be able to split your problem into smaller problems that can be solved independently of each other. Once you've worked out how you're going to do that, everything else is easier to think about and further questions of implementation (e.g. "parts of my calculation depend on other parts - how do I wait for them to have finished?") become concrete, specific things you can research or ask here about.
for java you can now look to Parallel Java Library or DPJ(deterministic Parallel Java!)
It will offer you great help in extracting parallelism from codes!!
Closed. This question needs to be more focused. It is not currently accepting answers.
Closed 9 years ago.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
I've searched on the Internet for comparisons between F# and Haskell but haven't found anything really definitive. What are the primary differences and why would I want to choose one over the other?
Haskell is a "pure" functional language, where as F# has aspects of both imperative/OO and functional languages. Haskell also has lazy evaluation, which is fairly rare amongst functional languages.
What do these things mean? A pure functional language, means there are no side effects (or changes in shared state, when a function is called) which means that you are guaranteed that if you call f(x), nothing else happens besides returning a value from the function, such as console output, database output, changes to global or static variables.. and although Haskell can have non pure functions (through monads), it must be 'explicitly' implied through declaration.
Pure functional languages and 'No side effect' programming has gained popularity recently as it lends itself well to multi core concurrency, as it is much harder to get wrong with no shared state, rather than myriad locks & semaphores.
Lazy evaluation is where a function is NOT evaluated until it is absolutely necessary required. meaning that many operation can be avoided when not necessary. Think of this in a basic C# if clause such as this:
if(IsSomethingTrue() && AnotherThingTrue())
{
do something;
}
If IsSomethingTrue() is false then AnotherThingTrue() method is never evaluated.
While Haskell is an amazing language, the major benefit of F# (for the time being), is that it sits on top of the CLR. This lends it self to polyglot programming. One day, you may write your web UI in ASP.net MVC, your business logic in C#, your core algorithms in F# and your unit tests in Ironruby.... All amongst the the .Net framework.
Listen to the Software Engineering radio with Simon Peyton Jones for more info on Haskell: Episode 108: Simon Peyton Jones on Functional Programming and Haskell
Big differences:
Platform
Object orientation
Laziness
The similarities are more important than the differences. Basically, you should use F# if you are on .NET already, Haskell otherwise. Also, OO and laziness mean that F# is closer to what you (probably) already know, so it is probably easier to learn.
Platform : Haskell has its own runtime, F# uses .NET. I don't know what the performance difference is, although I suspect the average code is about the same before optimisation. F# has the advantage if you need the .NET libraries.
Object orientation : F# has OO, and is very careful to make sure that .NET classes are easy to use even if your code isn't OO. Haskell has type classes which let you do something like OO, in a weird sort of way. They are like Ruby mixins crossed with Common Lisp generic functions. They're a little like Java/C# interfaces.
Laziness : Haskell is lazy, F# is not. Laziness enables some nice tricks and makes some things that look slow actually execute fast. But I find it a lot harder to guess how fast my code will run. Both languages let you use the other model, you just have to be explicit about it in your code.
Minor differences:
Syntax : Haskell has slightly nicer syntax in my opinion. It's a little more terse and regular, and I like declaring types on a separate line. YMMV.
Tools : F# has excellent Visual Studio integration, if you like that sort of thing. Haskell also has an older Visual Studio plugin, but I don't think it ever got out of beta. Haskell has a simple emacs mode, and you can probably use OCaml's tuareg-mode to edit F#.
Side effects : Both languages make it pretty obvious when you are mutating variables. But Haskell's compiler also forces you to mark side effects whenever you use them. The practical difference is that you have to be a lot more aware of when you use libraries with side effects as well.
F# is part of the ML family of languages and is very close to OCaml. You may want to read this discussion on the differences between Haskell and OCaml.
A major difference, which is probably a result ofthe purity but I less see mentioned, is the pervasive use of monads. As is frequently pointed out, monads can be built in most any language, but life changes greatly when they are used pervasively throughout the libraries, and you use them yourself.
Monads provide something seen in a much more limited way in other languages: abstraction of flow control. They're incredibly useful and elegant ways of doing all sorts of things, and a year of Haskell has entirely changed the way I program, in the same way that moving from imperative to OO programming many years ago changed it, or, much later, using higher-order functions did.
Unfortunately, there's no way in a space like this to provide enough understanding to let you see what the difference is. In fact, no amount of writing will do it; you simply have to spend enough time learning and writing code to gain a real understanding.
As well, F# sometimes may become slightly less functional or more awkward (from the functional programming point of view) when you interface with the .NET platform/libraries, as the libraries were obviously designed from an OO point of view.
So you might consider your decision this way: are you looking to try out one of these languages in order to get a quick, relatively small increment of improvement, or are you willing to put in more time and get less immediate benefit for something bigger in the long term. (Or, at least, if you don't get something bigger, the easy ability to switch to the other quickly?) If the former, F# is your choice, if the latter, Haskell.
A couple of other unrelated points:
Haskell has slightly nicer syntax, which is no suprise, since the designers of Haskell knew ML quite well. However, F#'s 'light' syntax goes a long way toward improving ML syntax, so there's not a huge gap there.
In terms of platforms, F# is of course .NET; how well that will work on Mono I don't know. GHC compiles to machine code with its own runtime, working well under both Windows and Unix, which compares to .NET in the same way, that, say, C++ does. This can be an advantage in some circumstances, especially in terms of speed and lower-level machine access. (I had no problem writing a DDE server in Haskell/GHC, for example; I don't think you could do that in any .NET language, and regardless, MS certainly doesn't want you doing that.)
Well, for one I'd say a main advantage is that F# compiles against the .NET platform which makes it easy to deploy on windows. I've seen examples which explained using F# combined with ASP.NET to build web applications ;-)
On the other hand, Haskell has been around for waaaaay longer, so I think the group of people who are real experts on that language is a lot bigger.
For F# I've only seen one real implementation so far, which is the Singularity proof of concept OS. I've seen more real world implementations of Haskell.