I am currently taking a couple of classes involving these concepts and in some circumstances, the terms 'concurrency' and 'parallelism' seem to be used interchangeably and in some circumstances there seems to be a difference. Are these terms synonymous? If not, what are the definitions of concurrency and parallelism and how are the concepts similar/different?
yes there is a difference.
Parallelism:
Ability to execute tasks in parallel
Concurrency can be done with even single processing unit using time sharing. This will give you the illusion of things performed in parallel but they are not.
http://www.haskell.org/haskellwiki/Parallelism_vs._Concurrency
Concurrency is about dealing with lots of things at once.
Parallelism is about doing lots of things at once.
Concurrency is searching the independent section in your code.
While Parallelism is about executing those sections.
Related
This is something of a follow-up to this other question of mine.
I would like to know if parallelized loops with a reduction operation, like a parallelized integration, belongs to the domain of applicability of C++11 threading or if OpenMP is best suited for tasks like this.
Now, consider the same setting but with threads executing computations that may throw exceptions. Does it change the scenario? Would now C++11 threading be best suited?
Thank you.
IMO, I would prefer OpenMP for any HPC / scientific and engineering computing codes. It more directly targets data parallelism. C++11 threading represents more task parallelism, which is preferable for other kinds of software (e.g., network server applications).
The situations might change in the future, there are some efforts to integrate more parallelism into C++, such as parallel STL algorithms. However, we now even do not know how this parallelism will look like.
You also rarely build codes from scratch. There are many performance-aware multi-threaded libraries that support OpenMP (sorting, linear algebra, ...), however few that support C++11 threads.
As best as I can determine, OpenMP represents greater performance potential, simply because there are a lot more tricks a compiler can use (particularly if your cpu supports vectorized computations) if it can be directly instructed to parallelize a construct. Host/dispatch threading models (like the threading models in Java and C++11) can't really do that without remarkably intelligent code analysis tools.
However, OpenMP does represent a tax on both code readability and design flexibility. Parallel execution of heterogeneous tasks is possible in OpenMP, but much more verbose to implement, and much more difficult to parse. And because it depends on preprocessor macros (which C++ purists don't like anyways) it's virtually impossible to set dynamic state about the threading model itself.
Personally, having worked on enterprise level code, I think I prefer Host/dispatch threading (aka, C++11 threads). It may represent a performance sacrifice, but as the saying goes: "Processor Cycles are much cheaper than Developer Cycles". And if you really, really are in a performance constrained environment, it either means an algorithm problem, and switching to OpenMP probably wouldn't fix it; or, it means you should probably be looking into compute cards or OpenCL/Cuda programming.
I am really struggling to formulate a good concept for this one. Is it along the lines of the idea that a Parallel program will execute truly independently without relying on another thread/process? I am also struggling to issues parallelism adds to sequential programming and what issues concurrency adds to parallelism?
My ideas so far are that parallelism creates issues with sequential programming because it requires a different mindset, therefore perhaps programs should be written in the first place to not be inherently reliant on the order of operations.
I fail to see major issues concurrency adds to parallelism? Any clarification or explanation on these concepts is greatly appreciated.
I have been reading the book The Art of Multiprocessor Programming and noticed that a lot of the algorithms mentioned assume a fixed number of threads (e.g. the combining tree algorithm). Is there a straightforward way of generalizing such algorithms to a scenario where threads are created and destroyed in an unpredictable manner?
As Zneak has said, there is no automatic way of determining the number of threads for an application. There are some patterns though that can help. The book may talk about some...here is an article that I was recently reading that I liked....hope it is useful for you too. Ref: Thread pools and Work queues
There's a plethora of paradigms and methods for concurrent programming in use today. Software transactional memory, actors, shared state concurrency, tuple spaces and many, many more.
What I find lacking, however, is a library of interesting test problems for concurrency. One well known example is the "Dining Philosophers Problem", which is neither a complex enough nor motivating nor realistic one. Then there are many parallel algorithms (matrix multiplication, rendering, general nested data parallelism) that just require distribution of work, but no real concurrency with communication between threads of execution.
So, can anyone point me to some interesting sets of problems that require real concurrency in an interactive, perhaps even distributed environment, that are simple enough to use as examples for concurrency paradigms? Ideally, I want to find a set of problems to serve as a "lackmus-test" for concurrency paradigms (or to highlight their differences, as every paradigm has its strengths and weaknesses).
Any help is much appreciated :)
I've previously considered this exact issue, having previously proposed some concurrent programing paradigms myself :p
The conclusion I reached then is that such a test set doesn't seem to really exist in a language-independent manner. While it might be helpful for it to exist, there seem to be some fairly good reasons it doesn't (to the best of my knowledge).
Most of the focus within concurrent programming tends to be on data parallelism, such that the same operation is applied in parallel to different pieces of the same data set. The kinds of task-level parallelism (i.e. different tasks being performed in parallel, possibly sharing data) that I think you're talking about is actually not done very much. I think this is because it's kinda hard. But I think it's also kinda hard because most problems don't lend themselves particularly well to this kind of concurrency. Describing a distributed system in terms of concurrency primitives may be helpful, but these systems tend to be decoupled such that there is a protocol (written or implied) moderating their communication. People tend not to think of these kinds of systems as obviously "concurrent" programming situations, even though they are when viewed within the right frame (i.e. considering the "client" and "server" as agents operating in parallel with synchronisation at some points).
The only places I think you could find some sources of inspiration would be within individual implementations. Erlang, Occam (and Occam-pi), Alice, CML, Concurrent Haskell etc all are likely to have small test corpuses, but both the problems and their implementations are going to be biased towards being implementable within a specific language (because they obviously are implementable within that language!). Perhaps you could also look to the communities working on multi-party session types, and various process calculi such as pi-calculus, CCS and CSP to see what kinds of systems they are using as example models. The idea of a standard language-agnostic set of problems for describing concurrent communicating systems is appealing, but somewhat elusive at this point, I think.
Are there any paradigm that give you a different mindset or have a different take to writing multi thread applications? Perhaps something that feels vastly different like procedural programming to function programming.
Concurrency has many different models for different problems. The Wikipedia page for concurrency lists a few models and there's also a page for concurrency patterns which has some good starting point for different kinds of ways to approach concurrency.
The approach you take is very dependent on the problem at hand. Different models solve various different issues that can arise in concurrent applications, and some build on others.
In class I was taught that concurrency uses mutual exclusion and synchronization together to solve concurrency issues. Some solutions only require one, but with both you should be able to solve any concurrency issue.
For a vastly different concept you could look at immutability and concurrency. If all data is immutable then the conventional approaches to concurrency aren't even required. This article explores that topic.
I don't really understand the question, but if you start doing some coding using CUDA give you some different way of thinking about multi-threading applications.
It differs from general multi-threading technics, like Semaphores, Monitors, etc. because you have thousands of threads concurrently. So the problem of parallelism in CUDA resides more in partitioning your data and mixing the chunks of data later.
Just a small example of a complete rethinking of a common serial problem is the SCAN algorithm. It is as simple as:
Given a SET {a,b,c,d,e}
I want the following set:
{a, a+b, a+b+c, a+b+c+d, a+b+c+d+e}
Where the symbol '+' in this case is any Commutattive operator (not only plus, you can do multiplication also).
How to do this in parallel? It's a complete rethink of the problem, it is described in this paper.
Many more implementations of different algorithms in CUDA can be found in the NVIDIA website
Well, a very conservative paradigm shift is from thread-centric concurrency (share everything) towards process-centric concurrency (address-space separation). This way one can avoid unintended data sharing and it's easier to enforce a communication policy between different sub-systems.
This idea is old and was propagated (among others) by the Micro-Kernel OS community to build more reliable operating systems. Interestingly, the Singularity OS prototype by Microsoft Research shows that traditional address spaces are not even required when working with this model.
The relatively new idea I like best is transactional memory: avoid concurrency issues by making sure updates are always atomic.
Have a looksee at OpenMP for an interesting variation.