What would the effectiveness be of multithreading with alpha beta pruning if:
The multithreading was used iteratively. For example, thread one would look at the first branch, the second thread would look at the second thread, etc. I believe this should only be done at the first depth (the next move the AI made), since the other depths could be cut off.
One thread was at the first "move" generated searching to half the moveset generated, and the second thread was at the last "move" generated and searching back to half the moveset. Here, I think there could be increased speedup, because the last move could be considered the best move, and as a result, the second thread could cause cutoffs the first thread couldn't.
The multithreading was used to think on the opponent's time. For example, say the opponent took some time to think and make a move. The AI could iteratively deepen its search and find results while the opponent is thinking, i'd imagine, not necessarily causing speedup, but having more time for minimax analysis.
There may be other optimizations, i'd imagine, but these were the few that came into mind. I don't know if they actually will improve anything, though.
If I understand your idea correctly, you plan to search moves in the root position in parallel. In comparison to a strictly sequential algorithm, it should be better but I would not expect it to scale well (with multiple CPUs).
For comparison, here is a summary of existing parallelization strategies in chess:
https://www.chessprogramming.org/Parallel_Search
As alpha-beta is a sequential algorithm, all parallelization strategies are speculative. So, you want to avoid spending time on searching parts of the search tree, which will eventually be cut by other moves. One relatively simple strategy to avoid searching irrelevant subtrees is called Young Brothers Wait Concept.
There are also algorithms with improved scalability but at the cost of being more difficult to understand and implement. For instance, supporting work-stealing should improve scalability.
Related
I'm trying to create a chess engine using an alpha beta minimax search algorithm, but the code is too slow. I've done all the optimisations I can think of, but it is still very slow in a single thread. I looked at the source code of some other engines to see how they do it and the chess programming wiki (https://www.chessprogramming.org/Parallel_Search#Parallel_Alpha-Beta), but the the code is beyond my level and I don't understand them. I couldn't find any written sources or code snippets either.
Can someone explain how to efficiently implement threading in an alpha-beta search algorithm? Thanks.
Alpha-beta is an inherently sequential algorithm as your alpha and beta values get updated continuously thorough the search and cutoffs are decided based on these values. For this reason getting any speedups by increasing the amount of threads is very hard, and the more threads you throw at it, the smaller the gains will be.
However there's still several ways to do it, most of them fairly complicated and they scale extremely poorly with more threads. The go to algorithm used to be the Young Brothers wait concept, it is a fairly complicated algo and it was used for example by Stockfish until a few years back. However with the increasing amount of cores available on modern computers the scaling was very poor and the code very complex. Today most modern engines use something called Lazy SMP. This algorithm is almost as simple as it can be and scales better than the others.
In lazy SMP all you have to do is start the exact same search as you would normally do, just on multiple threads. It relies on having a working transposition table through which the threads communicate with each other. The threads will never be exactly in sync and the randomness will lead each thread to explore slightly different parts of the search tree and then save their results into the transposition table, where it might be used by another thread. Of course there is a lot of repeating work done by each thread, however it is still better than trying to be clever about splitting the work and slowing down the algorithm, and this is especially true when you start scaling up the amount of threads.
I recommend you take a look at the chess programming wiki, where you can even find some pseudo code on how to implement it.
https://www.chessprogramming.org/Lazy_SMP
Though i should also point out that if what you are looking for is improving your time to depth, implementing multithreading won't do all that much for you (and in some extreme cases it might actually even slow it down!). What you need instead is more aggresive pruning of the search tree and more efficient implementation (eg. no memory allocations, so the garbage collector never has to run, etc.).
As the title states, what is the overhead of the different forms of parallelism, at least in the current implementation of Julia (v0.5, in case the implementation changes drastically in the future)? I am looking for some "practical measures", some general heuristics or ballparks to keep in my head for when it can be useful. For example, it's pretty obvious that multiprocessing won't give you gains in a loop like:
addprocs(4)
#parallel (+) for i=1:4
rand()
end
doesn't give you performance gains because each process is only taking one random number, but is there general heuristic for knowing when it will be worthwhile? Also, what about a heuristic for threading. It's surely a lower overhead than multiprocessing, but for example, with 4 threads, for what N is it a good idea to multithread:
A = rand(4)
Base.#threads (+) for i = 1:N
A[i%4+1]
end
(I know there isn't a threaded reduction right now, but let's act like there is, or edit with a better example). Sure, I can benchmark every example, but some good rules to keep in mind would go a long way.
In more concrete terms: what are some good rules of thumb?
How many numbers do you need to be adding/multiplying before threading gives performance enhancements, or before multiprocessing gives performance enhancements?
How much does the depend on Julia's current implementation?
How much does it depend on the number of threads/processes?
How much does the depend on the architecture? Are there good rules for knowing when the threshold should be higher/lower on a particular system?
What kinds of applications violate these heuristics?
Again, I'm not looking for hard rules, just general guidelines to guide development.
A few caveats: 1. I'm speaking from experience with version 0.4.6, (and prior), haven't played with 0.5 yet (but, as I hope my answer below demonstrates, I don't think this is essential vis-a-vis the response I give). 2. this isn't a fully comprehensive answer.
Nevertheless, from my experience, the overhead for multiple processes itself is very small provided that you aren't dealing with data movement issues. In other words, in my experience, any time that you ever find yourself in a situation of wishing something were faster than a single process on your CPU can manage, you're well past the point where parallelism will be beneficial. For instance, in the sum of random numbers example that you gave, I found through testing just now that the break-even point was somewhere around 10,000 random numbers. Anything more and parallelism was the clear winner. Generating 10,000 random number is trivial for modern computers, taking a tiny fraction of a second, and is well below the threshold where I'd start getting frustrated by the slowness of my scripts and want parallelism to speed them up.
Thus, I at least am of the opinion, that although there are probably even more wonderful things that the Julia developers could do to cut down on the overhead even more, at this point, anything pertinent to Julia isn't going to be so much of your limiting factor, at least in terms of the computation aspects of parallelism. I think that there are still improvements to be made in terms of enhancing both the ease and the efficiency of parallel data movement (I like the package that you've started on that topic as a good step. You and I would probably both agree there's still a ways more to go). But, the big limiting factors will be:
How much data do you need to be moving around between processes?
How much read/write to your memory do you need to be doing during your computations? (e.g. flops per read/write)
Aspect 1. might at times lean against using parallelism. Aspect 2. is more likely just to mean that you won't get so much benefit from it. And, at least as I interpret "overhead," neither of these really fall so directly into that specific consideration. And, both of these are, I believe, going to be far more heavily determined by your system hardware than by Julia.
I have been trying to understand how ACO optimization can be implemented with data parallelism. I have read some content after searching in Google. I only need the basic idea in simple way. Most of the papers are talking about everything else instead of the main thing in simple words.
What I understood so far is, we will make it work parallel by using multi-tasking(threading). But am not sure what each thread would do or how we could separate it into threads without causing trouble.
Does it means that we should create separate thread for each ants? But that would cause lots of threads to be created! So if there are 200 ants, then 200 threads?
Am still having confusion at this data parallelism topic in ACO. I would really love to hear in simple words on how we would implement it parallely.
A few simple ideas to run ACO in parallel
Since you have already read up on ACO, here are a few simple ideas on ways to run ACO in parallel. Rather than getting caught up in multiple-threads and mutli-tasking, it might be helpful to think in terms of 'parallel compute resources' at your disposal.
ACO is one case of Agent-Based Simulation (ABS), and ABS lends itself particularly well to parellization.
Simple Options
Option 1. Run a full version of ACO in each of the parallel resources.
Code your ACO algorithm, run it in parallel fashion. (Since there is a stochastic element to the algorithm, you can then look for the 'best' solution for your problem.)
Option 2. To explore effects of varying ACO parameters
Like any simulation approach, any ACO implementation has a large number of runtime parameters: Number of vertices, time to run, Number of ants, Pheromone evaporation rates, probability functions to choose path options and many more. When you mutliply these options, they add up to some large number of cases to be run. Divide up the work among your parallel compute resources.
The two options mentioned above are sometimes referred to as 'embarrassingly' parallel. Very easy to implement (think of it as a Design of Experiments) and you get back a whole matrix of results, and you can make conclusions by studying what effect the changes in the parameters had on the solution.
Option with solution sharing
Option 3: Master-Slave approach, with Partial Solution sharing
Going up one more level in complexity, we can use each node to contribute its 'knowledge/findings' to the overall problem solution. This is sometimes called a master-slave approach. The master is trying to solve the overall problem (Could be TSP, or some similar complex problem) and each 'slave' is solving some aspect of it, but with some fairly simple algorithm. The idea is that when combined they produce powerful results.
After a certain number of iterations, the solutions are passed back and forth, with 'bad' solutions thrown out. Some variant of the Map-Shuffle-Reduce paradigm would do that. The master evaluates the current best solution, and that is transferred back to each 'slave' node (Example: the latest overall pheromone levels are given to all the slave nodes). The next round of solving resumes.
Option 3 has tons of nuanced variations, and some people spend their entire lives improving various aspects of it.
Hope some of these ideas help.
This is a very general question that is mostly just conceptual. I was thinking about testing the random number generator to see its effectiveness at following a uniform distribution over some x values (ex 6 for a dice roll). Doing this in a simple loop is how I have it now, but I was thinking about multithreading the simulation.
I was wondering if this would give me any speedup since I would only have one random number generator shared between all threads with semaphore protection (needed to ensure no two threads access and generate random numbers at the same time meaning duplication of results).
Since each thread would hardly have other operations (just if statements for checking and incrementing x) would threading it even give me faster results, or will the dependency on one random number generator mean it would be essentially the same as a single thread?
I think you've answered your own question. Your plan will lead to a single-threaded use of the RNG, with different threads taking turns to be the single one in use. You'll probably achieve a speedup, but only a negative one.
In theory you should see an increase in performance, at least until the number of threads equals the number of cores in use. However, in reality, you'll be adding code (and therefore execution time) to handle the multithreading infrastructure, and if the bulk of each thread's time is spent waiting for a slow RNG you may see a decrease in performance.
On the other hand, you may be able to improve performance with some cleverness. For example, you might have one task dedicated to generating random numbers, and if you're only looking for values from 1 to 6, you may be able to generate more than one value from each result from the RNG. You could put those values into a queue and let the other tasks read from the queue. Of course, you'd have to be careful that your optimizations don't alter the distribution of the RNG.
Unless the idea of counting execution cycles excites you, the best way to find your answer is by trying it. And it's always educational to use a profiler to find out where most of the time is spent--this is notoriously difficult for humans to get right through intuition alone, and even experienced developers are often surprised by the results.
needed to ensure no two threads access and generate random numbers at
the same time
This mean that really only one worker thread would be run so you are not use benefits of multithreading. Or I've missed a point where you mentioned distributing some work across the multipel threads?
If you somehow improve overall design around RNG access from the multiple threads - consider using ReaderWriterLock technique rather than Semaphore.
According to Wikipedia, an "embarrassingly parallel" problem is one for which little or no effort is required to separate the problem into a number of parallel tasks. Raytracing is often cited as an example because each ray can, in principle, be processed in parallel.
Obviously, some problems are much harder to parallelize. Some may even be impossible. I'm wondering what terms are used and what the standard examples are for these harder cases.
Can I propose "Annoyingly Sequential" as a possible name?
Inherently sequential.
Example: The number of women will not reduce the length of pregnancy.
There's more than one opposite of an "embarrassingly parallel" problem.
Perfectly sequential
One opposite is a non-parallelizable problem, that is, a problem for which no speedup may be achieved by utilizing more than one processor. Several suggestions were already posted, but I'd propose yet another name: a perfectly sequential problem.
Examples: I/O-bound problems, "calculate f1000000(x0)" type of problems, calculating certain cryptographic hash functions.
Communication-intensive
Another opposite is a parallelizable problem which requires a lot of parallel communication (a communication-intensive problem). An implementation of such a problem will scale properly only on a supercomputer with high-bandwidth, low-latency interconnect. Contrast this with embarrassingly parallel problems, implementations of which run efficiently even on systems with very poor interconnect (e.g. farms).
Notable example of a communication-intensive problem: solving A x = b where A is a large, dense matrix. As a matter of fact, an implementation of the problem is used to compile the TOP500 ranking. It's a good benchmark, as it emphasizes both the computational power of individual CPUs and the quality of interconnect (due to intensity of communication).
In more practical terms, any mathematical model which solves a system of partial differential equations on a regular grid using discrete time stepping (think: weather forecasting, in silico crash tests), is parallelizable by domain decomposition. That means, each CPU takes care of a part of the grid, and at the end of each time step the CPUs exchange their results on region boundaries with "neighbour" CPUs. These exchanges render this class of problems communication-intensive.
Im having a hard time to not post this... cause I know it don't add anything to the discussion.. but for all southpark fans out there
"Super serial!"
"Stubbornly serial"?
The opposite of embarassingly parallel is Amdahl's Law, which says that some tasks cannot be parallel, and that the minimum time a perfectly parallel task will require is dictated by the purely sequential portion of that task.
"standard examples" of sequential processes:
making a baby: “Crash programs fail because they are based on theory that, with nine women pregnant, you can get a baby a month.” -- attributed to Werner von Braun
calculating pi, e, sqrt(2), and other irrational numbers to millions of digits: most algorithms sequential
navigation: to get from point A to point Z, you must first go through some intermediate points B, C, D, etc.
Newton's method: you need each approximation in order to calculate the next, better approximation
challenge-response authentication
key strengthening
hash chain
Hashcash
P-complete (but that's not known for sure yet).
I use "Humiliatingly Sequential"
Paul
If ever one should speculate what it would be like to have natural, incorrigibly sequential problems, try
blissfully sequential
to counter 'embarrassingly parallel'.
"Gladdengly Sequential"
It all has to do with data dependencies. Embarrassingly parallel problems are ones for which the solution is made up of many independent parts. Problems with the opposite of this nature would be ones that have massive data dependencies, where there is little to nothing that can be done in parallel. Degeneratively dependent?
The term I've heard most often is "tightly-coupled", in that each process must interact and communicate often in order to share intermediate data. Basically, each process depends on others to complete their computation.
For example, matrix processing often involves sharing boundary values at the edges of each array partition.
This is in contrast to embarassingly parallel (or loosely-coupled) problems where each part of the problem is completely self-contained, and no (or very little) IPC is needed. Think master/worker parallelism.
Boastfully sequential.
I've always preferred 'sadly sequential' ala the partition step in quicksort.
"Completely serial?"
It shouldn't really surprise you that scientists think more about what can be done than what cannot be done. Especially in this case, where the alternative to parallelizing is doing everything as one normally would.
Completely non-parallelizable?
Pessimally parallelizable?
The opposite is "disconcertingly serial".
taking into acount that parallelism is the act of doing many jobs in the same time step t. the opposite could be time-sequential problems
An example inherently sequential problem.
This is common in CAD packages and some kinds of engineering analysis.
Tree traversal with data dependencies between nodes.
Imagine traversing a graph and adding up weights of nodes.
You just can't parallelise it.
CAD software represents parts as a tree, and to render to object you have to traverse the tree.
For this reason, cad workstations use less cores and faster, rather than many cores.
Thanks for reading.
You could of course, however I think that both 'names' are a non-issue.
From a functional programming perspective you could say that the 'annoyingly sequential' part is the smallest more or less independent part of an algorithm.
While the 'embarrassingly parallel' if not indeed taking into a parallel approach is bad coding practice.
Thus I don't see a point in given these things a name if good coding practice is always to brake up your solution into independent pieces, even if you at that moment don't take advantage of parallelism.