I'm not quite sure as to what this term means. I saw it during a course where we are learning about concurrency. I've seen a lot of definitions for data interleaving, but I could find anything about process interleaving.
When looking at the term my instincts tell me it is the use of threads to run more than one process simultaneously, is that correct?
If you imagine a process as a (possibly infinite) sequence/trace of statements (e.g. obtained by loop unfolding), then the set of possible interleavings of several processes consists of all possible sequences of statements of any of those process.
Consider for example the processes
int i;
proctype A() {
i = 1;
}
proctype B() {
i = 2;
}
Then the possible interleavings are i = 1; i = 2 and i = 2; i = 1, i.e. the possible final values for i are 1 and 2. This can be of course more complex, for instance in the presence of guarded statements: Then the next possible statements in an interleaving sequence are not necessarily those at the position of the next program counter, but only those that are allowed by the guard; consider for example the proctype
proctype B() {
if
:: i == 0 -> i = 2
:: else -> skip
fi
}
Then the possible interleavings (given A() as before) are i = 1; skip and i = 2; i = 1, so there is only one possible final value for i.
Indeed the notion of interleavings is crucial for Spin's view of concurrency. In a trace semantics, the set of possible traces of concurrent processes is the set of possible interleavings of the traces of the individual processes.
It simply means performing (data access or execution or ... ) in an arbitrary order**(see the note). In the case of concurrency, it usually refers to action interleaving.
If the process P and Q are in parallel composition (P||Q) then the actions of these will be interleaved. Consider following processes:
PLAYING = (play_music -> stop_music -> STOP).
PERFORMING = (dance -> STOP).
||PLAY_PERFORM = (PLAYING || PERFORMING).
So each primitive process can be shown as: (generated by LTSA model-cheking tool)
Then the possible traces as the result of action interleaving will be:
dance -> play_music -> stop_music
play_music -> dance -> stop_music
play_music -> stop_music -> dance
Here is the LTSA tool generated output of this example.
**note: "arbitrary" here means arbitrary choice of process execution not their inner sequence of codes. The code execution in each process will be always followed sequentially.
If it is still something that you're not comfortable with you can take a look at: https://www.doc.ic.ac.uk/~jnm/book/firstbook/pdf/ch3.pdf
Hope it helps! :)
Operating Systems support Tasks (or Processes). But for now let's think of "Actitivities".
Activities can be executed in parallel. Here are two activities, P and Q:
P: abc
Q: def
a, b, c, d, e, f, are operations. *
Each operation has always the same effect independent of what other
operations may be executing at the same time (atomicity).
What is the effect of executing the two activities concurrently? We
do not know for sure, but we know that it will be the same as obtained
by executing sequentially an INTERLEAVING of the two activities
[interleavings are also called SCHEDULES]. Here are the possible
interleavings of these two activities:
abcdef
abdcef
abdecf
abdefc
adbcef
......
defabc
That is, the operations of the two activities are sequenced in all possible ways that preserve the order in which the operations appeared in the two activities. A serial interleaving [serial schedule] of two activities is one where all the operations of one activity precede all the operations of the other activity.
The importance of the concept of interleaving is that it allows us to express the meaning of concurrent programs: The parallel execution of activities is equivalent to the sequential execution of one of the interleavings of these activities.
For detailed information: https://cis.temple.edu/~ingargio/cis307/readings/interleave.html
Related
The code below represents a toy example of the problem I am trying to solve.
Imagine that we have an original stream of data originalStream and that the goal is to apply 2 very different data processing. As an example here, one data processing will multiply each element by 2 and sum the result (dataProcess1) and the other will multiply by 4 and sum the result (dataProcess2). Obviously the operation would not be so simple in real life....
The idea is to use jOOλ in order to duplicate the stream and apply both operations to the 2 streams. However, the trick is that I want to run both data processing in different threads. Since originalStream.duplicate() is not thread-safe out of the box, the code below will fail to give the right result which should be: result1 = 570; result2 = 180. Instead the code may unpredictably fail on NPE, yield the wrong result or (sometimes) even give the right result...
The question is how to minimally modify the code such that it will become thread-safe.
Note that I do not want to first collect the stream into a list and then generate 2 new streams. Instead I want to stay with streams until they are eventually collected at the end of the data process. It may not be the most efficient nor the most logical thing to want to do but I think it is nevertheless conceptually interesting. Note also that I wish to keep using org.jooq.lambda.Seq (group: 'org.jooq', name: 'jool', version: '0.9.12') as much as possible as the real data processing functions will use methods that are specific to this library and not present in regular Java streams.
Seq<Long> originalStream = seq(LongStream.range(0, 10));
Tuple2<Seq<Long>, Seq<Long>> duplicatedOriginalStream = originalStream.duplicate();
ExecutorService executor = Executors.newFixedThreadPool(2);
List<Future<Long>> res = executor.invokeAll(Arrays.asList(
() -> duplicatedOriginalStream.v1.map(x -> 2 * x).zipWithIndex().map(x -> x.v1 * x.v2).reduce((x, y) -> x + y).orElse(0L),
() -> duplicatedOriginalStream.v2.map(x -> 4 * x).reduce((x, y) -> x + y).orElse(0L)
));
executor.shutdown();
System.out.printf("result1 = %d\tresult2 = %d\n", res.get(0).get(), res.get(1).get());
I am not familiar with multi-thread and locks and atomic/nonatomic operations.
Recently I saw an interview question as below.
Put f1 and f2 in two separate threads and run them at the same time, when both of them return, what is the value of a?
int a = 2, b = 0, c = 0
func f1()
{
a = a * 2
a = b
}
func f2()
{
c = a + 11
a = c
}
I tried to implement the above code in objective c environment and what I got is a = 11. I'm not sure if this is right since what I did is put f1 in main queue and put f2 in a dispatch global queue and ran it async which could be incorrect.
If someone could give an answer and explain the process based on the level of register accessing, CPU processing, memory usage, that would be great.
The answer is - the result of A is random. It can be anything. Since access to A is not atomic and there is no synchronization, different threads might see a different value for a depending on random factors. If you manage to make a unaligned and run it on X86, you might even see a non-value for a.
I am just starting to learn the ins-and-outs of multithread programming and have a few basic questions that, once answered, should keep me occupied for quite sometime. I understand that multithreading loses its effectiveness once you have created more threads than there are cores (due to context switching and cache flushing). With that understood, I can think of two ways to employ multithreading of a recursive function...but am not quite sure what is the common way to approach the problem. One seems much more complicated, perhaps with a higher payoff...but thats what I hope you will be able to tell me.
Below is pseudo-code for two different methods of multithreading a recursive function. I have used the terminology of merge sort for simplicity, but it's not that important. It is easy to see how to generalize the methods to other problems. Also, I will personally be employing these methods using the pthreads library in C, so the thread syntax mildly reflects this.
Method 1:
main ()
{
A = array of length N
NUM_CORES = get number of functional cores
chunk[NUM_CORES] = array of indices partitioning A into (N / NUM_CORES) sized chunks
thread_id[NUM_CORES] = array of thread id’s
thread[NUM_CORES] = array of thread type
//start NUM_CORES threads on working on each chunk of A
for i = 0 to (NUM_CORES - 1) {
thread_id[i] = thread_start(thread[i], MergeSort, chunk[i])
}
//wait for all threads to finish
//Merge chunks appropriately
exit
}
MergeSort ( chunk )
{
MergeSort ( lowerSubChunk )
MergeSort ( higherSubChunk )
Merge(lowerSubChunk, higherSubChunk)
}
//Merge(,) not shown
Method 2:
main ()
{
A = array of length N
NUM_CORES = get number of functional cores
chunk = indices 0 and N
thread_id[NUM_CORES] = array of thread id’s
thread[NUM_CORES] = array of thread type
//lock variable aka mutex
THREADS_IN_USE = 1
MergeSort( chunk )
exit
}
MergeSort ( chunk )
{
lock THREADS_IN_USE
if ( THREADS_IN_USE < NUM_CORES ) {
FREE_CORE = find index of unused core
thread_id[FREE_CORE] = thread_start(thread[FREE_CORE], MergeSort, lowerSubChunk)
THREADS_IN_USE++
unlock THREADS_IN_USE
MergeSort( higherSubChunk )
//wait for thread_id[FREE_CORE] and current thread to finish
lock THREADS_IN_USE
THREADS_IN_USE--
unlock THREADS_IN_USE
Merge(lowerSubChunk, higherSubChunk)
}
else {
unlock THREADS_IN_USE
MergeSort( lowerSubChunk )
MergeSort( higherSubChunk )
Merge(lowerSubChunk, higherSubChunk)
}
}
//Merge(,) not shown
Visually, one can think of the differences between these two methods as follows:
Method 1: creates NUM_CORES separate recursion trees, each one having a single core traversing it.
Method 2: creates a single recursion tree but has all cores traversing it. In particular, whenever there is a free core, it is set to work on the "left child subtree" of the first node where MergeSort is called after the core is freed.
The problem with Method 1 is that if it is the case that the running time of the recursive function varies with the distribution of values within each initial subchunk (i.e. the chunk[i]), one thread could finish much faster leaving a core sitting idle while the others finish. With Merge Sort this is not likely to be the case since the work of MergeSort happens in Merge whose runtime isn't affected much by the distribution of values in the (sorted) subchunks. However, with a more involved recursive function, the running time on one subchunk could be much longer!
With Method 2 it is possible to have the same problem. Again, with merge sort its not clear since the running time for each subchunk is likely to be similar, but the line //wait for thread_id[FREE_CORE] and current thread to finish would also require one core to wait for the other. However, with Method 2, all calls to Merge run ASAP as opposed to Method 1 where one must wait for NUM_CORES calls to MergeSort to finish and then do NUM_CORES - 1 merges afterward (although you can multithread this as well...to an extent)
(though the syntax might not be completely correct)
Are both of these methods used in practice? Are there situations where one is more beneficial over the other? Is this the correct way to implement Method 2? (in this case, THREADS_IN_USE is a semaphore?)
Thanks so much for your help!
Structured programming languages typically have a few control structures, like while, if, for, do, switch, break, and continue that are used to express high-level structures in source code.
However, there are many other control structures that have been proposed over the years that haven't made their way into modern programming languages. For example, in Knuth's paper "Structured Programming with Go To Statements," page 275, he references a control structure that looks like a stripped-down version of exception handling:
loop until event1 or event2 or ... eventN
/* ... */
leave with event1;
/* ... */
repeat;
then event1 -> /* ... code if event1 occurred ... */
event2 -> /* ... code if event2 occurred ... */
/* ... */
eventN -> /* ... code if eventN occurred ... */
fi;
This seems like a useful structure, but I haven't seen any languages that actually implement it beyond as a special case of standard exception handling.
Similarly, Edsger Dijkstra often used a control structure in which one of many pieces of code is executed nondeterministically based on a set of conditions that may be true. You can see this on page 10 of his paper on smoothsort, among other places. Sample code might look like this:
do
/* Either of these may be chosen if x == 5 */
if x <= 5 then y = 5;
if x >= 5 then y = 137;
od;
I understand that historically C influenced many modern languages like C++, C#, and Java, and so many control structures we use today are based on the small set offered by C. However, as evidenced by this other SO question, we programmers like to think about alternative control structures that we'd love to have but aren't supported by many programming languages.
My question is this - are there common languages in use today that support control structures radically different from the C-style control structures I mentioned above? Such a control structure doesn't have to be something that can't be represented using the standard C structures - pretty much anything can be encoded that way - but ideally I'd like an example of something that lets you approach certain programming tasks in a fundamentally different way than the C model allows.
And no, "functional programming" isn't really a control structure.
Since Haskell is lazy, every function call is essentially a control structure.
Pattern-matching in ML-derived languages merges branching, variable binding, and destructuring objects into a single control structure.
Common Lisp's conditions are like exceptions that can be restarted.
Scheme and other languages support continuations which let you pause and resume or restart a program at any point.
Perhaps not "radically different" but "asynchronous" control structures are fairly new.
Async allows non-blocking code to be executed in parallel, with control returning to the main program flow once completed. Although the same could be achieved with nested callbacks, doing anything non-trivial in this way leads to fugly code very quickly.
For example in the upcoming versions of C#/VB, Async allows calling into asynchronous APIs without having to split your code across multiple methods or lambda expressions. I.e. no more callbacks. "await" and "async" keywords enable you to write asynchronous methods that can pause execution without consuming a thread, and then resume later where it left off.
// C#
async Task<int> SumPageSizesAsync(IList<Uri> uris)
{
int total = 0;
var statusText = new TextBox();
foreach (var uri in uris)
{
statusText.Text = string.Format("Found {0} bytes ...", total);
var data = await new WebClient().DownloadDataTaskAsync(uri);
total += data.Length;
}
statusText.Text = string.Format("Found {0} bytes total", total);
return total;
}
(pinched from http://blogs.msdn.com/b/visualstudio/archive/2011/04/13/async-ctp-refresh.aspx)
For Javascript, there's http://tamejs.org/ that allows you to write code like this:
var res1, res2;
await {
doOneThing(defer(res1));
andAnother(defer(res2));
}
thenDoSomethingWith(res1, res2);
C#/Python iterators/generators
def integers():
i = 0
while True:
yield i
i += 1
(I don't know a lot about the subject so I marked this a wiki)
Haskell's Pattern Matching.
Plain example:
sign x | x > 0 = 1
| x == 0 = 0
| x < 0 = -1
or, say, Fibonacci, which looks almost identical to the math equation:
fib x | x < 2 = 1
| x >= 2 = fib (x - 1) + fib (x - 2)
I need as an example how to program a parallel iter-function using ocaml-threads. My first idea was to have a function similiar to this:
let procs = 4 ;;
let rec _part part i lst = match lst with
[] -> ()
| hd::tl ->
let idx = i mod procs in
(* Printf.printf "part idx=%i\n" idx; *)
let accu = part.(idx) in
part.(idx) <- (hd::accu);
_part part (i+1) tl ;;
Then a parallel iter could look like this (here as process-based variant):
let iter f lst = let part = Array.create procs [] in
_part part 0 lst;
let rec _do i =
(* Printf.printf "do idx=%i\n" i; *)
match Unix.fork () with
0 -> (* Code of child *)
if i < procs then
begin
(* Printf.printf "child %i\n" i; *)
List.iter f part.(i)
end
| pid -> (* Code of father *)
(* Printf.printf "father %i\n" i; *)
if i >= procs then ignore (Unix.waitpid [] pid)
else _do (i+1)
in
_do 0 ;;
Because the usage of Thread-module is a little bit different, how would I code this using ocaml's thread module?
And there is another question, the _part() function must scan the whole list to split them into n parts and then each part will be piped through each own processes (here). Still exists there a solution without splitting a list first?
If you have a function which processes a list, and you want to run it on several lists independently, you can call Thread.create with that function and every list. If you store your lists in array part then:
let threads = Array.map (Thread.create (List.iter f)) part in
Array.iter Thread.join threads
INRIA OCaml threads are not actual threads: only one thread executes at any given time, which means if you have four processors and four threads, all four threads will use the same processor and the other three will remain unused.
Where threads are useful is that they still allow asynchronous programming: some Thread module primitives can wait for an external resource to become available. This can reduce the time your software spends blocked by an unavailable resource, because you can have another thread do something else in the mean time. You can also use this to concurrently start several external asynchronous processes (like querying several web servers through HTTP). If you don't have a lot of resource-related blocking, this is not going to help you.
As for your list-splitting question: to access an element of a list, you must traverse all previous elements. While this traversal could theoretically be split across several threads or processes, the communication overhead would likely make it a lot slower than just splitting things ahead of time in one process. Or using arrays.
Answer to a question from the comments. The answer does not quite fit in a comment itself.
There is a lock on the OCaml runtime. The lock is released when an OCaml thread is about to enter a C function that
may block;
may take a long time.
So you can only have one OCaml thread using the heap, but you can sometimes have non-heap-using C functions working in parallel with it.
See for instance the file ocaml-3.12.0/otherlibs/unix/write.c
memmove (iobuf, &Byte(buf, ofs), numbytes); // if we kept the data in the heap
// the GC might move it from
// under our feet.
enter_blocking_section(); // release lock.
// Another OCaml thread may
// start in parallel of this one now.
ret = write(Int_val(fd), iobuf, numbytes);
leave_blocking_section(); // take lock again to continue
// with Ocaml code.