I have a list like this:
[Header/Element]
[Element]
[Element]
[Header]
[Element]
[Element]
[Element]
[Header]
[Element]
...
[Element/Header]
So this list could or could not have a [Header] in the first position and might not contain also a [Header] element at the end.
I've been assigned to create an algorithm to group this elements under every header, so, the appearance of a header can start a new group with all elements below corresponding to this group. If the first element of the list is not a header (which can be a possibility) then a default group should be used, so all elements until the next header get in this group. The same for elements at the end: there might not be a header that tells you where to end/start a group. So far, not very difficult to do linearly iterating through the entire list.
The real question is, does anyone knows how can do this grouping algorithm but using multiple threads? The reason I want multiple threads is because this list of headers/elements can be very large so I thought that it would be a good idea to have many threads grouping at different parts of the list.
The problem is that I have no idea what could be the procedure to do this and how could I synchronize the threads, specially with the way the list has been layed out (using headers and then X quantity of elements below).
So, have any of you guys have solved a problem like this before? I'm not really interested in some specific implementation in an X programming language, but mostly in the procedure I could use to accomplish this task (and how should I synchronize these threads to prevent overlapping). I'm using C# just in case some of you really want to share some code.
Assuming there are n items in the list, start each thread i at index i*m,
where m = threadCount / n. Or, in simpler terms, split the list into parts and let each thread handle one part.
Now, let each thread read elements and store it in it's own list.
As soon as you read a header, store the elements you have so far (the previous thread will get these list at the end) and start a new list.
From here it's pretty straight-forward - just read the elements and split whenever you get a header.
When you're done, combine the list you're currently busy with with the first list from the next thread.
If a thread starts on a header, the first list will be empty.
If a thread ends on a header, the current list will be empty, so it will simply take the first list from the next thread.
There are some minor details you should look out for, like how you combine the lists at the end, and like knowing when a list is finalized, or whether it will be combined with other lists, but this should be easy enough.
Example:
Input:
A
B
C
Header
D
E
F
Header
With 4 threads, so each thread gets 2 each:
A
B
C
Header
D
E
F
Header
Then:
Thread Processes
1 A
2 C
3 D
4 F
Thread Processes
1 B
2 Header
3 E
4 Header
Here thread 2 will put C into its original list and thread 4 will put F into its original list, and each will start a new list.
Now we're done, so:
Thread 3 will combine its current list ({D,E}) with thread 4's original list ({F}), so thread 3 will end up with {D,E,F}.
Thread 2 will combine its current list ({}) with thread 3's original list (which is also the current list, since we found no header in thread 3 - {D,E,F}), so thread 2 will end up with {D,E,F}.
Thread 1 will combine its current list ({A,B}) with thread 2's original list ({C}), so thread 1 will end up with {A,B,C}.
Related
I am very confused why the concurrent.futures module is giving me different results each time. I have a function, say foo(), which runs on segments of a larger set of data d.
I consistently break this larger data set d into parts and make a list
d_parts = [d1, d2, d3, ...]
Then following the documentation, I do the following
results = [executor.submit(foo, d) for d in d_parts]
which is supposed to give me a list of "futures" objects in the order of foo(d1), foo(d2), and so on.
However, when I try to compile results with
done, _ = concurrent.futures.wait(results)
The list of results stored in done seem to be out of order, i.e. they are not the returns of foo(d1), foo(d2) but some different ordering. Hence, running this program on the same data set multiple times yields different results, as a result of the indeterminacy of which finishes first (the d1, d2... are roughly same size) Is there a reason why, since it seems that wait() should preserve the ordering in which the jobs were submitted?
Thanks!
I have a uniform 2D coordinate grid stored in a numpy array. The values of this array are assigned by a function that looks roughly like the following:
def update_grid(grid):
n, m = grid.shape
for i in range(n):
for j in range(m):
#assignment
Calling this function takes 5-10 seconds for a 100x100 grid, and it needs to be called several hundred times during the execution of my main program. This function is the rate limiting step in my program, so I want to reduce the process time as much as possible.
I believe that the assignment expression inside can be split up in a manner which accommodates multiprocessing. The value at each gridpoint is independent of the others, so the assignments can be split something like this:
def update_grid(grid):
n, m = grid.shape
for i in range (n):
for j in range (m):
p = Process(target=#assignment)
p.start()
So my questions are:
Does the above loop structure ensure each process will only operate
on a single gridpoint? Do I need anything else to allow each
process to write to the same array, even if they're writing to
different placing in that array?
The assignment expression requires a set of parameters. These are
constant, but each process will be reading at the same time. Is this okay?
To explicitly write the code I've structured above, I would need to
define my assignment expression as another function inside of
update_grid, correct?
Is this actually worthwhile?
Thanks in advance.
Edit:
I still haven't figured out how to speed up the assignment, but I was able to avoid the problem by changing my main program. It no longer needs to update the entire grid with each iteration, and instead tracks changes and only updates what has changed. This cut the execution time of my program down from an hour to less than a minute.
I have a main thread where I am getting a list of categories from there I am passing each category from inside the ForEach Controller to next thread. In this thread I get a number of categories inside the variable
In the second thread I am going through each category and using ForEach Controller to pass each product into the 3rd Thread. In this thread I get a number of products inside the variable.
Now I want to use the above number of categories and products into the second and third threads as a dynamic variable i.e if:
categories are 10 then the second thread numbers should be 10
products are 100 then I would like 100/10 = 10 threads for 3 thread group.
I am using the interthread communication processors successfully which works fine withe static number of threads but not when I pass as a variable via interthread communication processors
Please help me
In first Thread Group define the desired number of threads using __setProperty() function like:
${__setProperty(threads,10, )}
In second Thread Group read the value using __P() function like:
${__P(threads,)}
That's it, 2nd Thread Group will kick off as many threads as you define in the first one
More information: Knit One Pearl Two: How to Use Variables in Different Thread Groups
Also be aware that since JMeter 3.1 you should be using JSR223 Test Elements and Groovy language for any form of scripting so convert your Beanshell test elements into JSR223 and make sure to use Groovy
assume we have the following list and there are TWO writers only.
lst = [a, b, c]
One thread will modify items int the list; while another thread will keep appending new items to the end of the list.
Question> Is this thread-safe?
My guess is that this is thread-safe because the list append is atomic and the another thread only works on pre-existing items. Please correct me if I am wrong here.
Problem
Summary: Parallely apply a function F to each element of an array where F is NOT thread safe.
I have a set of elements E to process, lets say a queue of them.
I want to process all these elements in parallel using the same function f( E ).
Now, ideally I could call a map based parallel pattern, but the problem has the following constraints.
Each element contains a pair of 2 objects.( E = (A,B) )
Two elements may share an object. ( E1 = (A1,B1); E2 = (A1, B2) )
The function f cannot process two elements that share an object. so E1 and E2 cannot be processing in parallel.
What is the right way of doing this?
My thoughts are like so,
trivial thought: Keep a set of active As and Bs, and start processing an Element only when no other thread is already using A OR B.
So, when you give the element to a thread you add the As and Bs to the active set.
Pick the first element, if its elements are not in the active set spawn a new thread , otherwise push it to the back of the queue of elements.
Do this till the queue is empty.
Will this cause a deadlock ? Ideally when a processing is over some elements will become available right?
2.-The other thought is to make a graph of these connected objects.
Each node represents an object (A / B) . Each element is an edge connecting A & B, and then somehow process the data such that we know the elements are never overlapping.
Questions
How can we achieve this best?
Is there a standard pattern to do this ?
Is there a problem with these approaches?
Not necessary, but if you could tell the TBB methods to use, that'll be great.
The "best" approach depends on a lot of factors here:
How many elements "E" do you have and how much work is needed for f(E). --> Check if it's really worth it to work the elements in parallel (if you need a lot of locking and don't have much work to do, you'll probably slow down the process by working in parallel)
Is there any possibility to change the design that can make f(E) multi-threading safe?
How many elements "A" and "B" are there? Is there any logic to which elements "E" share specific versions of A and B? --> If you can sort the elements E into separate lists where each A and B only appears in a single list, then you can process these lists parallel without any further locking.
If there are many different A's and B's and you don't share too many of them, you may want to do a trivial approach where you just lock each "A" and "B" when entering and wait until you get the lock.
Whenever you do "lock and wait" with multiple locks it's very important that you always take the locks in the same order (e.g. always A first and B second) because otherwise you may run into deadlocks. This locking order needs to be observed everywhere (a single place in the whole application that uses a different order can cause a deadlock)
Edit: Also if you do "try lock" you need to ensure that the order is always the same. Otherwise you can cause a lifelock:
thread 1 locks A
thread 2 locks B
thread 1 tries to lock B and fails
thread 2 tries to lock A and fails
thread 1 releases lock A
thread 2 releases lock B
Goto 1 and repeat...
Chances that this actually happens "endless" are relatively slim, but it should be avoided anyway
Edit 2: principally I guess I'd just split E(Ax, Bx) into different lists based on Ax (e.g one list for all E's that share the same A). Then process these lists in parallel with locking of "B" (there you can still "TryLock" and continue if the required B is already used.