I'm using qt to make software that analyses a large amount of data. The data consist of individual "Uber" orders with information such as order time, start location, and end location, and I need to be able to evaluate the data such as plotting the graph of demand over time.
To do this, I have to check every record of the data and sum it onto a new data table according to its timestamp, this takes a long time so my initial solution is to use QtConcurrent::filterReduced to get my sum.
However, the filter function cannot take extra arguments to filter the data based on the time interval I want.
My question is, is there another quick and easy solution for this kind of problem? Or do I need to use QThread's low-level API for this, if so, any examples/tutorials on how I can achieve that?
Instead of passing a function, you can pass a function object which holds the "parameter".
Something like this (T is your datatype here):
struct FilterWithTime
{
FilterWithTime(const QString &filterPredicate)
: m_filterPredicate(filterPredicate) { }
typedef bool result_type;
bool operator()(const T &value)
{
... test value against filterPredicate
}
QString m_filterPredicate;
};
QtConcurrent::filterReduced<ResultType>(your-list-of-T, FilterWithTime(QString("10-12"), YourTransformationObject()));
Note the explicit instantiation with ResultType!!
Related
I am working on a multi-threaded module and need to implement map of map in golang - map[outer]map[inner]*some_struct. The outer key(map[outer]) will be accessed by multiple threads(goroutines) to add key to inner map. I have a doubt if multiple threads can concurrently add keys to inner map, for a common outer key - map[outer]. Is it thread safe and is sync.Map a better option ?
Also outer key- map[outer] and total number of outer keys are known at runtime so can't define locks beforehand.
To better understand the problem statement, we can take example of add information about different cities. We can group cities by states. Each thread represents a city. To add info about a city, first thread needs to check outer key - state,(map[state]) and then each thread will simply add info to map[state][city] = &some_struct{x:y,y:z}.
I have read few articles and found out sync.Map is suitable for concurrent map operations and these operations are performed atomically. But in documentation one of the use-case mentioned was - when multiple goroutines read, write, and overwrite entries for disjoint sets of keys.
It will be helpful if someone can suggest thread-safe approach for this problem statement.
You must thing in OO terms
What do you want to represent as map of map?
Map state, city make some sense. However what kind of operations do you want to do?
Write and Read, concurrent? Why?
Do you want to iterate over all cities? Do you need to delete cities/states?
Imagine the following interface
type DB interface {
Exists(state, city string) bool
Get(state, city string) *some_struct
Set(state, city string, data *some_struct)
Delete(state, city string)
DeleteState(state string)
ForeachCitiesInState(state string, func(city string, data *some_struct) bool)
Foreach(func(state, city…))
}
With this interface we can consider:
use a struct with a Mutex and map of maps to control the access on each read/write/delete
same as 1 but with Read Write Mutex if you have more reads than writes
if you don’t need loop over cities on a particular state, perhaps
you can create a map[ composite key ] struct like state:city to
simplify.
If you will load it from another place with a constant time interval, perhaps you should use atomic.Value to store the big map. Update is just a substitution for a more recent map.
Perhaps you can combine several rw locks. For instance one for state and another for city. You can split like
type states struct {
sync.Mutex
map[ stateName ]state
}
type state struct {
sync.Mutex
map[ cityFirstLetter ]cities
}
type cities struct {
sync.Mutex
map[ cityName ] *some_struct
}
Ideas:
Define the interface
Define (or measure) the real scenario of usage
Write benchmarks
Be careful by return a pointer to data. You can change the internal state. Consider return a copy or an interface
Hey how to get big amount of information like 1000 rows without stuck?
I try with this:
dispatch_async(dispatch_get_main_queue(), {
//here code
})
but when I executed the request self.context.executeFetchRequest it returns me fatal error: unexpectedly found nil while unwrapping an Optional value. I have an error and I have to add self. in front of the function.
let queue:dispatch_queue_t = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)
dispatch_async(queue, { () -> Void in
//code
})
but also I get the same error...
I use NSFetchRequest and I add the results in NSArray and I loop the results in for loop and in the loop I sort results in a dictionaries.
1000 records is not very much for Core Data. Just fetch them on the thread.
I would not advise to "sort results in a dictionaries". You should think how your app logic interacts with the data and simply fetch the objects you need from the Core Data persistent store.
For example, if you want to display 1000 lines in a table view, use `NSFetchedResultsController´ which is optimized for this sort of situation - so you will avoid memory and performance issues without any work.
If you really need threading with Core Data (which I doubt) I would advise not to start with GCD but to use Core Data's own concurrency APIs, such as performBlock and global queue child contexts. But most likely you won't have to worry about those.
Finally, your error is really referring to some code that you have not posted. It has to do with Swift's optionals. For example, if you declare a variable as type variable : String? (or you use an API that returns such a type), you can unwrap it with variable! if you are sure it is not nil . If it is nil you will get the above crash.
Is it possible to create a simple HashMap in OpenCL? E.g. one where all keys have type long and all values type int, and that never has to be modified (i.e. is passed read-only to the kernel).
Construction of the HashMap can take time (is it done once on the CPU and never has to be modified again), but read-access will be frequent, so get(long key, *hashmap H) should be cheap.
Are there any known implementations for this in OpenCL? I failed to find them. In case I'd have to write one from scratch, which HashMap implementation would be most suitable for this use?
I think that a simple hash table implementation using open addressing could fulfill your requirements here:
By its nature it is stored on a single buffer, and thus trivial to transfer to the kernels.
It's then easy to write the getter logic in the kernel, especially when you don't need any synchronization (read-only).
So, pass a buffer of long2 or a buffer of struct { long key; int val; }, when the first item is the key and the second the value, and also pass the buffer size; now write a regular open-address getter.
My question is in regards to a Java project I am currently working on. I have to build a simulation to show the workings on various different sorting algorithms conceptually/visually. These include: bubble sort, insertion sort, merge sort and some others. This must allow the user to either step through each step of the process or select a speed they wish it to execute at. This must be done with the use of a thread. The sorting algorithms are all inside one class and whichever one the user chooses from the from end will run within the thread.
e.g. sort.bubbleSort(objects); < objects being an array of objects to be sorted.
My problem is that I don't know how to manipulate the speed of the thread. For step by step I will need it to pause after each line of the algorithm is executed for example. This project also implements MVC. The thread is in the Contoller package and the sorting class is in Model. Anyone that can help me with this it would be much appreciated.
I don't have any magic bullet type solution. I think you are going to have to put Thread.sleep(...) calls in your code in various places.
public class BubbleSort {
private long sleepBetweenIterationsMillis;
public BubbleSort(long sleepBetweenIterationsMillis) {
this.sleepBetweenIterationsMillis = sleepBetweenIterationsMillis;
}
...
// iterate through the list bringing the highest value to the top
// wait a certain number of millis
Thread.sleep(sleepBetweenIterationsMillis);
// loop
...
}
Picking the points in your sorting algorithms to put those sleep calls depends on what you consider an "iteration" to be. Instead of injecting a sleep value, you could instead make a call to some sleep manager which could change the sleep values dynamically or something based on user input.
public interface SleepManager {
public void sleep();
}
public class BubbleSort {
private SleepManager sleepManager;
public BubbleSort(SleepManager sleepManager) {
this.sleepManager = sleepManager;
}
...
// iterate through the list bringing the highest value to the top
// call the manager which can dynamically slow or speed up the iterations
sleepManager.sleep();
// loop
...
}
I can't comment on the MVC questions. You are going to have to write another more specific question around what you have tried and what you want to accomplish.
Problem
I have such code
var ls = src.iter.toList
src.iter = ls.iterator
(this is part of copy constructor of my iterator-wrapper) which reads the source iterator, and in next line set it back. The problem is, those two lines have to be atomic (especially if you consider that I change the source of copy constructor -- I don't like it, but well...).
I've read about Actors but I don't see how they fit here -- they look more like a mechanism for asynchronous execution. I've read about Java solutions and using them in Scala, for example: http://naedyr.blogspot.com/2011/03/atomic-scala.html
My question is: what is the most Scala way to make some operations atomic? I don't want to use some heavy artillery for this, and also I would not like to use some external resources. In other words -- something that looks and feels "right".
I kind like the solution presented in the above link, because this is what I exactly do -- exchange references. And if I understand correctly, I would guard only those 2 lines, and other code does not have to be altered! But I will wait for definitive answer.
Background
Because every Nth question, instead of answer I read "but why do you use...", here:
How to copy iterator in Scala? :-)
I need to copy iterator (make a fork) and such solution is the most "right" I read about. The problem is, it destroys the original iterator.
Solutions
Locks
For example here:
http://www.ibm.com/developerworks/java/library/j-scala02049/index.html
The only problem I see here, that I have to put lock on those two lines, and every other usage on iter. It is minor thing now, but when I add some code, it is easy to forget to add additional lock.
I am not saying "no", but I have no experience, so I would like to get answer from someone who is familiar with Scala, to point a direction -- which solution is the best for such task, and in long-run.
Immutable iterator
While I appreciate the explanation by Paradigmatic, I don't see how such approach fits my problem. The thing is IteratorWrapper class has to wrap iterator -- i.e. raw iterator should be hidden within the class (usually it is done by making it private). Such methods as hasNext() and next() should be wrapped as well. Normally next() alters the state of the object (iterator) so in case of immutable IteratorWrapper it should return both new IteratorWrapper and status of next() (successful or not). Another solution would be returning NULL if raw next() fails, anyway, this makes using such IteratorWrapper not very handy.
Worse, there is still not easy way to copy such IteratorWrapper.
So either I miss something, or actually classic approach with making piece of code atomic is cleaner. Because all the burden is contained inside the class, and the user does not have to pay the price of they way IteratorWrapper handles the data (raw iterator in this case).
Scala approach is to favor immutability whenever it is possible (and it's very often possible). Then you do not need anymore copy constructors, locks, mutex, etc.
For example, you can convert the iterator to a List at object construction. Since lists are immutable, you can safely share them without having to lock:
class IteratorWrapper[A]( iter: Iterator[A] ) {
val list = iter.toList
def iteratorCopy = list.iterator
}
Here, the IteratorWrapper is also immutable. You can safely pass it around. But if you really need to change the wrapped iterator, you will need more demanding approaches. For instance you could:
Use locks
Transform the wrapper into an Actor
Use STM (akka or other implementations).
Clarifications: I lack information on your problem constraints. But here is how I understand it.
Several threads must traverse simultaneously an Iterator. A possible approach is to copy it before passing the reference to the threads. However, Scala practice aims at sharing immutable objects that do not need to be copied.
With the copy strategy, you would write something like:
//A single iterator producer
class Producer {
val iterator: Iterator[Foo] = produceIterator(...)
}
//Several consumers, living on different threads
class Consumer( p: Producer ) {
def consumeIterator = {
val iteratorCopy = copy( p.iterator ) //BROKEN !!!
while( iteratorCopy.hasNext ) {
doSomething( iteratorCopy.next )
}
}
}
However, it is difficult (or slow) to implement a copy method which is thread-safe. A possible solution using immutability will be:
class Producer {
val lst: List[Foo] = produceIterator(...).toList
def iteratorCopy = list.iterator
}
class Consumer( p: Producer ) {
def consumeIterator = {
val iteratorCopy = p.iteratorCopy
while( iteratorCopy.hasNext ) {
doSomething( iteratorCopy.next )
}
}
}
The producer will call produceIterator once at construction. It it immutable because its state is only a list which is also immutable. The iteratorCopy is also thread-safe, because the list is not modified when creating the copy (so several thread can traverse it simultaneously without having to lock).
Note that calling list.iterator does not traverse the list. So it will not decrease performances in any way (as opposed to really copying the iterator each time).