Perl Multithreading MySQL Data Fetch [closed] - multithreading

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I have several very large tables in mysql (Millions of rows) that I need to load into my perl script.
We then do some custom processing of the data, and aggregate it into a hash. Unfortunately, that custom processing can't be implemented in MySQL.
Heres a quick pseudocode.
my #data;
for my $table_num(#table_numbers){
my $sth = $dbh->prepare(...);
$sth->execute();
$sth->bind_columns(\my($a,$b,$c,...));
while(($sth->fetch()){
$data[$table_num]{black_box($a)}{secret_func($b)}+=$c;
}
}
my $x = $#data + 1;
for my $num (#table_numbers){
for my $a (keys %{$data[$num]}){
for my $b (keys %{$data[$num]{$a}){
$data[$x]{$a}{$b} += $data[$num]{$a}{$b};
}
}
}
Now, the first loop can take several minutes per iteration to run, so I am thinking of ways I can run them in parallel. I have looked at using Perl Threads before, but they seem to just run several perl interpreters at once, and my script is already using a lot of memory, and merging the data would seem t be problematic. Also, at this stage, the script is not using a lot of CPU.
I have been looking at possibly using Coro threads, but it seem like there would be a learning curve, plus a fairly complex integration of my current code. What I would like to know if I am likely to see any gains by going this route. Are there better ways of multithreading code like this. I can not afford to use any more memory then my code already uses. Is there something else I can do here?
Unfortunately doing the aggregation in MySQL is not an option, and rewriting the code in a different language would be too time consuming. I am aware that using arrays instead of hashes is likely to make my code faster/use less memory, but again that would require a major rewrite of a large script.
Edit: The above is pseudo code, the actual logic is a lot more complex. The bucketing is based on several db tables, and many more inputs then just $a and $b. Precomputing them is not practical, as the there are Trillions+ possible combinations. The main goal is how do I make the perl script run faster, not how to fix the SQL Part of things. That requires changes to how the data is stored and indexed in the actual server. Which would affect a lot of other code. There are other people working on doing those optimizations. My current goal is to attempt to make the code faster without changing any sql.

Answer 1

You could do it in mysql simply by making black_box and secret_func tables (temporary tables, if necessary) prepopulated with the results for every existing value of the relevant columns.
Short of that, measure how much time is spent in the calls to black_box and secret_func vs. execute/fetch. If a lot is in the former, you could memoize the results:
my %black_box;
my %secret_func;
for my $table_num...
...
$data[$table_num]{ $black_box{$a} //= black_box($a) }{ $secret_func{$b} //= secret_func($b) } += $c;

Answer 2

If you have memory concerns, using forks instead of threads may help. They use much less memory than the standard perl threads. There is going to be somewhat of a memory penalty for multi-threading, and YMMV as far as performance goes, but you might want to try something like:
use forks;
use Thread::Queue;
my $inQueue = Thread::Queue->new;
my $outQueue = Thread::Queue->new;
$inQueue->enqueue(#table_numbers);
# create the worker threads
my $numThreads = 4;
for(1 .. $numThreads) {
threads->create(\&doMagic);
}
# wait for the threads to finish
$_->join for threads->list;
# collect the data
my #data;
while(my $result = $outQueue->dequeue_nb) {
# merge $result into #data
}
sub doMagic {
while(my $table_num = $inQueue->dequeue_nb) {
my #data;
# your first loop goes here
$outQueue->enqueue(\#data);
}
return;
}