Perl6 multithreaded spurt problems - multithreading

Question

I write program which generates "random" text files where 3 words substituted with keys stored in $keysfilename file. Keys file has very simple structure, like
ASD123ASD
QWE123QWE
XZC123ZXC
Problem happens when I use more than one thread, for example
my #threads = (^32).map({
If fails on arbitrary file, with error
started
Thread<17>(14) got 1
Thread<18>(15) got 2
Thread<20>(17) got 17
Thread<5>(2) got 3
Thread<16>(13) got 4
Thread<21>(18) got 5
Thread<3>(0) got 6
Thread<8>(5) got 7
Thread<12>(9) got 10
Thread<11>(8) got 8
Thread<9>(6) got 9
Thread<14>(11) got 11
Thread<15>(12) got 12
Unhandled exception: Failed to open file C:\c\perltests\00000017.txt: no such file or directory
Thread<10>(7) got 13
Thread<13>(10) got 14
Thread<7>(4) got 15
Thread<19>(16) got 16
Thread<4>(1) got 0
Thread<6>(3) got 18
Thread<22>(19) got 19
at <unknown>:1 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:4294967295)
Thread<23>(20) got 20
Thread<24>(21) got 21
from gen/moar/m-CORE.setting:22337 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:34)
Thread<26>(23) got 22
Thread<25>(22) got 23
Thread<27>(24) got 24
from gen/moar/m-CORE.setting:22374 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:FALLBACK:35)
Thread<28>(25) got 25
Thread<29>(26) got 26
from gen/moar/m-Metamodel.nqp:3041 (C:\rakudo\share\nqp\lib/Perl6/Metamodel.moarvm::13)
Thread<30>(27) got 27
Thread<16>(13) got 28
Thread<17>(14) got 29
Thread<5>(2) got 30
Thread<18>(15) got 31
Thread<14>(11) got 32
Thread<15>(12) got 33
from gen/moar/m-CORE.setting:25189 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:226)
Thread<30>(27) got 58
Thread<29>(26) got 57
Unhandled exception: Failed to open file C:\c\perltests\00000028.txt: no such file or directory
Thread<28>(25) got 56
from gen/moar/m-CORE.setting:25203 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:92)
Thread<25>(22) got 55
from gen/moar/m-CORE.setting:25199 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:44)
Thread<27>(24) got 54
from gen/moar/m-CORE.setting:25506 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:96)
Thread<26>(23) got 53
from gentexts.pl:54 (<ephemeral file>::189)
Thread<24>(21) got 52
Thread<23>(20) got 51
Unhandled exception: Failed to open file C:\c\perltests\00000058.txt: no such file or directory
Thread<6>(3) got 50
Thread<22>(19) got 49
at <unknown>:1 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:4294967295)
Thread<34>(31) got 48
from gen/moar/m-CORE.setting:22337 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:34)
Thread<33>(30) got 47
Thread<4>(1) got 46
from gen/moar/m-CORE.setting:22374 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:FALLBACK:35)
Thread<7>(4) got 45
from gen/moar/m-Metamodel.nqp:3041 (C:\rakudo\share\nqp\lib/Perl6/Metamodel.moarvm::13)
Thread<19>(16) got 44
Thread<11>(8) got 43
at <unknown>:1 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:4294967295)
Thread<8>(5) got 42
from gen/moar/m-CORE.setting:22337 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:throw:34)
Thread<12>(9) got 41
Thread<10>(7) got 40
from gen/moar/m-CORE.setting:25189 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:226)
Thread<13>(10) got 39
from gen/moar/m-CORE.setting:25203 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:92)
Thread<31>(28) got 37
from gen/moar/m-CORE.setting:25199 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:44)
Thread<32>(29) got 38
Thread<9>(6) got 36
from gentexts.pl:44 (<ephemeral file>::15)
Thread<3>(0) got 35
Thread<21>(18) got 34
from gen/moar/m-CORE.setting:22374 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:FALLBACK:35)
Thread<17>(14) got 59
from gen/moar/m-CORE.setting:25506 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:spurt:96)
Thread<5>(2) got 60
Thread<18>(15) got 61
from gentexts.pl:54 (<ephemeral file>::189)
Thread<9>(6) got 85
Thread<32>(29) got 84
from gen/moar/m-CORE.setting:30638 (C:\rakudo/share/perl6/runtime/CORE.setting.moarvm:THREAD-ENTRY:44)
System Win 10 x32, Rakudo 6.c
my $keysfilename := 'C:/c/keysfile.txt';
my $output := 'C:/c/perltests';
my #keys = ();
for $keysfilename.IO.words {
#keys.push($_);
}
my $len := elems #keys;
my $lorem = q:to/END/;
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec malesuada purus vel tincidunt eleifend. Fusce sollicitudin augue augue, et gravida dolor varius a. Vestibulum iaculis, dui iaculis iaculis molestie, tellus ante hendrerit massa, at volutpat risus metus vitae nisi. Integer neque magna, ultrices eu erat at, efficitur sollicitudin sem. Aliquam sed purus malesuada, porta est eu, rutrum neque. Quisque dolor leo, condimentum non mollis eget, tristique eget odio. Donec dignissim magna nec imperdiet iaculis. Vestibulum lorem ligula, euismod ac porttitor faucibus, rutrum eu ex.
Donec scelerisque nisi eget risus condimentum ultrices. Integer porta maximus quam, in lobortis quam fermentum eu. Morbi eu ligula consequat, aliquam sem eget, sollicitudin eros. Suspendisse potenti. Cras finibus metus et eros accumsan, id vehicula libero lobortis. Aenean vulputate lacinia urna at fringilla. Nulla id tincidunt lectus, quis accumsan lorem. In posuere magna non purus hendrerit rutrum. Maecenas in mattis tellus. Maecenas vel feugiat enim. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin convallis dapibus tellus vitae euismod. Nam eleifend dui quam, eget lobortis quam pulvinar id. Cras euismod posuere dolor non ultricies.
Ut dapibus porta faucibus. Duis velit ante, tincidunt id velit id, imperdiet egestas velit. Morbi efficitur enim dignissim interdum egestas. Vivamus eu urna condimentum, aliquam orci non, ullamcorper est. Phasellus egestas at tellus nec tristique. Fusce feugiat commodo faucibus. In hac habitasse platea dictumst. Quisque dignissim, mauris a pellentesque dictum, mauris velit tincidunt lorem, sed tincidunt libero enim vitae orci. Nam interdum, ante nec consequat vulputate, nisi turpis euismod nibh, sit amet elementum nunc diam non eros. Proin quis viverra risus. Vestibulum vestibulum diam in velit consectetur, eu elementum lacus sagittis. Morbi accumsan ac ante eget faucibus. In nec elit bibendum, tristique enim non, sodales ex. Donec sodales erat vitae odio cursus commodo.
Vestibulum felis lacus, mattis eget porta eget, mattis ut felis. Pellentesque aliquet, purus eu semper suscipit, sem ipsum euismod nunc, sed dapibus augue sem vel elit. Etiam tincidunt arcu ut nisi tempor commodo. Mauris at eros tincidunt, fringilla erat nec, sagittis ante. Integer et malesuada quam. Cras vel porta erat, sit amet efficitur erat. Praesent blandit purus quis urna consectetur, eget ultricies ipsum pulvinar. Phasellus ac molestie elit. Vestibulum et tincidunt velit. Aliquam a venenatis ipsum, nec commodo libero. Nullam eget consectetur lectus. Morbi placerat interdum erat nec interdum.
Morbi bibendum dui eu turpis pretium, eget aliquet augue aliquam. Aliquam eu dignissim mauris, vitae placerat augue. Ut sed tortor sit amet augue imperdiet rutrum. Aliquam erat volutpat. Morbi a turpis in sapien ultrices tristique. Proin quis vestibulum lorem, ut pharetra ex. Quisque tempor bibendum purus ac vehicula. Suspendisse tellus ipsum, imperdiet id sodales vel, congue a leo. Nulla gravida tincidunt nisi eu tempor. Mauris imperdiet tempor ante eget rutrum. Nam ut dui at augue laoreet mollis. Sed metus elit, viverra ac fringilla vel, fermentum et magna. Nam ligula purus, pretium vel dignissim vitae, fermentum at urna. Nullam ac ullamcorper felis. Maecenas dapibus consequat mi.
END
my #words = $lorem.split(' ');
my $wordlen = #words.elems;
my &getNext = sub {
my $counter = 0;
my Lock $lock .= new;
return sub (#filename) {
$lock.lock;
if ($counter < 100_000) {
#filename[0] = $counter;
$counter++;
}
$lock.unlock;
};
}();
say "started";
my #threads = (^1).map({
Thread.start(
name => $_,
sub {
loop {
my #counter = (-1);
getNext(#counter);
last if #counter[0] < 0;
say $*THREAD ~ " got " ~ #counter[0];
my #copy = #words.clone;
for (^3) {
my $pos = $wordlen.rand.round;
#copy[$pos] = #keys[$len.rand.round];
}
spurt sprintf($output ~ '/%08d.txt', #counter[0]), #copy.join(' ');
}
}
);
}).join;

Answer 1

A few notes.
This:
my #keys = ();
for $keysfilename.IO.words {
#keys.push($_);
}
is really just this: my #keys = $keysfilename.IO.words;
and this: #keys[$len.rand.round]
is really just this:#keys.pick
I'm not quite sure what you're doing with the counter array, it seems
like you just want files 0..99999.
You are making locks inside your subroutine for each thread. If you
want to keep multiple threads from accessing something, make one lock
outside and have each one of them use it to lock the exclusive code.
You're also using Thread to make threads, but that is really a
pretty low level interface. Try to use the higher level constructs
before falling to that level if possible.
In this case, it seems like you just want to split up the 100,000 file
creations into multiple threads just to get some parallelism. There
are a few constructs that make that really easy, hyper and
race. Since you don't even really care what order the files are
created in, you can just use race and it will churn them out as fast
as possible.
You can use the 'degree' parameter to race to say how many
threads you want to use. (You might need to set RAKUDO_MAX_THREADS to
get the behavior you want).
my $keysfilename := 'C:/c/keysfile.txt';
my $output := 'C:/c/perltests';
my #keys = $keysfilename.IO.words;
my $lorem = q:to/END/;
Lorem ipsum ...
END
my #words = $lorem.split(' ');
my $wordlen = #words.elems;
[^100000].race(degree => 32).map({
# say $*THREAD ~ " got " ~ $_;
my #copy = #words;
for (^3) {
my $pos = $wordlen.rand.round;
#copy[$pos] = #keys.pick;
}
spurt sprintf($output ~ '/%08d.txt', $_), #copy.join(' ');
});

Answer 2

There is almost never a reason to use Threads directly in Perl 6, there are plenty of features that do the hard work for you. In most cases the simplest improvement is to just use start instead of Thread.start, and await LIST instead of LIST».join or LIST.map(*.join).
You were calling List.join not Thread.join, which may be at least partially why you were having problems.
There are many more things that make this difficult to read and understand.
by how you wrote getNext it seems like you know only or mostly only JavaScript. This is how I would write it.
sub get-next (#filename) {
state Int $counter = 0; # set to zero only the first time it is called
state Lock $lock .= new;
$lock.protect: ->{
if $counter < 100_000 {
#filename[0] = $counter++;
}
}
}
except that you are only using #filename to bypass the default readonly semantics of parameters.
sub get-next ($filename is rw) {
state Int $counter = 0;
state Lock $lock .= new;
$lock.protect: ->{
if $counter < 100_000 {
$filename = $counter++;
}
}
}
except that it would make more sense to just return the value.
sub get-next () {
state Int $counter = 0;
state Lock $lock .= new;
$lock.protect: ->{
$counter++ if $counter < 100_000
} // Nil
# the `if` will return `Empty` when $counter gets too big
# but we want `Nil` instead
}
This either returns the counter or Nil which is an undefined value so you can use it like this:
loop {
my $counter = get-next() orelse last;
…
}
Since you only ever use the sub in a loop, it could just call last itself.
Or better yet if you instead changed the line that declared the counter to
state Int $counter = 0 but True;
Then you could change the loop to a while loop, with no need to add last anywhere.
while get-next() -> $counter { … }
Now that I showed you how you can improve getNext, I'm going to throw it away and use much nicer features of Perl 6.
(It would have made more sense to use a Channel by the way)
# will probably still work with `use v6.c;`
# but v6.d has a better system backing `await`
use v6.d.PREVIEW;
# override the original default number of threads
# (16 threads currently)
BEGIN %*ENV<RAKUDO_MAX_THREADS> //= 32;
# the "correct" way to do this is setting $*SCHEDULER
# but this is easier
my $keys-filename = 'C:/c/keysfile.txt';
my $output-dir = 'C:/c/perltests';
my #keys = $keys-filename.IO.words;
my #lorem = q:to/EOF/.split(' ');
…
EOF
say 'started';
END say 'finished';
for race ^100_000 -> $counter {
say $*THREAD, " got ", $counter;
my #copy = #lorem; # no need for .clone
for (^+#copy).pick(3) -> $pos {
#copy[$pos] = #keys.roll;
}
spurt $*SPEC.catfile($output-dir, $counter.fmt('/%08d.txt')), #copy.join(' ');
}
( I ran a test, and it never gave a Thread id higher than 6 )
I used race rather than hyper because the return values are thrown away anyway.
If it didn't work, you have a buggy version of Rakudo. If so I would recommend updating to the latest release v2017.02
(say $*PERL.compiler.version;)
+#array is the same as #array.elems
^ NUMBER is the same as 0 ..^ NUMBER which is sugar for
Range.new( 0, NUMBER, :excludes-max )
This still has the problem that some of the “words” have extra characters.
amet, elit. and ex.␤␤Donec for example
…
my #pos = ($lorem ~~ m:ex/ « \w+: /).map: { .from, .chars }
# #pos looks something like [(0,5),(6,5),(12,5),(18,3),…]
…
for race ^100_000 -> $counter {
say $*THREAD, " got ", $counter;
# vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
my $copy = $lorem;
# sort so that the transforms are done from the end of the string
# towards the beginning of the string
for #pos.pick(3).sort.reverse {
$copy.substr-rw( |$_ ) = #keys.roll;
}
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
spurt $*SPEC.catfile($output-dir, $counter.fmt('/%08d.txt')), $copy;
}