Running self-feeding channels in Perl 6 - multithreading

I would like to set up a number of threads operating concurrently on a channel, and every one of those threads should be also feeding the channel. One of the threads would decide when to stop. However, this is the closest I have come to doing that:
use Algorithm::Evolutionary::Simple;
my $length = 32;
my $supplier = Supplier.new;
my $supply = $supplier.Supply;
my $channel-one = $supply.Channel;
my $pairs-supply = $supply.batch( elems => 2 );
my $channel-two = $pairs-supply.Channel;
my $single = start {
react {
whenever $channel-one -> $item {
say "via Channel 1:", max-ones($item);
}
}
}
my $pairs = start {
react {
whenever $channel-two -> #pair {
my #new-chromosome = crossover( #pair[0], #pair[1] );
say "In Channel 2: ", #new-chromosome;
$supplier.emit( #new-chromosome[0]);
$supplier.emit( #new-chromosome[1]);
}
}
}
await (^10).map: -> $r {
start {
sleep $r/100.0;
$supplier.emit( random-chromosome($length) );
}
}
$supplier.done;
This stops after a number of emissions. And it's probably not running concurrently anyway. I am using channels instead of supplies and taps because these are not run concurrently, but asynchronously. I need supplies because I want to have a seudo-channel that takes the elements in pairs, as it's done above; I haven't seen the way of doing that with pure channels.
There is no difference above if I change the supply's emit to channel's send.
So several questions here
Are these react blocks run in different threads? If not, what would be the way of doing that?
Even if they are not, why does it stop even if $pairs is emitting to the channel all the time?
Could I have "batch" channels created automatically from single-item channels?
Update 1: if I eliminate $supplier.done from the end, it will just block. If I create a promise in whenever, one for each read, it just blocks and does nothing.

The answer is here, stripped down to the minimum necessary
my Channel $c .= new;
my Channel $c2 = $c.Supply.batch( elems => 2).Channel;
my Channel $output .= new;
my $count = 0;
$c.send(1) for ^2;
my $more-work = start react whenever $c2 -> #item {
if ( $count++ < 32 ) {
$c.send( #item[1]);
my $sum = sum #item;
$c.send( $sum );
$output.send( $sum );
} else {
$c.close;
}
}
await $more-work;
loop {
if my $item = $output.poll {
$item.say
} else {
$output.close;
}
if $output.closed { last };
}
A second channel that batches the first channel every two elements is used via the creation of a supply from a channel ($c.Supply), batching that supply in batches of two (batch( elems => 2)) and turning it back into a channel. A third channel is created for output.
In order to not exhaust the supply and hang the channel, every second element that is read from the first (and actually, only) channel is put back there. So the second channel that reads in twos is never hanged or waiting for new elements.
An output channel is created for every new element, and an external counter to finish the operation when it's needed; that output channel is read in a non-blocking way, and closed when there's nothing left to read in the last line.
To answer precisely to my original questions:
Yes, they are, only they are stealing elements from each other.
Because the two threads were reading from the same channel. The first one to stumble into an element, reads it.
Yes, by turning channels into supplies, batching them and turning them back into channels. Only bear in mind that they are not copies, they will be sharing the self same elements.

Related

No Instance Found When Using "always"

I'm trying to visualise a spec for a Payment object where it moves from "queued" to "processing" to "complete". I have come up with the following:
enum State {Queued, Processing, Complete}
sig Payment {
var state: State
}
pred processPayment[p: Payment] {
p.state = Queued // guard
p.state' = Processing // action
}
pred completePayment[p: Payment] {
p.state = Processing // guard
p.state' = Complete // action
}
fact init {
Payment.state = Queued
}
fact next {
always (some p : Payment | processPayment[p] or completePayment[p])
}
run {} for 1 Payment
Unfortunately, I get no instances found for this spec. From my understanding, a Payment with state "Queued" initially and a next state where it's in "Processing" should be allowed with the always (some p : Payment | processPayment[p] or completePayment[p])" formula according to the tutorial at https://haslab.github.io/formal-software-design/overview/index.html. Am I missing something?
The issue turned out to be a missing terminating predicate, adding the below(or a stutter) fixes it.
pred afterComplete[p: Payment] {
p.state = Complete // guard
p.state' = Complete // action
}
I am not an expert but I believe that it is impossible for either of your predicates to be true at t0. I think you need a holdPayment predicate like this:
pred holdPayment[p:Payment] {
p.state = p.state'
}
fact next {
always (some p : Payment | processPayment[p] or completePayment[p] or holdPayment[p])
}
run {} for 1 Payment

How to safely select across channels where some may get concurrently closed?

While answering a question I attempted to implement a setup where the main thread joins the efforts of the CommonPool to execute a number of independent tasks in parallel (this is how java.util.streams operates).
I create as many actors as there are CommonPool threads, plus a channel for the main thread. The actors use rendezvous channels:
val resultChannel = Channel<Double>(UNLIMITED)
val poolComputeChannels = (1..commonPool().parallelism).map {
actor<Task>(CommonPool) {
for (task in channel) {
task.execute().also { resultChannel.send(it) }
}
}
}
val mainComputeChannel = Channel<Task>()
val allComputeChannels = poolComputeChannels + mainComputeChannel
This allows me to distribute the load by using a select expression to find an idle actor for each task:
select {
allComputeChannels.forEach { chan ->
chan.onSend(task) {}
}
}
So I send all the tasks and close the channels:
launch(CommonPool) {
jobs.forEach { task ->
select {
allComputeChannels.forEach { chan ->
chan.onSend(task) {}
}
}
}
allComputeChannels.forEach { it.close() }
}
Now I have to write the code for the main thread. Here I decided to serve both the mainComputeChannel, executing the tasks submitted to the main thread, and the resultChannel, accumulating the individual results into the final sum:
return runBlocking {
var completedCount = 0
var sum = 0.0
while (completedCount < NUM_TASKS) {
select<Unit> {
mainComputeChannel.onReceive { task ->
task.execute().also { resultChannel.send(it) }
}
resultChannel.onReceive { result ->
sum += result
completedCount++
}
}
}
resultChannel.close()
sum
}
This gives rise to the situation where mainComputeChannel may be closed from a CommonPool thread, but the resultChannel still needs serving. If the channel is closed, onReceive will throw an exception and onReceiveOrNull will immediately select with null. Neither option is acceptable. I didn't find a way to avoid registering the mainComputeChannel if it's closed, either. If I use if (!mainComputeChannel.isClosedForReceive), it will not be atomic with the registration call.
This leads me to my question: what would be a good idiom to select over channels where some may get closed by another thread while others are still live?
The kotlinx.coroutines library is currently missing a primitive to make it convenient. The outstanding proposal is to add receiveOrClose function and onReceiveOrClosed clause for select that would make writing code like this possible.
However, you will still have to manually track the fact that your mainComputeChannel was closed and stop selecting on it when it was. So, using a proposed onReceiveOrClosed clause you'll write something like this:
// outside of loop
var mainComputeChannelClosed = false
// inside loop
select<Unit> {
if (!mainComputeChannelClosed) {
mainComputeChannel.onReceiveOrClosed {
if (it.isClosed) mainComputeChannelClosed = true
else { /* do something with it */ }
}
}
// more clauses
}
See https://github.com/Kotlin/kotlinx.coroutines/issues/330 for details.
There are no proposals on the table to further simplify this kind of pattern.

baconjs: throttle consecutive events with criteria

I'm coding a messaging app with Node.js and I need to detect when the same user sends N consecutive messages in a group (to avoid spammers). I'm using a bacon.js Bus where I push the incoming messages from all users.
A message looks like this:
{
"text": "My message",
"user": { "id": 1, name: "Pep" }
}
And this is my working code so far:
const Bacon = require('baconjs')
const bus = new Bacon.Bus();
const CONSECUTIVE_MESSAGES = 5;
bus.slidingWindow(CONSECUTIVE_MESSAGES)
.filter((messages) => {
return messages.length === MAX_CONSECUTIVE_MESSAGES &&
_.uniqBy(messages, 'user.id').length === 1;
})
.onValue((messages) => {
console.log(`User ${__.last(messages).user.id}`);
});
// ... on every message
bus.push(message);
It creates a sliding window, to keep only the number on consecutive messages I want to detect. On every event, it filters the array to let the data flow to the next step only if all the messages in the window belong to the same user. Last, in the onValue, it takes the last message to get the user id.
The code looks quite dirty/complex to me:
The filter doesn't look very natural with streams. Is there a better way to emit an event when N consecutive events match some criteria? .
Is there a better way to receive just a single event with the user (instead of an array of messages) in the onValue function.
It doesn't really throttle. If a user sends N messages in one year, he or she shouldn't be detected. The stream should forget old events somehow.
Any ideas to improve it? I'm open to migrating it to rxjs if that helps.
Maybe start with
latestMsgsP = bus.slidingWindow(CONSECUTIVE_MESSAGES)
.map(msgs => msgs.filter(msg => msgAge(msg) < AGE_LIMIT))
See if we should be blockking someone
let blockedUserIdP = latestMsgsP.map(getUserToBlock)
Where you can use something shamelessly imperative such as
function getUserToBlock(msgs) {
if (msgs.length < CONSECUTIVE_MESSAGES) return
let prevUserId;
for (var i = 0; i < msgs.length; i++) {
let userId = msgs[i].user.id
if (prevUserId && prevUserId != userId) return
prevUserId = userId
}
return prevUserId
}
Consider mapping the property you’re interested in as early as possible, then the rest of the stream can be simpler. Also, equality checks on every item in the sliding window won’t scale well as you increase the threshold. Consider using scan instead, so you simply keep a count which resets when the current and previous values don’t match.
bus
.map('.user.id')
.scan([0], ([n, a], b) => [a === b ? n + 1 : 1, b])
.filter(([n]) => n >= MAX_CONSECUTIVE_MESSAGES)
.onValue(([count, userId]) => void console.log(`User ${userId}`));

Make update loop in sync with music and notes in Phaser

I'm trying to make some notes appearing when the music hits a certain time in Phaser, but when I log the "hit times" in the console, it only show up sometimes.
I have an object of "notes", the key being the time I expect the note to show :
{
1377: {
jam: 1,
duration: 0.40
}
}
with 1464 notes.
But, in the update loop, if I do something like this :
update () {
if (music && music.currentTime) {
if (notes[music.currentTime]) {
console.log('notes[music.currentTime].jam', notes[music.currentTime].jam)
}
}
}
It logs only some of the notes, randomly.
Do you have any idea why ?
This is probably because music.currentTime is incrementing by ~16 ms on each update, so it can skip specific time keys from your notes object. Other than that I believe that the time can also be a floating point value, so it won't exactly match your keys in the notes variable.
An alternate way to implement what you want would be to change format of the notes variable to an array so it could be accessed later in a different manner:
var notes = [
...
{'start': 1377, 'jam': 1, 'duration': 0.40},
{'start': 2456, 'jam': 1, 'duration': 0.30},
...
];
// Index of the first note that will be played.
// Will be incremented by 1 or more with some update() calls,
// depending on the time that passed.
var nextNote = 0;
function update() {
// Process all notes that are now in the past
// compared to the current time of the playing music,
// but skip notes that have been already played before.
while (music && nextNote < notes.length && notes[nextNote].start <= music.currentTime) {
console.log(notes[nextNote]);
nextNote += 1;
}
}
For this method to work, the notes array must hold start times in an ascending order.

Why does a shared perl hash not share updates with other threads?

I'm trying to make a web server whose requests are farmed out to a set of interpreters hidden behind open2(), based on which 'device' is indicated in the cgi parameters.
The trouble is, I want it multi-threaded but the hash I'm using to try to keep track of the event queue relating to each device doesn't remember the new device created for each request: the server below only prints this sort of thing:
Did not find default-device in (alreadyThere)...
Added default-device with Sun Oct 27 20:43:35 2013 to alreadyThere, default-device
Now... does (alreadyThere, default-device) persist for the next request?
Here is the script:
#!/usr/bin/perl -w
use strict;
use threads;
use threads::shared;
use base qw(Net::Server::HTTP);
our $monkeys = shared_clone({ alreadyThere => { 'a' => 'b' } });
sub process_http_request {
require CGI;
my $cgi = CGI->new;
my $device = $cgi->param('device') || 'default-device';
print "HTTP/1.0 200 OK\r\nContent-type: text/html\r\n\r\n<pre>";
unless (exists $monkeys->{$device}) {
print "Did not find $device in (".join(", ", sort keys %$monkeys).")...\n";
lock $monkeys;
unless (exists $monkeys->{$device}) {
my $t = localtime;
$monkeys->{$device} = $t;
print "\nAdded $device with ".$t." to ".join(", ", sort keys %$monkeys);
} else {
print "\nSurprise device... ".$device;
}
} else {
print "\nFound device... ".$device;
}
print "\nNow... does (".join(", ", sort keys %$monkeys).") persist for the next request?</pre>";
}
__PACKAGE__->run(port => 8080);
It's not the $t bit - that was previously shared_clone({ id => $t }), but I'm darned if I can see why $monkeys never seems to update.
The different requests are served by different processes, not threads.
Net::Server doesn't have a multi-threaded "personality"[1], so you're going to have to use a different sharing mechanism.
Notes:
"in the near future, we would like to add a 'Thread' personality"
Building on Ikegami's answer, I'm trying with this additional code to fake a 'threaded' personality with some success (and some problems with 'open3' misbehaving):
sub default_server_type { 'Single' }
sub loop {
my $self = shift;
while( $self->accept ){
async {
$self->run_client_connection;
};
last if $self->done;
}
}
a) Is there any reason to use Net::Server::HTTP instead of the higher level and easier to use Plack?
b) I've had to solve a problem not unlike this one recently, and settled on using event-based httpd with AnyEvent (or higher abstraction, Coro). There's Net::Server::Coro if you need a drop-in replacement for your code, or even a plethora of canned AnyEvent-based httpds like Twiggy, Feersum, etc.

Resources