I am working in a boss worker crew multithreaded scenario with Thread::Queue in Perl.
The boss enqueues tasks and the workers dequeue from the queue.
I need to achieve that the worker crew sends downstream ping messages in case the boss does not send a task via the queue for x seconds.
Unfortunately there seems to be no dequeue method with a timeout.
Have I missed something or would you recommend a different approach/different data structure?
You can add the functionality yourself, knowing that a Thread::Queue object is a blessed reference to a shared array (which I believe is the implementation from 5.8 through 5.16):
package Thread::Queue::TimedDequeue;
use parent 'Thread::Queue';
use threads::shared qw(lock cond_timedwait);
sub timed_dequeue {
my ($q, $patience) = #_; # XXX revert to $q->dequeue() if $patience is negative?
# $q->dequeue_nb() if $patience is zero?
my $timelimit = time() + $patience;
lock(#$q);
until (#$q) {
last if !cond_timedwait(#$q, $timelimit);
}
return shift if #$q; # We got an element
# else we timed out.
}
1;
Then you'd do something like:
# main.pl
use threads;
use strict; use warnings;
use Thread::Queue::TimedDequeue;
use constant WORKER_PATIENCE => 10; # 10 seconds
my $queue = Thread::Queue::TimedDequeue->new();
...
sub worker {
my $item = $queue->dequeue(WORKER_PATIENCE);
timedout() unless $item;
...
}
Note that the above approach assumes you do not enqueue undef or an otherwise false value.
There is nothing wrong with your approach/structure, you just need to add some timeout control over your "Thread::Queue". That is either:
create some "yield" based loop to check your queue from the child side while using a time reference to detect timeout.
use the "Thread::Queue::Duplex" or "Thread::Queue::Multiplex" modules which might be a bit overill but do implement timeout controls.
Related
I am trying to build small application with Yew (Rustwasm) . I would like to put sleep function in Yew app.when I use use std::thread::sleep , I am getting below error
I am using sleep as below
let mut index = 0;
sleep(Duration::new(1, 0));
if col < 3 {
index = row * 4 + (col + 1);
if self.cellule[index].value == 1 {
sleep(Duration::new(1, 0));
wasm.js:314 panicked at 'can't sleep', src/libstd/sys/wasm/thread.rs:26:9
Stack:
Error
at imports.wbg.__wbg_new_59cb74e423758ede (http://127.0.0.1:8080/wasm.js:302:19)
at console_error_panic_hook::hook::hd38f373f442d725c (http://127.0.0.1:8080/wasm_bg.wasm:wasm-function[117]:0x16a3e)
at core::ops::function::Fn::call::hf1476807b3d9587d (http://127.0.0.1:8080/wasm_bg.wasm:wasm-function[429]:0x22955)
at std::panicking::rust_panic_with_hook::hb07b303a83b6d242 (http://127.0.0.1:8080/wasm_bg.wasm:wasm-function[211]:0x1ed0d)
at std::panicking::begin_panic::h97f15f2442acdda4 (http://127.0.0.1:8080/wasm_bg.wasm:wasm-function[321]:0x21ee0)
at std::sys::wasm::thread::Thread::sleep::hdd97a2b229644713 (http://127.0.0.1:8080/wasm_bg.wasm:wasm-function[406]:0x22829)
The methods like thread::sleep doesn't work, because in the JS environment you have a single thread only. If you call that sleep you will block the app completely.
If you want to use an interval you should "order" a callback. You can check the following example how to use TimeoutService or IntervalService for that: yew/examples/timer
The core idea to create a service this way:
let handle = TimeoutService::spawn(
Duration::from_secs(3),
self.link.callback(|_| Msg::Done),
);
// Keep the task or timer will be cancelled
self.timeout_job = Some(handle);
Now you can use handler of Msg::Done to react to that timer elapsed.
Threads are actually available, but it's a complex topic and you have to use Web Workers API reach them. Anyway it's useless for your case. Also there are some proposals in standards, but they aren't available in the browsers yet.
How do I achieve Javascript setTimeout functionality in Perl? This is the piece of javascript code I'm trying to write in Perl. Is this possible using threads?
alert("Event 1 occured");
setTimeout(function(){ alert("3 seconds elapsed"); }, 3000);
alert("Event 2 occured");
The output is:
Event 1 occured
Event 2 occured
3 seconds elapsed
I have perl 5.18.2 and I'm on Mac OSX
There's no need for threads, and they're not great anyway in Perl. You can use an event loop just like JavaScript does, there's just not any in the Perl core. Two popular and well supported event loop ecosystems are IO::Async and Mojo::IOLoop (the event loop behind the Mojolicious web framework). The main difference is that unlike in JavaScript, the event loop isn't running until something starts it.
use strict;
use warnings;
use IO::Async::Loop;
print "Event 1 occurred\n";
my $future = IO::Async::Loop->new->delay_future(after => 3)->on_done(sub { print "3 seconds elapsed\n" });
print "Event 2 occurred\n";
$future->await; # run event loop until Future has been resolved
use strict;
use warnings;
use Mojo::IOLoop;
print "Event 1 occurred\n";
Mojo::IOLoop->timer(3 => sub { print "3 seconds elapsed\n" });
print "Event 2 occurred\n";
Mojo::IOLoop->start; # run event loop until no more events to wait for
Check out the Mojolicious cookbook for a very high level overview of event loops and non-blocking code.
See the alarm function. Otherwise, consider looking at threads.pm or , if you want to get feature parity with Javascript, one of the event loops, like AnyEvent or Mojo::IOLoop or IO::Async.
With threads , your example would be:
use strict;
use threads;
print("Event 1 occured\n");
async {
sleep 3;
print "3 seconds elapsed\n";
};
print("Event 2 occured\n");
$_->join for threads->list; # to wait until all threads have finished
I need some help, I can't figure out why my thread doesn't want to start. I don't have experience with perl, and was asked to make a script that will process a file row by row. Depending on the row, the process should execute other functions (not in snippet), call the same function on a new file or call the same function on a new file in parallel (thread).
Below, I pasted a snippet of the actual code (removed the non-relevant code).
I'm testing the multithreading part on a function called "test" which should print "ok".
The process executes correctly, "start" is printed, but then it gets stuck and after a brief delay, the process stops executing completely.
A thousand thanks to whoever may help me!
use strict;
use warnings;
use IO::Prompter;
use Getopt::Long;
use Log::Message::Simple;
use File::Basename;
use File::Spec;
use IO::Socket::INET;
use UUID::Tiny ':std';
use threads;
use threads::shared;
# *bunch of code deleted*
process_file( $cmdline{csvfile}, 1 );
sub test {
print "ok\n";
}
sub process_file {
# get parameters
my ( $input_file, $flowid ) = #_;
# init variables
# open input file
open( my $fh, '<:encoding(UTF-8)', $input_folder . $input_file )
or die "Could not open file '$input_file' $!";
# process file
while ( my $row = <$fh> ) {
chomp $row;
#request = split ";", $row;
$flow_type = $request[0];
$flow = $request[1];
# *bunch of code deleted*
$filename = "$flow.csv";
$keep_flowid = $request[2]; # keep flowid?
$tmp_flowid = $keep_flowid ? $flowid : undef; # set flowid
$thread = $request[3];
if ( $thread == 1 ) {
### Create new thread
print "start\n";
my $process_thread = threads->create("test");
$process_thread->join();
}
elsif ( $thread == 0 ) {
# wait on process to complete
process_file( $filename, $tmp_flowid );
}
# *bunch of code deleted*
}
# close file
close $fh or die "Couldn't close inputfile: $input_file";
}
It's hard to say exactly why you're having this problem - the major possiblity seems to be:
$thread = $request[3];
if ($thread == 1){
This is input from your filehandle, so a real possiblity is that "$request[3]" isn't actually 1.
I am a bit suspicious though - your code as use strict; use warnings at the top, but you're not declaring e.g. $thread, $flow etc. with my. That either means you're not using strict, or you're reusing variables - which is a good way to end up with annoying glitches (like this one).
But as it stands - we can't tell you for sure, because we cannot reproduce the problem to test it. In order to do this, we would need some sample input and a MCVE
To expand on the point about threads made in the comments - you may see warnings that they are "Discouraged". The major reason for this, is because perl threads are not like threads in other languages. They aren't lightweight, where in other languages they are. They're perfectly viable solutions to particular classes of problems - specifically, the ones where you need parallelism with more IPC than a fork based concurrency model would give you.
I suspect you are experiencing this bug, fixed in Perl 5.24.
If so, you could work around it by performing your own decoding rather than using an encoding layer.
I'd like to have Parallel::ForkManager use a callback to get something back from a child process and then also restart it. Is that possible? The following is from the Parallel::ForkManager docs:
use strict;
use Parallel::ForkManager;
my $max_procs = 5;
my #names = qw( Fred Jim Lily Steve Jessica Bob Dave Christine Rico Sara );
# hash to resolve PID's back to child specific information
my $pm = new Parallel::ForkManager($max_procs);
# Setup a callback for when a child finishes up so we can
# get it's exit code
$pm->run_on_finish(
sub { my ($pid, $exit_code, $ident) = #_;
print "** $ident just got out of the pool ".
"with PID $pid and exit code: $exit_code\n";
}
);
$pm->run_on_start(
sub { my ($pid,$ident)=#_;
print "** $ident started, pid: $pid\n";
}
);
$pm->run_on_wait(
sub {
print "** Have to wait for one children ...\n"
},
0.5
);
foreach my $child ( 0 .. $#names ) {
my $pid = $pm->start($names[$child]) and next;
# This code is the child process
print "This is $names[$child], Child number $child\n";
sleep ( 2 * $child );
print "$names[$child], Child $child is about to get out...\n";
sleep 1;
$pm->finish($child); # pass an exit code to finish
#####here is where I'd like each child process to restart
}
So when $pm->finish happens, the callback confirms the "child" is "out of the pool." How can I both get the callback to fire and immediately put the child back in the pool as they come out, so that it runs forever?
I think you're misunderstanding what's happening. Under the covers, what Parallel::ForkManager is doing is calling a fork(). Two processes exist at this point, with only a single difference - different PIDs.
Your child process goes and runs some stuff, then exits, generating an exit status, which your parent then reaps.
Restarting the process... well, you just need to fork again and run your code.
Now, what you're doing - a foreach loop, that - foreach array element, forks and then the fork exits.
So really - all your need to do, is call $pm -> start again. How you figure out which one exited (and thus the child name) is more difficult though - your callback runs in the parent process, so data isn't being passed back aside from the exit status of your child. You'll need to figure out some sort of IPC to notify the necessary details.
Although - I'd point out #names isn't a hash, so treating it like one is going to have strange behaviour :).
Have you considered threading as an alternative? Threads are good for shared memory operations passing keyed subprocesses is something it's good at.
I am trying to take a Perl program I wrote and thread it. The problem is I read that some modules aren't "thread safe". How do I know if a module is thread safe? I've looked around for a list and cannot locate one.
To test out one module I use frequently (Text::CSV_XS) I tried the following code out:
use strict;
use warnings;
use threads;
use threads::shared;
require Text::CSV_XS;
my $CSV = Text::CSV_XS->new ({ binary => 1, eol => "\n" }) or die("Cannot use CSV: ".Text::CSV->error_diag());
open my $OUTPUT , ">:encoding(utf8)", "test.csv" or die("test.csv: $!");
share($CSV);
my $thr1 = threads->create(\&sayHello('1'));
my $thr2 = threads->create(\&sayHello('2'));
my $thr3 = threads->create(\&sayHello('3'));
sub sayHello
{
my($num) = #_;
print("Hello thread number: $num\n");
my #row = ($num);
lock($CSV);{
$CSV->print($OUTPUT, \#row);
$OUTPUT->autoflush(1);
}#lock
}#sayHello
The output I receive is the following:
Hello thread number: 1
Segmentation fault
Does this mean the module is not thread safe, or is it another problem?
Thanks
Generally speaking, core and high-visibility modules are thread-safe unless their documentation says otherwise.
That said, there are a few missteps in your post:
share($CSV)
This clears $CSV (a blessed hashref), just as documented in threads. Generally, you want to share() complex objects prior to initialization or, perhaps in this case, share() some dumb $lock variable between threads.
Since $CSV holds state for the underlying XS, this might lead to undefined behavior.
But this isn't your segfault.
threads->create(\&sayHello('1'));
You are mistakenly invoking sayHello(1) in the main thread and passing a reference to its return value to threads->create() as a (bogus) start routine.
You meant to say:
threads->create(\&sayHello, '1');
But this isn't your segfault.
(EDIT Just to clarify -- a bad start routine here doesn't risk a SEGV in any case. threads::create properly complains if an unrecognized subroutine name or non-CODE ref is passed in. In your case, however, you are segfaulting too quickly to reach this error handling.)
Encodings are not thread-safe.
Again as documented in encodings, the encoding module is not thread-safe.
Here's the smallest possible code I could get to reproduce your symptoms:
use threads;
open my $OUTPUT , ">:encoding(utf8)", "/dev/null" or die $!;
threads->create( sub {} )->join;
That's perl 5.12.1 with threads-1.77 on i686-linux-thread-multi, if you're interested. Drop the "utf8" magic, and it works just fine.
This is your segfault