I am trying to search information if inter process communication can happen with tcl threads. I am biginner on this topic so right now just collecting information. I understand that sender and receiver mechanism to be coded to pass data between processess. And tcl thread package provides send command. Also thread can be used as timer for spawn process inside the same.
Is it possible to recieve data from thread to another thread?
Thanking you.
#contains of test.tcl
puts stdout "hello from wish"
# end of file
# set cmd
set exe {wish85.exe}
set exepath [list $exe test.tcl]
# This next line is slightly magical
set f [open |$exepath r+]
# Use the next line or you'll regret it!
puts $f {fconfigure stdout -buffering line}
fileevent $f readable "getline $f"
proc getline f {
if {[gets $f line]<0} {
close $f ;
return ;
}
puts "line=$line"
}
You need to be much clearer in your mind about what you are looking for. Threads are not processes! With Tcl, every Tcl interpreter context (the thing you make commands and variables in) is bound to a single thread, and every thread is coupled to a single process.
Tcl has a Thread package for managing threads (it should be shipped with any proper distribution of Tcl 8.6) and that provides a mechanism for sending messages between threads, thread::send. Those messages? They're executable scripts, which means that they are really flexible.
For communication between processes, things are much more complicated because you have to consider both discovery of the other processes and security (because processes are a security boundary by design). Here are some of the options:
Tcl is very good at running subprocesses and talking with them via pipes. For example, you can run a subordinate interpreter in just a couple of lines using open:
# This next line is slightly magical
set mypipeline [open |[list [info nameofexecutable]] r+]
# Use the next line or you'll regret it!
puts $mypipeline {fconfigure stdout -buffering line}
It even works well with the fileevent command, so you can do asynchronous processing within each interpreter. (That's really quite uncommon in language runtimes, alas.)
The send command in Tk lets you send scripts to other processes using the same display (I'm not sure if this works on Windows) much as thread::send does with threads in the same process.
The comm package in Tcllib does something very similar, but uses general sockets as a communication fabric.
On Windows, you can use the dde command in Tcl to communicate with other processes. I don't think Tcl registers a DDE server by default, but it's pretty easy to do (provided you are running the event loop, but that's a common requirement for most of the IPC mechanisms to work at their best).
More generally, you can think in terms of running webservices and so on, but that's getting quite complicated!
Related
I have a small sample program that hangs on perl 5.16.3. I am attempting to use an alarm to trigger if two threads don't finish working in time in a much more complicated program, but this boils down the gist of it. I know there's plenty of other ways to do this, but for the sake of argument, let's say I'm stuck with the code the way it is. I'm not sure if this is a bug in perl, or something that legitimately shouldn't work.
I have researched this on the Internet, and it seems like mixing alarms and threads is generally discouraged, but I've seen plenty of examples where people claim that this is a perfectly reasonable thing to do, such as this other SO question, Perl threads with alarm. The code provided in the accepted answer on that question also hangs on my system, which is why I'm wondering if maybe this is something that's now broke, at least as of 5.16.3.
It appears that in the code below, if I call join before the alarm goes off, the alarm never triggers. If I replace the join with while(1){} and go into a busy-wait loop, then the alarm goes off just fine, so it appears that join is blocking the SIGALRM for some reason.
My expectation is that the join happens, and then a few seconds later I see "Alarm!" printed on the screen, but this never happens, so long as that join gets called before the alarm goes off.
#!/usr/bin/env perl
use strict;
use warnings;
use threads;
sub worker {
print "Worker thread started.\n";
while(1){}
}
my $thread = threads->create(\&worker);
print "Setting alarm.\n";
$SIG{ALRM} = sub { print "Alarm!\n" };
alarm 2;
print "Joining.\n";
$thread->join();
The problem has nothing to do with threads. Signals are only processed between Perl ops, and join is written in C, so the signal will only be handled when join returns. The following demonstrates this:
#!/usr/bin/env perl
use strict;
use warnings;
use threads;
sub worker {
print "Worker thread started.\n";
for (1..5) {
sleep(1);
print(".\n");
}
}
my $thread = threads->create(\&worker);
print "Setting alarm.\n";
$SIG{ALRM} = sub { print "Alarm!\n" };
alarm 2;
print "Joining.\n";
$thread->join();
Output:
Setting alarm.
Joining.
Worker thread started.
.
.
.
.
.
Alarm!
join is essentially a call to pthread_join. Unlike other blocking system calls, pthread_join does not get interrupted by signals.
By the way, I renamed $tid to $thread since threads->create returns a thread object, not a thread id.
I'm going to post an answer to my own question to add some detail to ikegami's response above, and summarize our conversation, which should save future visitors from having to read through the huge comment trail it collected.
After discussing things with ikegami, I went and did some more reading on perl signals, consulted some other perl experts, and discovered the exact reason why join isn't being "interrupted" by the interpreter. As ikegami said, signals only get delivered in between perl operations. In perl, this is called Deferred Signals, or Safe Signals.
Deferred Signals were released in 5.8.0, back in 2002, which could be one of the reasons I was seeing older posts on the Net which don't appear to work. They probably worked with "unsafe signals", which act more like signal delivery that we're used to in C. In fact, as of 5.8.1, you can turn off deferred signal delivery by setting the environment variable PERL_SIGNALS=unsafe before executing your script. When I do this, the threads::join call is indeed interrupted as I was expecting, just as pthread_join is interrupted in C in this same scenario.
Unlike other I/O operations, like read, which returns EINTR when a signal interrupts it, threads::join doesn't do this. Under the hood it's a call to the C library call pthread_join, which the man page confirms does not return EINTR. Under deferred signals, when the interpreter gets the SIGALRM, it schedules delivery of the signal, deferring it, until the threads::join->pthread_join library call returns. Since pthread_join doesn't "interrupt" and return EINTR, my SIGALRM is effectively being swallowed by the threads::join. With other I/O operations, they would "interrupt" and return EINTR, giving the perl interpreter a chance to deliver the signal and then restart the system call via SA_RESTART.
Obviously, running in unsafe signals mode is probably a Bad Thing, so as an alternative, according to perlipc, you can use the POSIX module to install a signal handler directly via sigaction. This then makes the one particular signal "unsafe".
I've got an application that needs to send a stream of data from one process to multiple readers, each of which needs to see its own copy of the stream. This is reasonably high-rate (100MB/s is not uncommon), so I'd like to avoid duplication if possible. In my ideal world, linux would have named pipes that supported multiple readers, with a fast path for the common single-reader case.
I'd like something that provides some measure of namespace isolation (eg: broadcasting on 127.0.0.1 is open to any process I believe...). Unix domain sockets don't support broadcast, and UDP is "unreliable" anyways (server will drop packets instead of blocking in my case). I supposed I could create a shared-memory segment and store the common buffers there, but that feels like reinventing the wheel. Is there a canonical way to do this in linux?
I supposed I could create a shared-memory segment and store the common buffers there, but that feels like reinventing the wheel. Is there a canonical way to do this in linux?
The short answer: No
The long answer: Yes [and you're on the right track]
I've had to do this before [for even higher speeds], so I had to research this. The following is what I came up with.
In the main process, create a pool of shared buffers [use SysV shm or private mmap as you chose]. Assign ID numbers to them (e.g. 1,2,3,...). Now there is a mapping from bufid to buffer memory address. To make this accessible to child processes, do this before you fork them. The children also inherit the shared memory mappings, so not much work
Now fork the children. Give them each a unique process id. You can just incrementally start with a number: 2,3,4,... [main is 1] or just use regular pids.
Open up a SysV msg channel (msgget et. al.). Again, if you do this in the main process before the fork, they are available to the children [IIRC].
Now here's how it works:
main finds an unused buffer and fills it. For each child, main sends an IPC message via msgsnd (on the single common IPC channel) where the message payload [mtext] is the bufid number. Each message has the standard header's mtype field set to the destination child's pid.
After doing this, main remembers the buffer as "in flight" and not yet reusable.
Each child does a msgrcv with the mtype set to its pid. It then extracts the bufid from mtext and processes the buffer. When it's done, it sends an IPC message [again on the same channel] with mtype set to main's pid with an mtext of the bufid it just processed.
main's loop does an non-blocking msgrcv, noting all "release" messages for a given bufid. When all children have released the buffer, it's put back on the buffer "free queue". In main's service loop, it may fill new buffers and send more messages as appropriate [intersperse with the waits].
The child then does an msgrcv and the cycle repeats.
So, we're using [large] shared memory buffers and short [a few bytes] bufid descriptor IPC messages.
Okay, so the question you may be asking: "Why SysV IPC for the comm channel?" [vs. multiple pipes or sockets].
You already know that a shared buffer avoids sending multiple copies of your data.
So, that's the way to go. But, why not send the above bufid messages across sockets or pipes [or shared queues, condition variables, mutexes, etc]?
The answer is speed and the wakeup characteristics of the target process.
For a highly realtime response, when main sends out the bufid messages, you want the target process [if it's been sleeping] to wake up immediately and start processing the buffer.
I examined the linux kernel source and the only mechanism that has that characteristic is SysV IPC. All others have a [scheduling] lag.
When process A does msgsnd on a channel that process B has done msgrcv on, three things will happen:
process B will be marked runnable by the scheduler.
[IIRC] B will be moved to the front of its scheduling queue
Also, more importantly, this then causes an immediate reschedule of all processes.
B will start right away [as opposed to next timer interrupt or when some other process just happens to sleep]. On a single core machine, A will be put to sleep and B will run in its stead.
Caveat: All my research was done a few years back before the CFS scheduler, but, I believe the above should still hold. Also, I was using the RT scheduler, which may be a possible option if CFS doesn't work as intended.
UPDATE:
Looking at the POSIX message queue source, I think that the same immediate-wakeup behavior you discussed with the System V queues is going on, which gives the added benefit of POSIX compatibility.
The timing semantics are possible [and desirable] so I wouldn't be surprised. But, SysV is actually more standard and ubiquitous than POSIX mqueues. And, there are some semantic differences [See below].
For timing, you can build a unit test program [just using msgs] with nsec timestamps. I used TSC stamps, but clock_gettime(CLOCK_REALTIME,...) might also work. Stamp departure time and arrival/wakeup time to see. Compare both SysV and mq
With either SysV or mq you may need to bump up the max # of msgs, max msg size, max # of queues via /proc/*. The default values are relatively small. If you don't, you may find tasks blocked waiting for a msg but master can't send one [is blocked] due to a msg queue maximum parameter being exceeded. I actually had such a bug, so I changed my code to bump up these values [it was running as root] during startup. So, you may need to do this as an RC boot script (or whatever the [atrocious ;-)] systemd equivalent is)
I looked at using mq to replace SysV in my own code. It didn't have the same semantics for a many-to-one return-to-free-pool msg. In my original answer, I had forgotten to mention that two msg queues are needed: master-to-children (e.g. work-to-do) and children-to-master (e.g. returning a now available buffer).
I had several different types of buffers (e.g. compressed video, compressed audio, uncompressed video, uncompressed audio) that had varying types and struct descriptors.
Also, multiple different buffer queues as these buffers got passed from thread to thread [different processing stages].
With SysV you can use a single msg queue for multiple buffer lists/queues, the buffer list ID is the msg mtype. A child msgrcv waits with mtype set to the ID value. The master waits on the return-to-free msg queue with mtype of 0.
mq* requires a separate mqd_t for each ID because it doesn't allow a wait on a msg subtype.
msgrcv allows IPC_NOWAIT on each call, but to get the same effect with mq_receive you have to open the queue with O_NONBLOCK or use the timed version. This gets used during the "shutdown" or "restart" phase (e.g. send a msg to children that no more data will arrive and they should terminate [or reconfigure, etc.]). The IPC_NOWAIT is handy for "draining" a queue during program startup [to get rid of stale messages from a prior invocation] or drain stale messages from a prior configuration during operation.
So, instead of just two SysV msg queues to handle an arbitrary number of buffer lists, you'll need a separate mqd_t for each buffer list/type.
I have a shell script background process that runs "nohupped". This process shall receive signals in a trap, but when playing around with some code, I noticed that some signals are ignored if the interval between them is too small. The execution of the trap function takes too much time and therefore the subsequent signal goes
unserved. Unfortunately, the trap command doesn't have some kind of signal queue, that's why I am asking: What is the best way to solve this problem?
A simple example:
function receive_signal()
{
local TIMESTAMP=`date '+%Y%m%d%H%M%S'`
echo "some text" > $TIMESTAMP
}
trap receive_signal USR1
while :
do
sleep 5
done
The easiest change, without redesigning your approach, is to use realtime signals, which queue.
This is not portable. Realtime signals themselves are an optional extension, and shell and utility support for them are not required by the extension in any case. However, it so happens that the relevant GNU utilities on Linux — bash(1) and kill(1) — do support realtime signals in a commonsense way. So, you can say:
trap sahandler RTMIN+1
and, elsewhere:
$ kill RTMIN+1 $pid_of_my_process
Did you consider multiple one line trap statements? One for each signal you want to block or process?
trap dosomething 15
trap segfault SEGV
Also you want to have the least possible code in a signal handler for the reason you just encountered.
Edit - for bash you can code your own error handling / signal handling in C, or anything else using modern signal semantics if you want with dynamically loadable modules:
http://cfajohnson.com/shell/articles/dynamically-loadable/
I'm trying to write a Perl script to capture system log output while a loop runs a system command at intervals. I want the script to do the equivalent of something I often do on the (unix) command line: taking Java process thread dumps by tailing /var/log/jbossas/default/console.log into a new file in the background, while running kill -QUIT [PID] an arbitrary number of times at intervals, in the foreground. I do not need to examine or process the log file output while it's being tailed, I just want it to go to a new file while my loop runs; once the loop exits, the background task should exit too.
# basic loop
# $process is PID given as argument
my $duration = 6;
my $dumps = 0;
until ($dumps == $duration) {
system "kill -QUIT $process";
$dumps++;
print STDOUT "$dumps of $duration thread dumps sent to log.\n";
print STDOUT "sleeping for $frequency seconds\n";
sleep 30;
}
Somehow I need to wrap this loop in another loop that will know when this one exits, and then exit the background log tailing task. I realize that this should be trivial in Perl, but I am not sure of how to proceed, and other questions or examples I've found are not doing quite what I'm trying to do here. Seems like using Perl's system blocks my proceeding into the inner loop; exec forks off the tail job so I'm not sure how I'd exit it after my inner loop runs. I'd strongly prefer to use only core Perl modules, and not File::Tail or any additional CPAN modules.
Thanks in advance for any feedback, and feel free to mock my Perlessness. I've looked for similar questions answered here, but if I've missed one that seems to address my problem, I'd appreciate your linking me to it.
This is probably best suited with an event loop. Read up on the answer to Making a Perl daemon that runs 24/7 and reads from named pipes, that'll give you an intro on reading a filehandle in an event loop. Just open a pipe to the tail output, print it off to the file, run the kill on a timer event, then once the timer events are done just signal an exit.
I'm using the Snack audio processing kit along with TCL.
I want to cut up part of the sound and give this section to another thread to work with.
My question is how to pass something by reference, between threads in TCL.
proc a {} {
snack::sound snd
thread::send -async $Thread [list B snd]
}
set Thread [thead::create {
proc B{snd} {
... do something with snd
}
}
That's not going to work. Tcl threads are designed to be strongly isolated from each other since it massively reduces the amount of locking required for normal processing. The down-side of this is that passing things between threads is non-trivial (other than for short messages containing commands, which audio data isn't!) But there is a way forward…
If you can send the data as a chunk of bytes (at the script level) then I recommend transferring it between threads using the tsv package, which is parceled up with the thread package so you'll already have it. That will let you transport the data between threads relatively simply. Be aware that the snack package is not thread-aware in its script-level interface, so the data transfers are still going to involve copying, and Tk (like a great many GUI toolkits, FWIW) does not support multi-threaded use (well, not without techniques for another time) so if you're doing waveform visualization you've got some work ahead. (OTOH, modern CPUs have loads of time to spare too.)