A multi-threaded echo server - multithreading

Which "thread" does this error relate to?
thufir#tleilax:~/wunderground$
thufir#tleilax:~/wunderground$ tclsh 21.12.tcl
can't read "thread": no such variable
while executing
"thread::send -async $thread {
thread::attach $sock
fconfigure $sock -buffering line -blocking 0
fileevent $sock readable [list ReadLine $sock]
SendMes..."
(file "21.12.tcl" line 71)
thufir#tleilax:~/wunderground$
example code 21-12:
package require Tcl 8.4
package require Thread 2.5
set sock 12345
set host 127.0.0.1
#set port 7777
if {$argc > 0} {
set port [lindex $argv 0]
} else {
set port 9001
}
socket -server _ClientConnect $port
proc _ClientConnect {sock host port} {
}
#Tcl holds a reference to the client socket during
#this callback, so we can't transfer the channel to our
#worker thread immediately. Instead, we'll schedule an
#after event to create the worker thread and transfer
#the channel once we've re-entered the event loop.
after 0 [list ClientConnect $sock $host $port]
proc ClientConnect {sock host port} {
#Create a separate thread to manage this client. The
#thread initialization script defines all of the client
#communication procedures and puts the thread in its
#event loop.
set thread [thread::create {
proc ReadLine {sock} {
if {[catch {gets $sock line} len] || [eof $sock]} {
catch {close $sock}
thread::release
} elseif {$len >= 0} {
EchoLine $sock $line
}
}
proc EchoLine {sock line} {
if {[string equal -nocase $line quit]} {
SendMessage $sock \
"Closing connection to Echo server"
catch {close $sock}
thread::release
} else {
SendMessage $sock $line
}
}
proc SendMessage {sock msg} {
if {[catch {puts $sock $msg} error]} {
puts stderr "Error writing to socket: $error"
catch {close $sock}
thread::release
}
}
# Enter the event loop
thread::wait
}]
#Release the channel from the main thread. We use
#thread::detach/thread::attach in this case to prevent
#blocking thread::transfer and synchronous thread::send
#commands from blocking our listening socket thread.
# Copy the value of the socket ID into the
# client's thread
thread::send -async $thread [list set sock $sock]
# Attach the communication socket to the client-servicing
# thread, and finish the socket setup.
}
thread::send -async $thread {
thread::attach $sock
fconfigure $sock -buffering line -blocking 0
fileevent $sock readable [list ReadLine $sock]
SendMessage $sock "Connected to Echo server"
}
vwait forever
from:
This Chapter is from Practical Programming in Tcl and Tk, 4th Ed.
Copyright 2003 © Brent Welch, Ken Jones
http://www.beedub.com/book/
I had to add the sock and and host variables which makes me think that I'm missing more. The name "thread" for a thread variable seems questionable (?).

You've managed to chew the ordering of the lines in the book just enough to make the code really not work. If we look at the relevant chapter, we see that the code actually says this things like this:
proc _ClientConnect {sock host port} {
# Tcl holds a reference to the client socket during
# this callback, so we can't transfer the channel to our
# worker thread immediately. Instead, we'll schedule an
# after event to create the worker thread and transfer
# the channel once we've re-entered the event loop.
after 0 [list ClientConnect $sock $host $port]
}
That's much less strange! Similarly, you've moved this block:
thread::send -async $thread {
thread::attach $sock
fconfigure $sock -buffering line -blocking 0
fileevent $sock readable [list ReadLine $sock]
SendMessage $sock "Connected to Echo server"
}
outside the ClientConnect procedure, despite it needing to run inside there because that's where the thread variable is defined. (This is why you are getting that error message, BTW.) If you fix these things (and check whether you've done other silly mistakes in the same vein), the code should work.
If you're going to copy examples from elsewhere, please copy them correctly. Randomly changing the context in which code fragments run is likely to not work well in any programming language…

There's something odd about copy/paste from at least this PDF, it literally puts lines in the wrong order. I ran it through an online PDF to text converter with the same result.
Badly indented code to follow (sans comments, which don't copy well):
package require Tcl 8.4
package require Thread 2.5
if {$argc > 0} {
set port [lindex $argv 0]
} else {
set port 9001
}
socket -server _ClientConnect $port
proc _ClientConnect {sock host port} {
after 0 [list ClientConnect $sock $host $port]
}
proc ClientConnect {sock host port} {
set thread [thread::create {
proc ReadLine {sock} {
if {[catch {gets $sock line} len] || [eof $sock]} {
catch {close $sock}
thread::release
} elseif {$len >= 0} {
EchoLine $sock $line
}
}
proc EchoLine {sock line} {
if {[string equal -nocase $line quit]} {
SendMessage $sock \
"Closing connection to Echo server"
catch {close $sock}
thread::release
} else {
SendMessage $sock $line
}
}
proc SendMessage {sock msg} {
if {[catch {puts $sock $msg} error]} {
puts stderr "Error writing to socket: $error"
catch {close $sock}
thread::release
}
}
thread::wait
}]
thread::detach $sock
thread::send -async $thread [list set sock $sock]
thread::send -async $thread {
thread::attach $sock
fconfigure $sock -buffering line -blocking 0
fileevent $sock readable [list ReadLine $sock]
SendMessage $sock "Connected to Echo server"
}
}
vwait forever
Too bad komodo doesn't auto-indent. Anyhow, it seems to at least run without error. Looking into naglfar capabilities.

Related

How to automate ctrl+d action in the ssh expect script in bash?

I have a usecase where I need to execute a command after connecting to the host through ssh. After the command execution, I need to perform Ctrl-D and Ctrl-M so that I can issue other commands.
I tried with using EOF but It is completely closing the session.
expect << EOF
spawn ssh -o StrictHostKeyChecking=no LocalCOMUser#$nodeIp -p $ssh_port
expect {
"password:" {}
timeout { send_user "Timed out in ssh connection" ;exit 1}
}
send "p#ssword\r"
expect {
">" {}
timeout { send_user "Timed out in ssh connection" ;exit 1}
}
set timeout 120
send "mml\r"
expect {
"<" {}
timeout { send_user "Timed out in ssh connection" ;exit 1}
}
send "$command1\r"
expect {
"<" {}
}
send "exit;\r"
expect {
">" {}
timeout { send_user "Timed out in ssh connection" ;exit 1}
}
send "exit\r"
EOF
Need a command that does action as Ctrl-d.
You can simulate pressing Ctrl-D by sending the ^D/␄ character:
send "\x04"

tpool use to speed up computation

I try to use tpool to speed up the shasum computation of any files with size between 30 and 300 MiB in some folders.
I use tcl-tk version 8.6.6 from homebrew on macOS Sierra 10.12.5 (the tcl-tk OS X version is 8.5.9 and the result is same but terminate with an error).
As I can understand the result of command “brew info tcl-tk”, tcl is compiled with option :
—with-threads
"Build with multithreading support"
Without thread usage, it’s clear the CPU load (Activity Monitor) isn’t very busy.
With threading of tpool, not more, and time spend to compute a set of shasum files is globally the same.
Depending of -maxworkers number, the global % CPU of perl5.18 processes (shasum is a Perl script) is the same. Everything happens as if all the threads that work at the same time do it on the same processor core.
See below the main script based on an example from Donald Fellows answer:
#!/bin/sh
# -*- tcl -*-
# The next line is executed by /bin/sh, but not tcl \
exec /usr/local/Cellar/tcl-tk/bin/tclsh "$0" ${1+"$#"}
package require Thread
set pool [tpool::create -maxworkers 16 -initcmd {
source myProcedures.tcl
}]
proc TREE_parse_big_files {path} {
foreach f [ glob -nocomplain ${path}/* ] {
puts "Parsing ${f}"
if { [ file type ${f} ] == "link" } { return }
if { [ file isdirectory ${f} ] } {
TREE_parse_big_files ${f}
} else {
tpool::post $::pool [list computeSHA ${f}]
}
}
}
TREE_parse_big_files “/tmp/BigFiles” $pool
# Dispose of the pool
tpool::release $pool
The file myProcedures.tcl contains:
proc computeSHA {bigFile} {
puts [eval exec shasum -a 256 -b {$bigFile}]
}
Is there something I didn't understood?

Expect ignoring pattern matching and not exiting

I'm new using expect and is puzzling me big time. It works perfectly with one pattern but when the second case comes up it just ignores the exit completely. First, this is my code.
#!/usr/bin/expect
#Usage migration_test.xpct <ssh_password> <vmname> <no_migraciones>
set timest [ timestamp -format %Y-%m-%d_%H-%M ]
set vmname [lindex $argv 1]
log_file migtest_${vmname}_${timest}.log ;
set password [lindex $argv 0]
set num [lindex $argv 2]
set failureMsg "Status: Failure\n\r"
set timeout 60
spawn ssh admin#localhost -p 10000
expect "yes/no" {
send "yes\r"
expect "*?assword" { send "$password\r" }
} "*?assword" { send "$password\r" }
for {set i 0} {$i < $num} {incr i 1} {
expect "OVM> " {
send "show Vm name=$vmname\r"
expect {
$failureMsg { }
-re "Status = Running\n\r" {
exp_continue
}
-re "Server = .*? \\\[(.*?)(1|2)?\\\]\n\r" {
set destserver $expect_out(2,string);
if { $destserver == 1 } {
send_user "\n\nMIGRATION [ expr $i+1 ] of $num\n\n"
send "migrate Vm name=$vmname destServer=serv_prod02\r"
expect {
-re "JobId: (.*?)\n\r" {
set jobid $expect_out(1,string);
send "show Job id=$jobid\r";
expect {
-re "Command:(.*?)\n\r" { send_user "\n\nWaiting 30secs before next migration\n\n";
sleep 30; }
}
}
-re "Status: Failure\n\r" { send_user "\n\nExiting\n"; exit 1 }
}
} else {
send_user "\n\nMIGRATION [expr $i+1] of $num\n\n"
send "migrate Vm name=$vmname destServer=serv_prod01\r"
expect {
-re "JobId: (.*?)\n\r" {
set jobid $expect_out(1,string);
send "show Job id=$jobid\r";
expect {
-re "Command:(.*?)\n\r" { send_user "\n\nWaiting 30secs before next migration\n\n";
sleep 30; }
}
}
-re "Status: Failure\n\r" { send_user "\n\nExiting\n"; exit 1 }
}
}
}
}
}
}
send "exit\r"
expect eof
The problem comes when it reaches the "migrate vm" section. That's a job I'm sending to a CLI (oracle ovm cli to be precise) and the job can either fail or success. I want to print the job details when it success but finish the entire execution if the job fails (since it already shows the reason and I don't have to expand the job details).
Here is how the output of a successful job looks:
MIGRATION 5 of 12
migrate Vm name=slestest_temp_share_vm destServer=serv_prod01
Command: migrate Vm name=slestest_temp_share_vm
destServer=serv_prod01
Status: Success
Time: 2016-04-13 10:45:24,174
JobId: 12345678978
OVM> show Job id=12345678978
Command: show Job id=12345678978
Status: Success Time: 2016-04-13 10:45:24,188
Data:
Run State = Success
Summary State = Success
Done = Yes
Summary Done = Yes
Job Group = No
Username = admin
Creation Time = Apr 13, 2016 10:44:45 am
Start Time = Apr 13, 201 10:44:45 am
End Time = Apr 13, 2016 10:45:23 am
Duration = 37s
Id = 12345678978 [Migrate Vm: slestest_temp_share_vm to Server: serv_prod01]
Name = Migrate Vm: slestest_temp_share_vm to Server:serv_prod01
Description = Migrate Vm: slestest_temp_share_vm to
Server: serv_prod01 Locked = false
OVM>
Waiting 30secs before next migration
And here is how a failured job looks like:
MIGRATION 4 of 12
migrate Vm name=slestest_temp_share_vm destServer=serv_prod01
Command: migrate Vm name=slestest_temp_share_vm destServer=serv_prod01
Status: Failure
Time: 2016-04-13 11:31:08,819
JobId: 1460564963372
Error Msg: Job failed on Core: OVMAPI_5001E Job: 1460564963372/Migrate Vm: slestest_temp_share_vm to Server: serv_prod01/Migrate Vm: slestest_temp_share_vm serv_prod01, failed. Job Failure Event: 1460565064570/Server Async Command Failed/OVMEVT_00C014D_001 Async command failed serv_prod02. Object: slestest_temp_share_vm, PID: 1724,
Server error: Command: ['xm', 'migrate', '--live', '0004fb00000600009f354416bab38df6', '8.8.8.1'] failed (1): stderr: Error: ti
stdout: Usage: xm migrate
Migrate a domain to another machine.
Options:
-h, --help Print this help.
-l, --live Use live migration.
-p=portnum, --port=portnum
Use specified port for migration.
-n=nodenum, --node=nodenum
Use specified NUMA node on target.
-s, --ssl Use ssl connection for migration.
-c, --change_home_server
Change home server for managed domains.
, on server: serv_prod02, associated with object: 0004fb00000600009f354416bab38df6 [Wed Apr 13 11:31:04 2016]
Why does the Status: Failure is ignored? Also, when that happens it seems it jumps an iteration of the loop, if it was in the 5th it then shows "Migration 7 of 12" for example.
Thanks everyone
I can suggest two things, one you can rewrite code to avoid duplicacy. Second, I think you are matching for both \n\r at the end of pattern. Try with \n alone or use \n?\r? which will match zero, one, or both line endings.
-re "Server = .*? \\\[(.*?)(1|2)?\\\]\n" {
set destserver $expect_out(2,string);
send_user "\n\nMIGRATION [ expr $i+1 ] of $num\n\n"
if { $destserver == 1 } {
send "migrate Vm name=$vmname destServer=serv_prod02\r"
} else {
send "migrate Vm name=$vmname destServer=serv_prod01\r"
}
expect {
-re "JobId: (.*?)\n" {
set jobid $expect_out(1,string);
send "show Job id=$jobid\r";
expect {
-re "Command:(.*?)$" {
send_user "\n\nWaiting 30secs before next migration\n\n";
sleep 30;
}
}
}
-re "Status: Failure\n" { send_user "\n\nExiting\n"; exit 1 }
}
}
Well, after some tests I found the problem. It seems I didn't understand how the timeout worked in expect. Every time a failured migration was performed it exceeded the timeout.
This wasn't evident for me because, although the timeout was exceeded, the script still kept waiting for the answer and printed it anyways, just none of the patterns I was expecting to get were being checked.
The solution was either use the "timeout" command or set it higher. I did the later and everything is running fine now.

#include in Expect script

So I am having trouble making TestCases with expect scripts, I have like 10 TestCases which all starts and ends with the same "functions" like login and logout or turning some flags off, is there a possibility to include them or execute them remotely from my script, like spawn login.exp or even better to put them in functions ?
TC01.exp
#!/usr/bin/expect -f
set timeout 5
#example of getting arguments passed from command line..
#not necessarily the best practice for passwords though...
set server [lindex $argv 0]
set user [lindex $argv 1]
set pass [lindex $argv 2]
set no [lindex $argv 3]
set counter 0
# connect to server via ssh, login, and su to root
send_user "connecting to $server\n"
spawn ssh $user#$server
#login handles cases:
# login with keys (no user/pass)
# user/pass
# login with keys (first time verification)
expect {
"> " { }
"$ " { }
"assword: " {
send "$pass\n"
expect {
"> " { }
"$ " { }
"assword: " {
send_user "\nLogin failed\n"
incr counter 1
exit 5
}
}
}
"(yes/no)? " {
send "yes\n"
expect {
"> " { }
"$ " { }
}
}
default {
send_user "Login failed\n"
incr counter 1
exit
}
}
#TEST CASE HERE
#login out
send "exit\n"
expect {
"> " {}
default {}
}
if { $counter > 0 } {
send_user "\nTestCase finished with some errors!\nFAILED!!!\nERRORS $counter\n";
exit 4;
}
send_user "\nTestCase finished with SUCCESS!\nERRORS: $counter\n";
So i would like to have login and count_error as functions, so I would be able to create my test cases just like this:
TC01.exp
#!/usr/bin/expect -f
set timeout 5
set server [lindex $argv 0]
set user [lindex $argv 1]
set pass [lindex $argv 2]
set no [lindex $argv 3]
set counter 0
login($server, $user, $pass)
#TestCase
Errors($counter)
exit
Expect is actually Tcl with some bindings to pty's and fork() thrown in. All very well described on http://tcl.tk
Functions in Tcl are done with proc (look here) eg:
lib.tcl
proc login {server user pass} {
# Your expect code goes here
return $errorCount
}
proc errors {errorCount} {
if {$errorCount > 0} {
# Your error handling code here
}
}
test:
#!/usr/bin/env expect
source lib.tcl
set errors [login $server $user $pass]
# your test case here
errors $errors

using external redis server for testing tcl scripts

I am running Ubuntu 11.10.
i am trying to run TCL test scripts using external redis server.
using the following :
sb#sb-laptop:~/Redis/redis$ tclsh tests/test_helper.tcl --host 192.168.1.130 --port 6379
Getting the following error :
Testing unit/type/list
[exception]: Executing test client: couldn't open socket: connection refused.
couldn't open socket: connection refused
while executing
"socket $server $port"
(procedure "redis" line 2)
invoked from within
"redis $::host $::port"
(procedure "start_server" line 9)
invoked from within
"start_server {tags {"protocol"}} {
test "Handle an empty query" {
reconnect
r write "\r\n"
r flush
assert_equal "P..."
(file "tests/unit/protocol.tcl" line 1)
invoked from within
"source $path"
(procedure "execute_tests" line 4)
invoked from within
"execute_tests $data"
(procedure "test_client_main" line 9)
invoked from within
"test_client_main $::test_server_port "
the redis.conf is set to default binding, but it is commented out.
If this is possible, what i am doing wrong?
Additional Information:
Below is the tcl code that is responsible for starting the server
proc start_server {options {code undefined}} {
# If we are runnign against an external server, we just push the
# host/port pair in the stack the first time
if {$::external} {
if {[llength $::servers] == 0} {
set srv {}
dict set srv "host" $::host
dict set srv "port" $::port
set client [redis $::host $::port]
dict set srv "client" $client
$client select 9
# append the server to the stack
lappend ::servers $srv
}
uplevel 1 $code
return
}
# setup defaults
set baseconfig "default.conf"
set overrides {}
set tags {}
# parse options
foreach {option value} $options {
switch $option {
"config" {
set baseconfig $value }
"overrides" {
set overrides $value }
"tags" {
set tags $value
set ::tags [concat $::tags $value] }
default {
error "Unknown option $option" }
}
}
set data [split [exec cat "tests/assets/$baseconfig"] "\n"]
set config {}
foreach line $data {
if {[string length $line] > 0 && [string index $line 0] ne "#"} {
set elements [split $line " "]
set directive [lrange $elements 0 0]
set arguments [lrange $elements 1 end]
dict set config $directive $arguments
}
}
# use a different directory every time a server is started
dict set config dir [tmpdir server]
# start every server on a different port
set ::port [find_available_port [expr {$::port+1}]]
dict set config port $::port
# apply overrides from global space and arguments
foreach {directive arguments} [concat $::global_overrides $overrides] {
dict set config $directive $arguments
}
# write new configuration to temporary file
set config_file [tmpfile redis.conf]
set fp [open $config_file w+]
foreach directive [dict keys $config] {
puts -nonewline $fp "$directive "
puts $fp [dict get $config $directive]
}
close $fp
set stdout [format "%s/%s" [dict get $config "dir"] "stdout"]
set stderr [format "%s/%s" [dict get $config "dir"] "stderr"]
if {$::valgrind} {
exec valgrind --suppressions=src/valgrind.sup src/redis-server $config_file > $stdout 2> $stderr &
} else {
exec src/redis-server $config_file > $stdout 2> $stderr &
}
# check that the server actually started
# ugly but tries to be as fast as possible...
set retrynum 100
set serverisup 0
if {$::verbose} {
puts -nonewline "=== ($tags) Starting server ${::host}:${::port} "
}
after 10
if {$code ne "undefined"} {
while {[incr retrynum -1]} {
catch {
if {[ping_server $::host $::port]} {
set serverisup 1
}
}
if {$serverisup} break
after 50
}
} else {
set serverisup 1
}
if {$::verbose} {
puts ""
}
if {!$serverisup} {
error_and_quit $config_file [exec cat $stderr]
}
# find out the pid
while {![info exists pid]} {
regexp {\[(\d+)\]} [exec cat $stdout] _ pid
after 100
}
# setup properties to be able to initialize a client object
set host $::host
set port $::port
if {[dict exists $config bind]} { set host [dict get $config bind] }
if {[dict exists $config port]} { set port [dict get $config port] }
# setup config dict
dict set srv "config_file" $config_file
dict set srv "config" $config
dict set srv "pid" $pid
dict set srv "host" $host
dict set srv "port" $port
dict set srv "stdout" $stdout
dict set srv "stderr" $stderr
# if a block of code is supplied, we wait for the server to become
# available, create a client object and kill the server afterwards
if {$code ne "undefined"} {
set line [exec head -n1 $stdout]
if {[string match {*already in use*} $line]} {
error_and_quit $config_file $line
}
while 1 {
# check that the server actually started and is ready for connections
if {[exec cat $stdout | grep "ready to accept" | wc -l] > 0} {
break
}
after 10
}
# append the server to the stack
lappend ::servers $srv
# connect client (after server dict is put on the stack)
reconnect
# execute provided block
set num_tests $::num_tests
if {[catch { uplevel 1 $code } error]} {
set backtrace $::errorInfo
# Kill the server without checking for leaks
dict set srv "skipleaks" 1
kill_server $srv
# Print warnings from log
puts [format "\nLogged warnings (pid %d):" [dict get $srv "pid"]]
set warnings [warnings_from_file [dict get $srv "stdout"]]
if {[string length $warnings] > 0} {
puts "$warnings"
} else {
puts "(none)"
}
puts ""
error $error $backtrace
}
# Don't do the leak check when no tests were run
if {$num_tests == $::num_tests} {
dict set srv "skipleaks" 1
}
# pop the server object
set ::servers [lrange $::servers 0 end-1]
set ::tags [lrange $::tags 0 end-[llength $tags]]
kill_server $srv
} else {
set ::tags [lrange $::tags 0 end-[llength $tags]]
set _ $srv
}
}
Either there's nothing listening on host 192.168.1.130, port 6379 (well, at a guess) or your firewall configuration is blocking the connection. Impossible to say which, since all the code is really seeing is “the connection didn't work; something said ‘no’…”.

Resources