I try to use tpool to speed up the shasum computation of any files with size between 30 and 300 MiB in some folders.
I use tcl-tk version 8.6.6 from homebrew on macOS Sierra 10.12.5 (the tcl-tk OS X version is 8.5.9 and the result is same but terminate with an error).
As I can understand the result of command “brew info tcl-tk”, tcl is compiled with option :
—with-threads
"Build with multithreading support"
Without thread usage, it’s clear the CPU load (Activity Monitor) isn’t very busy.
With threading of tpool, not more, and time spend to compute a set of shasum files is globally the same.
Depending of -maxworkers number, the global % CPU of perl5.18 processes (shasum is a Perl script) is the same. Everything happens as if all the threads that work at the same time do it on the same processor core.
See below the main script based on an example from Donald Fellows answer:
#!/bin/sh
# -*- tcl -*-
# The next line is executed by /bin/sh, but not tcl \
exec /usr/local/Cellar/tcl-tk/bin/tclsh "$0" ${1+"$#"}
package require Thread
set pool [tpool::create -maxworkers 16 -initcmd {
source myProcedures.tcl
}]
proc TREE_parse_big_files {path} {
foreach f [ glob -nocomplain ${path}/* ] {
puts "Parsing ${f}"
if { [ file type ${f} ] == "link" } { return }
if { [ file isdirectory ${f} ] } {
TREE_parse_big_files ${f}
} else {
tpool::post $::pool [list computeSHA ${f}]
}
}
}
TREE_parse_big_files “/tmp/BigFiles” $pool
# Dispose of the pool
tpool::release $pool
The file myProcedures.tcl contains:
proc computeSHA {bigFile} {
puts [eval exec shasum -a 256 -b {$bigFile}]
}
Is there something I didn't understood?
Related
When i am passing value of a variable declared in jenkins Groovy script its value is not retained in for loop which is running on a remote server. Strange thing is i am able to access the same value outside the for loop.
Here is the sample code i am trying to use
#!/usr/bin/env groovy
def config
def COMMANDS_TO_CHECK='curl grep hello awk tr mkdir bc'
pipeline {
agent {
label "master"
}
stages {
stage ('Validation of commands') {
steps {
script {
sh """
#!/bin/bash
/usr/bin/sshpass -p passwrd ssh user#host << EOF
hostname
echo $COMMANDS_TO_CHECK ---> This is printed
for CURRENT_COMMAND in \$COMMANDS_TO_CHECK
do
echo ${CURRENT_COMMAND} ---> Why This is not printed?
echo \${CURRENT_COMMAND} ----> Why This is not printed?
done
hostname
EOF
exit
"""
}
}
}
}
}
Output
[workspace#3] Running shell script
+ /usr/bin/sshpass -p passwrd ssh user#host
Pseudo-terminal will not be allocated because stdin is not a terminal.
illinsduck01
curl grep hello awk tr mkdir bc
illinsduck01
+ exit
You can wrap sh in """ ... """ as below
#!/usr/bin/env groovy
def config
pipeline {
agent {
label "master"
}
stages {
stage ('Validation of commands') {
steps {
script {
sh """#!/bin/sh
/usr/bin/sshpass -p password ssh username#hostname << EOF
COMMANDS_TO_CHECK="curl grep hello awk tr mkdir bc"
hostname
echo \$COMMANDS_TO_CHECK
for CURRENT_COMMAND in \$COMMANDS_TO_CHECK
do
echo \$CURRENT_COMMAND
which \$CURRENT_COMMAND
status=\$?
if [ \${status} -eq 0 ]
then
echo \${CURRENT_COMMAND} command is OK
else
echo "Failed to find the \${CURRENT_COMMAND} command"
fi
done
hostname
EOF
exit
"""
}
}
}
}
}
I am trying to comprehend how, if even it can be done, can I avoid subshell?
Is this the only way the code can be written or is there another way?
I tried to use braces { ... }, but it won't pass shellcheck and won't run.
is_running_interactively ()
# test if file descriptor 0 = standard input is connected to the terminal
{
[ -t 0 ]
}
is_tput_available ()
# check if tput coloring is available
{
command -v tput > /dev/null 2>&1 &&
tput bold > /dev/null 2>&1 &&
tput setaf 1 > /dev/null 2>&1
}
some_other_function ()
# so far unfinished function
{
# is this a subshell? if so, can I avoid it somehow?
( is_running_interactively && is_tput_available ) || # <-- HERE
{
printf '%b' "${2}"
return
}
...
}
It is a compound-list, and yes those commands are run in a subshell. To avoid it, use curly braces instead of parentheses:
{ is_running_interactively && is_tput_available; } || ...
We want to split up our project into smaller pieces. Our current CI process goes through a short test phase and then runs a deployment script. However, if nothing has changed in one of the sub project, we do not want to go through the build for this.
Jenkins without pipelines supports exclusions in the SCM configuration (we use git) and based on this, you can configure a specific job to run. However, when using a pipeline, how can I know, if I should build this part or not? How do I get access to the paths that were affected by the last push?
At the moment our script is very simple, and we would like to keep it as simple as possible.
We were playing around with the scripted and the declarative syntax, but could not find a good solution.
Declarative:
#!groovy
pipeline {
agent any
tools {
nodejs '8.1'
}
stages {
stage('Checkout') {
steps {
checkout scm
}
}
# Only continue, if something has changed
stage('Install') {
steps {
sh 'npm install'
}
}
stage('Test') {
steps {
sh 'npm run test-jenkins'
}
post {
always {
junit "artifacts/test/report.xml"
}
}
}
}
}
Scripted:
#!groovy
node {
def nodejs = tool name: '8.1', type: 'jenkins.plugins.nodejs.tools.NodeJSInstallation'
env.PATH = "${nodejs}/bin:${env.PATH}"
stage('Checkout') {
checkout scm
}
# Only continue, if something has changed
stage('Install') {
sh 'npm install'
}
stage('Test') {
try {
sh 'npm run test-jenkins'
} finally {
junit "artifacts/test/report.xml"
}
}
}
Thanks to ElpieKay's fast comment on my question, we now have an elegant solution:
Make a tag to the current commit on a successful build
In the next build compare the new commit and the tag for changes
We are using a multi-branch pipeline and a parallel build for the multiple projects we have under the same source root. We iterate through the projects (serviceX) and check in the corresponding directory for a change:
def projects = ['service1', 'service2']
def builders = [:]
for (p in projects) {
def label = p
builders[label] = {
def tag = "${BRANCH_NAME}_last"
node {
echo "Checking for changes compared to ${tag} in directory ${label}"
try {
sh "./check-for-changes ${tag} ${label}"
} catch (ignored) {
echo "Nothing to do"
return
}
dir (label) {
stage(label + ": Install") {
sh "npm install"
}
stage(label + ": Test") {
try {
sh "npm run test-jenkins"
} finally {
junit 'artifacts/test/report.xml'
}
}
echo "Setting tag for the last build on this branch"
sh "git tag -f ${tag}"
}
}
}
}
parallel builders
... and the script to check for changes:
#!/bin/bash
SHA_PREV=$1
if [ -z ${SHA_PREV} ]; then
echo "Usage: `basename $0` <tag> <path>"
exit 1
fi
CHECK_PATH=$2
if [ -z ${CHECK_PATH} ]; then
echo "Usage: `basename $0` <tag> <path>"
exit 1
fi
if `git rev-parse ${SHA_PREV} >/dev/null 2>&1`; then
echo "Found previous tag: ${SHA_PREV}"
else
SHA_PREV=`git rev-list --max-parents=0 HEAD`
echo "Using initial commit: ${SHA_PREV}"
fi
changes=`git diff --name-only ${SHA_PREV} HEAD | grep ${CHECK_PATH}/`
if [ ! -n "${changes}" ]; then
echo "No changes found"
exit 2 # no changes found
fi
I'm new using expect and is puzzling me big time. It works perfectly with one pattern but when the second case comes up it just ignores the exit completely. First, this is my code.
#!/usr/bin/expect
#Usage migration_test.xpct <ssh_password> <vmname> <no_migraciones>
set timest [ timestamp -format %Y-%m-%d_%H-%M ]
set vmname [lindex $argv 1]
log_file migtest_${vmname}_${timest}.log ;
set password [lindex $argv 0]
set num [lindex $argv 2]
set failureMsg "Status: Failure\n\r"
set timeout 60
spawn ssh admin#localhost -p 10000
expect "yes/no" {
send "yes\r"
expect "*?assword" { send "$password\r" }
} "*?assword" { send "$password\r" }
for {set i 0} {$i < $num} {incr i 1} {
expect "OVM> " {
send "show Vm name=$vmname\r"
expect {
$failureMsg { }
-re "Status = Running\n\r" {
exp_continue
}
-re "Server = .*? \\\[(.*?)(1|2)?\\\]\n\r" {
set destserver $expect_out(2,string);
if { $destserver == 1 } {
send_user "\n\nMIGRATION [ expr $i+1 ] of $num\n\n"
send "migrate Vm name=$vmname destServer=serv_prod02\r"
expect {
-re "JobId: (.*?)\n\r" {
set jobid $expect_out(1,string);
send "show Job id=$jobid\r";
expect {
-re "Command:(.*?)\n\r" { send_user "\n\nWaiting 30secs before next migration\n\n";
sleep 30; }
}
}
-re "Status: Failure\n\r" { send_user "\n\nExiting\n"; exit 1 }
}
} else {
send_user "\n\nMIGRATION [expr $i+1] of $num\n\n"
send "migrate Vm name=$vmname destServer=serv_prod01\r"
expect {
-re "JobId: (.*?)\n\r" {
set jobid $expect_out(1,string);
send "show Job id=$jobid\r";
expect {
-re "Command:(.*?)\n\r" { send_user "\n\nWaiting 30secs before next migration\n\n";
sleep 30; }
}
}
-re "Status: Failure\n\r" { send_user "\n\nExiting\n"; exit 1 }
}
}
}
}
}
}
send "exit\r"
expect eof
The problem comes when it reaches the "migrate vm" section. That's a job I'm sending to a CLI (oracle ovm cli to be precise) and the job can either fail or success. I want to print the job details when it success but finish the entire execution if the job fails (since it already shows the reason and I don't have to expand the job details).
Here is how the output of a successful job looks:
MIGRATION 5 of 12
migrate Vm name=slestest_temp_share_vm destServer=serv_prod01
Command: migrate Vm name=slestest_temp_share_vm
destServer=serv_prod01
Status: Success
Time: 2016-04-13 10:45:24,174
JobId: 12345678978
OVM> show Job id=12345678978
Command: show Job id=12345678978
Status: Success Time: 2016-04-13 10:45:24,188
Data:
Run State = Success
Summary State = Success
Done = Yes
Summary Done = Yes
Job Group = No
Username = admin
Creation Time = Apr 13, 2016 10:44:45 am
Start Time = Apr 13, 201 10:44:45 am
End Time = Apr 13, 2016 10:45:23 am
Duration = 37s
Id = 12345678978 [Migrate Vm: slestest_temp_share_vm to Server: serv_prod01]
Name = Migrate Vm: slestest_temp_share_vm to Server:serv_prod01
Description = Migrate Vm: slestest_temp_share_vm to
Server: serv_prod01 Locked = false
OVM>
Waiting 30secs before next migration
And here is how a failured job looks like:
MIGRATION 4 of 12
migrate Vm name=slestest_temp_share_vm destServer=serv_prod01
Command: migrate Vm name=slestest_temp_share_vm destServer=serv_prod01
Status: Failure
Time: 2016-04-13 11:31:08,819
JobId: 1460564963372
Error Msg: Job failed on Core: OVMAPI_5001E Job: 1460564963372/Migrate Vm: slestest_temp_share_vm to Server: serv_prod01/Migrate Vm: slestest_temp_share_vm serv_prod01, failed. Job Failure Event: 1460565064570/Server Async Command Failed/OVMEVT_00C014D_001 Async command failed serv_prod02. Object: slestest_temp_share_vm, PID: 1724,
Server error: Command: ['xm', 'migrate', '--live', '0004fb00000600009f354416bab38df6', '8.8.8.1'] failed (1): stderr: Error: ti
stdout: Usage: xm migrate
Migrate a domain to another machine.
Options:
-h, --help Print this help.
-l, --live Use live migration.
-p=portnum, --port=portnum
Use specified port for migration.
-n=nodenum, --node=nodenum
Use specified NUMA node on target.
-s, --ssl Use ssl connection for migration.
-c, --change_home_server
Change home server for managed domains.
, on server: serv_prod02, associated with object: 0004fb00000600009f354416bab38df6 [Wed Apr 13 11:31:04 2016]
Why does the Status: Failure is ignored? Also, when that happens it seems it jumps an iteration of the loop, if it was in the 5th it then shows "Migration 7 of 12" for example.
Thanks everyone
I can suggest two things, one you can rewrite code to avoid duplicacy. Second, I think you are matching for both \n\r at the end of pattern. Try with \n alone or use \n?\r? which will match zero, one, or both line endings.
-re "Server = .*? \\\[(.*?)(1|2)?\\\]\n" {
set destserver $expect_out(2,string);
send_user "\n\nMIGRATION [ expr $i+1 ] of $num\n\n"
if { $destserver == 1 } {
send "migrate Vm name=$vmname destServer=serv_prod02\r"
} else {
send "migrate Vm name=$vmname destServer=serv_prod01\r"
}
expect {
-re "JobId: (.*?)\n" {
set jobid $expect_out(1,string);
send "show Job id=$jobid\r";
expect {
-re "Command:(.*?)$" {
send_user "\n\nWaiting 30secs before next migration\n\n";
sleep 30;
}
}
}
-re "Status: Failure\n" { send_user "\n\nExiting\n"; exit 1 }
}
}
Well, after some tests I found the problem. It seems I didn't understand how the timeout worked in expect. Every time a failured migration was performed it exceeded the timeout.
This wasn't evident for me because, although the timeout was exceeded, the script still kept waiting for the answer and printed it anyways, just none of the patterns I was expecting to get were being checked.
The solution was either use the "timeout" command or set it higher. I did the later and everything is running fine now.
I am using the nagios to monitor gearman and getting error "CRITICAL - function 'xxx' is not registered in the server"
Script that nagios execute to check the gearman is like
#!/usr/bin/env perl
# taken from: gearmand-0.24/libgearman-server/server.c:974
# function->function_name, function->job_total,
# function->job_running, function->worker_count);
#
# this code give following result with gearadmin --status
#
# FunctionName job_total job_running worker_count
# AdsUpdateCountersFunction 0 0 4
use strict;
use warnings;
use Nagios::Plugin;
my $VERSION="0.2.1";
my $np;
$np = Nagios::Plugin->new(usage => "Usage: %s -f|--flist <func1[:threshold1],..,funcN[:thresholdN]> [--host|-H <host>] [--port|-p <port>] [ -c|--critworkers=<threshold> ] [ -w|--warnworkers=<threshold>] [-?|--usage] [-V|--version] [-h|--help] [-v|--verbose] [-t|--timeout=<timeout>]",
version => $VERSION,
blurb => 'This plugin checks a gearman job server, expecting that every function in function-list arg is registered by at least one worker, and expecting that job_total is not too much high.',
license => "Brought to you AS IS, WITHOUT WARRANTY, under GPL. (C) Remi Paulmier <remi.paulmier\#gmail.com>",
shortname => "CHECK_GEARMAN",
);
$np->add_arg(spec => 'flist|f=s',
help => q(Check for the functions listed in STRING, separated by comma. If optional threshold is given (separated by :), check that waiting jobs for this particular function are not exceeding that value),
required => 1,
);
$np->add_arg(spec => 'host|H=s',
help => q(Check the host indicated in STRING),
required => 0,
default => 'localhost',
);
$np->add_arg(spec => 'port|p=i',
help => q(Use the TCP port indicated in INTEGER),
required => 0,
default => 4730,
);
$np->add_arg(spec => 'critworkers|c=i',
help => q(Exit with CRITICAL status if fewer than INTEGER workers have registered a particular function),
required => 0,
default => 1,
);
$np->add_arg(spec => 'warnworkers|w=i',
help => q(Exit with WARNING status if fewer than INTEGER workers have registered a particular function),
required => 0,
default => 4,
);
$np->getopts;
my $ng = $np->opts;
# manage timeout
alarm $ng->timeout;
my $runtime = {'status' => OK,
'message' => "Everything OK",
};
# host & port
my $host = $ng->get('host');
my $port = $ng->get('port');
# verbosity
my $verbose = $ng->get('verbose');# look for gearadmin, use nc if not found
my #paths = grep { -x "$_/gearadmin" } split /:/, $ENV{PATH};
my $cmd = "gearadmin --status -h $host -p $port";
if (#paths == 0) {
print STDERR "gearadmin not found, using nc\n" if ($verbose != 0);
# $cmd = "echo status | nc -w 1 $host $port";
$cmd = "echo status | nc -i 1 -w 1 $host $port";
}
foreach (`$cmd 2>/dev/null | grep -v '^\\.'`) {
chomp;
my ($fname, $job_total, $job_running, $worker_count) =
split /[[:space:]]+/;
$runtime->{'funcs'}{"$fname"} = {job_total => $job_total,
job_running => $job_running,
worker_count => $worker_count };
# print "$fname : $runtime->{'funcs'}{\"$fname\"}{'worker_count'}\n";
}
# get function list
my #flist = split /,/, $ng->get('flist');
foreach (#flist) {
my ($fname, $fthreshold);
if (/\:/) {
($fname, $fthreshold) = split /:/;
} else {
($fname, $fthreshold) = ($_, -1);
}
# print "defined for $fname: $runtime->{'funcs'}{\"$fname\"}{'worker_count'}\n";
# if (defined($runtime->{'funcs'}{"$fname"})) {
# print "$fname is defined\n";
# } else {
# print "$fname is NOT defined\n";
# }
if (!defined($runtime->{'funcs'}{"$fname"}) &&
$runtime->{'status'} <= CRITICAL) {
($runtime->{'status'}, $runtime->{'message'}) =
(CRITICAL, "function '$fname' is not registered in the server");
} else {
if ($runtime->{'funcs'}{"$fname"}{'worker_count'} <
$ng->get('critworkers') && $runtime->{'status'} <= CRITICAL) {
($runtime->{'status'}, $runtime->{'message'}) =
(CRITICAL,
"less than " .$ng->get('critworkers').
" workers were found having function '$fname' registered.");
}
if ($runtime->{'funcs'}{"$fname"}{'worker_count'} <
$ng->get('warnworkers') && $runtime->{'status'} <= WARNING) {
($runtime->{'status'}, $runtime->{'message'}) =
(WARNING,
"less than " .$ng->get('warnworkers').
" workers were found having function '$fname' registered.");
}
if ($runtime->{'funcs'}{"$fname"}{'job_total'} > $fthreshold
&& $fthreshold != -1 && $runtime->{'status'}<=WARNING) {
($runtime->{'status'}, $runtime->{'message'}) =
(WARNING,
$runtime->{'funcs'}{"$fname"}{'job_total'}.
" jobs for $fname exceeds threshold $fthreshold");
}
}
}
$np->nagios_exit($runtime->{'status'}, $runtime->{'message'});
When the script is executed simply by command line it says "everything ok"
But in nagios it shows error "CRITICAL - function 'xxx' is not registered in the server"
Thanks in advance
After spending long time on this, finally got the answer all that have to do is.
yum install nc
nc is what that was missing from the system.
With Regards,
Bankat Vikhe
Not easy to say but it could be related to your script not being executable as embedded Perl.
Try with # nagios: -epn at the beginning of the script.
#!/usr/bin/env perl
# nagios: -epn
use strict;
use warnings;
Be sure to check all the hints in the Perl Plugins section of the Nagios Plugin Development Guidelines