What are some simple fixes you would make for the obvious vulnerabilities in this script?
#!/bin/tcsh
# foreachfile command
# a shell script to apply command to each file in the current directory
set ListOfFiles = `ls`
set Count = 1
set ListLength = $#ListOfFiles
while ($Count <= $ListLength)
$argv $ListOfFiles[$Count]
# Count = $Count + 1
end
#!/bin/tcsh
# foreachfile command <<<< You gave away the ending!
# a shell script to apply command to each file in the current directory
foreach f (*)
"$argv" "$f"
end
You might want to check $argv against a whitelist of permitted commands.
Related
I am working on updating a script to append the loop number iteration to a filename. However I can't find any documentation that will help me update the variable value. Instead, it appends the operation as a string.
(base) me#axoneme:~$ tcsh
axoneme:~> set j=100
axoneme:~> set j=$j+1
axoneme:~> echo $j
100+1
How can I do arithmetic in tcsh such that the 102nd loop will just be set j= $j + 1 where $j =101
# sign seems to be used in tcsh
axoneme:~> set j = 100
axoneme:~> # j = 100 + 1
axoneme:~> echo $j
101
The following Perl script generates an .xls file from a text file. It runs great in our linux test environment, but generates an empty spreadsheet (.xls) in our production environment when run via cron (cron works in test, as well.) Nothing jumps out at our sys admins in terms of system level settings that might account for this behavior. Towards the bottom of the script in the import_data subroutine, the correct number of lines is reported, but nothing is written to the spreadsheet and no errors are returned at either the script or system level. I ran it through the perl debugger but my skills fell short of being able to interactively watch it populate the file. The cron entry looks like this:
cd <script directory>; cvs2xls input.txt output.xls 2>&1
Any debugging tips would be appreciated, as well as potential system settings that I can forward on to our sysadmins.
#!/usr/bin/perl
use strict;
use warnings;
use lib '/apps/tu01688/perl5/lib/perl5';
use Spreadsheet::WriteExcel;
use Text::CSV::Simple;
BEGIN {
unshift #INC, "/apps/tu01688/jobs/mayo-expert";
};
my $infile = shift;
usage() unless defined $infile && -f $infile;
my $parser = Text::CSV::Simple->new;
my #data = $parser->read_file($infile);
my $headers = shift #data;
my $outfile = shift || $infile . ".xls";
my $subject = shift || 'worksheet';
sub usage {
print "csv2xls infile [outfile] [subject]\n";
exit;
}
my $workbook = Spreadsheet::WriteExcel->new($outfile);
my $bold = $workbook->add_format();
$bold->set_bold(1);
import_data($workbook, $subject, $headers, \#data);
# Add a worksheet
sub import_data {
my $workbook = shift;
my $base_name = shift;
my $colums = shift;
my $data = shift;
my $limit = shift || 50_000;
my $start_row = shift || 1;
my $worksheet = $workbook->add_worksheet($base_name);
$worksheet->add_write_handler(qr[\w], \&store_string_widths);
#$worksheet->add_write_handler(qr[\w]| \&store_string_widths);
my $w = 1;
$worksheet->write('A' . $start_row, $colums, ,$bold);
my $i = $start_row;
my $qty = 0;
for my $row (#$data) {
$qty++;
if ($i > $limit) {
$i = $start_row;
$w++;
$worksheet = $workbook->add_worksheet("$base_name - $w");
$worksheet->write('A1', $colums,$bold);
}
$worksheet->write($i++, 0, $row);
}
autofit_columns($worksheet);
warn "Converted $qty rows.";
return $worksheet;
}
###############################################################################
###############################################################################
#
# Functions used for Autofit.
#
###############################################################################
#
# Adjust the column widths to fit the longest string in the column.
#
sub autofit_columns {
my $worksheet = shift;
my $col = 0;
for my $width (#{$worksheet->{__col_widths}}) {
$worksheet->set_column($col, $col, $width) if $width;
$col++;
}
}
###############################################################################
#
# The following function is a callback that was added via add_write_handler()
# above. It modifies the write() function so that it stores the maximum
# unwrapped width of a string in a column.
#
sub store_string_widths {
my $worksheet = shift;
my $col = $_[1];
my $token = $_[2];
# Ignore some tokens that we aren't interested in.
return if not defined $token; # Ignore undefs.
return if $token eq ''; # Ignore blank cells.
return if ref $token eq 'ARRAY'; # Ignore array refs.
return if $token =~ /^=/; # Ignore formula
# Ignore numbers
#return if $token =~ /^([+-]?)(?=\d|\.\d)\d*(\.\d*)?([Ee]([+-]?\d+))?$/;
# Ignore various internal and external hyperlinks. In a real scenario
# you may wish to track the length of the optional strings used with
# urls.
return if $token =~ m{^[fh]tt?ps?://};
return if $token =~ m{^mailto:};
return if $token =~ m{^(?:in|ex)ternal:};
# We store the string width as data in the Worksheet object. We use
# a double underscore key name to avoid conflicts with future names.
#
my $old_width = $worksheet->{__col_widths}->[$col];
my $string_width = string_width($token);
if (not defined $old_width or $string_width > $old_width) {
# You may wish to set a minimum column width as follows.
#return undef if $string_width < 10;
$worksheet->{__col_widths}->[$col] = $string_width;
}
# Return control to write();
return undef;
}
###############################################################################
#
# Very simple conversion between string length and string width for Arial 10.
# See below for a more sophisticated method.
#
sub string_width {
return length $_[0];
}
Uhmm.. don't put chained commands in cron, use an external script instead. Anyway: some suggestions that may help you:
Debugging cron commands
Check the mail! By default cron will mail any output from the command to the user it is running the command as. If there is no output there will be no mail. If you want cron to send mail to a different account then you can set the MAILTO environment variable in the crontab file e.g.
MAILTO=user#somehost.tld
1 2 * * * /path/to/your/command
Capture the output yourself
1 2 * * * /path/to/your/command &>/tmp/mycommand.log
which captures stdout and stderr to /tmp/mycommand.log
Look at the logs; cron logs its actions via syslog, which (depending on your setup) often go to /var/log/cron or /var/log/syslog.
If required you can filter the cron statements with e.g.
grep CRON /var/log/syslog
Now that we've gone over the basics of cron, where the files are and how to use them let's look at some common problems.
Check that cron is running
If cron isn't running then your commands won't be scheduled ...
ps -ef | grep cron | grep -v grep
should get you something like
root 1224 1 0 Nov16 ? 00:00:03 cron
or
root 2018 1 0 Nov14 ? 00:00:06 crond
If not restart it
/sbin/service cron start
or
/sbin/service crond start
There may be other methods; use what your distro provides.
cron runs your command in a restricted environment.
What environment variables are available is likely to be very limited. Typically, you'll only get a few variables defined, such as $LOGNAME, $HOME, and $PATH.
Of particular note is the PATH is restricted to /bin:/usr/bin. The vast majority of "my cron script doesn't work" problems are caused by this restrictive path. If your command is in a different location you can solve this in a couple of ways:
Provide the full path to your command.
1 2 * * * /path/to/your/command
Provide a suitable PATH in the crontab file
PATH=/usr:/usr/bin:/path/to/something/else
1 2 * * * command
If your command requires other environment variables you can define them in the crontab file too.
cron runs your command with cwd == $HOME
Regardless of where the program you execute resides on the filesystem, the current working directory of the program when cron runs it will be the user's home directory. If you access files in your program, you'll need to take this into account if you use relative paths, or (preferably) just use fully-qualified paths everywhere, and save everyone a whole lot of confusion.
The last command in my crontab doesn't run
Cron generally requires that commands are terminated with a new line. Edit your crontab; go to the end of the line which contains the last command and insert a new line (press enter).
Check the crontab format
You can't use a user crontab formatted crontab for /etc/crontab or the fragments in /etc/cron.d and vice versa. A user formatted crontab does not include a username in the 6th position of a row, while a system formatted crontab includes the username and runs the command as that user.
I put a file in /etc/cron.{hourly,daily,weekly,monthly} and it doesn't run
Check that the filename doesn't have an extension see run-parts
Ensure the file has execute permissions.
Tell the system what to use when executing your script (eg. put #!/bin/sh at top)
Cron date related bugs
If your date is recently changed by a user or system update, timezone or other, then crontab will start behaving erratically and exhibit bizarre bugs, sometimes working, sometimes not. This is crontab's attempt to try to "do what you want" when the time changes out from underneath it. The "minute" field will become ineffective after the hour is changed. In this scenario, only asterisks would be accepted. Restart cron and try it again without connecting to the internet (so the date doesn't have a chance to reset to one of the time servers).
Percent signs, again
To emphasise the advice about percent signs, here's an example of what cron does with them:
# cron entry
* * * * * cat >$HOME/cron.out%foo%bar%baz
will create the ~/cron.out file containing the 3 lines
foo
bar
baz
This is particularly intrusive when using the date command. Be sure to escape the percent signs
* * * * * /path/to/command --day "$(date "+\%Y\%m\%d")"
Thanks so much for the extensive feedback, everyone. I'm certainly taking a lot more away from this than I put into it. In any event, I ran across the answer. In my perl5 lib folder I found that somehow the IO and OLE libraries were missing on production. Copying those over from development resulted in everything working fine. The fact that I was unable to determine/capture this through conventional debugging efforts as opposed to merely comparing directory listings out of exasperation speaks to how much more I have to learn along these lines. But I'm confident that the great feedback I received will go a long ways towards getting me there. Thanks again, everyone.
I have a file file.dat as follow:
1.1,2.1 1.4
3.1,2.1 2.4
2.4,4.5 11.5
..
And I want to select each time the whole line (string) and replace it in another file. So far I tried the following
#!/bin/csh
set FILENAME = 'file.dat' # file in which the strings are
set str = "229.8,230.9 230.36" # initialize the first string
set n = 1
while ( $n <= 3 ) # number of lines in the FILENAME
echo Testing the first string $str
set rep = $(head -n $n "$FILENAME")
# n++ # increment the index
end
When I tried to launch the script csh launch.sh I obtained the follow error message
Testing the first string 229.8,230.9 230.36
Illegal variable name. # connect with the rep definition(?)
The file in which I want to change the string str is as follow (this is btw a secondary problem which I could figure out by myself once I understand what's wrong in the first lines):
# Name Type Par
Mi FI 154.2355189465
So UN 229.8,230.9 230.36 # line to be changed
Za FI 0.8000020209
May somebody help me, please?
$(...) is Bash syntax for command substitution in Bash.
In C-shell you have to use backticks instead (yuck).
I am trying to read to id_rsa file into a variable var( set var=`cat id_rsa`) in tcsh to provide input to a program. But when i echo the variable ( echo "$var")new lines are gone, its a one line file content. So how do i correctly store and print the variable?
Don't use tcsh for this task, getting the output of a command into a variable in verbatim is unnecessarily difficult:
Some workarounds if you have to use tcsh are:
use redirection
% yourtool < id_rsa
Store the variable as base-16 (or something else) encoded stuff, so that it doesn't contain any newline characters that will get mangled by tcsh.
% set hex_contents = `<id_rsa xxd -l 16 -p`
Use a tempfile?
% set tempfile = `mktemp`
% program > tempfile
... later
% <tempfile other-program
I asked a similar question almost a year ago; https://unix.stackexchange.com/questions/284220/tcsh-preserve-newlines-in-command-substitution
In case you're curious this is how you get the verbatim contents (credit Stéphane Chazelas).
set temp = "`(some command; echo .) | paste -d . - /dev/null`"
set var = ""
set nl = '\
'
foreach i ($temp:q)
set var = $var:q$i:r:q$nl:q
end
set var = $var:r:q
It is very strange, i have many script like following and all running in crontab but following script running on command (./load.pl) line but not inside crontab
crontab:
0-59/5 * * * * /home/spatel/rrd/load.pl >> /tmp/load.out
Notes:
I also tried following method
0-59/5 * * * * /usr/bin/perl /home/spatel/rrd/load.pl >> /tmp/load.out
0-59/5 * * * * root /usr/bin/perl /home/spatel/rrd/load.pl >> /tmp/load.out
Somewhere i read cron ignore newline end of the script so i look care of that too
I have put print in script and redirect to /tmp/load.out i can see that output in load.out when cron execute but somehow it is not updating data in side load.rrd file.
If i run script on command like ./load.pl it works! but not inside crontab.
I have set crontab PATH whatever root use has. I tried all possible way to debug but it is not running inside cron. Here is the place where i get this script, all other script working file in crontab only following one has issue :( https://github.com/mmitch/rrd
Script:
#!/usr/bin/env perl
#
# RRD script to display system load
# 2003,2011 (c) by Christian Garbs <mitch#cgarbs.de>
# Licensed under GNU GPL.
#
# This script should be run every 5 minutes.
#
# *ADDITIONALLY* data aquisition is done externally every minute:
# rrdtool update $datafile N:$( PROCS=`echo /proc/[0-9]*|wc -w|tr -d ' '`; read L1 L2 L3 DUMMY < /proc/loadavg ; echo ${L1}:${L2}:${L3}:${PROCS} )
#
use strict;
use warnings;
#use 5.010;
use RRDs;
# parse configuration file
my %conf;
eval(`/bin/cat /home/spatel/rrd/rrd-conf.pl`);
die $# if $#;
# set variables
my $datafile = "/home/spatel/rrd/db/load.rrd";
my $picbase = "/var/www/mrtg/rrd/load-";
# global error variable
my $ERR;
# whoami?
my $hostname = `/bin/hostname`;
chomp $hostname;
# generate database if absent
if ( ! -e $datafile ) {
# max 70000 for all values
RRDs::create($datafile,
"--step=60",
"DS:load1:GAUGE:120:0:70000",
"DS:load2:GAUGE:120:0:70000",
"DS:load3:GAUGE:120:0:70000",
"DS:procs:GAUGE:120:0:70000",
"RRA:AVERAGE:0.5:1:120",
"RRA:AVERAGE:0.5:5:600",
"RRA:AVERAGE:0.5:30:700",
"RRA:AVERAGE:0.5:120:775",
"RRA:AVERAGE:0.5:1440:797",
"RRA:MAX:0.5:1:120",
"RRA:MAX:0.5:5:600",
"RRA:MAX:0.5:6:700",
"RRA:MAX:0.5:120:775",
"RRA:MAX:0.5:1440:797",
"RRA:MIN:0.5:1:120",
"RRA:MIN:0.5:5:600",
"RRA:MIN:0.5:6:700",
"RRA:MIN:0.5:120:775",
"RRA:MIN:0.5:1440:797"
);
$ERR=RRDs::error;
die "ERROR while creating $datafile: $ERR\n" if $ERR;
print "created $datafile\n";
}
# data aquisition is done externally every minute:
my #procs = glob '/proc/[0-9]*';
my $file = '/proc/loadavg';
open my $fh, '<', $file or die "Failed to open '$file': $!";
my $load = <$fh>;
my $p = (scalar #procs);
my $l = (join ':', (split ' ', $load)[0..2]);
print "${l}:${p}";
# update rrd
RRDs::update($datafile,
"N:${l}:${p}"
);
$ERR=RRDs::error;
die "ERROR while updating $datafile: $ERR\n" if $ERR;
# draw pictures
foreach ( [3600, "hour"], [86400, "day"], [604800, "week"], [31536000, "year"] ) {
my ($time, $scale) = #{$_};
RRDs::graph($picbase . $scale . ".png",
"--start=-${time}",
'--lazy',
'--imgformat=PNG',
"--title=${hostname} system load (last $scale)",
"--width=$conf{GRAPH_WIDTH}",
"--height=$conf{GRAPH_HEIGHT}",
'--slope-mode',
'--alt-autoscale',
"DEF:load1=${datafile}:load1:AVERAGE",
"DEF:load2=${datafile}:load2:AVERAGE",
"DEF:load3=${datafile}:load3:AVERAGE",
"DEF:procsx=${datafile}:procs:AVERAGE",
"DEF:procminx=${datafile}:procs:MIN",
"DEF:procmaxx=${datafile}:procs:MAX",
'CDEF:procs=procsx,100,/',
'CDEF:procmin=procminx,100,/',
'CDEF:procrange=procmaxx,procminx,-,100,/',
'AREA:procmin',
'STACK:procrange#E0E0E0',
'AREA:load3#000099:loadavg3',
'LINE2:load2#0000FF:loadavg2',
'LINE1:load1#9999FF:loadavg1',
'COMMENT:\n',
'LINE1:procs#000000:processes/100',
);
$ERR=RRDs::error;
die "ERROR while drawing $datafile $time: $ERR\n" if $ERR;
}
Update:
Here is the output of script:
[root#spatel tmp]# pwd
/tmp
[root#spatel tmp]# /home/spatel/rrd/load.pl
0.15:0.06:0.01:664
Its most likely an environment problem.
A "cron" scripts starts of with the default system environment (Default PATH, LANG etc.) your .profile, .rc are NOT executed.
So you need to provide all the environment variables PATHs etc. your program requires in the script. This is a bit of a pain for a pure perl script so its probably better to wrap it in a shell script which sets whatever your ".profile" script sets.
Solved:
You won't believe how problem got resolved, I use perl instead of /usr/bin/perl in crontab, I though its best practice to use full PATH in crontab but it prove wrong, Still i don't know why and How?
0-59/5 * * * * perl /home/spatel/rrd/load.pl