I am trying to work around a situation where I need to login to a multitude
of servers without being able to utilize ssh keys. As a result I am
formulating a "that is a bad practice"-expect script:
#!/usr/bin/expect
set arg1 [lindex $argv 0]
spawn ssh $arg1 -l user "hostname; env x='() { :;}; echo vulnerable' bash -c \"echo this is a test\"; echo"
expect " password:"
send "my_supersecret_password\n"
interact
Running it works fine:
$ ./ssh.expect server
spawn ssh server -l user hostname; env x='() { :;}; echo vulnerable' bash -c "echo this is a test"; echo
user#server's password:
server
this is a test
$
But I need a better formatted list when running on more than one system, so I attempt to let perl reformat the data:
$ ./ssh.expect server | perl -e '$dump = <>; $dump = <>; chomp($line1 = <>); chomp($line2 = <>); $dump = <>; print "$line1:$line2\n";'
:this is a test
The server name is printed as if it ends with a \r. I don't think that it should. Do you agree? How can I get the system to not return to column 0 after printing the server name?
I can verify that both variables contain data by adding a newline to my print:
$ ./ssh.expect server | perl -e '$dump = <>; $dump = <>; chomp($line1 = <>); chomp($line2 = <>); $dump = <>; print "$line1\n:$line2\n";'
server
:this is a test
EDIT:
As commented, the following works.
./ssh.expect server | tr -d '\r' | perl -e '$dump = <>; $dump = <>; chomp($line1 = <>); chomp($line2 = <>); $dump = <>; print "$line1:$line2\n";'
server:this is a test
Shouldn't chomp make the tr redundant?
Expect uses a pseudo-TTY to communicate with the command that it spawns, which in this case is the ssh process. TTYs and PTYs by default will translate \n to \r\n (LF to CRLF) in output text. So the output of your expect script would contain CRLF sequences unless you took the effort to remove the CR's. You can see this by running expect interactively:
$ expect -c 'spawn echo foo; interact' | od -a
0000000 s p a w n sp e c h o sp f o o cr nl
0000020 f o o cr nl
0000025 ^^--^^--note
TTY LF->CRLF conversion is controlled by the TTY "onlcr" flag. I played around with turning that flag off in the expect script but wasn't successful.
Perl's chomp command removes sequences of the perl input record separator (the $/ variable), which will just be \n on Unix systems. In other words, \r isn't special to chomp on unix by default. You could alter $/ in your perl script to make chomp remove the carriage returns, if you like. Or you could pipe the output of your expect script through tr, as discussed.
\r is used by terminals; the Enter key on your keyboard actually sends \r and the Linux kernel translates it to \n; \r\n is the line ending for output to the terminal and the Linux kernel translates \n to it on output. It's possible to disable those translations, putting the terminal into 'raw mode', in which the program can talk directly to the terminal (see http://www.linusakesson.net/programming/tty/). This is needed for full-screen programs, such as vi, nethack, etc.
My guess is that ssh is putting the terminal into raw mode on the client (your end) so the kernel on the server end can do the \n -> \r\n translation on output. This is what allows you to run vi under ssh, but it means that when you run ssh under a program need to pretend to be a terminal, which means sending \r as EOL on output and reading \r\n as EOL on input.
Related
I am wanting to pass a string variable in a ssh command. You can see in the code below I ssh to a server then cd to a directory that I pass a variable to. (cd $orig)
The variable is pulled from a file that I read in and put into an array.
I think that is where my error is because there might be unwanted hidden characters after I used the split command to read in from the file.
Here is the error I get:
ksh: /OnSight/jetplan/scripts/release/jscripts^M: not found
Can't open perl script "AddAlias.pl": No such file or directory
/OnSight/users/onsadm
SSHing to densbp53
/OnSight//scripts/release/jscripts
It can't find my script because the CD to the folder fails.
Sometimes the error says that 'end of file' can't be found. Like I'm doing a CD command with a EOF hidden symbol.
And here is the code:
for(my $j=0; $j < $#servName+1; $j++)
{
print "\nSSHing to $servName[$j]\n\n";
my $orig = $scriptfileLoc[$j];
#my $chopped = chop($orig);
chop($orig);
chomp($orig);
print ("\n$orig\n");
$sshstart = `ssh $servName[$j] "cd $orig; pwd; perl AddAlias.pl $aliasName $aliasCommand $addperl $servProfileLoc[$j]"`;
print $sshstart;
}
It outputs the $orig variable and it looks fine after the chop and chomp. (Which I've done both by themselves and still got the same error) So I pass it in my SSH command and it doesnt work.
I have a server file that holds all the server information, and yes it looks repetative I know.
densbp40:/export/home/.profile:/OnSight/scripts/release/jscripts
densbp41:/export/home/.profile:/OnSight/scripts/release/jscripts
densbp42:/export/home/.profile:/OnSight/scripts/release/jscripts
densbp43:/export/home/.profile:/OnSight/scripts/release/jscripts
densbp50:/export/home/.profile:/OnSight/scripts/release/jscripts
densbp51:/export/home/.profile:/OnSight/scripts/release/jscripts
densbp52:/export/home/.profile:/OnSight/scripts/release/jscripts
densbp53:/export/home/.profile:/OnSight/scripts/release/jscripts
densbp60:/export/home/.profile:/OnSight/scripts/release/jscripts
densbp61:/export/home/.profile:/OnSight/scripts/release/jscripts
densbp62:/export/home/.profile:/OnSight/scripts/release/jscripts
tulsbp40:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
tulsbp41:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
tulsbp42:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
tulsbp43:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
tulsbp50:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
tulsbp51:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
tulsbp52:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
tulsbp53:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
densbcp1:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
densbcp2:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
densmsv1:/OnSight/.profile:/OnSight/scripts/jscripts
denamdp1:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
denamap1:/OnSight/users/profile:/OnSight/scripts/release/jscripts
denamap2:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
denfpev1:/OnSight/users/.profile:/OnSight/scripts/release/jscripts
This script asks the user to choose to send a file to ALL servers or just one.
To remove CR (^M) from the end of lines, use the following regex:
$orig =~ s/\r$//gm;
Anchoring at the line end guarantees that any other carriage return characters are not removed from your input. (You probably don't them there either, but to normalize line endings, it's better to not touch other characters).
g enables global matches (not only the first) and m enables multiline mode, so that $ matches the end of each line in a multiline string, not only the end of the string.
"^M" is carriage return a.k.a "\r". Use regex to remove it:
$orig =~ s/\r//g;
For personal development and projects I work on, we use four spaces instead of tabs.
However, I need to use a heredoc, and I can't do so without breaking the indention flow.
The only working way to do this I can think of would be this:
usage() {
cat << ' EOF' | sed -e 's/^ //';
Hello, this is a cool program.
This should get unindented.
This code should stay indented:
something() {
echo It works, yo!;
}
That's all.
EOF
}
Is there a better way to do this?
Let me know if this belongs on the Unix/Linux Stack Exchange instead.
(If you are using bash 4, scroll to the end for what I think is the best combination of pure shell and readability.)
For heredocs, using tabs is not a matter of preference or style; it's how the language is defined.
usage () {
⟶# Lines between EOF are each indented with the same number of tabs
⟶# Spaces can follow the tabs for in-document indentation
⟶cat <<-EOF
⟶⟶Hello, this is a cool program.
⟶⟶This should get unindented.
⟶⟶This code should stay indented:
⟶⟶ something() {
⟶⟶ echo It works, yo!;
⟶⟶ }
⟶⟶That's all.
⟶EOF
}
Another option is to avoid a here document altogether, at the cost of having to use more quotes and line continuations:
usage () {
printf '%s\n' \
"Hello, this is a cool program." \
"This should get unindented." \
"This code should stay indented:" \
" something() {" \
" echo It works, yo!" \
" }" \
"That's all."
}
If you are willing to forego POSIX compatibility, you can use an array to avoid the explicit line continuations:
usage () {
message=(
"Hello, this is a cool program."
"This should get unindented."
"This code should stay indented:"
" something() {"
" echo It works, yo!"
" }"
"That's all."
)
printf '%s\n' "${message[#]}"
}
The following uses a here document again, but this time with bash 4's readarray command to populate an array. Parameter expansion takes care of removing a fixed number of spaces from the beginning of each lie.
usage () {
# No tabs necessary!
readarray message <<' EOF'
Hello, this is a cool program.
This should get unindented.
This code should stay indented:
something() {
echo It works, yo!;
}
That's all.
EOF
# Each line is indented an extra 8 spaces, so strip them
printf '%s' "${message[#]# }"
}
One last variation: you can use an extended pattern to simplify the parameter expansion. Instead of having to count how many spaces are used for indentation, simply end the indentation with a chosen non-space character, then match the fixed prefix. I use : . (The space following
the colon is for readability; it can be dropped with a minor change to the prefix pattern.)
(Also, as an aside, one drawback to your very nice trick of using a here-doc delimiter that starts with whitespace is that it prevents you from performing expansions inside the here-doc. If you wanted to do so, you'd have to either leave the delimiter unindented, or make one minor exception to your no-tab rule and use <<-EOF and a tab-indented closing delimiter.)
usage () {
# No tabs necessary!
closing="That's all"
readarray message <<EOF
: Hello, this is a cool program.
: This should get unindented.
: This code should stay indented:
: something() {
: echo It works, yo!;
: }
: $closing
EOF
shopt -s extglob
printf '%s' "${message[#]#+( ): }"
shopt -u extglob
}
geta() {
local _ref=$1
local -a _lines
local _i
local _leading_whitespace
local _len
IFS=$'\n' read -rd '' -a _lines ||:
_leading_whitespace=${_lines[0]%%[^[:space:]]*}
_len=${#_leading_whitespace}
for _i in "${!_lines[#]}"; do
printf -v "$_ref"[$_i] '%s' "${_lines[$_i]:$_len}"
done
}
gets() {
local _ref=$1
local -a _result
local IFS
geta _result
IFS=$'\n'
printf -v "$_ref" '%s' "${_result[*]}"
}
This is a slightly different approach which requires Bash 4.1 due to printf's assigning to array elements. (for prior versions, substitute the geta function below). It deals with arbitrary leading whitespace, not just a predetermined amount.
The first function, geta, reads from stdin, strips leading whitespace and returns the result in the array whose name was passed in.
The second, gets, does the same thing as geta but returns a single string with newlines intact (except the last).
If you pass in the name of an existing variable to geta, make sure it is already empty.
Invoke geta like so:
$ geta hello <<'EOS'
> hello
> there
>EOS
$ declare -p hello
declare -a hello='([0]="hello" [1]="there")'
gets:
$ unset -v hello
$ gets hello <<'EOS'
> hello
> there
> EOS
$ declare -p hello
declare -- hello="hello
there"
This approach should work for any combination of leading whitespace characters, so long as they are the same characters for all subsequent lines. The function strips the same number of characters from the front of each line, based on the number of leading whitespace characters in the first line.
The reason all the variables start with underscore is to minimize the chance of a name collision with the passed array name. You might want to rewrite this to prefix them with something even less likely to collide.
To use in OP's function:
gets usage_message <<'EOS'
Hello, this is a cool program.
This should get unindented.
This code should stay indented:
something() {
echo It works, yo!;
}
That's all.
EOS
usage() {
printf '%s\n' "$usage_message"
}
As mentioned, for Bash older than 4.1:
geta() {
local _ref=$1
local -a _lines
local _i
local _leading_whitespace
local _len
IFS=$'\n' read -rd '' -a _lines ||:
_leading_whitespace=${_lines[0]%%[^[:space:]]*}
_len=${#_leading_whitespace}
for _i in "${!_lines[#]}"; do
eval "$(printf '%s+=( "%s" )' "$_ref" "${_lines[$_i]:$_len}")"
done
}
Trying to execute remotely a bunch of commands in a perl script
This looks like that :
$CMD1 = "/usr/sbin/mminfo -av -q \"savetime>'-1 day 18:00:00',savetime<'17:59:59'\" -r \"ssid,totalsize,nfiles,pool\"|grep \"xxxxx\"|/usr/bin/awk '!seen[\$1]++'";
print Dumper $CMD1;
$CMD = "/usr/bin/ssh xxxx\#$SRV \'$CMD1\' 2>&1";
print Dumper $CMD;
But I still have problem with the $1 in the awk command, It seems to be cancelled when running.
What I can see :
$VAR1 = '/usr/sbin/mminfo -av -q "savetime>\'-1 day 18:00:00\',savetime<\'17:59:59\'" -r "ssid,totalsize,nfiles,pool"|grep "xxxxxx"|/usr/bin/awk \'!seen[$1]++\'';
$VAR1 = '/usr/bin/ssh xxxxx#\'xxxxxx\' \'/usr/sbin/mminfo -av -q "savetime>\'-1 day 18:00:00\',savetime<\'17:59:59\'" -r "ssid,totalsize,nfiles,pool"|grep "xxxxx"|/usr/bin/awk \'!seen[$1]++\'\' 2>&1';
So the '$1' of the awk command is passed correctly to the remote but when running :
#RESU = `$CMD`;
print Dumper #RESU;
I can see that my $1 is missing (or interpretated by the remote shell as a null value) :
$VAR1 = 'awk: ligne de commande:1: !seen[]++
';
$VAR2 = 'awk: ligne de commande:1: ^ syntax error
';
$VAR3 = 'awk: ligne de commande:1: error: expression indice non valide
';
I've tried many things like quoting or double-quoting the string, creating the string with perl 'qq' function, putting value of $CMD1 directly in $CMD and escaping quotes but no way.
And of course, my awk is piped to another awk (not provided here).
I don't want a solution which runs awk localy since I've millions lines returned from the 'mminfo' command.
Any clue (or a better way to do that !) ?
You might want to break it into smaller pieces for readability, and use the multi-arg invocation of system to avoid perl having to spawn a shell. The q() function goes a long way toward avoiding quoting hell.
$mminfo = q{/usr/sbin/mminfo -av -q "savetime>'-1 day 18:00:00',savetime<'17:59:59'" -r "ssid,totalsize,nfiles,pool"};
$awk = q{/usr/bin/awk '/xxxxx/ && !seen[$1]++');
print Dumper [$mminfo, $awk];
#cmd = ( "/usr/bin/ssh", "xxxx\#$SRV", "$mminfo | $awk" );
print Dumper \#cmd;
system #cmd;
Even if you can not use modules in your final environment, you may be able to use them in your local machine. In that case you can use them to quote the command programmatically and then just copy and paste the quoted string into the script you are developing. For instance:
use strict;
use warnings;
use Net::OpenSSH;
my $quoted_cmd1 = Net::OpenSSH->shell_quote('/usr/sbin/mminfo', '-av',
-q => q(savetime>'-1 day 18:00:00',savetime<'17:59:59'),
-r => 'ssid,totalsize,nfiles,pool',
\\'|',
'grep', 'xxxxx',
\\'|',
'/usr/bin/awk', '!seen[$1]++');
my $SRV = "foo";
my $quoted_cmd = Net::OpenSSH->shell_quote('/usr/bin/ssh', "xxxx\#$SRV",
$quoted_cmd1,
\\'2>&1');
print "$quoted_cmd\n";
Which outputs...
/usr/bin/ssh xxxx#foo '/usr/sbin/mminfo -av -q '\''savetime>'\''\'"''"'-1 day 18:00:00'\''\'"''"',savetime<'\''\'\''17:59:59\'\'' -r ssid,totalsize,nfiles,pool | grep xxxxx | /usr/bin/awk '\''!seen[$1]++'\' 2>&1
I don't have much experience with perl, and would appreciate any/all feedback....
[Before I start: I do not have access/authority to change the existing perl scripts.]
I run a couple perl scripts several times a day, but I would like to begin capturing their output in a file.
The first perl script does not take any arguments, and I'm able to "tee" its output without issue:
/asdf/loc1/rebuild-stuff.pl 2>&1 | tee $mytmpfile1
The second perl script hangs with this command:
/asdf/loc1/create-site.pl --record=${newsite} 2>&1 | tee $mytmpfile2
FYI, the following command does NOT hang:
/asdf/loc1/create-site.pl --record=${newsite} 2>&1
I'm wondering if /asdf/loc1/create-site.pl is trying to process the | tee $mytmpfile2 as additional command-line arguments? I'm not permitted to share the entire script, but here's the beginning of its main routine:
...
my $fullpath = $0;
$0 =~ s%.*/%%;
# Parse command-line options.
...
Getopt::Long::config ('no_ignore_case','bundling');
GetOptions ('h|help' => \$help,
'n|dry-run|just-print' => \$preview,
'q|quiet|no-mail' => \$quiet,
'r|record=s' => \$record,
'V|noverify' => \$skipverify,
'v|version' => \$version) or exit 1;
...
Does the above code provide any clues? Other than modifying the script, do you have any tips for allowing me to capture its output in a file?
It's not hanging. You are "suffering from buffering". Like most programs, Perl's STDOUT is buffered by default. Like most programs, Perl's STDOUT is flushed by a newline when connected to a terminal, and block buffered otherwise. When STDOUT isn't connected to a terminal, you won't get any output until 4 KiB or 8 KiB of output is accumulated (depending on your version of Perl) or the program exits.
You could add $| = 1; to the script to disable buffering for STDOUT. If your program ends with a true value or exits using exit, you can do that without changing the .pl file. Simply use the following wrapper:
perl -e'
$| = 1;
$0 = shift;
do($0);
my $e = $# || $! || "$0 didn\x27t return a true value\n";
die($e) if $e;
' -- prog args | ...
Or you could fool the program into thinking it's connected to a terminal using unbuffer.
unbuffer prog args | ...
I'm looking for a standard tool capable of taking all of its arguments and turning it into a single string suitable for use as multiple arguments in an automatically generated bash/sh/zsh script. Such a command is extremely useful in various disciplines of script-fu. An example of its usage:
% shsafe 'A big \nasty string '\'' $HOME $PATH' 'another string \\'
'A big \nasty string '\'' $HOME $PATH' 'another string \\'
Using it in another script:
% sshc host rm 'file/with spaces and $special chars'
where sshc contains
#!/bin/bash
# usage: sshc host command [arg ...]
# Escapes its arguments so that the command may contain special
# characters. Assumes the remote shell is sh-like.
host=$1
shift
exec ssh "$host" "$(shsafe "$#")"
Another example:
#!/bin/bash
# Run multiple commands in a single sudo session. The arguments of
# this script are passed as arguments to the first command. Useful if
# you don't want to have to type the password for both commands and
# the first one takes a while to run.
sudo bash -c "pacman -Syu $(shsafe "$#") && find /etc -name '*.pacnew'"
I couldn't find a suitable solution to this problem in the pre-existing commands, so I made up my own, called shsafe. It uses the fact that single quotes, '', turn off absolutely all shell expansion, except for ' itself.
shsafe:
#!/usr/bin/env python
from sys import *
n = len(argv)
if n == 1:
exit(0)
i = 1
while True:
stdout.write("'" + argv[i].replace("'", "'\\''") + "'")
i += 1
if i == n:
break
stdout.write(' ')
stdout.write('\n')
Is there any standard tool capable of doing this to its arguments?
Note that the printf command with a format string consisting of just the %q formatter is not good enough for this, because it won't keep multiple arguments separated:
% printf %q arg1 arg2
arg1arg2
I did eventually figure out a decent way of doing this:
% printf "$'%q' " 'crazy string \ $HOME' 'another\ string'
$'crazy\ string\ \\\ \$HOME' $'another\\\ string'
It's a little error prone what with the quotes everywhere, so it's not ideal, IMO, but it's a solid solution that should work anywhere. If it's being used a lot, you could always turn it into a shell function:
shsafe () {
printf "$'%q' " "$#"
}