Variables with ssh and awk with perl - linux

Trying to execute remotely a bunch of commands in a perl script
This looks like that :
$CMD1 = "/usr/sbin/mminfo -av -q \"savetime>'-1 day 18:00:00',savetime<'17:59:59'\" -r \"ssid,totalsize,nfiles,pool\"|grep \"xxxxx\"|/usr/bin/awk '!seen[\$1]++'";
print Dumper $CMD1;
$CMD = "/usr/bin/ssh xxxx\#$SRV \'$CMD1\' 2>&1";
print Dumper $CMD;
But I still have problem with the $1 in the awk command, It seems to be cancelled when running.
What I can see :
$VAR1 = '/usr/sbin/mminfo -av -q "savetime>\'-1 day 18:00:00\',savetime<\'17:59:59\'" -r "ssid,totalsize,nfiles,pool"|grep "xxxxxx"|/usr/bin/awk \'!seen[$1]++\'';
$VAR1 = '/usr/bin/ssh xxxxx#\'xxxxxx\' \'/usr/sbin/mminfo -av -q "savetime>\'-1 day 18:00:00\',savetime<\'17:59:59\'" -r "ssid,totalsize,nfiles,pool"|grep "xxxxx"|/usr/bin/awk \'!seen[$1]++\'\' 2>&1';
So the '$1' of the awk command is passed correctly to the remote but when running :
#RESU = `$CMD`;
print Dumper #RESU;
I can see that my $1 is missing (or interpretated by the remote shell as a null value) :
$VAR1 = 'awk: ligne de commande:1: !seen[]++
';
$VAR2 = 'awk: ligne de commande:1: ^ syntax error
';
$VAR3 = 'awk: ligne de commande:1: error: expression indice non valide
';
I've tried many things like quoting or double-quoting the string, creating the string with perl 'qq' function, putting value of $CMD1 directly in $CMD and escaping quotes but no way.
And of course, my awk is piped to another awk (not provided here).
I don't want a solution which runs awk localy since I've millions lines returned from the 'mminfo' command.
Any clue (or a better way to do that !) ?

You might want to break it into smaller pieces for readability, and use the multi-arg invocation of system to avoid perl having to spawn a shell. The q() function goes a long way toward avoiding quoting hell.
$mminfo = q{/usr/sbin/mminfo -av -q "savetime>'-1 day 18:00:00',savetime<'17:59:59'" -r "ssid,totalsize,nfiles,pool"};
$awk = q{/usr/bin/awk '/xxxxx/ && !seen[$1]++');
print Dumper [$mminfo, $awk];
#cmd = ( "/usr/bin/ssh", "xxxx\#$SRV", "$mminfo | $awk" );
print Dumper \#cmd;
system #cmd;

Even if you can not use modules in your final environment, you may be able to use them in your local machine. In that case you can use them to quote the command programmatically and then just copy and paste the quoted string into the script you are developing. For instance:
use strict;
use warnings;
use Net::OpenSSH;
my $quoted_cmd1 = Net::OpenSSH->shell_quote('/usr/sbin/mminfo', '-av',
-q => q(savetime>'-1 day 18:00:00',savetime<'17:59:59'),
-r => 'ssid,totalsize,nfiles,pool',
\\'|',
'grep', 'xxxxx',
\\'|',
'/usr/bin/awk', '!seen[$1]++');
my $SRV = "foo";
my $quoted_cmd = Net::OpenSSH->shell_quote('/usr/bin/ssh', "xxxx\#$SRV",
$quoted_cmd1,
\\'2>&1');
print "$quoted_cmd\n";
Which outputs...
/usr/bin/ssh xxxx#foo '/usr/sbin/mminfo -av -q '\''savetime>'\''\'"''"'-1 day 18:00:00'\''\'"''"',savetime<'\''\'\''17:59:59\'\'' -r ssid,totalsize,nfiles,pool | grep xxxxx | /usr/bin/awk '\''!seen[$1]++'\' 2>&1

Related

A strange error in shell script while combining two string variables

I will post my script here
#!/bin/tcsh
echo 'Running'
set fileN = '2021-02-07-0448-04S.JKH_RR.SAC'
set fileE = '2021-02-07-0448-04S.JKH_RR_BHE.SAC'
set compR=BHR
set compT=BHT
set compR_name=BHR.SAC
set compT_name=BHT.SAC
set fileN_rot = `echo $fileN | awk '{split($0,a,".SAC"); print a[1]}'`
set fileE_rot = `echo $fileE | awk '{split($0,a,".SAC"); print a[1]}'`
echo 'output1'
echo $fileN
echo $fileE
echo 'output2'
echo $fileN_rot
echo $fileE_rot
echo 'output3'
echo $fileE_rot-$compR_name
echo $fileN_rot-$compT_name
The output is:
Running
output1
2021-02-07-0448-04S.JKH_RR_BHN.SAC 2021-02-07-0448-04S.JKH_RR_BHE.SAC
output2
2021-02-07-0448-04S.JKH_RR_BHN
2021-02-07-0448-04S.JKH_RR_BHE
output3
2021-02-07-0448-04S.JKH_RR_BHN
-BHR.SAC
2021-02-07-0448-04S.JKH_RR_BHE-BHT.SAC
echo $fileE_rot-$compR_name giving wrong output.
Here the out is copy-pasted from the output file,so -BHR.SAC showing in new line.
But in shell terminal it is showing -BHR.SAC07-0448-04S.JKH_RR_BHN.
I find it strange.
Looks like you have some control chars in your strings. Run cat -Ev script to see them and if you see ^Ms in the output then read Why does my tool output overwrite itself and how do I fix it? for how to deal with them.
Don't write scripts in [t]csh, though, as it wasn't designed for that. Writing a script in csh is like digging a hole with a toothbrush - sure you CAN kinda get there in the end but there are better alternatives. See https://www.google.com/search?q=google+csh+why+not.
Having said that, it's not obvious why you're trying to manipulate text in any shell. Shells exist to manipulate (create/destroy) files and processes and sequence calls to tools. The people who invented shell also invented tools such as awk for shell to call when appropriate to manipulate text. So, here is how to really write a shell script to do what you want (the shell part is to call awk to manipulate the text):
$ cat tst.sh
#!/usr/bin/env bash
awk '
BEGIN {
print "Running"
fileN = "2021-02-07-0448-04S.JKH_RR.SAC"
fileE = "2021-02-07-0448-04S.JKH_RR_BHE.SAC"
compR = "BHR"
compT = "BHT"
compR_name = "BHR.SAC"
compT_name = "BHT.SAC"
fileN_rot = fileN
sub(/\.SAC$/,"",fileN_rot)
fileE_rot = fileE
sub(/\.SAC$/,"",fileE_rot)
print "output1"
print fileN
print fileE
print "output2"
print fileN_rot
print fileE_rot
print "output3"
print fileE_rot "-" compR_name
print fileN_rot "-" compT_name
}
'
$ ./tst.sh
Running
output1
2021-02-07-0448-04S.JKH_RR.SAC
2021-02-07-0448-04S.JKH_RR_BHE.SAC
output2
2021-02-07-0448-04S.JKH_RR
2021-02-07-0448-04S.JKH_RR_BHE
output3
2021-02-07-0448-04S.JKH_RR_BHE-BHR.SAC
2021-02-07-0448-04S.JKH_RR-BHT.SAC
or if there really was some reason to want to do it directly in a shell (e.g. this code is in some loop manipulating files named based on these variables) then:
$ cat tst.sh
#!/usr/bin/env bash
fileN='2021-02-07-0448-04S.JKH_RR.SAC'
fileE='2021-02-07-0448-04S.JKH_RR_BHE.SAC'
compR='BHR'
compT='BHT'
compR_name='BHR.SAC'
compT_name='BHT.SAC'
fileN_rot="${fileN%*.SAC}"
fileE_rot="${fileE%*.SAC}"
echo 'output1'
echo "$fileN"
echo "$fileE"
echo 'output2'
echo "$fileN_rot"
echo "$fileE_rot"
echo 'output3'
echo "${fileE_rot}-${compR_name}"
echo "${fileN_rot}-${compT_name}"
$ ./tst.sh
output1
2021-02-07-0448-04S.JKH_RR.SAC
2021-02-07-0448-04S.JKH_RR_BHE.SAC
output2
2021-02-07-0448-04S.JKH_RR
2021-02-07-0448-04S.JKH_RR_BHE
output3
2021-02-07-0448-04S.JKH_RR_BHE-BHR.SAC
2021-02-07-0448-04S.JKH_RR-BHT.SAC

Inserting the date into a "MAIL" message body in a BASH script

I have a relatively simple BASH script to send mail from my Raspberry Pi. The first argument is the Subject line and the second is a string of data files to be attached.
It is basically working when I specify the message body as a file (line 6). But if I try to create a text sting containing the date as the message body it fails (line7). Here is my script:
#!/bin/bash
#echo $2
# To
TO="me#hotmail.com"
# Message
MESSAGE="output/MessageBody.txt"
MESSAGEx="Midnight `date '+%Y-%m-%d %H:%M:%S %Z'` Pi report"
echo $MESSAGE
echo $MESSAGEx
temp=$(echo $2 | tr ";" "\n")
declare -a attargs
for att in $temp; do
attargs+=( "-A" "$att" )
done
# Sending email using /bin/mail
/usr/bin/mail -s "$1" "$TO" ${attargs[#]} < $MESSAGEx
Here is the output from this command
/usr/pgms/sendtome.sh "test message" "/mnt/usbdrive/output/JSONstart.txt;/mnt/usbdrive/output/Outback_error.log;/mnt/usbdrive/output/OutbackReaderPrint.txt"
when I specify MESSAGEx as the message body:
/mnt/usbdrive/output/MessageBody.txt
Midnight 2019-08-14 07:40:31 MDT Pi report
/usr/pgms/sendtome.sh: line 22: $MESSAGEx: ambiguous redirect
If I use MESSAGE, ie the text file reference, it works.
How can it create a message body text paragraph which contains the date or some other item? Thanks....RDK
There's a number of issues here.
You should generally quote strings. Without quoting, the string after < is split (hence the error message) and the array you took so much care to collect will lose its purpose.
The thing after < needs to be the name of a file. In Bash you can use a here string <<<"$MESSAGEx" but the common and simple portable solution is to echo (or better printf) its value into a pipe.
You should prefer lower case for your private variable names, but this is mainly a stylistic recommendation. (There are reserved variables like PATH and SHELL which you really don't want to clobber; POSIX reserves upper case variable names for system use.)
Here's a refactoring which attempts to address these concerns.
#!/bin/bash
to="me#hotmail.com"
# Message
#msgfile="output/MessageBody.txt"
msgbody="Midnight `date '+%Y-%m-%d %H:%M:%S %Z'` Pi report"
#echo "$msgfile"
#echo "$msgbody"
declare -a attargs
for att in $(echo "$2" | tr ";" "\n"); do
attargs+=( "-A" "$att" )
done
/usr/bin/mail -s "$1" "${attargs[#]}" "$to"<<< "$msgbody"
Perhaps a better design would be to just shift the first argument and then use "$#" as the list of files to attach.

SED command inside a loop

Hello: I have a lot of files called test-MR3000-1.txt to test-MR4000-1.nt, where the number in the name changes by 100 (i.e. I have 11 files),
$ ls test-MR*
test-MR3000-1.nt test-MR3300-1.nt test-MR3600-1.nt test-MR3900-1.nt
test-MR3100-1.nt test-MR3400-1.nt test-MR3700-1.nt test-MR4000-1.nt
test-MR3200-1.nt test-MR3500-1.nt test-MR3800-1.nt
and also a file called resonancia.kumac which in a couple on lines contains the string XXXX.
$ head resonancia.kumac
close 0
hist/delete 0
vect/delete *
h/file 1 test-MRXXXX-1.nt
sigma MR=XXXX
I want to execute a bash file which substitutes the strig XXXX in a file by a set of numbers obtained from the command ls *MR* | cut -b 8-11.
I found a post in which there are some suggestions. I try my own code
for i in `ls *MR* | cut -b 8-11`; do
sed -e "s/XXXX/$i/" resonancia.kumac >> proof.kumac
done
however, in the substitution the numbers are surrounded by sigle qoutes (e.g. '3000').
Q: What should I do to avoid the single quote in the set of numbers? Thank you.
This is a reproducer for the environment described:
for ((i=3000; i<=4000; i+=100)); do
touch test-MR${i}-1.nt
done
cat >resonancia.kumac <<'EOF'
close 0
hist/delete 0
vect/delete *
h/file 1 test-MRXXXX-1.nt
sigma MR=XXXX
EOF
This is a script which will run inside that environment:
content="$(<resonancia.kumac)"
for f in *MR*; do
substring=${f:7:3}
echo "${content//XXXX/$substring}"
done >proof.kumac
...and the output looks like so:
close 0
hist/delete 0
vect/delete *
h/file 1 test-MR300-1.nt
sigma MR=300
There are no quotes anywhere in this output; the problem described is not reproduced.
or if it could be perl:
#!/usr/bin/perl
#ls = glob('*MR*');
open (FILE, 'resonancia.kumac') || die("not good\n");
#cont = <FILE>;
$f = shift(#ls);
$f =~ /test-MR([0-9]*)-1\.nt/;
$nr = $1;
#out = ();
foreach $l (#cont){
if($l =~ s/XXXX/$nr/){
$f = shift(#ls);
$f =~ /test-MR([0-9]*)-1\.nt/;
$nr = $1;
}
push #out, $l;
}
close FILE;
open FILE, '>resonancia.kumac' || die("not good\n");
print FILE #out;
That would replace the first XXXX with the first filename, what seemed to be the question before change.

String Concatenation error in Cygwin bash shell

I am having problems concatenate two strings in BASH (I am using Cygwin)
When I am doing it step by step in the cygwin window, it works.
i.e by defining dt=2012-12-31 and c=.txt explicitly and then concatenating in filename=${dt}${c}.
It doesn't seem to work when i am running it through my script where these variables are defined by cutting and assigning values from content of a file.
Though the variables are assigned with the same values as above, the concatenation in this case doesn't work.
instead of 2012-12-31.txt i am getting .txt-12-31 as result.
The code is:
for x in {0..11}
do
IFS=$'\n'
filename=date_list.txt
file=($(<"$filename"))
IFS=$'\t\n'
dt=${file[$x]}
echo $dt
for y in {0..85}
do
IFS=$'\n'
filename=SQL_Mnemonics.txt
file=($(<"$filename"))
IFS=$'\t\n'
Mn=${file[$y]}
for k in {3..502}
do
IFS=$'\n'
c=.txt
filename=${dt}${c}
file=($(<"$filename"))
IFS=$'\t\n'
echo ${file[$k]} > temp_file.txt
cusip=`cut -c11-19 temp_file.txt`
result=$(sh ctest.sh $Mn, $dt, $cusip)
echo "$result" > tmp1.txt
t1=`cut -c18-40 tmp1.txt`
echo $t1 | sed 's/[[:space:]]//g' > temp_file.txt
cat tst.txt | sed 's/-----//g' >> ForFame/${Mn}.${dt}.txt
done
done
done

Replacing environment variables in a properties file

In Linux, say I have the following file (e.g. conf.properties):
HOST_URL=http://$HOSTNAME
STD_CONFIG=http://$HOSTNAME/config
USER_CONFIG=http://$HOSTNAME/config/$unconfigured
I want to create another file with all the environment variables replaced...e.g. say the environment variable $HOSTNAME is 'myhost' and $unconfigured is not set, a script should produce the following output:
HOST_URL=http://myhost
STD_CONFIG=http://myhost/config
USER_CONFIG=http://myhost/config/
I was thinking this could be done in a simple one-liner with some sort of sed/awk magic, but I'm no expert and my searches have been in vein, so appreciate any help.
Edit:
I should mention that the file can really be any format text file, for example xml. I just want to replace anything that looks like an env variable with whatever is currently set in the environment.
This is what envsubst is for.
echo 'Hello $USER'
Hello $USER
echo 'Hello $USER' | envsubst
Hello malvineous
You would probably use it more like this though:
envsubst < input.txt > output.txt
envsubst seems to be part of GNU gettext.
sed 's/$HOSTNAME/myhost/g;s/$unconfigured//g' yourfile.txt > another_file.txt
update:
Based on updates in your question, this won't be a good solution.
update2 :
This is based on an answer to a related question. I've hacked at it (I'm unfamiliar with perl) to remove undefined vars.
perl -p -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg; s/\$\{([^}]+)\}//eg' yourfile.txt
Should work for any input text file, however you will need to define vars using the ${...} format which simplifies the string matching.
(rant regarding the evilness of eval moved to a separate post so as not to confuse readers)
"eval is evil"
This is not an answer, but a warning in response to using eval for this task. You really really really don't want to do that.
Exhibit 1: a malicious template file:
HOST_URL=http://$HOSTNAME
STD_CONFIG=http://$HOSTNAME/config
USER_CONFIG=http://$HOSTNAME/config/$unconfigured
&& cat /etc/redhat-release
An unsuspecting user:
[lsc#aphek]$ cat somefile | while read line; do echo $(eval echo `echo $line`); done
HOST_URL=http://xyz
STD_CONFIG=http://xyz/config
USER_CONFIG=http://xyz/config/
Red Hat Enterprise Linux WS release 4 (Nahant Update 9)
Note the last line!
Now, imagine the possibilities....
I'd do it like this:
# Set the $HOSTNAME and other variables
# Now evaluate the properties file as a shell script.
. config.properties
# Write the values
cat >somefile <<EOF
HOST_URL=$HOST_URL
STD_CONFIG=$STD_CONFIG
USER_CONFIG=$USER_CONFIG
EOF
Edit: Or this very nasty thing (I'm sure there's a better way)
for name in HOST_URL STD_CONFIG USER_CONFIG
echo "$name=$(eval echo `echo '$'$name`)" >>somefile
end
Thanks to #DarkDust I came up with this:
cat somefile | while read line; do echo $(eval echo `echo $line`); done > somefile.replaced
I used this oneliner to replace ${VARIABLE} style variables in a file:
TARGET_FILE=/etc/apache2/apache2.conf; for VARNAME in $(grep -P -o -e '\$\{\S+\}' ${TARGET_FILE} | sed -e 's|^\${||g' -e 's|}$||g' | sort -u); do sed -i "s|\${$(echo $VARNAME)}|${!VARNAME}|g" ${TARGET_FILE}; done
I'm pretty sure someone can do this in 1/3rd of the length using awk… feel challenged! ;)
Here is a snippet of Javascript that I like to have around for solving this exact problem:
// A Javascript version of envsubst for our builds
// Purpose: replace all ocurrences of ${VAR} with the equivalent var from the environment from stdin
var readline = require('readline');
var rl = readline.createInterface({
input: process.stdin,
output: process.stdout,
terminal: false
});
const environment = process.env;
rl.on('line', function(line) {
const newLine = line.replace(/\$\{([a-zA-Z0-9_]+)\}/g, function(_match, variable) {
const envVar = environment[variable];
return envVar ? envVar : '';
});
process.stdout.write(`${newLine}\n`);
});
Hopefully this helps somebody else.
Here's a short one-liner that uses python's curly brace formatting to safely do the magic:
contents=\"\"\"`cat $file`\"\"\"; python -c "import os;print $contents.format(**os.environ)"
avoids evil eval
allows outputting curly braces: use {{ instead of {
no need to specify vars explicitly when calling the script
For example, given properties file settings.properties:
# my properties file
someVar = {MY_ENV_VAR}
curlyBraceVar = has {{curly braces}}
Then, do the substitution with:
$ export MY_ENV_VAR="hello"
$ file=settings.properties
$ contents=\"\"\"`cat $file`\"\"\"; python -c "import os;print $contents.format(**os.environ)"
# my properties file
someVar = hello
curlyBraceVar = has {curly braces}
A script is here: https://raw.githubusercontent.com/aneilbaboo/machome/master/bin/substenv
if you have installed nodejs you can run
npx #utft/tt -e FOO=bar /path/to/input /path/to/output
or you can run it programmatically
https://github.com/utftufutukgyftryidytftuv/tt

Resources