Parse args passed from aws System Manager send_command - aws-ssm

I have a long running job that cannot finish within lambda 15 minutes limit. So, I decided to use an EC2 worker instance to run the job. The job needs to be kicked off from a lambda function. I am using the following Python code to send the command to EC2 instance.
ssm.send_command(
InstanceIds=['*****'],
DocumentName="AWS-RunShellScript",
Parameters={'commands': [f'/home/ssm-user/get_cert_attributes.sh --doc_id={doc_id}']})
Shell script is getting called. However, I am unable to parse the args --doc_id. I am using the below code block to parse the arg. doc_id is coming blank. Any help in this regard would be highly appreciated.
#!/bin/bash
while [ "${1:-}" != "" ]; do
case "$1" in
"-d" | "--doc_id")
shift
doc_id=$1
;;
esac
shift
done
echo $doc_id

I resolved the issue by creating my own ssm document:
schemaVersion: "2.2"
description: "Get document attributes"
parameters:
docid:
type: "String"
description: "Document id to be processed"
mainSteps:
- action: "aws:runShellScript"
name: "GetDocAttr"
inputs:
runCommand:
- "/home/ssm-user/get_cert_attributes.sh --doc_id {{docid}}"
On the shell script side, I had to use export doc_id to export the environment variable to use it in subsequent child sessions.
#!/bin/bash
while [ "${1:-}" != "" ]; do
case "$1" in
"-d" | "--doc_id")
shift
doc_id=$1
;;
esac
shift
done
echo $doc_id
export doc_id

I'm able to run a shell script with arguments like this. You don't need all the messaging and other information, but it's there just in case.
commands = ['sudo -u ec2-user ./p1_consolidate.sh 0 1 0']
instanceid = 'i-0780a999exxxdxxx'
ssmc = boto3.client('ssm')
response_send = ssmc.send_command(
DocumentName="AWS-RunShellScript",
Parameters={'commands': commands,
'workingDirectory': ['/home/ec2-user'],
'executionTimeout': ['14400']},
OutputS3BucketName='xxxxx-data-files-for-functions',
OutputS3KeyPrefix='ssm-outputfiles-automation/',
InstanceIds=[instanceid],
ServiceRoleArn='arn:aws:iam::xxxxx6657583:role/SNS-Publish-SSM-Statuses',
NotificationConfig={
'NotificationArn': 'arn:aws:sns:us-east-1:xxxxx6657583:your-sns',
'NotificationEvents': ['All'],
'NotificationType': 'Command'}
)
p1_consolidate.sh stored in /home/ec2-user/ directory
Just takes the three arguments sent via commands below. Then the python file runs with those arguments.
s=$1
e=$2
q=$3
nohup python /home/ec2-user/code/mypythonfile.py -s $s -e $e -q $q &

Related

Parsing long and short args in ksh using loop

I am trying to parse arguments in ksh. Can't do getopt for the same as in short options I have two/three characters. Currently I am using for loop. Its stupid but am unable to find something better.
Question: How do I set option+value as one unit in order to parse?
Also if eval set -- $option will help me then how do I use it? echo on option does not show the expected "--" at the end. Am I assuming something wrong?
I am thinking of using a variable to keep track of when an option is found but this method seems too confusing and unnecessary.
Thanks for your time and help.
Update 1:
Adding code as pointed out. Thanks to markp, Andre Gelinas and random down-voter in making this question better. Trying to execute the script as given in line 2 and 3 of code - or any other combination of short and long options passed together.
#!/bin/ksh
# bash script1.sh --one 123 --two 234 --three "some string"
# bash script1.sh -o 123 -t 234 -th "some string"
# the following creates problems for short options.
#options=$(getopt -o o:t:th: -l one:two:three: "--" "$#")
#Since the below `eval set -- "$options"` did not append "--" at the end
#eval set -- "$options"
for i in $#; do
options="$options $i"
done
options="$options --"
# TODO capture args into variables
Attempted code below TODO until now:
for i in $options; do
echo $i
done
Will be capturing the args using:
while true; do
case $1 in
--one|-o) shift; ONE=$1
;;
--two|-t) shift; TWO=$1
;;
--three|-th) shift; THREE=$1
;;
--) shift; break
;;
esac
done
Try something like this :
#!/bin/ksh
#Default value
ONE=123
TWO=456
# getopts configuration
USAGE="[-author?Andre Gelinas <andre.gelinas#foo.bar>]"
USAGE+="[-copyright?2018]"
USAGE+="[+NAME?TestGetOpts.sh]"
USAGE+="[+DESCRIPTION?Try out for GetOps]"
USAGE+="[o:one]#[one:=$ONE?First.]"
USAGE+="[s:second]#[second:=$TWO?Second.]"
USAGE+="[t:three]:[three?Third.]"
USAGE+=$'[+SEE ALSO?\aman\a(1), \aGetOpts\a(1)]'
while getopts "$USAGE" optchar ; do
case $optchar in
o) ONE=$OPTARG ;;
s) TWO=$OPTARG ;;
t) THREE=$OPTARG ;;
esac
done
print "ONE = "$ONE
print "TWO = "$TWO
print "THREE = "$THREE
You can use either --one or -o. Using --man or --help are also working. Also -o and -s are numeric only, but -t will take anything. Hope this help.

Write output of subprocess launched by a `screen` CLI Command to a log file?

I am launching a bunch of the same script (generate_records.php) into screens. I am doing this to easily parallelize the processes. I would like to write the output of each of the PHP processes to a log file using something like &> log_$i (StdOut an StdErr).
My shell scripting is weak sauce, and I can't get the syntax correct. I keep getting the output of the screen, which is empty.
Exmaple: launch_processes_in_screens.sh
max_record_id=300000000
# number of parallel processors to run
total_processors=10
# max staging companies per processor
(( num_records_per_processor = $max_record_id / $total_processors))
i=0
while [ $i -lt $total_processors ]
do
(( starting_id = $i * $num_records_per_processor + 1 ))
(( ending_id = $starting_id + $num_records_per_processor - 1 ))
printf "\n - Starting processor #%s starting at ID:%s and ending at ID: %s" "$i" "$starting_id" "$ending_id"
screen -d -m -S "process_$i" php generate_records.php "$starting_id" "$num_records_per_processor" "FALSE"
((i++))
done
If the only reason you're using screen is to launch many processes in parallel, you can avoid it entirely and use & to start them in the background:
php generate_records.php "$starting_id" "$num_records_per_processor" FALSE &
You may also be able to remove some code by using parallel.

Run cron job in non-silent mode?

I created a simple linux script that essentially calls sqlplus and puts the results in variable X. I then analyze X and determine whether or not I need to send out a syslog message.
The script works perfectly when I run it from the command line as "oracle"; however when I use crontab as "oracle" and add it to my job, X isn't getting filled.
I could be wrong, but I believe the issue is since cron runs things in silent mode, X isn't actually getting filled, but when I run it manually it is.
Here's my crontab -l result (as oracle):
0,30 * * * * /scripts/isOracleUp.sh syslog
Here's my full script:
#Created by: hatguy
#Created date: May 8, 2012
#File Attributes: Must be executable by "oracle"
#Description: This script is used to determine if Oracle is up
# and running. It does a simple select on dual to check this.
DATE=`date`
USER=$(whoami)
if [ "$USER" != "oracle" ]; then
#note: $0 is the full path of whatever script is being run.
echo "You must run this as oracle. Try \"su - oracle -c $0\" instead"
exit;
fi
X=`sqlplus -s '/ as sysdba'<<eof
set serveroutput on;
set feedback off;
set linesize 1000;
select count(*) as count_col from dual;
EXIT;
eof`
#This COULD be more elegant. The issue I'm having is that I can't figure out
#which hidden characters are getting fed into X, so instead what I did was
#check the string legth (26) and checked that COUNT_COL and 1 were where I
#expected.
if [ ${#X} -eq 26 ] && [ ${X:1:10} = "COUNT_COL" ] && [ ${X:24:3} = "1" ] ; then
echo "Connected"
#log to a text file that we checked and confirmed connection
if [ "$1" == "syslog" ]; then
echo "$DATE: Connected" >> /scripts/log/isOracleUp.log
fi
else
echo "Not Connected"
echo "Details: $X"
if [ "$1" == "syslog" ]; then
echo "Sending this to syslog"
echo "==========================================================" >> /scripts/log/isOracleUp.log
echo "$DATE: Disconnected" >> /scripts/log/isOracleUp.log
echo "Message from sqlplus: $X" >> /scripts/log/isOracleUp.log
/scripts/sendMessageToSyslog.sh "PROD Oracle is DOWN!!!"
/scripts/sendMessageToSyslog.sh "PROD Details: $X"
fi
fi
Here's output when run as oracle from terminal:
Wed May 9 10:03:07 MDT 2012: Disconnected
Message from sqlplus: select count(*) as count_col from dual
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
Here's my log output when run through oracle's crontab job:
Wed May 9 11:00:04 MDT 2012: Disconnected
Message from sqlplus:
And to syslog:
PROD Details:
PROD Oracle is DOWN!!!
Any help would be appreciated as I'm a new linux user and this is my first linux script.
Thanks!
My Oracle db skills are pretty limited but dont you need to set ORACLE_SID and ORACLE_HOME ?
Check these variables from the command lines and set these variables within cron and retry.

Makefile errors while executing in Linux

I have one requirement in my automation.
I need to pass values like 1, 2, 3 to MY_IMAGE in command line in Linux.
I had defines activities for all these inputs in other make file.
The code similar to below i wrote for my requirement. Issue was whenever I passes values like MY_IMAGE=1, MY_IMAGE=2, MY_IMAGE=3
it's printing only echo ACT_DO=XYZ;
It's not displaying the other info whenever I selected 2 or 3. Can anyone check and correct my code.
export MY_IMAGE
MY_IMAGE=$img_value;
if [ $img_value :="1" ]
then
echo ACT_DO=XYZ;
else
if [ $img_value :="2 ]
then
echo ACT_DO=ABC;
else
if [ $img_value :=3 ]
then
echo ACT_DO=ETC;
else
echo ""$img_value" is unsupported";
exit 1;
fi
fi
fi
Your code has a quote in the wrong place, and uses := which doesn't mean anything, as far as I know. It's also implemented confusingly.
Try this:
export MY_IMAGE
MY_IMAGE=$img_value
case "$img_value" in
1 ) echo ACT_DO=XYZ ;;
2 ) echo ACT_DO=ABC ;;
3 ) echo ACT_DO=ETC ;;
* )
echo "\"$img_value\" is unsupported"
exit 1
;;
esac
The first two lines are not required for this code, but I presume you wanted that for something else.

Is there a "goto" statement in bash?

Is there a "goto" statement in bash ? I know It is considered bad practice, but I need specifically "goto".
If you are using it to skip part of a large script for debugging (see Karl Nicoll's comment), then if false could be a good option (not sure if "false" is always available, for me it is in /bin/false):
# ... Code I want to run here ...
if false; then
# ... Code I want to skip here ...
fi
# ... I want to resume here ...
The difficulty comes in when it's time to rip out your debugging code. The "if false" construct is pretty straightforward and memorable, but how do you find the matching fi? If your editor allows you to block indent, you could indent the skipped block (then you'll want to put it back when you're done). Or a comment on the fi line, but it would have to be something you'll remember, which I suspect will be very programmer-dependent.
No, there is not; see ยง3.2.4 "Compound Commands" in the Bash Reference Manual for information about the control structures that do exist. In particular, note the mention of break and continue, which aren't as flexible as goto, but are more flexible in Bash than in some languages, and may help you achieve what you want. (Whatever it is that you want . . .)
It indeed may be useful for some debug or demonstration needs.
I found that Bob Copeland solution http://bobcopeland.com/blog/2012/10/goto-in-bash/ elegant:
#!/bin/bash
# include this boilerplate
function jumpto
{
label=$1
cmd=$(sed -n "/$label:/{:a;n;p;ba};" $0 | grep -v ':$')
eval "$cmd"
exit
}
start=${1:-"start"}
jumpto $start
start:
# your script goes here...
x=100
jumpto foo
mid:
x=101
echo "This is not printed!"
foo:
x=${x:-10}
echo x is $x
results in:
$ ./test.sh
x is 100
$ ./test.sh foo
x is 10
$ ./test.sh mid
This is not printed!
x is 101
You can use case in bash to simulate a goto:
#!/bin/bash
case bar in
foo)
echo foo
;&
bar)
echo bar
;&
*)
echo star
;;
esac
produces:
bar
star
If you're testing/debugging a bash script, and simply want to skip forwards past one or more sections of code, here is a very simple way to do it that is also very easy to find and remove later (unlike most of the methods described above).
#!/bin/bash
echo "Run this"
cat >/dev/null <<GOTO_1
echo "Don't run this"
GOTO_1
echo "Also run this"
cat >/dev/null <<GOTO_2
echo "Don't run this either"
GOTO_2
echo "Yet more code I want to run"
To put your script back to normal, just delete any lines with GOTO.
We can also prettify this solution, by adding a goto command as an alias:
#!/bin/bash
shopt -s expand_aliases
alias goto="cat >/dev/null <<"
goto GOTO_1
echo "Don't run this"
GOTO_1
echo "Run this"
goto GOTO_2
echo "Don't run this either"
GOTO_2
echo "All done"
Aliases don't usually work in bash scripts, so we need the shopt command to fix that.
If you want to be able to enable/disable your goto's, we need a little bit more:
#!/bin/bash
shopt -s expand_aliases
if [ -n "$DEBUG" ] ; then
alias goto="cat >/dev/null <<"
else
alias goto=":"
fi
goto '#GOTO_1'
echo "Don't run this"
#GOTO1
echo "Run this"
goto '#GOTO_2'
echo "Don't run this either"
#GOTO_2
echo "All done"
Then you can do export DEBUG=TRUE before running the script.
The labels are comments, so won't cause syntax errors if disable our goto's (by setting goto to the ':' no-op), but this means we need to quote them in our goto statements.
Whenever using any kind of goto solution, you need to be careful that the code you're jumping past doesn't set any variables that you rely on later - you may need to move those definitions to the top of your script, or just above one of your goto statements.
Although others have already clarified that there is no direct goto equivalent in bash (and provided the closest alternatives such as functions, loops, and break), I would like to illustrate how using a loop plus break can simulate a specific type of goto statement.
The situation where I find this the most useful is when I need to return to the beginning of a section of code if certain conditions are not met. In the example below, the while loop will run forever until ping stops dropping packets to a test IP.
#!/bin/bash
TestIP="8.8.8.8"
# Loop forever (until break is issued)
while true; do
# Do a simple test for Internet connectivity
PacketLoss=$(ping "$TestIP" -c 2 | grep -Eo "[0-9]+% packet loss" | grep -Eo "^[0-9]")
# Exit the loop if ping is no longer dropping packets
if [ "$PacketLoss" == 0 ]; then
echo "Connection restored"
break
else
echo "No connectivity"
fi
done
This solution had the following issues:
Indiscriminately removes all code lines ending in a :
Treats label: anywhere on a line as a label
Here's a fixed (shell-check clean and POSIX compatible) version:
#!/bin/sh
# GOTO for bash, based upon https://stackoverflow.com/a/31269848/5353461
goto() {
label=$1
cmd=$(sed -En "/^[[:space:]]*#[[:space:]]*$label:[[:space:]]*#/{:a;n;p;ba};" "$0")
eval "$cmd"
exit
}
start=${1:-start}
goto "$start" # GOTO start: by default
#start:# Comments can occur after labels
echo start
goto end
# skip: # Whitespace is allowed
echo this is usually skipped
# end: #
echo end
There is one more ability to achieve a desired results: command trap. It can be used to clean-up purposes for example.
There is no goto in bash.
Here is some dirty workaround using trap which jumps only backwards:)
#!/bin/bash -e
trap '
echo I am
sleep 1
echo here now.
' EXIT
echo foo
goto trap 2> /dev/null
echo bar
Output:
$ ./test.sh
foo
I am
here now.
This shouldn't be used in that way, but only for educational purposes. Here is why this works:
trap is using exception handling to achieve the change in code flow.
In this case the trap is catching anything that causes the script to EXIT. The command goto doesn't exist, and hence throws an error, which would ordinarily exit the script. This error is being caught with trap, and the 2>/dev/null hides the error message that would ordinarily be displayed.
This implementation of goto is obviously not reliable, since any non-existent command (or any other error, for that manner), would execute the same trap command. In particular, you cannot choose which label to go-to.
Basically in real scenario you don't need any goto statements, they're redundant as random calls to different places only make your code difficult to understand.
If your code is invoked many times, then consider to use loop and changing its workflow to use continue and break.
If your code repeats it-self, consider writing the function and calling it as many times as you want.
If your code needs to jump into specific section based on the variable value, then consider using case statement.
If you can separate your long code into smaller pieces, consider moving it into separate files and call them from the parent script.
I found out a way to do this using functions.
Say, for example, you have 3 choices: A, B, and C. A and Bexecute a command, but C gives you more info and takes you to the original prompt again. This can be done using functions.
Note that since the line containg function demoFunction is just setting up the function, you need to call demoFunction after that script so the function will actually run.
You can easily adapt this by writing multiple other functions and calling them if you need to "GOTO" another place in your shell script.
function demoFunction {
read -n1 -p "Pick a letter to run a command [A, B, or C for more info] " runCommand
case $runCommand in
a|A) printf "\n\tpwd being executed...\n" && pwd;;
b|B) printf "\n\tls being executed...\n" && ls;;
c|C) printf "\n\toption A runs pwd, option B runs ls\n" && demoFunction;;
esac
}
demoFunction
This is a small correction of the Judy Schmidt script put up by Hubbbitus.
Putting non-escaped labels in the script was problematic on the machine and caused it to crash. This was easy enough to resolve by adding # to escape the labels. Thanks to Alexej Magura and access_granted for their suggestions.
#!/bin/bash
# include this boilerplate
function goto {
label=$1
cmd=$(sed -n "/$#label#:/{:a;n;p;ba};" $0 | grep -v ':$')
eval "$cmd"
exit
}
start=${1:-"start"}
goto $start
#start#
echo "start"
goto bing
#boom#
echo boom
goto eof
#bang#
echo bang
goto boom
#bing#
echo bing
goto bang
#eof#
echo "the end mother-hugger..."
A simple searchable goto for the use of commenting out code blocks when debugging.
GOTO=false
if ${GOTO}; then
echo "GOTO failed"
...
fi # End of GOTO
echo "GOTO done"
Result is-> GOTO done
My idea for creating something like "goto" is to use select with case and assign a variable, which I then check in an if statement. Not perfect, but may help in some cases
Example:
#!/usr/bin/env bash
select goto in Ubuntu Debian Quit ; do
case $goto in
Ubuntu) { CHOICE="Ubuntu" ; break ; } ;;
Debian) { CHOICE="Debian" ; break ; } ;;
Quit) { echo "Bye" ; exit ; } ;;
*) { echo "Invalid selection, please try again..." ; } ;;
esac
done
if [ "$CHOICE" == "Ubuntu" ]; then
echo "I'm in Ubuntu"
fi
if [ "$CHOICE" == "Debian" ]; then
echo "I'm in Debian"
fi
Why don't anyone just use functions directly ?
BTW functions are easier to deal with than making a new thing
My style :
#!/bin/bash
# Your functions
function1 ()
{
commands
}
function2 ()
{
commands
}
:
:
functionn ()
{
commands
}
# Execute 1 to n in order
for i in {1..n}
do
function$i
done
# with conditions
for i in {1..n}
do
[ condition$i ] && function$i
done
# Random order
function1
functionn
function5
:
:
function3
Example for above style :
#!/bin/bash
# Your functions
function1 ()
{
echo "Task 1"
}
function2 ()
{
echo "Task 2"
}
function3 ()
{
echo "Task 3"
}
function1
function3
function2
Output :
Task 1
Task 3
Task 2
Drawbacks :
Script in an organized way.
Less problems and not prone to errors.
You can make function inside a existing function.
Move back and forth without any problems.

Resources