The parallel execution is working properly as expected. But i need to get the states of each file. Here I need to check it by using the if condition. This is just a sample and i have many scenarios like this in my code.
I'm a beginner at #help shell scripts & linux but actually i love linux now.
I'm expecting an alternative method or a help to resolve this. Thank you everyone.
( echo $FILE_PATH/import_csv_location_data_into_hive.sh; echo $FILE_PATH/import_csv_order_data_into_hive.sh; echo $FILE_PATH/import_csv_order_with_delivery_info_into_hive.sh; ) | parallel bash
wait
IMPORTED_LOCATIONS_STATUS=$? IMPORTED_ORDERS_STATUS=$? IMPORTED_ORDERS_WITH_DELIVERY_INFO_STATUS=$?
if [[ "$IMPORTED_LOCATIONS_STATUS" == "0" && "$IMPORTED_ORDERS_STATUS" == "0" && "$IMPORTED_ORDERS_WITH_DELIVERY_INFO_STATUS" == "0" ]];
then
echo "COMPLETED."
exit 0
else
STATUS="FAILED."
exit 1
fi
Sample code
enter image description here
(echo Your-Special-Header; \
echo whoami; \
echo cat nothere \
) | parallel --header : --results all.csv bash -c
The file nothere does not exist. Here is the csv output:
Seq,Host,Starttime,JobRuntime,Send,Receive,Exitval,Signal,Command,Your-Special-Header,Stdout,Stderr
1,:,1662823208.481,0.001,0,5,0,0,"bash -c whoami",whoami,"root
",
2,:,1662823208.481,0.003,0,0,1,0,"bash -c cat\ nothere","cat nothere",,"cat: nothere: No such file or directory
"
The 7th column is Exitval.
If you only need to know if all finished with no errors:
(
echo $FILE_PATH/import_csv_location_data_into_hive.sh;
echo $FILE_PATH/import_csv_order_data_into_hive.sh;
echo $FILE_PATH/import_csv_order_with_delivery_info_into_hive.sh;
) | parallel bash
if [[ "$?" == "0" ]]; then
echo "COMPLETED."
exit 0
else
STATUS="FAILED."
exit 1
fi
Shorter, but less readable IMO:
(
echo $FILE_PATH/import_csv_location_data_into_hive.sh;
echo $FILE_PATH/import_csv_order_data_into_hive.sh;
echo $FILE_PATH/import_csv_order_with_delivery_info_into_hive.sh;
) | if parallel bash ; then
echo "COMPLETED."
exit 0
else
STATUS="FAILED."
exit 1
fi
Related
I have a pretty simple bash script that coordinates running a couple python scripts. What I am having trouble figuring out is why after running the bash script (. bash_script.sh), the terminal hangs. I can't ctrl+c, ctrl+z or do anything except restart the SSH session. All I see is just a blinking cursor. Checking all the log files indicates a 0 status exit code with no errors in the scripts themselves. Running ps aux | grep bash_script.sh does not show any anything running either. Is there anyway to debug this?
#!/bin/bash
exec >> <DIR>/logfile.log 2>&1
script_message () {
status_arg=$1
if [[ $status_arg = "pass" ]]; then
printf "Script Done\n"
printf '=%.0s' {1..50}
printf "\n"
elif [[ $status_arg = "fail" ]]; then
printf "Script Failed\n"
printf '=%.0s' {1..50}
printf "\n"
else
:
fi
}
current_date=$(date '+%Y-%m-%d %H:%M:%S')
day=$(date +%u)
hour=$(date +%H)
printf "RUN DATE: $current_date\n"
# activate virtual env
source /<VENV DIR/bin/activate>
python <PYTHON SCRIPT>.py >> <DIR>/logfile2.log 2>&1
retVal=$?
if [[ $retVal -eq 0 && $day -eq 4 ]]; then
python <PYTHON SCRIPT 2>.py >> <DIR>/logfile3.log 2>&1
script_message pass
elif [[ $retVal -eq 0 ]]; then
script_message pass
else
#:
script_message fail
fi
echo $?
i have a question about simple shell script.
this is the source code of rand.sh below
#!/bin/bash
n=$(( RANDOM % 100 ))
if [[ n -eq 42 ]]; then
echo "Something went wrong"
>&2 echo "The error was using magic numbers"
exit 1
fi
echo "Everything went accrding to plan"
and i'm going to make a new shell script, let me call it quiz.sh.
quiz.sh should loop until n==42. if n==42, save the stdout("Something went wrong") and stderr("The error was using magic numbers")
and it finally terminated with printing out those stdout,stderr and Total execution count.
here is my quiz.sh
#!/bin/bash
cnt=0
while [[ "${n}" -ne 42 ]]
do
(( cnt = "${cnt}"+1 ))
source ./rand.sh &> error.txt
done
cat error.txt
echo "${cnt}"
but this is not working. because of exit 1 in rand.sh, the program is terminated before executing cat and echo which is at the end two line.
how can i fix it?? please let me know!
I want to make happen cat error.txt and echo "${cnt}" as well
Run the loop in a subshell
(
while something; do
something
exit 1 # exits only from the subshell
done
)
Note: parent shell doesn't access/inherit child process environment. So cnt is going to be empty in parent shell. Transfer it some other way.
(
cnt=0
while ((n != 42)); do
((cnt++))
echo "$cnt" > cntfile.txt
# >& is deprecated
source myrand > error.txt 2>&1
done
)
cnt=$(<cntfile.txt)
cat error.txt
echo "$cnt"
Reference Bash manual command grouping.
As KamilCuk pointed out correctly, you should use $n instead of n.
Furthermore, I personally would add that using source ./rand.sh &> error.txt is kind of weird in this case. If you want to run it as a background process, use:
./rand.sh &> error.txt &
wait $! # $! is the pid
Otherwise, just make a function out of it:
#!/bin/bash
function myrand {
n=$(( RANDOM % 100 ))
if [[ n -eq 42 ]]; then
echo "Something went wrong"
>&2 echo "The error was using magic numbers"
return 1
fi
echo "Everything went accrding to plan"
return 0
}
cnt=0
while [[ "${n}" -ne 42 ]]
do
(( cnt = "${cnt}"+1 ))
myrand() &> error.txt
done
cat error.txt
echo "${cnt}"
p.s. code not tested, but I guess it works.
I want to check for file in directory if there then push it to ssh server checing server connection if file not there then try 3 times with each 1min interval and in between if it comes ( on 2nd attend for example) then try again to connect ssh and push. else check for 3 attempts and exit
Please check my below code it is halting after 1st attempt ( during 2nd attempt I am making file available)
#!/bin/sh
echo "OK, start pushing the Userdetails to COUPA now..."
cd /usr/App/ss/outbound/usrdtl/
n=0
until [ $n -ge 3 ] || [ ! -f /usr/App/ss/outbound/usrdtl/USERS_APPROVERS_*.csv ]
do
if [ -f /usr/App/ss/outbound/usrdtl/USERS_APPROVERS_*.csv ] ;
then
pushFiles()
else
n=$[$n+1]
sleep 60
echo " trying " $n "times "
fi
done
pushFiles()
{
echo "File present Now try SSH connection"
while [ $? -eq 0 ];
do
echo $(date);
scpg3 -v /usr/App/ss/outbound/usrdtl/USERS_APPROVERS_*.csv <sshHost>:/Incoming/Users/
if [ $? -eq 0 ]; then
echo "Successfull"
echo $(date);
echo "Successfull" >> /usr/App/ss/UserApproverDetails.log
exit 1;
else
echo $(date);
echo "Failed" >> /usr/App/ss/UserApproverDetails.log
echo "trying again to push file.."
scpg3 -v /usr/App/sg/outbound/usrdtl/USERS_APPROVERS_*.csv <ssh Host>:/Incoming/Users/
echo $(date);
exit 1;
fi
done
}
I've tried to simplify this code for you. I hope it helps:
#!/bin/bash
outdir="/usr/App/ss/outbound/usrdtl"
logfile="/usr/App/ss/UserApproverDetails.log"
file_prefix="USERS_APPROVERS_"
function push_files() {
echo "File present now try SSH connection"
local attempts=1
local retries=2
date
while [[ ${attempts} -lt ${retries} ]]; do
if scp ${outdir}/${file_prefix}*.csv <sshHost>:/Incoming/Users/ ; then
echo "Successful" | tee -a ${logfile}
date
exit 0
else
echo "Failed" >> ${logfile}
fi
attempts=$((attempts+1))
do
echo "scp failed twice" | tee -a ${logfile}
exit 2
}
echo "OK, start pushing the Userdetails to COUPA now..."
cd ${outdir}
attempts=1
retries=3
while [[ ${attempts} -lt ${retries} ]]; do
echo "looking for files...attempt ${attempts}"
if test -n "$(shopt -s nullglob; echo ${outdir}/${file_prefix}*.csv)"; then
push_files()
fi
attempts=$((attempts+1))
sleep 60
done
echo "Files were never found" | tee -a ${logfile}
exit 1
Look at this code and tell me how it's not doing what you're trying to do. The most complicated part here is the nullglob stuff, which is a handy trick to see if any file in a glob matches
Also, I generally used bashisms.
have some problem with shell script.
In our office we set up only few commands, that available for devs when they are trying ssh to server. It is configured with help of .ssh/authorized_keys file and available command for user there is bash script:
#!/bin/sh
if [[ $1 == "--help" ]]; then
cat <<"EOF"
This script has the purpose to let people remote execute certain commands without logging into the system.
For this they NEED to have a homedir on this system and uploaded their RSA public key to .ssh/authorized_keys (via ssh-copy-id)
Then you can alter that file and add some commands in front of their key eg :
command="/usr/bin/dev.sh",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty
The user will do the following : ssh testuser#server tail testserver.example.com/2017/01/01/user.log
EOF
exit 0;
fi
# set global variable
set $SSH_ORIGINAL_COMMAND
# set the syslog path where the files can be found
PATH="/opt/syslog/logs"
# strip ; or any other unwanted signs out of the command, this prevents them from breaking out of the setup command
if [[ $1 != "" ]]; then
COMMAND=$1
COMMAND=${COMMAND//[;\`]/}
fi
if [[ $2 != "" ]]; then
ARGU1=$2
ARGU1=${ARGU1//[;\`]/}
fi
if [[ $3 != "" ]]; then
ARGU2=$3
ARGU2=${ARGU2//[;\`]/}
fi
if [[ $4 != "" ]]; then
ARGU3=$4
ARGU3=${ARGU3//[;\`]/}
fi
# checking for the commands
case "$COMMAND" in
less)
ARGU2=${ARGU1//\.\./}
FILE=$PATH/$ARGU1
if [ ! -f $FILE ]; then
echo "File doesn't exist"
exit 1;
fi
#echo " --------------------------------- LESS $FILE"
/usr/bin/less $FILE
;;
grep)
if [[ $ARGU2 == "" ]]; then
echo "Pls give a filename"
exit 1
fi
if [[ $ARGU1 == "" ]]; then
echo "Pls give a string to search for"
exit 1
fi
ARGU2=${ARGU2//\.\./}
FILE=$PATH/$ARGU2
/usr/bin/logger -t restricted-command -- "------- $USER Executing grep $ARGU1 \"$ARGU2\" $FILE"
if [ ! -f $FILE ]; then
echo "File doesn't exist"
/usr/bin/logger -t restricted-command -- "$USER Executing $#"
exit 1;
fi
/bin/grep $ARGU1 $FILE
;;
tail)
if [[ $ARGU1 == "" ]]; then
echo "Pls give a filename"
exit 1
fi
ARGU1=${ARGU1//\.\./}
FILE=$PATH/$ARGU1
if [ ! -f $FILE ]; then
echo "File doesn't exist"
/usr/bin/logger -t restricted-command -- "$USER Executing $# ($FILE)"
exit 1;
fi
/usr/bin/tail -f $FILE
;;
cat)
ARGU2=${ARGU1//\.\./}
FILE=$PATH/$ARGU1
if [ ! -f $FILE ]; then
echo "File doesn't exist"
exit 1;
fi
/bin/cat $FILE
;;
help)
/bin/cat <<"EOF"
# less LOGNAME (eg less testserver.example.com/YYYY/MM/DD/logfile.log)
# grep [ARGUMENT] LOGNAME
# tail LOGNAME (eg tail testserver.example.com/YYYY/MM/DD/logfile.log)
# cat LOGNAME (eg cat testserver.example.com/YYYY/MM/DD/logfile.log)
In total the command looks like this : ssh user#testserver.example.com COMMAND [ARGUMENT] LOGFILE
EOF
/usr/bin/logger -t restricted-command -- "$USER HELP requested $#"
exit 1
;;
*)
/usr/bin/logger -s -t restricted-command -- "$USER Invalid command $#"
exit 1
;;
esac
/usr/bin/logger -t restricted-command -- "$USER Executing $#"
The problem is next:
when i try to exec some command, it takes only first argument, if i do recursion in files by using {n,n1,n2} - it doesn't work:
[testuser#local ~]$ ssh testuser#syslog.server less srv1838.example.com/2017/02/10/local1.log |grep 'srv2010' | wc -l
0
[testuser#local ~]$ ssh testuser#syslog.server less srv2010.example.com/2017/02/10/local1.log |grep 'srv2010' | wc -l
11591
[testuser#local ~]$ ssh testuser#syslog.server less srv{1838,2010}.example.com/2017/02/10/local1.log |grep 'srv2010' | wc -l
0
[testuser#local ~]$ ssh testuser#syslog.server less srv{2010,1838}.example.com/2017/02/21/local1.log |grep 'srv2010' | wc -l
11591
Could someone help me, how can i parse\count command arguments to make it work?
Thank you and have a nice day!
The number of arguments for a bash script would be $#. As a quick example:
#!/bin/bash
narg=$#
typeset -i i
i=1
while [ $i -le $narg ] ; do
echo " $# $i: $1"
shift
i=$i+1
done
gives, for bash tst.sh a b {c,d}
4 1: a
3 2: b
2 3: c
1 4: d
In your script, the command to execute (cat, less, ...) gets explicitly only the second argument to the script. If you want to read all arguments, you should do something like this (note: only a hint, removed all sorts of checks etc..)
command="$1"
shift
case $command in
(grep) pattern="$1"
shift
while [ $# -gt 0 ] ; do
grep "$pattern" "$1"
shift
done
;;
esac
note: added some quotes as comment suggested, but, being only a hint, you should carefully look at quoting and your checks in your own script.
Less command working now:
case "$COMMAND" in
less)
if [[ $ARGU1 == "" ]]; then
echo "Pls give a filename"
exit 1
fi
FILES_LIST=${#:2}
FILE=(${FILES_LIST//\.\./})
for v in "${FILE[#]}";do
v=${v//[;\']/}
if [ ! -f $v ]; then
echo "File doesn't exist"
fi
/usr/bin/less $PATH/$v
done;;
tail command works too with 2 and more files, but i can't execute tail -f command on two files unfortunately.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
Task 1:
If the Time is AM, print "It is Morning". If it is PM, print "It is not morning".
Task 2:
Given 2 arguments passed to the script.
Validate that 2 arguments have been submitted. If three arguments have not been provided, print "Must Supply 2 Arguments" and exit the script
Argument 1: Should be a directory (needs validated). If this does not exist, print "Directory: directory, does not exist" and exit the script
Argument 2: Should be a file (needs validated). If this does not exist, print "File: filename, does not exist" and exit the script.
If all arguments are valid, print "Given valid: filename and directory"
This is what I have so far
echo "James DuBois: 555555 - Task 1"
TIME=$(date "+%H")
if [ $TIME -lt 12 ]; then
echo "morning"
else
echo "not morning"
fi
echo "Task 2"
[ -d "$1" ] || exit
[ -d "$2" ] || exit
[ $# == 2 ] || exit
echo "arg1: $1"
echo "arg2: $2"
James, BASH is a wonderful, flexible shell. It has its warts, but if you need to do anything having to do with Linux system administration, etc.., you can do it in bash. Your tasks are to familiarize you with using conditional expressions (tests). There are tests for just about anything you need. That's why I pointed you to the CONDITIONAL EXPRESSIONS part of man bash.
Your second task requires the input of a filename so you can test it. I presume it is intended to be passed as an argument to your script (called positional parameters). Here is one way to approach the test. Note: I have interchanged output routines echo and printf intentionally for your benefit (printf being a bit more robust). Take a look at the following and let me know what questions you have:
#!/bin/bash
# My first script
#
# echo & printf are used at random below -- intentionally
#
[ -z $1 ] && { # validate at least 1 argument given on command line
printf "error: insufficient input. usage: %s filename\n" "${0##*/}"
exit 1
}
printf "\nJames DuBois: 5555555\n\n Task 1\n\n"
TIME=$(date "+%H")
## test for time of date: morning/not morning
if [ $TIME -lt 12 ]; then
printf " morning - time for coffee\n"
else
echo " not morning - time for scotch"
fi
echo -e "\n Task 2\n"
printf "Testing whether '%s' is a valid file.\n\n" "$1"
## test for file using compound commands
[ -f "$1" ] && echo -e " file found: '$1'\n" || printf " file not found: '%s'\n\n" "$1"
echo -e "Second test whether '$1' is a valid file.\n"
## test for file using if; then; else; fi
if [ -f "$1" ]; then
printf " file found: '%s'\n\n" "$1"
else
echo -e " file not found: '$1'\n"
fi
exit 0
Use/Output
$ bash ~/scr/tmp/stack/morningfile.sh
error: insufficient input. usage: morningfile.sh filename
$ bash ~/scr/tmp/stack/morningfile.sh mtrx_simple_dyn.c
James DuBois: 5555555
Task 1
not morning - time for scotch
Task 2
Testing whether 'mtrx_simple_dyn.c' is a valid file.
file found: 'mtrx_simple_dyn.c'
Second test whether 'mtrx_simple_dyn.c' is a valid file.
file found: 'mtrx_simple_dyn.c'
$ bash ~/scr/tmp/stack/morningfile.sh dog.c
James DuBois: 5555555
Task 1
not morning - time for scotch
Task 2
Testing whether 'dog.c' is a valid file.
file not found: 'dog.c'
Second test whether 'dog.c' is a valid file.
file not found: 'dog.c'
This will have some additional advantages while doing the same thing in task 2:
echo "Task 2"
[[ -d $1 && ! -L $1 && -f $2 && ! -L $2 ]] || exit
printf "%s\n" "arg1: $1" "arg2: $2"
It checks for symbolic links with the -L option.
[[ ]] has some advantages over [ ] like dealing with white spaces without the need for quoting variables.
If you want to print messages while checking for directory and file:
[[ ! -d $1 || -L $1 ]] && echo "Directory doesn't exist" && exit
[[ ! -f $2 || -L $2 ]] && echo "File doesn't exist" && exit
printf "%s\n" "arg1: $1" "arg2: $2"