How to validate if the file not passed in - linux

So I have script that takes file from user with option -y or -n and -d is required
script runs ./example.sh -d file -y or -n
variable
file=""
I need to do validation on if the user did not pass file in for example
./example.sh -y --- it should give error "must provide file"`
./example.sh -n --- it should give error "must provide file"`
./example.sh -d --- it should give error "must provide file"
./example.sh -y -n --- it should give error "must provide file"
i came up with this but its not working
#checking if the file is provided
if [ $file -eq 1 ]; then
echo "No file provided"
exit 1
fi
# Does the file exist
which works for checking if the file exists but it only check if the file in there
if [[ ! -f $file ]]; then
echo
echo "Error: The file $file does not exist."
exit 1
fi

Why not simply loop over the arguments and test the -d, -n and -y options in a case statement. The filename must follow the -d option as I understand it. When the -d option is processed in your case statement check that the next argument holds a filename that exists on the system and is non-empty, if so, set the filename, e.g. filename="${argv[i+1]}".
When you exit the option processing loop, simply test if filename is set and you have your answer.
A short example could be:
#!/bin/bash
argc=$# ## argument count
argv=("$#") ## array holding arguments (argument vector)
filename= ## filename is empty
## loop over all arguments, filename must follow -d
for ((i = 0; i < argc; i++)); do
case "${argv[i]}" in
## file exists and non-empty, set filename to argument
-d ) [ -s "${argv[i+1]}" ] && { filename="${argv[i+1]}"; };;
-n ) # do whatever for option n
;;
-y ) # do whatever for option y
;;
esac
done
[ -n "$filename" ] || { ## check filename set or exit
printf "error: file empty or not readable.\n" >&2
exit 1
}
## valid filename received
printf "file '%s' found - script running.\n" "$filename"
Only if a valid filename is given after the -d option will the script run. Otherwise it will give an error message and exit.
Examples
$ ./process-opts.sh -y -n
error: file empty or not readable.
or
$ ./process-opts.sh -y -d -n
error: file empty or not readable.
Finally
$ ./process-opts.sh -y -d README -n
file 'README' found - script running.
or in any order:
$ ./process-opts.sh -y -n -d README
file 'README' found - script running.

Related

Linux Shell Script * or ???? is replace automatically

I have some files stored on a server. I have to get all those files using a pattern and exclude the file which contains the current date as the file name.
Files are given below
/var/tomcat/logs/catalina.2022-05-11.log
/var/tomcat/logs/catalina.2022-05-13.log
/var/tomcat/logs/catalina.2022-05-14.log
/var/tomcat/logs/catalina.2022-05-16.log
/var/tomcat/logs/error_1.log
/var/tomcat/logs/error_2.log
/var/tomcat/logs/error_3.log
/var/tomcat/logs/error_4.log
For this, I have stored patterns in a file and I want to read the pattern of that file and find all files with the help of those patterns.
Pattern Input File content is given below, in below I have used '%Y-%m-%d' to identify the date format so that I can exclude the current date file.
/var/tomcat/logs/catalina.*.log;%Y-%m-%d
/var/tomcat/logs/error_*.log
I have developed a shell script which is given below
#!/bin/sh
pattern_filepath=$1
while IFS= read -r line || [ -n "$line" ]; do
pattern_var="$line"
echo pattern: "$pattern_var"
filepath=""
date_format=""
if [[ $pattern_var == *";"* ]];
then
echo "Semicolons ; separator is there"
filepath=($(echo "$pattern_var" | cut -f1 -d ';'))
echo filepath: "$filepath"
date_format=($(echo "$pattern_var" | cut -f2 -d ';'))
else
echo "Semicolons ; separator is not there"
filepath=$pattern_var
fi
echo "date_format: "$date_format
done < "$pattern_filepath"
Command to run the script
sh /var/monitoring/test.sh "/var/monitoring/pattern" > /var/monitoring/test.log
Inside the log file, I can see, that in the file path variable I am getting the value as a date but that should be with an asterisk instead of a date.
Log file 'test.log'
pattern: /var/tomcat/logs/catalina.*.log;%Y-%m-%d
Semicolons ; separator is there
filepath: /var/tomcat/logs/catalina.2022-05-11.log
date_format: %Y-%m-%d
pattern: /var/tomcat/logs/error_*.log
Semicolons ; separator is not there
date_format:
Please help me with this, how can I achieve this?
Simple, straight to the point. Achieves the requirement, without the complexity of patterns, or lists, or ...
#!/bin/bash
sourcedir="/var/tomcat/logs"
if [[ ! -d "$sourcedir" ]]
then
echo "ERROR: source directory $sourcedir does not exist."
exit 1
fi
targetdir="/tmp/somedir"
if [[ ! -d "$targetdir" ]]
then
echo "ERROR: target directory $targetdir does not exist."
exit 1
fi
# Get the catalina logs
cp "$sourcedir"/catalina.*.log "$targetdir"
# Remove the catalina log for today
filetoremove="$targetdir"/catalina.$(date +%Y-%m-%d).log
if [[ -f "$filetoremove" ]]
then
rm -f "$filetoremove"
fi
# Get the error logs
cp "$sourcedir"/error_*.log "$targetdir"
You can add error checking and recovery for the cp and rm commands.

Shell Script to check value in simple typescript file

I have a simple TS file
env.ts
Which contains
export const BUILD_SDK = false
export const DUMMY_PRESET = false
and I have build.sh script which builds our SDk.
I want to write a shell script (build.sh) which throws an error if value in env.ts is false before building.
Can someone please help me out in figuring out a way to do it?
Update based on the answer from Flashtube
So I tried this
#!/bin/bash
# With head -n 2 env.ts we get the first line of env.ts
# With cut -d " " we split the line at every space
# With -f 5 we specify we want the 5th spit value, in this case the boolean value
# If that is equals to true, we echo an error and pipe it to the error output with 1>&2 and exit with exit status 64
if [[ $(head -n 2 App/env/index.tsx | cut -d " " -f 5) = "false" ]]; then
echo "Error! check env variables and set it to true" 1>&2;
exit 64;
fi
Where my env file looks like this
//Please inform if you do any changes in this file so that we can update build script
export const SDK_BUILD = false
but this didn't work
If you want to check whether any const of a set of specific consts is false, you can do it in bash this way:
#!/bin/bash
FILE="/path/to/file/env.ts"
CONST_TO_CHECK="(BUILD_SDK|DUMMY_PRESET)"
if grep -Eq 'export\s*const\s*'"${CONST_TO_CHECK}"'\s*=\s*false' "$FILE"; then
echo "Error"
fi
Simply add your const names to CONST_TO_CHECK, separated with a pipe.
If you want to see whether any const of a long list of consts is false, use this:
#!/bin/bash
FILE="/path/to/file/env.ts"
CONST_NAME_REGEX='[a-zA-Z_]*'
if grep -Eq 'export\s*const\s*'"${CONST_NAME_REGEX}"'\s*=\s*false' "$FILE"; then
echo "Error"
fi
Here you might need to update CONST_NAME_REGEX to reflect you naming conventions.
You can use following bash script
#!/bin/bash
# With head -n 1 env.ts we get the first line of env.ts
# With cut -d " " we split the line at every space
# With -f 5 we specify we want the 5th spit value, in this case the boolean value
# If that is equals to true, we echo an error and pipe it to the error output with 1>&2 and exit with exit status 64
if [[ $(head -n 1 env.ts | cut -d " " -f 5) = "false" ]]; then
echo "Error!" 1>&2;
exit 64;
fi
Now we can detect wheter the value is false or true.

Mishandling of input using "getopts" and "case"

ISSUE: I need to place a users cmd-line argument ( in this case a directory path ) into an associative array using declare -a DIR. The directory path will be stored in the array for later use in the script. However, at the moment that when the proper option & arg is provided by the user nothig is stored in the DIR array. The script is executed as such ./selectDir -d <dir_path> [file1] [file2]
DESIRED OUTPUT: I want the user's selected directory choice to be stored in the array "DIR". I would then like to be able to print the contents of the array once i've stored them so that I can verfiy the storage.
CURRENT CODE:
!/bin/bash -x
#
# Description:
# Usage:
#
# Issue: current problem seems to be with the execution of the scipt
# --syntax ./parseargs -d [$path] [file1] [file2] ... but this is keeps snagging on error
#
totalArgs="$#"
function usageErr()
{
echo 'usage: baseline.sh [ -d path ] file1 [file2]'
echo 'creates or compares a baseline from path '
echo 'default for path is /'
exit 2
} >&2 #rename the error output file
function parseArgs ()
{
# need to figure out how to trigger the conditional
while getopts "c:" MYOPT
do
case "${MYOPT}" in
d) # provide a custom directory
selectPath=YES
DIR+=( "$OPTARG" )
;;
*) # any other option provided will result in error
usageErr
;;
esac
done
shift $((OPTIND-1))
#no arguments? too many?
(( $# == 0 || $# > 2 )) && usageErr
(( ${#DIR[*]} == 0 )) && DIR=( "/" )
}
declare -a DIR
parseArgs # see about passing the cmd args at this point
echo "DIR: ${DIR[*]}"
echo "TEST: $test"
echo "PATH: $selectPath"

Using a for loop in a bash scrip to iterate over extentions held in a csv file to download data from NASA

I am currently trying to download data from (https://ladsweb.modaps.eosdis.nasa.gov/search/order). They provide a bash file (https://ladsweb.modaps.eosdis.nasa.gov/tools-and-services/data-download-scripts/) which I need to try and edit. I have a .csv file with all of the file extensions I need to download (specifically I need all VNP46A1 data for China from 2015 until now.
In pseudo-code form I would like to add the following:
FOR url_path IN url_list:
recurse "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5000/VNP46A1/“+”$url_path"+”.h5" “your_directory”+”$url_path" "TOKEN_HERE"
I need to edit this bash to iterate over the files in the csv and download them into a folder for use later.
The bash file is as follows:
#!/bin/bash
function usage {
echo "Usage:"
echo " $0 [options]"
echo ""
echo "Description:"
echo " This script will recursively download all files if they don't exist"
echo " from a LAADS URL and stores them to the specified path"
echo ""
echo "Options:"
echo " -s|--source [URL] Recursively download files at [URL]"
echo " -d|--destination [path] Store directory structure to [path]"
echo " -t|--token [token] Use app token [token] to authenticate"
echo ""
echo "Dependencies:"
echo " Requires 'jq' which is available as a standalone executable from"
echo " https://stedolan.github.io/jq/download/"
}
function recurse {
local src=$1
local dest=$2
local token=$3
echo "Querying ${src}.json"
for dir in $(curl -s -H "Authorization: Bearer ${token}" ${src}.json | jq '.[] | select(.size==0) | .name' | tr -d '"')
do
echo "Creating ${dest}/${dir}"
mkdir -p "${dest}/${dir}"
echo "Recursing ${src}/${dir}/ for ${dest}/${dir}"
recurse "${src}/${dir}/" "${dest}/${dir}"
done
for file in $(curl -s -H "Authorization: Bearer ${token}" ${src}.json | jq '.[] | select(.size!=0) | .name' | tr -d '"')
do
if [ ! -f ${dest}/${file} ]
then
echo "Downloading $file to ${dest}"
# replace '-s' with '-#' below for download progress bars
curl -s -H "Authorization: Bearer ${token}" ${src}/${file} -o ${dest}/${file}
else
echo "Skipping $file ..."
fi
done
}
POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
-s|--source)
src="$2"
shift # past argument
shift # past value
;;
-d|--destination)
dest="$2"
shift # past argument
shift # past value
;;
-t|--token)
token="$2"
shift # past argument
shift # past value
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
if [ -z ${src+x} ]
then
echo "Source is not specified"
usage
exit 1
fi
if [ -z ${dest+x} ]
then
echo "Destination is not specified"
usage
exit 1
fi
if [ -z ${token+x} ]
then
echo "Token is not specified"
usage
exit 1
fi
recurse "$src" "$dest" "$token"
and a shortened (for testing purposes) csv file is given as:
/archive/allData/5000/VNP46A1/2015/001/VNP46A1.A2015001.h30v05.001.2019135185504.h5
/archive/allData/5000/VNP46A1/2015/002/VNP46A1.A2015002.h30v05.001.2019136091632.h5
/archive/allData/5000/VNP46A1/2015/003/VNP46A1.A2015003.h30v05.001.2019136075625.h5
/archive/allData/5000/VNP46A1/2015/004/VNP46A1.A2015004.h30v05.001.2019136081706.h5
/archive/allData/5000/VNP46A1/2015/005/VNP46A1.A2015005.h30v05.001.2019136084155.h5
/archive/allData/5000/VNP46A1/2015/006/VNP46A1.A2015006.h30v05.001.2019136084128.h5
/archive/allData/5000/VNP46A1/2015/007/VNP46A1.A2015007.h30v05.001.2019136085336.h5
/archive/allData/5000/VNP46A1/2015/008/VNP46A1.A2015008.h30v05.001.2019136103147.h5
/archive/allData/5000/VNP46A1/2015/009/VNP46A1.A2015009.h30v05.001.2019136100110.h5
Any help or suggestions will be much appreciated.
Kind regards
You want to create a bash script that will loop over each line to download the data that you need, using the script provided by NASA.
For example say the below script was a file called save-data.sh:
#!/bin/bash
while read p; do
./laads-data-download.sh -s $P -d "destination" -t "token"
echo "$p"
done <paths.txt
Example tree structure:
nasa-satellite-data
├── laads-data-download.sh
├── paths.txt
└── save-data.sh

updating a file using tee randomly fails in linux bash script

when using sed -e to update some parameters of a config file and pipe it to | tee (to write the updated content into the file), this randomly breaks and causes the file to be invalid (size 0).
In Summary, this code is used for updating parameters:
# based on the provided linenumber, add some comments, add the new value, delete old line
sed -e "$lineNr a # comments" -e "$lineNr a $newValue" -e "$lineNr d" $myFile | sudo tee $myFile
I set up an script which calls this update command 100 times.
In a Ubuntu VM (Parallels Desktop) on a shared Directory with OSX this
behaviour occurs up to 50 times
In a Ubuntu VM (Parallels Desktop) on the
Ubuntu partition this behaviour occurs up to 40 times
On a native System (IntelNUC with Ubuntu) this behaviour occurs up to 15 times
Can someone explain why this is happening?
Here is a fully functional script where you can run the experiment as well. (All necessary files are generated by the script, so you can simply copy/paste it into a bashscriptfile and run it)
#!/bin/bash
# main function at bottom
#====================
#===HELPER METHOD====
#====================
# This method updates parameters with a new value. The replacement is performed linewise.
doUpdateParameterInFile()
{
local valueOfInterest="$1"
local newValue="$2"
local filePath="$3"
# stores all matching linenumbers
local listOfLines=""
# stores the linenumber which is going to be replaced
local lineToReplace=""
# find value of interest in all non-commented lines and store related lineNumber
lineToReplace=$( grep -nr "^[^#]*$valueOfInterest" $filePath | sed -n 's/^\([0-9]*\)[:].*/\1/p' )
# Update parameters
# replace the matching line with the desired value
oldValue=$( sed -n "$lineToReplace p" $filePath )
sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" $filePath | sudo tee $filePath >/dev/null
# Sanity check to make sure file did not get corrupted by updating parameters
if [[ ! -s $filePath ]] ; then
echo "[ERROR]: While updating file it turned invalid."
return 31
fi
}
#===============================
#=== Actual Update Function ====
#===============================
main_script()
{
echo -n "Update Parameter1 ..."
doUpdateParameterInFile "Parameter1" "Parameter1 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 33 ; fi
echo -n "Update Parameter2 ..."
doUpdateParameterInFile "Parameter2" "Parameter2=90" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 34 ; fi
echo -n "Update Parameter3 ..."
doUpdateParameterInFile "Parameter3" "Parameter3 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 35 ; fi
}
#=================
#=== Main Loop ===
#=================
#generate file config.txt
printf "# Configfile with 3 Parameters\n#[Parameter1]\n#only takes YES or NO\nParameter1 NO \n\n#[Parameter2]\n#Parameter2 takes numbers\nParameter2 = 100 \n\n#[Parameter3]\n#Parameter3 takes YES or NO \nParameter3 YES\n" > config.txt
cp config.txt config.txt.bkup
# Start the experiment and let it run 100 times
cnt=0
failSum=0
while [[ $cnt != "100" ]] ; do
echo "==========run: $cnt; fails: $failSum======="
main_script
if [[ $? != "0" ]] ; then cp config.txt.bkup config.txt ; failSum=$(($failSum+1)) ; fi
cnt=$((cnt+1))
sleep 0.5
done
regards
DonPromillo
The problem is that you're using tee to overwrite $filepath at the same time as sed is trying to read from it. If tee truncates it first then sed gets an empty file and you end up with a 0 length file at the other end.
If you have GNU sed you can use the -i flag to have sed modify the file in place (other versions support -i but require an argument to it). If your sed doesn't support it you can have it write to a temp file and move it back to the original name like
tmpname=$(mktemp)
sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" "$filePath" > "$tmpname"
sudo mv "$tmpname" "$filePath"
or if you want to preserve the original permissions you could do
sudo sh -c "cat '$tmpname' > '$filePath'"
rm "$tmpname"
or use your tee approach like
sudo tee "$filePath" >/dev/null <"$tmpname"
rm "$tmpname"

Resources