Bash- Count Newlines via terminal command - linux

i'm trying to make a bash script that counts the newlines in an input. The first if statement (switch $0) works fine but the problem I'm having is trying to get it to read the WC of a file in a terminal argument.
e.g.
~$ ./script.sh
1
2
3
4
(User presses CTRL+D)
display word count here # answer is 5 - works fine
e.g.
~$ .script1.sh < script1.sh
WC here -(5)
~$ succesfully redirects the stdin from a file
but
e.g.
~$ ./script1.sh script1.sh script2.sh
WC displayed here for script1.sh
WC displayed here for script2.sh
NOTHING
~$
the problem I believe is the second if statement, instead of executing the script in the terminal it goes to the if statement and waits for a user input and its not giving back the echo statement.
Any help would be greatly appreciated since I cannot figure out why it won't work without the ~$ < operator.
#!/bin/bash
#!/bin/sh
read filename ## read provided filename
USAGE="Usage: $0 $1 $2..." ## switch statement
if [ "$#" == "0" ]; then
declare -i lines=0 words=0 chars=0
while IFS= read -r line; do
((lines++))
array=($line)
((words += ${#array[#]}))
((chars += ${#line} + 1)) # add 1 for the newline
done < /dev/stdin
fi
echo "$lines $words $chars $filename" ## filename doesn't print, just filler
### problem if statement####
if [ "$#" != "0" ]; then # space between [] IS VERY IMPORTANT
declare -i lines=0 words=0 chars=0
while IFS= read -r line; do
lines=$( grep -c '\n'<"filename") ##should use grep -c to compare only new lines in the filename. assign to variable line
words=$( grep -c '\n'<"filename")
chars=$( grep -c '\n'<"filename")
echo "$lines $words $chars"
#lets user press CTRL+D to end script and count the WC
fi

#!/bin/sh
set -e
if [ -t 0 ]; then
# We are *not* reading stdin from a pipe or a redirection.
# Get the counts from the files specified on the cmdline
if [ "$#" -eq 0 ]; then
echo "no files specified" >&2
exit 1
fi
cat "$#" | wc
else
# stdin is attached to a pipe or redirected from a file
wc
fi | { read lines words chars; echo "lines=$lines words=$words chars=$chars"; }
The variables from the read command only exist within the braces, due to the way the shell (some shells anyway) use subshells for commands in a pipeline. Typically, the solution for that is to redirect from a process substitution (bash/ksh).
This can be squashed down to
#!/bin/bash
[[ -t 0 ]] && files=true || files=false
read lines words chars < <({ ! $files && cat || cat "$#"; } | wc)
echo "lines=$lines words=$words chars=$chars"
a very quick demo of cmd | read x versus read x < <(cmd)
$ x=foo; echo bar | read x; echo $x
foo
$ x=foo; read x < <(echo bar); echo $x
bar

Use wc.
Maybe the simplest is to replace the second if block with a for.
$: cat tst
#! /bin/env bash
declare -i lines=0 words=0 chars=0
case "$#" in
0) wc ;;
*) for file in $*
do read lines words chars x <<< "$( wc $file )"
echo "$lines $words $chars $file"
done ;;
esac
$: cat foo
hello
world
and
goodbye cruel world!
$: tst < foo
6 6 40
$: tst foo tst
6 6 40 foo
9 38 206 tst

Related

bash/ksh grep script take more than one argument

#!/bin/ksh
if [ -n "$1" ]
then
if grep -w -- "$1" codelist.lst
then
true
else
echo "Value not Found"
fi
else
echo "Please enter a valid input"
fi
This is my script and it works exactly how I want at the moment, I want to add if I add more arguments It will give me the multiple outputs, How can I do that?
So For Example I do ./test.sh apple it will grep apple in codelist.lst and Give me the output : Apple
I want to do ./test.sh apple orange and will do:
Apple
Orange
You can do that with shift and a loop, something like (works in both bash and ksh):
for ((i = $#; i > 0 ; i--)) ; do
echo "Processing '$1'"
shift
done
You'll notice I've also opted not to use the [[ -n "$1" ]] method as that would terminate the loop early with an empty string (such as with ./script.sh a b "" c stopping without doing c).
To iterate over the positional parameters:
for pattern in "$#"; do
grep -w -- "$pattern" codelist.lst || echo "'$pattern' not Found"
done
For a more advanced usage, which only invokes grep once, use the -f option with a shell process substitution:
grep -w -f <(printf '%s\n' "$#") codelist.lst

Compare a list of strings in Bash

I have a list of rpm files in a file called rpmlist.txt which I have to compare with another list, newlist.txt, and see if they are same in Bash. For example, this is my requirement:
files inside rpmlist.txt
bash-4.4-9.10.1_x86_64
binutils-2.32-7.8.1_x86_64
bison-3.0.4-1.268_x86_64
files inside newlist.txt
bash-5.4-9.10.1_x86_64
binutils-2.32-7.8.1_x86_64
bison-6.0.4-1.268_x86_64
And print if they are matching or not. Any help would be appreciated
Try with:
#!/bin/bash
# Load files into arrays
readarray source_list < rpmlist.txt
readarray target_list < newlist.txt
# Check files size
source_size=${#source_list[#]}
target_size=${#target_list[#]}
if [ ${source_size} -ne ${target_size} ]; then
echo "File lines count not matching!" >&2
exit 1
fi
# Enum files
for (( i=0; i < ${source_size}; i++ )); do
# Get file name
source_file=${source_list[$i]}
target_file=${target_list[$i]}
# Remove CR/LF
source_file=$(echo "${source_file}" | sed 's:\r$::')
target_file=$(echo "${target_file}" | sed 's:\r$::')
# Check if files exist
if [ ! -f ${source_file} ] || [ ! -f ${target_file} ]; then
echo "Source and/or Target does not exist." >&2
exit 2
fi
# Compare files
diff -q "${source_file}" "${target_file}"
done
PS: I tested it and it works.
Edit (1)
Based on comments, I think you should replace my script with the following simple command:
cat rpmlist.txt | xargs -I "{}" grep "{}" newlist.txt
Edit (2) - Unmatched list
cat rpmlist.txt | xargs -I "{}" grep -v "{}" newlist.txt
This will cross-compare an arbitrary number of such files.
#!/bin/bash
set -e
declare -ar list_names=("$#")
declare -Ai "${list_names[#]}"
for list in "${list_names[#]}"; do
declare -n set="$list"
while IFS= read -r line; do
((++set["$line"]))
done < "${list}.txt"
done
compare_lists() {
local -rn set1="$1"
local -rn set2="$2"
local name
echo "Lines in ${1}, but not in ${2}:"
for name in "${!set1[#]}"; do
((set2["${name}"])) || printf ' %s\n' "$name"
done
}
declare -i idx jdx
for ((idx = 0; idx < ${#list_names[#]}; ++idx)); do
for ((jdx = idx + 1; jdx < ${#list_names[#]}; ++jdx)); do
compare_lists "${list_names[idx]}" "${list_names[jdx]}"
compare_lists "${list_names[jdx]}" "${list_names[idx]}"
done
done
Example (when the above script is called listdiff.sh):
$ ./listdiff.sh rpmlist newlist
Lines in rpmlist, but not in newlist:
bison-3.0.4-1.268_x86_64
bash-4.4-9.10.1_x86_64
Lines in newlist, but not in rpmlist:
bison-6.0.4-1.268_x86_64
bash-5.4-9.10.1_x86_64
And it can get more arguments than 2.

updating a file using tee randomly fails in linux bash script

when using sed -e to update some parameters of a config file and pipe it to | tee (to write the updated content into the file), this randomly breaks and causes the file to be invalid (size 0).
In Summary, this code is used for updating parameters:
# based on the provided linenumber, add some comments, add the new value, delete old line
sed -e "$lineNr a # comments" -e "$lineNr a $newValue" -e "$lineNr d" $myFile | sudo tee $myFile
I set up an script which calls this update command 100 times.
In a Ubuntu VM (Parallels Desktop) on a shared Directory with OSX this
behaviour occurs up to 50 times
In a Ubuntu VM (Parallels Desktop) on the
Ubuntu partition this behaviour occurs up to 40 times
On a native System (IntelNUC with Ubuntu) this behaviour occurs up to 15 times
Can someone explain why this is happening?
Here is a fully functional script where you can run the experiment as well. (All necessary files are generated by the script, so you can simply copy/paste it into a bashscriptfile and run it)
#!/bin/bash
# main function at bottom
#====================
#===HELPER METHOD====
#====================
# This method updates parameters with a new value. The replacement is performed linewise.
doUpdateParameterInFile()
{
local valueOfInterest="$1"
local newValue="$2"
local filePath="$3"
# stores all matching linenumbers
local listOfLines=""
# stores the linenumber which is going to be replaced
local lineToReplace=""
# find value of interest in all non-commented lines and store related lineNumber
lineToReplace=$( grep -nr "^[^#]*$valueOfInterest" $filePath | sed -n 's/^\([0-9]*\)[:].*/\1/p' )
# Update parameters
# replace the matching line with the desired value
oldValue=$( sed -n "$lineToReplace p" $filePath )
sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" $filePath | sudo tee $filePath >/dev/null
# Sanity check to make sure file did not get corrupted by updating parameters
if [[ ! -s $filePath ]] ; then
echo "[ERROR]: While updating file it turned invalid."
return 31
fi
}
#===============================
#=== Actual Update Function ====
#===============================
main_script()
{
echo -n "Update Parameter1 ..."
doUpdateParameterInFile "Parameter1" "Parameter1 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 33 ; fi
echo -n "Update Parameter2 ..."
doUpdateParameterInFile "Parameter2" "Parameter2=90" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 34 ; fi
echo -n "Update Parameter3 ..."
doUpdateParameterInFile "Parameter3" "Parameter3 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 35 ; fi
}
#=================
#=== Main Loop ===
#=================
#generate file config.txt
printf "# Configfile with 3 Parameters\n#[Parameter1]\n#only takes YES or NO\nParameter1 NO \n\n#[Parameter2]\n#Parameter2 takes numbers\nParameter2 = 100 \n\n#[Parameter3]\n#Parameter3 takes YES or NO \nParameter3 YES\n" > config.txt
cp config.txt config.txt.bkup
# Start the experiment and let it run 100 times
cnt=0
failSum=0
while [[ $cnt != "100" ]] ; do
echo "==========run: $cnt; fails: $failSum======="
main_script
if [[ $? != "0" ]] ; then cp config.txt.bkup config.txt ; failSum=$(($failSum+1)) ; fi
cnt=$((cnt+1))
sleep 0.5
done
regards
DonPromillo
The problem is that you're using tee to overwrite $filepath at the same time as sed is trying to read from it. If tee truncates it first then sed gets an empty file and you end up with a 0 length file at the other end.
If you have GNU sed you can use the -i flag to have sed modify the file in place (other versions support -i but require an argument to it). If your sed doesn't support it you can have it write to a temp file and move it back to the original name like
tmpname=$(mktemp)
sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" "$filePath" > "$tmpname"
sudo mv "$tmpname" "$filePath"
or if you want to preserve the original permissions you could do
sudo sh -c "cat '$tmpname' > '$filePath'"
rm "$tmpname"
or use your tee approach like
sudo tee "$filePath" >/dev/null <"$tmpname"
rm "$tmpname"

Cannot echo the changes made into the files

I am checking if the files have been modified and I need to echo what new strings have been added. When I try this script for a single file it works, but when I iterate through multiple files in a directory it does not work as it should. Any suggestion?
#! /bin/bash
GAP=5
while :
do
FILES=/home/Desktop/*
for f in $FILES
do
len=`wc -l $f | awk '{ print $1 }'`
if [ -N $f ]; then
echo "`date`: New entries in $f:"
newlen=`wc -l $f | awk '{ print $1 }'`
newlines=`expr $newlen - $len`
tail -$newlines $f
len=$newlen
fi
sleep $GAP
done
done
Continuing from the comments, here is the original solution I envisioned using inotifywait (from the inotify-tools package) and an associative array. The benefit here is inotifywait will block and will not waste resources endlessly checking the line count of each file on each loop iteration. I'll work on a solution using a temporary file, but when you go that route you open yourself up to a change occurring in between loop iterations. Here is the first solution:
#!/bin/bash
watchdir="${1:-$PWD}"
events="-e modify -e attrib -e close_write -e create -e delete -e move"
declare -A lines
for i in "$watchdir"/*; do
[ -f "$i" ] && lines[$i]=$(wc -l <"$i")
done
while :; do ## watch for changes in chosen dir
fname="${watchdir}/$(inotifywait -q $events --format '%f' "$watchdir")"
newlc=$(wc -l <"$fname") ## get line count for changed file
if [ "${lines[$fname]}" -ne "$newlc" ]; then ## if changed, print
printf " lines chanaged : %s -> %s (%s)\n" \
"${lines[$fname]}" "$newlc" "$fname"
lines[$fname]=$newlc ## update saved line count for file
fi
done
Original testfile.txt
$ cat dat/tmp/testfile.txt
1 1.2
2 2.2
Example Use/Output
Script saved in watchdir.sh. Start watchdir.sh so inotifywait is watching the dat/tmp directory
$ ./watchdir.sh dat/tmp
Using a second terminal, modify file in the dat/tmp directory
$ echo "newline" >> ~/scr/tmp/stack/dat/tmp/testfile.txt
$ echo "newline" >> ~/scr/tmp/stack/dat/tmp/testfile.txt
Output of watchdir.sh running in separate terminal (or background)
$ ./watchdir.sh dat/tmp
lines chanaged : 2 -> 3 (dat/tmp/testfile.txt)
lines chanaged : 3 -> 4 (dat/tmp/testfile.txt)
Resulting testfile.txt
$ cat dat/tmp/testfile.txt
1 1.2
2 2.2
newline
newline
Second Solution Using [ -N file ]
Here is a second solution a bit closer to your first attempt. It is a less robust way to approach the solution (it will miss multiple changes between tests, etc.). Look it over and let me know if you have questions
#!/bin/bash
watchdir="${1:-$PWD}"
gap=5
tmpfile="$TMPDIR/watchtmp" ## temp file in system $TMPDIR (/tmp)
:>"$tmpfile"
trap 'rm $tmpfile' SIGTERM EXIT ## remove tmpfile on exit
for i in "$watchdir"/*; do ## populate tmpfile with line counts
[ -f "$i" ] && echo "$i,$(wc -l <"$i")" >> "$tmpfile"
done
while :; do ## loop every $gap seconds
for i in "$watchdir"/*; do ## for each file
if [ -N "$i" ]; then ## check changed
cnt=$(wc -l <"$i") ## get new line count
oldcnt=$(grep "$i" "$tmpfile") ## get old count
oldcnt=${oldcnt##*,}
if [ "$cnt" -ne "$oldcnt" ]; then ## if not equal, print
printf " lines chanaged : %s -> %s (%s)\n" \
"$oldcnt" "$cnt" "$i"
## update tmpfile with new count
sed -i "s|^${i}[,][0-9][0-9]*.*$|${i},$cnt|" "$tmpfile"
fi
fi
done
sleep $gap
done
Use/Output
Start watchdir.sh
$ ./watchdir2.sh dat/tmp
In second terminal modify file
$ echo "newline" >> ~/scr/tmp/stack/dat/tmp/testfile.txt
wait for $gap to expire (if changed twice - it will not register)
$ echo "newline" >> ~/scr/tmp/stack/dat/tmp/testfile.txt
Results
$ ./watchdir2.sh dat/tmp
lines chanaged : 10 -> 11 (dat/tmp/testfile.txt)
lines chanaged : 11 -> 12 (dat/tmp/testfile.txt)

Create new file but add number if filename already exists in bash

I found similar questions but not in Linux/Bash
I want my script to create a file with a given name (via user input) but add number at the end if filename already exists.
Example:
$ create somefile
Created "somefile.ext"
$ create somefile
Created "somefile-2.ext"
The following script can help you. You should not be running several copies of the script at the same time to avoid race condition.
name=somefile
if [[ -e $name.ext || -L $name.ext ]] ; then
i=0
while [[ -e $name-$i.ext || -L $name-$i.ext ]] ; do
let i++
done
name=$name-$i
fi
touch -- "$name".ext
Easier:
touch file`ls file* | wc -l`.ext
You'll get:
$ ls file*
file0.ext file1.ext file2.ext file3.ext file4.ext file5.ext file6.ext
To avoid the race conditions:
name=some-file
n=
set -o noclobber
until
file=$name${n:+-$n}.ext
{ command exec 3> "$file"; } 2> /dev/null
do
((n++))
done
printf 'File is "%s"\n' "$file"
echo some text in it >&3
And in addition, you have the file open for writing on fd 3.
With bash-4.4+, you can make it a function like:
create() { # fd base [suffix [max]]]
local fd="$1" base="$2" suffix="${3-}" max="${4-}"
local n= file
local - # ash-style local scoping of options in 4.4+
set -o noclobber
REPLY=
until
file=$base${n:+-$n}$suffix
eval 'command exec '"$fd"'> "$file"' 2> /dev/null
do
((n++))
((max > 0 && n > max)) && return 1
done
REPLY=$file
}
To be used for instance as:
create 3 somefile .ext || exit
printf 'File: "%s"\n' "$REPLY"
echo something >&3
exec 3>&- # close the file
The max value can be used to guard against infinite loops when the files can't be created for other reason than noclobber.
Note that noclobber only applies to the > operator, not >> nor <>.
Remaining race condition
Actually, noclobber does not remove the race condition in all cases. It only prevents clobbering regular files (not other types of files, so that cmd > /dev/null for instance doesn't fail) and has a race condition itself in most shells.
The shell first does a stat(2) on the file to check if it's a regular file or not (fifo, directory, device...). Only if the file doesn't exist (yet) or is a regular file does 3> "$file" use the O_EXCL flag to guarantee not clobbering the file.
So if there's a fifo or device file by that name, it will be used (provided it can be open in write-only), and a regular file may be clobbered if it gets created as a replacement for a fifo/device/directory... in between that stat(2) and open(2) without O_EXCL!
Changing the
{ command exec 3> "$file"; } 2> /dev/null
to
[ ! -e "$file" ] && { command exec 3> "$file"; } 2> /dev/null
Would avoid using an already existing non-regular file, but not address the race condition.
Now, that's only really a concern in the face of a malicious adversary that would want to make you overwrite an arbitrary file on the file system. It does remove the race condition in the normal case of two instances of the same script running at the same time. So, in that, it's better than approaches that only check for file existence beforehand with [ -e "$file" ].
For a working version without race condition at all, you could use the zsh shell instead of bash which has a raw interface to open() as the sysopen builtin in the zsh/system module:
zmodload zsh/system
name=some-file
n=
until
file=$name${n:+-$n}.ext
sysopen -w -o excl -u 3 -- "$file" 2> /dev/null
do
((n++))
done
printf 'File is "%s"\n' "$file"
echo some text in it >&3
Try something like this
name=somefile
path=$(dirname "$name")
filename=$(basename "$name")
extension="${filename##*.}"
filename="${filename%.*}"
if [[ -e $path/$filename.$extension ]] ; then
i=2
while [[ -e $path/$filename-$i.$extension ]] ; do
let i++
done
filename=$filename-$i
fi
target=$path/$filename.$extension
Use touch or whatever you want instead of echo:
echo file$((`ls file* | sed -n 's/file\([0-9]*\)/\1/p' | sort -rh | head -n 1`+1))
Parts of expression explained:
list files by pattern: ls file*
take only number part in each line: sed -n 's/file\([0-9]*\)/\1/p'
apply reverse human sort: sort -rh
take only first line (i.e. max value): head -n 1
combine all in pipe and increment (full expression above)
Try something like this (untested, but you get the idea):
filename=$1
# If file doesn't exist, create it
if [[ ! -f $filename ]]; then
touch $filename
echo "Created \"$filename\""
exit 0
fi
# If file already exists, find a similar filename that is not yet taken
digit=1
while true; do
temp_name=$filename-$digit
if [[ ! -f $temp_name ]]; then
touch $temp_name
echo "Created \"$temp_name\""
exit 0
fi
digit=$(($digit + 1))
done
Depending on what you're doing, replace the calls to touch with whatever code is needed to create the files that you are working with.
This is a much better method I've used for creating directories incrementally.
It could be adjusted for filename too.
LAST_SOLUTION=$(echo $(ls -d SOLUTION_[[:digit:]][[:digit:]][[:digit:]][[:digit:]] 2> /dev/null) | awk '{ print $(NF) }')
if [ -n "$LAST_SOLUTION" ] ; then
mkdir SOLUTION_$(printf "%04d\n" $(expr ${LAST_SOLUTION: -4} + 1))
else
mkdir SOLUTION_0001
fi
A simple repackaging of choroba's answer as a generalized function:
autoincr() {
f="$1"
ext=""
# Extract the file extension (if any), with preceeding '.'
[[ "$f" == *.* ]] && ext=".${f##*.}"
if [[ -e "$f" ]] ; then
i=1
f="${f%.*}";
while [[ -e "${f}_${i}${ext}" ]]; do
let i++
done
f="${f}_${i}${ext}"
fi
echo "$f"
}
touch "$(autoincr "somefile.ext")"
without looping and not use regex or shell expr.
last=$(ls $1* | tail -n1)
last_wo_ext=$($last | basename $last .ext)
n=$(echo $last_wo_ext | rev | cut -d - -f 1 | rev)
if [ x$n = x ]; then
n=2
else
n=$((n + 1))
fi
echo $1-$n.ext
more simple without extension and exception of "-1".
n=$(ls $1* | tail -n1 | rev | cut -d - -f 1 | rev)
n=$((n + 1))
echo $1-$n.ext

Resources