I have a few text files with numbers (structure as below). I'd like to sum up every line form one file with ever line form other files (line1 from file1 + line1 from file2 etc.). I have written the bash script as following but this gives me the expr error.
function countHourly () {
for i in {1..24}
do
for file in $PLACE/*.dailycount.txt
do
SECBUFF=`head -n $i $file`
VAL=`expr $VAL + $SECBUFF` ## <-- this cause expr error
done
echo line $i form all files counts: $VAL
done
}
file structure *.dailycount.txt:
1
0
14
56
45
0
3
45
23
23
9 (every number in new line).
Assuming your files each contain exactly 24 lines, you could solve this problem with a simple one-liner:
counthourly() {
paste -d+ $PLACE/*.dailycount.txt | bc
}
The head -n NUMBER FILE command outputs the first NUMBER lines. This means that SECBUFF ends up being 1 0 on the second run of the loop, and something like expr 1 + 2 3 is not a valid expression so you get an error from expr.
You can use sed to pick only the nth line from a file, but I wonder if you shouldn't restructure the program somehow.
SECBUFF=`sed -ne ${i}p $file`
This could help. With that variation you could check very input so that only numbers would be added for the sum, even if there are lines that invalid.
function countHourly {
local NUMBERS TOTAL=0 I
readarray -t NUMBERS < <(cat "$PLACE"/*.dailycount.txt)
for I in "${NUMBERS[#]}"; do
[[ $I =~ ^[[:digit:]]+$ ]] && (( TOTAL += I ))
done
echo "Total: $TOTAL"
}
Or
function countHourly {
local NUMBERS TOTAL=0 I
while read I; do
[[ $I =~ ^[[:digit:]]+$ ]] && (( TOTAL += I ))
done < <(cat "$PLACE"/*.dailycount.txt)
echo "Total: $TOTAL"
}
Related
I'm new in bash scripting and I'm trying to make a script which split a large file in multiple files.
I succeeded with case statement but how can I make it without case statement? For example If I have a file with 30 millions of lines (some database file).
Thank you in advance!
echo File which one you want to split
read pathOfFile
echo
countLines=`wc -l < $pathOfFile`
echo The file has $countLines lines
echo
echo In how many files do you want to split?
echo -e "a = 2 files\nb = 3 files\nc = 4 files\nd = 5 files\ne = 10 files\nf = 25 files"
read numberOfFiles
echo
echo The files name with should start:
read nameForFiles
echo
#Split the file
case $numberOfFiles in
a) split -l $(($countLines / 2)) $pathOfFile $nameForFiles;;
b) split -l $(($countLines / 3)) $pathOfFile $nameForFiles;;
c) split -l $(($countLines / 4)) $pathOfFile $nameForFiles;;
d) split -l $(($countLines / 5)) $pathOfFile $nameForFiles;;
e) split -l $(($countLines / 10)) $pathOfFile $nameForFiles;;
f) split -l $(($countLines / 25)) $pathOfFile $nameForFiles;;
*) echo Invalid choice.
esac
You can just use an array to store values then convert your character to an integer to use as an index:
# ...
z=('2' '3' '4' '5' '10' '25')
x=$(( $(printf '%d' "'$numberOfFiles") -97 ))
if [[ $x -lt "${#z[#]}" ]] && [[ $x -ge '0' ]] ; then
split -l $(($countLines / ${z[x]})) $pathOfFile $nameForFiles
else
echo "Invalid choice"
fi
As you can see just convert character to ascii then minus 97 will ensure index lines up within the range of array z.
I found another way to resolve this one, check below:
echo File which one you want to split
read pathOfFile
echo
countLines=`wc -l < $pathOfFile`
echo The file has $countLines lines
echo
echo In how many files do you want to split?
read numberOfFiles
echo
echo The files name with should start:
read nameForFiles
echo
#Split the file
if [[ -n ${numberOfFiles//[0-9]/} ]];
then
echo You type something else than a number. - Bye
exit 1
else
split -l $(($countLines / $numberOfFiles)) -a 3 -d $pathOfFiles $nameForFiles
fi
Is it possible to write a script that reads the file containing numbers (one per line) and writes their maximum, minimum and sum. If the file is empty, it will print an appropriate message. The name of the file is to be given as the parameter of the script. I mange to create below script, but there are 2 errors:
./4.3: line 20: syntax error near unexpected token `done'
./4.3: line 20: `done echo "Max: $max" '
Is it possible to add multiple files as parameter?
lines=`cat "$1" | wc -l`
if [ $lines -eq 0 ];
then echo "File $1 is empty!"
exit fi min=`cat "$1" | head -n 1`
max=$min sum=0
while [ $lines -gt 0 ];
do num=`cat "$1" |
tail -n $lines`
if [ $num -gt $max ];
then max=$num
elif [ $num -lt $min ];
then min=$num fiS
sum=$[ $sum + $num] lines=$[ $lines - 1 ]
done echo "Max: $max"
echo "Min: number $min"
echo "Sum: $sum"
Pretty compelling use of GNU datamash here:
read sum min max < <( datamash sum 1 min 1 max 1 < "$1" )
[[ -z $sum ]] && echo "file is empty"
echo "sum=$sum; min=$min; max=$max"
Or, sort and awk:
sort -n "$1" | awk '
NR == 1 { min = $1 }
{ sum += $1 }
END {
if (NR == 0) {
print "file is empty"
} else {
print "min=" min
print "max=" $1
print "sum=" sum
}
}
'
Here's how I'd fix your original attempt, preserving as much of the intent as possible:
#!/usr/bin/env bash
lines=$(wc -l "$1")
if [ "$lines" -eq 0 ]; then
echo "File $1 is empty!"
exit
fi
min=$(head -n 1 "$1")
max=$min
sum=0
while [ "$lines" -gt 0 ]; do
num=$(tail -n "$lines" "$1")
if [ "$num" -gt "$max" ]; then
max=$num
elif [ "$num" -lt "$min" ]; then
min=$num
fi
sum=$(( sum + num ))
lines=$(( lines - 1 ))
done
echo "Max: $max"
echo "Min: number $min"
echo "Sum: $sum"
The dealbreakers were missing linebreaks (can't use exit fi on a single line without ;); other changes are good practice (quoting expansions, useless use of cat), but wouldn't have prevented your script from working; and others are cosmetic (indentation, no backticks).
The overall approach is a massive antipattern, though: you read the whole file for each line being processed.
Here's how I would do it instead:
#!/usr/bin/env bash
for fname in "$#"; do
[[ -s $fname ]] || { echo "file $fname is empty" >&2; continue; }
IFS= read -r min < "$fname"
max=$min
sum=0
while IFS= read -r num; do
(( sum += num ))
(( max = num > max ? num : max ))
(( min = num < min ? num : min ))
done < "$fname"
printf '%s\n' "$fname:" " min: $min" " max: $max" " sum: $sum"
done
This uses the proper way to loop over an input file and utilizes the ternary operator in the arithmetic context.
The outermost for loop loops over all arguments.
You can do the whole thing in one while loop inside a shell script. Here's the bash version:
s=0
while read x; do
if [ ! $mi ]; then
mi=$x
elif [ $mi -gt $x ]; then
mi=$x
fi
if [ ! $ma ]; then
ma=$x
elif [ $ma -lt $x ]; then
ma=$x
fi
s=$((s+x))
done
if [ ! $ma ]; then
echo "File is empty."
else
echo "s=$s, mi=$mi, ma=$ma"
fi
Save that script into a file, and then you can use pipes to send as many input files into it as you wish, like so (assuming the script is called "mysum"):
cat file1 file2 file3 | mysum
or for a single file
mysum < file1
(Make sure, the script is executable and on the $PATH, otherwise use "./mysum" for the script in the current directory or indeed "bash mysum" if it isn't executable.)
The script assumes that the numbers are one per line and that there's nothing else on the line. It gives a message if the input is empty.
How does it work? The "read x" will take input from stdin line-by-line. If the file is empty, the while loop will never be run, and thus variables mi and ma won't be set. So we use this at the end to trigger the appropriate message. Otherwise the loop checks first if the mi and ma variables exist. If they don't, they are initialised with the first x. Otherwise it is checked if the next x requires updating the mi and ma found thus far.
Note that this trick ensures that you can feed-in any sequence of numbers. Otherwise you have to initialise mi with something that's definitely too large and ma with something that's definitely too small - which works until you encounter a strange number list.
Note further, that this works for integers only. If you need to work with floats, then you need to use some other tool than the shell, e.g. awk.
Just for fun, here's the awk version, a one-liner, use as-is or in a script, and it will work with floats, too:
cat file1 file2 file3 | awk 'BEGIN{s=0}; {s+=$1; if(length(mi)==0)mi=$1; if(length(ma)==0)ma=$1; if(mi>$1)mi=$1; if(ma<$1)ma=$1} END{print s, mi, ma}'
or for one file:
awk 'BEGIN{s=0}; {s+=$1; if(length(mi)==0)mi=$1; if(length(ma)==0)ma=$1; if(mi>$1)mi=$1; if(ma<$1)ma=$1} END{print s, mi, ma}' < file1
Downside: if doesn't give a decent error message for an empty file.
a script that reads the file containing numbers (one per line) and writes their maximum, minimum and sum
Bash solution using sort:
<file sort -n | {
read -r sum
echo "Min is $sum"
while read -r num; do
sum=$((sum+num));
done
echo "Max is $num"
echo "Sum is $sum"
}
Let's speed up by using some smart parsing using tee, tr and calculating with bc and if we don't mind using stderr for output. But we could do a little fifo and synchronize tee output. Anyway:
{
<file sort -n |
tee >(echo "Min is $(head -n1)" >&2) >(echo "Max is $(tail -n1)" >&2) |
tr '\n' '+';
echo 0;
} | bc | sed 's/^/Sum is /'
And there is always datamash. The following willl output 3 numbers, being sum, min and max:
<file datamash sum 1 min 1 max 1
You can try with a shell loop and dc
while [ $# -gt 0 ] ; do
dc -f - -e '
['"$1"' is empty]sa
[la p q ]sZ
z 0 =Z
# if file is empty
dd sb sc
# populate max and min with the first value
[d sb]sY
[d lb <Y ]sM
# if max keep it
[d sc]sX
[d lc >X ]sN
# if min keep it
[lM x lN x ld + sd z 0 <B]sB
lB x
# on each line look for max, min and keep the sum
[max for '"$1"' = ] n lb p
[min for '"$1"' = ] n lc p
[sum for '"$1"' = ] n ld p
# print summary at end of each file
' <"$1"
shift
done
This is my first question on StackOverflow, I hope it's not too noob for this forum. Thanks for your help in advance!!!
[PROBLEM]
I have a Linux bash variable in my bash script with the below content:
[split]
this is a test 1
[split]
this is a test 2
[split]
this is a test 3
this is a test 4
this is a test 5
How can I split this file on the string "[split]" and return the last section after the split?
this is a test 3
this is a test 4
this is a test 5
The last section can vary in length but it is always at the end of the "string" / "file"
Using awk, set record separator to the regular expression representing the split string, print the last record at END.
gawk 'BEGIN{ RS="[[]split[]]" } END{ print $0 }' tmp/test.txt
Result assuming input coming from a file:
this is a test 3
this is a test 4
this is a test 5
How about this ? :)
FILE="test.txt"
NEW_FILE="test_result.txt"
SPLIT="split"
while read line
do
if [[ $line == $SPLIT ]]
then
$(rm ${NEW_FILE})
else
$(echo -e "${line}" >> ${NEW_FILE})
fi
done < $FILE
#!/bin/bash
s="[split]
this is a test 1
[split]
this is a test 2
[split]
this is a test 3
this is a test 4
this is a test 5"
a=()
i=0
while read -r line
do
a[i]="${a[i]}${line}"$'\n'
if [ "$line" == "[split]" ]
then
let ++i
fi
done <<< "$s"
echo ${a[-1]}
I simply read each line from the string into an array and when I encounter [split] ,I increment the array index.At last,I echo the last element.
EDIT:
if you just want the last part no need for an array too.You could do something like
while read -r line
do
a+="${line}"$'\n'
if [ "$line" == "[split]" ]
then
a=""
fi
done <<< "$s"
echo $a
I want to add a special value to the value of a variable. this is my script:
x 55;
y 106;
now I want to change the value of x from 55 to 60.
Generally, how can we apply a math expression on the values of variables in a script?
Others might come up with something simpler (ex: sed, awk, ...), but this quick and dirty script works. It assumes your input file is exactly like you posted:
this is my script.
x 55;
y 106;
And the code:
#!/bin/bash
#
if [ $# -ne 1 ]
then
echo "ERROR: usage $0 <file>"
exit 1
else
inputfile=$1
if [ ! -f $inputfile ]
then
echo "ERROR: could not find $inputfile"
exit 1
fi
fi
tempfile="/tmp/tempfile.$$"
>$tempfile
while read line
do
firstelement=$(echo $line | awk '{print $1}')
if [ "$firstelement" == 'x' ]
then
secondelement=$(echo $line | awk '{print $2}' | cut -d';' -f1)
(( secondelement = secondelement + 5 ))
echo "$firstelement $secondelement;" >>$tempfile
else
echo "$line" >>$tempfile
fi
done <$inputfile
mv $tempfile $inputfile
So it reads the input file line per line. If the line starts with variable x, it takes the number that follows, does +5 to it and outputs it to a temp file. If the line does not start with x, it outputs the line, unchanged, to the temp file. Lastly the temp file overwrite the input file.
Copy this code in a file, make it executable and run it with the input file as an argument.
I want to find number or words in file. As first parameter it gets file name and second number you are looking for.
For example I write in command line:
bash script.sh file.txt 6
And i get on output
Number 6 repeats 4 time
This is content in file.txt
5 4 5 6 2 4 6 3 6 6
This is the code what I came up and stuck
para2=$2
while read line
do
array=($line)
echo "Value of third element in my array : ${array[3]} "
done < $1
I dont know how to compare parameter 2 with every array. I know that in code above I print out third array but I dont know how to go through every array and compare them with parameter two. I mean i want to go through all numbers and compare with input parameter. Pleas help
Try this:
numOccurences=0
while read line
do
array=($line)
for i in "${array[#]}"
do
if [ "$2" = "$i" ]
then
numOccurences=`expr $numOccurences + 1`
fi
done
done < $1
echo "$2 occurs $numOccurences times in $1"
The program will read a line, iterate through the array formed by the line, and then it will compare the value to the target character. A counter is updated for every match, and the result is printed at the end.
Example input (file.txt):
5 4 5 6 2 4 6 3 6 6 6 6
6 6
Command:
/Users/Robert/Desktop/Untitled.sh /Users/Robert/Desktop/file.txt 6
Output:
6 occurs 8 times in /Users/Robert/Desktop/file.txt
para2=$2
counter=0
while read line
do
for num in $line
do
if [[ $num -eq $para2 ]]
then let counter = ((counter + 1))
fi
done
done < "$1"
echo Number $para2 repeats $counter times
#!/bin/bash
echo "Number to be searched $2 "
echo "File name passed : $1"
filename=$1
count=0
while read line
do
for word in $line; do
#echo "Number = $word"
if [ "$2" == "$word" ]; then
count=$(expr $count + 1)
fi
done
done < $filename
echo $2 is observed $count times