I have a problem with bash script. I have a list of files in specific location. I have to take only a date from it and compare it with another date.
for i in *.gz; do
echo $i | grep -Eo '[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}'
done
The above is greping date from filenames correctly but only when I use echo. In another cases I have errors. I have tried:
tmp=$(echo $i | grep -Eo '[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}')
Also not working. Any suggestions? I would be grateful for small help!
I wouldn't use grep at all here; use bash's built-in regular-expression handling.
for i in *.gz; do
[[ $i =~ [[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2} ]]
echo "${BASH_REMATCH[0]}"
done
One way around it, could be using stat command
for i in *.gz; do
tmp=$(stat "$i" | awk '/Modify/ { print $2}' )
done
or if you want an array
declare -a tmp
tmp+=$(
for i in *.gz; do
stat "$i" | awk '/Modify/ { print $2}'
done
)
The advantage is, that it is independent of the file names
edit:
I cannot comment on others answwers yet. So this is how you compare date
sixago=$(date --date='-6 month' +%s)
tmp=$(date --date="$tmp" +%s)
if [ "$tmp" -gt "$sixago" ];then
...
fi
Related
My goal is to write a shell script take the users that I have already filtered out of a file and check whether those users have a certain string, and if they do, label them as major, if not, nonmajor. My trouble is coming from my first if statement, and I'm not sure if grep is the right way to go in an if statement. Here is what I have:
(
while read i
do
username=`echo $i | grep -v 'CMPSC 1513' | grep -P -v '(?!.*CPSMA 2923)CPSMA' | cut -d'|' -f2`
fullname=`echo $i | grep -v 'CMPSC 1513' | grep -P -v '(?!.*CPSMA 2923)CPSMA' | cut -d'|' -f3`
id=`echo $i | grep -v 'CMPSC 1513' | grep -P -v '(?!.*CPSMA 2923)CPSMA' | cut -d'|' -f4`
if [ $username ]
then
if grep -q "|0510"
then
echo $username":(password):(UID):(GID):"$fullname"+"$id":/home/STUDENTS/majors:/bin/bash"
else
echo $username":(password):(UID):(GID):"$fullname"+"$id":/home/STUDENTS/nonmajors:/bin/bash"
fi
fi
done
)<./cs_roster.txt
Just some info, this is contained in a while loop. In the while loop, i determine whether the person listed should even be major or nonmajor, and my if [ $username ] has been tested and does return all the correct users. At this point the while loop is only running once and then stopping.
Just remove the square brackets and pass $i to grep:
if echo $i | grep -q "|0510"
In your code sample, grep does not have anything to work on.
The "binary operator expected" occurs because you are invoking the command [ with the arguments "grep" and "-q" (you are not invoking grep at all), and [ expects a binary operator where you have specified -q. [ is a command, treated no differently that grep or ls or cat. It is better (IMO) to spell it test, and when invoked by the name test it does not require that its last argument be ]. If you want to use grep in an if statement, just do something like:
if echo "$username" | grep -q "|0510"; then ...
(Although I suspect, depending on the context, there are better ways to accomplish your goal.)
The basic syntax of an if statement is if pipeline; then.... In the common case, the pipeline is the simple command test, and at some point in pre-history, the decision was made to provide the name [ for the test command with the added caveat that its final argument must be ]. I believe this was done in an effort to make if statements look more natural, as if the [ is an operator in the language. Just ignore [ and always use test and much confusion will be avoided.
You can use this code as an exercise. Write an awk script for it, or start with something like
while IFS='|' read -r f1 username fullname id otherfields; do
# I don't know which field you want to test. I will rest with id
if [[ $id =~ ^0510 ]]; then
subdir=majors
else
subdir=nonmajors
fi
echo "${username}:(password):(UID):(GID):${fullname}+${id}:/home/STUDENTS/${subdir}:/bin/bash"
done < <( grep -v 'CMPSC 1513' ./cs_roster.txt | grep -P -v '(?!.*CPSMA 2923)CPSMA' )
This is nice for learning some bash syntax, but consider an awk script for avoiding a while-loop.
I apologize if this is a trivial question. I am learning how to use linux bash and this little task is giving me a headache...
So I need to write a script, let's call it count.sh. I want that: for each file in the working directory, prints the filename, the number of lines, and the number of words to the console:
test.txt 100 1023
someOtherfiles 10 233
So far, I know that the following gives me all the files names in the directory. And thanks for all who helped me, I get this working version:
for f in *; do
echo -n "$f"
cat "$f" | wc -wl
done
I would really appreciate your help! Thanks ahead!
P.s. If you know great resources (links for tutorials) for learning about script and you are willing to share it with me. I think I really need to know these basics. Thanks again!
If you must have the file name as the first field in your output, try this:
for f in *; do
if [ -f "$f" ]; then
echo -n "$f"
cat "$f" | wc -wl
fi
done
for f in *; do
if [[ -f $f ]]; then
echo "$f $(wc -wl < "$f")"
fi
done
[[ -f $f ]] processes only files (excludes subdirectories) and also handles the case where the directory is empty (in which case * is (by default) left unexpanded, i.e. assigned to $f as is).
echo "$f $(wc -wl < "$f")" uses command substitution ($( ... )) to directly include the output from the enclosed command in the output string passed to echo.
Note that the reason that < is used to direct the content of file $f to wc via stdin is that wc would otherwise append the name of the input file to its output (thanks, #R Sahu).
I want to search for patterns in a file and remove the lines containing the pattern. To do this, am using:
originalLogFile='sample.log'
outputFile='3.txt'
temp=$originalLogFile
while read line
do
echo "Removing"
echo $line
grep -v "$line" $temp > $outputFile
temp=$outputFile
done <$whiteListOfErrors
This works fine for the first iteration. For the second run, it throws :
grep: input file ‘3.txt’ is also the output
Any solutions or alternate methods?
The following should be equivalent
grep -v -f "$whiteListOfErrors" "$originalLogFile" > "$outputFile"
originalLogFile='sample.log'
outputFile='3.txt'
tmpfile='tmp.txt'
temp=$originalLogFile
while read line
do
echo "Removing"
echo $line
grep -v "$line" $temp > $outputFile
cp $outputfile $tmpfile
temp=$tmpfile
done <$whiteListOfErrors
Use sed for this:
sed '/.*pattern.*/d' file
If you have multiple patterns you may use the -e option
sed -e '/.*pattern1.*/d' -e '/.*pattern2.*/d' file
If you have GNU sed (typical on Linux) the -i option is comfortable as it can modify the original file instead of writing to a new file. (But handle with care, in order to not overwrite your original)
Used this to fix the problem:
while read line
do
echo "Removing"
echo $line
grep -v "$line" $temp | tee $outputFile
temp=$outputFile
done <$falseFailures
Trivial solution might be to work with alternating files; e.g.
idx=0
while ...
let next='(idx+1) % 2'
grep ... $file.$idx > $file.$next
idx=$next
A more elegant might be the creation of one large grep command
args=( )
while read line; do args=( "${args[#]}" -v "$line" ); done < $whiteList
grep "${args[#]}" $origFile
I have a problem with my bash script. I want to excluding from processing files that are listed in the exclude.log. After a file is processed it is written in to the exclude log.
for I in `ls $1 | grep ./exclude.log -v`
do
echo "Procesing ...."
echo $I >> ./exclude.log
done
$I is not assigned a value.
Also your grep is not right formulated.
You possibly want
LIST=$( grep -v -f /path/to/exclude.log * )
for I in $LIST
do
echo "Procesing ...."
echo $I >> /path/to/exclude.log
done
Make sure you don't have any empty lines in exclude.log
You can use this while loop:
while read -r l; do
echo "$l";
done < <(fgrep -v -wf exclude.log <(printf "%s\n" "$1"/*))
I run bash scripts from time to time on my servers, I am trying to write a script that monitors log folders and compress log files if folder exceeds defined capacity. I know there are better ways of doing what I am currently trying to do, your suggestions are more than welcome. The script below is throwing an error "unexpected end of file" .Below is my script.
dir_base=$1
size_ok=5000000
cd $dir_base
curr_size=du -s -D | awk '{print $1}' | sed 's/%//g' zipname=archivedate +%Y%m%d
if (( $curr_size > $size_ok ))
then
echo "Compressing and archiving files, Logs folder has grown above 5G"
echo "oldest to newest selected."
targfiles=( `ls -1rt` )
echo "rocess files."
for tfile in ${targfiles[#]}
do
let `du -s -D | awk '{print $1}' | sed 's/%//g' | tail -1`
if [ $curr_size -lt $size_ok ];
then
echo "$size_ok has been reached. Stopping processes"
break
else if [ $curr_size -gt $size_ok ];
then
zip -r $zipname $tfile
rm -f $tfile
echo "Added ' $tfile ' to archive'date +%Y%m%d`'.zip and removed"
else [ $curr_size -le $size_ok ];
echo "files in $dir_base are less than 5G, not archiving"
fi
Look into logrotate. Here is an example of putting it to use.
With what you give us, you lack a "done" to end the for loop and a "fi" to end the main if. Please reformat your code and You will get more precise answers ...
EDIT :
Looking at your reformatted script, it is as said : The "unexpected end of file" comes from the fact you have not closed your "for" loop neither your "if"
As it seems that you mimick the logrotate behaviour, check it as suggested by #Hank...
my2c
My du -s -D does not show % sign. So you can just do.
curr_size=$(du -s -D)
set -- $curr_size
curr_size=$1
saves you a few overheads instead of du -s -D | awk '{print $1}' | sed 's/%//g.
If it does show % sign, you can get rid of it like this
du -s -D | awk '{print $1+0}'. No need to use sed.
Use $() syntax instead of backticks whenever possible
For targfiles=(ls -1rt) , you can omit the -1. So it can be
targfiles=( $(ls -rt) )
Use quotes around your variables whenever possible. eg "$zipname" , "$tfile"