I'm currently monitoring a number of dirs for log files; specifically those just created. It's been a long time since my Linux and after some trial and error I've hacked together what I need but it takes a full 20secs or more to return. I'm hoping I can have an expert look at it and advise me on something a little more streamlined.
find . -type f -follow -print | xargs ls -ltr 2>/dev/null | grep '2\?10' | tail
So for example find the last 10 files matching the name. Optimally I'd like to turn this into a bash script that accepts one argument and replaces the grep expression but I figure one thing at a time.
Thanks for your help in advance!
I bit the bullet and wrote the script; I'll tinker with it more later.
#!/bin/bash
if [ $# != 2 ]; then
echo findLog Usage: findLog [3 digit cluster] [pick 1: main message service detail soap]
exit 0
fi
if [ "$2" == "service" ]; then
file="$2-time-"
elif [ "$2" == "detail" ]; then
file="$2-time-"
else file="$2-"
fi
cluster="$1"
#store logpaths for readability
a="/pathto/A"
b="/pathto/B"
c="/pathto/C"
d="/pathto/D"
e="/pathto/E"
f="/pathto/F"
g="/pathto/G"
h="/pathto/H"
logpaths=( $a $b $c $d $e $f $g $h )
for i in "${logpaths[#]}"
do
ls -ltr "$i"/*.log | grep "$file"${cluster:0:1}${i: -1}${cluster: -2}
done
Related
I have a quite simple script I'd like to write just using bash.
Given a folder with 0..N *.XML files; I want to sort those by name and remove N-10 files (leave the last 10 in place).
I've been tinkering with find and tail/head but couldn't figure a way
find /mnt/user/Temporary/1 -name *.xml | tail -n +10 | rm
Please read up. It is about keeping the last 10. If there are 10 or less files, none should be deleted!
EDIT:
As someone closed, but did not repoen the question, here is the solution for those getting here with the same question.
#!/bin/bash
files=()
while IFS= read -r -d $'\0'; do
files+=("$REPLY")
done < <(find . -name *.xml -print0 | sort)
Limit=$((${#files[#]}-10))
count=0
while [ $Limit -gt $count ]; do
rm "${files[count]}"
let count=count+1
done
Maybe some linux "pro" can optimize it or give it some parameters (like limit, path and file pattern) to make it callable anywhere.
EDIT: New answer
#!/usr/bin/env bash
files=$(find *.xml | wc -l)
[ "$files" -lt 10 ] && echo "Files are less than 10..." && exit 1
count=$(($files-10))
for i in $(find *.xml | sort -V); do
[ $count -eq 0 ] && echo "Done" && exit 1
rm $i
((count--))
done
$files stores the number of *.xml in the folder
if the number is less or equal to 10 exit
set a counter that of the number of files to delete
loop through each file in order
if the counter is equal to 0 exit
if not remove the file and increment the counter
i want to compute all *bin files inside a given directory. Initially I was working with a for-loop:
var=0
for i in *ls *bin
do
perform computations on $i ....
var+=1
done
echo $var
However, in some directories there are too many files resulting in an error: Argument list too long
Therefore, I was trying it with a piped while-loop:
var=0
ls *.bin | while read i;
do
perform computations on $i
var+=1
done
echo $var
The problem now is by using the pipe subshells are created. Thus, echo $var returns 0.
How can I deal with this problem?
The original Code:
#!/bin/bash
function entropyImpl {
if [[ -n "$1" ]]
then
if [[ -e "$1" ]]
then
echo "scale = 4; $(gzip -c ${1} | wc -c) / $(cat ${1} | wc -c)" | bc
else
echo "file ($1) not found"
fi
else
datafile="$(mktemp entropy.XXXXX)"
cat - > "$datafile"
entropy "$datafile"
rm "$datafile"
fi
return 1
}
declare acc_entropy=0
declare count=0
ls *.bin | while read i ;
do
echo "Computing $i" | tee -a entropy.txt
curr_entropy=`entropyImpl $i`
curr_entropy=`echo $curr_entropy | bc`
echo -e "\tEntropy: $curr_entropy" | tee -a entropy.txt
acc_entropy=`echo $acc_entropy + $curr_entropy | bc`
let count+=1
done
echo "Out of function: $count | $acc_entropy"
acc_entropy=`echo "scale=4; $acc_entropy / $count" | bc`
echo -e "===================================================\n" | tee -a entropy.txt
echo -e "Accumulated Entropy:\t$acc_entropy ($count files processed)\n" | tee -a entropy.txt
The problem is that the while loop is part of a pipeline. In a bash pipeline, every element of the pipeline is executed in its own subshell [ref]. So after the while loop terminates, the while loop subshell's copy of var is discarded, and the original var of the parent (whose value is unchanged) is echoed.
One way to fix this is by using Process Substitution as shown below:
var=0
while read i;
do
# perform computations on $i
((var++))
done < <(find . -type f -name "*.bin" -maxdepth 1)
Take a look at BashFAQ/024 for other workarounds.
Notice that I have also replaced ls with find because it is not good practice to parse ls.
A POSIX compliant solution would be to use a pipe (p file). This solution is very nice, portable, and POSIX, but writes something on the hard disk.
mkfifo mypipe
find . -type f -name "*.bin" -maxdepth 1 > mypipe &
while read line
do
# action
done < mypipe
rm mypipe
Your pipe is a file on your hard disk. If you want to avoid having useless files, do not forget to remove it.
So researching the generic issue, passing variables from a sub-shelled while loop to the parent. One solution I found, missing here, was to use a here-string. As that was bash-ish, and I preferred a POSIX solution, I found that a here-string is really just a shortcut for a here-document. With that knowledge at hand, I came up with the following, avoiding the subshell; thus allowing variables to be set in the loop.
#!/bin/sh
set -eu
passwd="username,password,uid,gid
root,admin,0,0
john,appleseed,1,1
jane,doe,2,2"
main()
{
while IFS="," read -r _user _pass _uid _gid; do
if [ "${_user}" = "${1:-}" ]; then
password="${_pass}"
fi
done <<-EOT
${passwd}
EOT
if [ -z "${password:-}" ]; then
echo "No password found."
exit 1
fi
echo "The password is '${password}'."
}
main "${#}"
exit 0
One important note to all copy pasters, is that the here-document is setup using the hyphen, indicating that tabs are to be ignored. This is needed to keep the layout somewhat nice. It is important to note, because stackoverflow doesn't render tabs in 'code' and replaces them with spaces. Grmbl. SO, don't mangle my code, just cause you guys favor spaces over tabs, it's irrelevant in this case!
This probably breaks on different editor(settings) and what not. So the alternative would be to have it as:
done <<-EOT
${passwd}
EOT
This could be done with a for loop, too:
var=0;
for file in `find . -type f -name "*.bin" -maxdepth 1`; do
# perform computations on "$i"
((var++))
done
echo $var
I need help with the following:
I have these files with names: ABC.rt1, ABC.rt2, ... , ABC.rt8
I should rename all of them in the following way: if the number, let us say i at the end .rt${i} is odd, then replace the extension by appendig AB, else CD. For example I would expect these output:
ABC.rt1 --> ABC.rt1-AB
ABC.rt2 --> ABC.rt1-CD ... etc.
I tried to use this simple script, but does not work:
for i in `seq 1 8`; do mv -v ./ABC.rt${i} ./ABC.rt${if [ $(( $i % 2 )) -eq 0 ] ; then echo ${i}-CD; else echo ${i}-AB;fi};done
Can you help me?
In this specific example it might be easier to make two loops?
for i in *[1357];do mv -v $i $i-AB;done
for i in *[2468];do mv -v $i $i-CD;done
If its a requirement to create a general solution please let me know and I'll help more.
Try:
for i in `seq 1 8`; do
if [ $(( $i % 2 )) -eq 0 ] ; then
newext=${i}-CD
else
newext=${i}-AB
fi
mv -v ./ABC.rt${i} ./ABC.rt$newext
done
Don't try and cram all that in a single line, it's unreadable and your syntax ends up being incorrect.
perl -e'while(<*.rt?>) { rename $_, $_.($1 % 2 ? "-AB" : "-CD") if /\.rt(\d)$/ }'
Before:
a.rt1 a.rt2
After:
a.rt1-AB a.rt2-CD
Note: it is different from the example in your question.
Here you go, all bash, short and concise:
for f in *[1-8]
do
mv $f $f-`echo $f | grep -q '[1357]$' && echo AB || echo CD`
done
I am looking for a command (or way of doing) the following:
echo -n 6 | doif -criteria "isgreaterthan 4" -command 'do some stuff'
The echo part would obviously come from a more complicated string of bash commands. Essentially I am taking a piece of text from each line of a file and if it appears in another set of files more than x (say 100) then it will be appended to another file.
Is there a way to perform such trickery with awk somehow? Or is there another command.. I'm hoping that there is some sort of xargs style command to do this in the sense that the -I% portion would be the value with which to check the criteria and whatever follows would be the command to execute.
Thanks for thy insight.
It's possible, though I don't see the reason why you would do that...
function doif
{
read val1
op=$1
val2="$2"
shift 2
if [ $val1 $op "$val2" ]; then
"$#"
fi
}
echo -n 6 | doif -gt 3 ls /
if test 6 -gt 4; then
# do some stuff
fi
or
if test $( echo 6 ) -gt 4; then : ;fi
or
output=$( some cmds that generate text)
# this will be an error if $output is ill-formed
if test "$output" -gt 4; then : ; fi
I'm trying to write a function that will traverse the file directory and give me the value of the deepest directory. I've written the function and it seems like it is going to each directory, but my counter doesn't seem to work at all.
dir_depth(){
local olddir=$PWD
local dir
local counter=0
cd "$1"
for dir in *
do
if [ -d "$dir" ]
then
dir_depth "$1/$dir"
echo "$dir"
counter=$(( $counter + 1 ))
fi
done
cd "$olddir"
}
What I want it to do is feed the function a directory, say /home, and it'll go down each subdirectory within and find the deepest value. I'm trying to learn recursion better, but I'm not sure what I'm doing wrong.
Obviously find should be used for this
find . -type d -exec bash -c 'echo $(tr -cd / <<< "$1"|wc -c):$1' -- {} \; | sort -n | tail -n 1 | awk -F: '{print $1, $2}'
At the end I use awk to just print the output, but if that were the output you wanted it would be better just to echo it that way to begin with.
Not that it helps learn about recursion, of course.
Here's a one–liner that's pretty fast:
find . -type d -printf '%d:%p\n' | sort -n | tail -1
Or as a function:
depth()
{
find $1 -type d -printf '%d:%p\n' | sort -n | tail -1
}
Here is a version that seems to work:
#!/bin/sh
dir_depth() {
cd "$1"
maxdepth=0
for d in */.; do
[ -d "$d" ] || continue
depth=`dir_depth "$d"`
maxdepth=$(($depth > $maxdepth ? $depth : $maxdepth))
done
echo $((1 + $maxdepth))
}
dir_depth "$#"
Just a few small changes to your script. I've added several explanatory comments:
dir_depth(){
# don't need olddir and counter needs to be "global"
local dir
cd -- "$1" # the -- protects against dirnames that start with -
# do this out here because we're counting depth not visits
((counter++))
for dir in *
do
if [ -d "$dir" ]
then
# we want to descend from where we are rather than where we started from
dir_depth "$dir"
fi
done
if ((counter > max))
then
max=$counter # these are what we're after
maxdir=$PWD
fi
((counter--)) # decrement and test to see if we're back where we started
if (( counter == 0 ))
then
echo $max $maxdir # ta da!
unset counter # ready for the next run
else
cd .. # go up one level instead of "olddir"
fi
}
It prints the max depth (including the starting directory as 1) and the first directory name that it finds at that depth. You can change the test if ((counter > max)) to >= and it will print the last directory name it finds at that depth.
The AIX (6.1) find command seems to be quite limited (e.g. no printf option). If you like to list all directories up to a given depth try this combination of find and dirname. Save the script code as maxdepth.ksh. In comparison to the Linux find -maxdepth option, AIX find will not stop at the given maximum level which results in a longer runtime, depending on the size/depth of the scanned direcory:
#!/usr/bin/ksh
# Param 1: maxdepth
# Param 2: Directoryname
max_depth=0
netxt_dir=$2
while [[ "$netxt_dir" != "/" ]] && [[ "$netxt_dir" != "." ]]; do
max_depth=$(($max_depth + 1))
netxt_dir=$(dirname $netxt_dir)
done
if [ $1 -lt $max_depth ]; then
ret=1
else
ret=0
ls -d $2
fi
exit $ret
Sample call:
find /usr -type d -exec maxdepth.ksh 2 {} \;
The traditional way to do this is to have dir_depth return the maximum depth too. So you'll return both the name and depth.
You can't return an array, struct, or object in bash, so you can return e.g. a comma-separated string instead..
dir_depth(){
local dir
local max_dir="$1"
local max_depth=0
for dir in $1/*
do
if [ -d "$dir" ]
then
cur_ret=$(dir_depth "$dir")
cur_depth=$(expr "$cur_ret" : '\([^,]*\)')
cur_dir=$(expr "$cur_ret" : '.*,\(.*\)')
if [[ "$cur_depth" -gt "$max_depth" ]]; then
max_depth="$cur_depth"
max_dir="$cur_dir"
fi
fi
done
max_depth=$(($max_depth + 1))
echo "$max_depth,$max_dir"
}
EDIT: Fixed now. It starts with the directory you passed in as level 1, then counts upwards. I removed the cd, as it isn't necessary. Note that this will fail if filenames contain commas.
You might want to consider using a programming language with more built-in data structures, like Python.