I need to write a script for a web server that will clean out files/folders older than 14 days, but keep the last 7 files/directories. I've been doing my research so far and here is what I came up with (I know the syntax and commands are incorrect but just so you get an idea):
ls -ldt /data/deployments/product/website.com/*/ | tail -n +8 | xargs find /data/deployments/product/website.com/ -type f -type d -mtime +14 -exec rm -R {} \;
This is my thought process as to how the script should behave (I'm more a windows batch guy):
List the directory contents
If contents is less than or equal to 7, goto END
If contents is > 7 goto CLEAN
:CLEAN
ls -ldt /data/deployments/product/website.com/*/
keep last 7 entries (tail -n +8)
output of that "tail" -> find -type f -type d (both files and directories) -mtime +14 (not older than 14 days) -exec rm -R (delete)
I've seen a bunch of examples, using xargs and sed but I just can't figure out how to put it all together.
#!/bin/bash
find you_dir -mindepth 1 -maxdepth 1 -printf "%T# %p\n" | \
sort -nrk1,1 |sed '1,7d' | cut -d' ' -f2 | \
xargs -n1 -I fname \
find fname -maxdepth 0 -mtime +14 -exec echo rm -rf {} \;
remove the echo if your happy with the output...
Explanation (line-by-line):
find in exactly in your_dir and print seconds_since_Unix_epoch (%T#) and file(/dir)name for each file/dir on a separate line
sort by first field (seconds_since_Unix_epoch) descending, throw the first seven lines away - from the rest extract just the name (second field)
xargs passes on to new find process argument-by-argument (-n1) and uses fname to represent argument
-maxdepth 0 limits find to just fname
You could store the minNrOfFiles and the ageLimit in Bash-Variables or pass in to the script with just few changes:
minNrOfFiles=7 # or $1
ageLimit=14 # or $2
change: sed '1,'"$minNrOfFiles"'d' and -mtime +"$ageLimit"
Related
I want to remove files in directory except a specific file and print a message if files remove.
find . ! -name 'name' -type f -exec echo 'removed files:' rm -v -f {} +
When I run this command it prints:
removed files: rm -v -f ./aaavl ./aaavlpo
I want to print putput like:
removed files:
./aaavl
./aaavlpo
How should I do this?
Just use find in a Bash loop to modify the output.
Given:
$ ls -1
file 1
file 2
file 3
file 4
file 5
You can still loop and negate with find as desired. Just use Bash to delete and report:
$ find . ! -name *3 -type f | while read fn; do echo "removing: $fn"; rm "$fn"; done
removing: ./file 1
removing: ./file 2
removing: ./file 5
removing: ./file 4
$ ls -1
file 3
That loop will work for filenames with spaces OTHER THAN \n.
If there is a possibility of file names with \n in them, use xargs with a NUL delimiter:
$ find . ! -name *3 -type f -print0 | xargs -0 -n 1 -I% bash -c ' echo "%" ; rm "%" ;'
And add the header echo "removed files:" above the loop or xargs pipe as desired.
maybe it's not a better way, but ...
1 - Save the file names in .txt:
find -name 'name' -type f -exec echo >> test.txt {} \;
2 after saving the files, executing the deletion of the files
find. ! -name 'name' -type f -exec rm -f {} +
I am trying to get the total number of lines in all files in a directory.
I tried to do the following:
cat * | wc -1
to get the total number of lines in the directory, but it gives me a message that some of the files are directories. ('cat : some_dir: Is a directory')
How can I exclude directories when concatenating all files?
To get just sum you can try something like below, get count of each file and sum it
find . -type f -exec wc -l {} \; | awk '{ SUM += $1} END { print SUM }'
add -maxdepth 1 which skips scanning further
-type f for filtering only files
Here is test results :
$ seq 1 4 >file1
$ seq 1 5 >file2
$ cat file1
1
2
3
4
$ cat file2
1
2
3
4
5
$ find . -type f -exec wc -l {} \;
5 ./file2
4 ./file1
$ find . -type f -exec wc -l {} \; | awk '{ SUM += $1} END { print SUM }'
9
$ find . -type f -exec wc -l {} +
5 ./file2
4 ./file1
9 total
$ find . -type f -exec wc -l {} + | awk 'END{print $1}'
9
Under bash:
shopt -s globstar
wc -l **/*
From bash's man page:
globstar
If set, the pattern ** used in a pathname expansion con‐
text will match all files and zero or more directories
and subdirectories. If the pattern is followed by a /,
only directories and subdirectories match.
But care: this will take symlinks too!
Some tricks about find
If you want to read files only:
find . -type f -exec wc -l {} +
and, for total only:
find . -type f -exec wc -l {} + | tail -n 1
Huge dirs:
Syntax find ... -exec ... + will limit groupping args to maximum line length. So if your tree is really big, the previous command will generate more than only one fork to wc.
-exec command {} +
This variant of the -exec action runs the specified command on
the selected files, but the command line is built by appending
each selected file name at the end; the total number of invoca‐
tions of the command will be much less than the number of
matched files. The command line is built in much the same way
that xargs builds its command lines. Only one instance of `{}'
is allowed within the command, and (when find is being invoked
from a shell) it should be quoted (for example, '{}') to protect
it from interpretation by shells. The command is executed in
the starting directory. If any invocation returns a non-zero
value as exit status, then find returns a non-zero exit status.
If find encounters an error, this can sometimes cause an immedi‐
ate exit, so some pending commands may not be run at all. This
variant of -exec always returns true.
find . -type f -exec wc -l {} + | awk 'BEGIN{t=0};/total/{t+=$1};END{print t}'
This will compute sum of total lines.
Alternative: using bash for summarize
For fun, as there is no real improvment:
tot=0
while read val nam;do
[ "$nam" = "total" ] && ((tot+=val))
done < <( find . -type f -exec wc -l {} + )
echo $tot
I have a folder and I want count all regular files in it, and for this I use this bash command:
find pathfolder -type f 2> err.txt | wc -l
In the folder there are 3 empty text files and a subfolder with inside it other text files.
For this reason I should get 3 as a result, but I get 6 and I don't understand why. Maybe there is some options that I did not set.
If I remove the subfolder I get 4 as result
To grab all the files and directories in current directory without dot files:
shopt -u dotglob
all=(*)
To grab only directories:
dirs=(*/)
To count only non-dot files in current directory:
echo $(( ${#all[#]} - ${#dirs[#]} ))
To do this with find use:
find . -type f -maxdepth 1 ! -name '.*' -exec printf '%.0s.\n' {} + | wc -l
Below solutions ignore the filenames starting with dot.
To count the files in pathfolder only:
find pathfolder -maxdepth 1 -type f -not -path '*/\.*' | wc -l
To count the files in ALL child directories of pathfolder:
find pathfolder -mindepth 2 -maxdepth 2 -type f -not -path '*/\.*' | wc -l
UPDATE: Converting comments into an answer
Based on the suggestions received from anubhava, by creating a dummy file using the command touch $'foo\nbar', the wc -l counts this filename twice, like in below example:
$> touch $'foo\nbar'
$> find . -type f
./foo?bar
$> find . -type f | wc -l
2
To avoid this, get rid of the newlines before calling wc (anubhava's solution):
$> find . -type f -exec printf '%.0sbla\n' {} +
bla
$> find . -type f -exec printf '%.0sbla\n' {} + | wc -l
1
or avoid calling wc at all:
$> find . -type f -exec sh -c 'i=0; for f; do ((i++)); done; echo $i' sh {} +
1
I am looking to combine the output of the Linux find and head commands (to derive a list of filenames) with output of another Linux/bash command and save the result in a file such that each filename from the "find" occurs with the other command output on a separate line.
So for example,
- if a dir testdir contains files a.txt, b.txt and c.txt,
- and the output of the other command is some number say 10, the desired output I'm looking for is
10 a.txt
10 b.txt
10 c.txt
On searching here, I saw folks recommending paste for doing similar merging but I couldn't figure out how to do it in this scenario as paste seems to be expecting files . I tried
paste $(find testdir -maxdepth 1 -type f -name "*.text" | head -2) $(echo "10") > output.txt
paste: 10: No such file or directory
Would appreciate any pointers as to what I'm doing wrong. Any other ways of achieving the same thing are also welcome.
Note that if I wanted to make everything appear on the same line, I could use xargs and that does the job.
$find testdir -maxdepth 1 -type f -name "*.text" | head -2 |xargs echo "10" > output.txt
$cat output.txt
10 a.txt b.txt
But my requirement is to merge the two command outputs as shown earlier.
Thanks in advance for any help!
find can handle both the -exec and -print directives, you just need to merge the output:
$ find . -maxdepth 1 -type f -name \*.txt -exec echo hello \; -print | paste - -
hello ./b.txt
hello ./a.txt
hello ./all.txt
Assuming your "command" requires the filename (here's a very contrived example):
$ find . -maxdepth 1 -type f -name \*.txt -exec sh -c 'wc -l <"$1"' _ {} \; -print | paste - -
4 ./b.txt
4 ./a.txt
7 ./all.txt
Of course, that's executing the command for each file. To restrict myself to your question:
cmd_out=$(echo 10)
for file in *.txt; do
echo "$cmd_out $file"
done
Try this,
$find testdir -maxdepth 1 -type f -name "*.text" | head -2 |tr ' ' '\n'|sed -i 's/^/10/' > output.txt
You can make xargs operate on one line at a time using -L1:
find testdir -maxdepth 1 -type f -name "*.text" | xargs -L1 echo "10" > output.txt
I have a requirement where I need to display the last lines of all the files under a directory in the format
filename: lastline
I found the following code
find /user/directory/* -name "*txt" -mtime 0 -type f -exec awk '{s=$0};END{print FILENAME, ": ",s}' {} \;
But I read this reads the entire file each time. The files in my directory are huge so I cannot afford this. Do I have any alternatives?
find /user/directory/* -name "*txt" -mtime 0 -type f | while IFS= read -r file
do
echo -n "$file: "
tail -1 "$file"
done
The important change is that tail -1 won't read the whole file, but reads small portions from the end and increases them until it has found the complete last line.
If you know the directory name:
for f in $(/bin/ls directory/*.txt); do
echo "$f: $(tail -1 $f)"
done
will do the trick. More generally,
for f in $(find /user/directory -type f -name "*.txt"); do
echo "$f: $(tail -1 $f)"
done
will work as well. The program tail will start reading the file from the end, and tail -n will only read the last n lines of a specified file.
Using tail as in the other answers is good. Now, you can wrap all this into the find command.
If your find supports the -printf command:
find /user/directory/ -name "*txt" -mtime 0 -type f -printf '%p: ' -exec tail -1 {} \;
If your find doesn't support the -printf command:
find /user/directory/ -name "*txt" -mtime 0 -type f -exec printf '%s: ' {} \; -exec tail -1 {} \;