I have this list of files on a Linux server:
abc.log.2012-03-14
abc.log.2012-03-27
abc.log.2012-03-28
abc.log.2012-03-29
abc.log.2012-03-30
abc.log.2012-04-02
abc.log.2012-04-04
abc.log.2012-04-05
abc.log.2012-04-09
abc.log.2012-04-10
I've been deleting selected log files one by one, using the command rm -rf see below:
rm -rf abc.log.2012-03-14
rm -rf abc.log.2012-03-27
rm -rf abc.log.2012-03-28
Is there another way, so that I can delete the selected files at once?
Bash supports all sorts of wildcards and expansions.
Your exact case would be handled by brace expansion, like so:
$ rm -rf abc.log.2012-03-{14,27,28}
The above would expand to a single command with all three arguments, and be equivalent to typing:
$ rm -rf abc.log.2012-03-14 abc.log.2012-03-27 abc.log.2012-03-28
It's important to note that this expansion is done by the shell, before rm is even loaded.
Use a wildcard (*) to match multiple files.
For example, the command below will delete all files with names beginning with abc.log.2012-03-.
rm -f abc.log.2012-03-*
I'd recommend running ls abc.log.2012-03-* to list the files so that you can see what you are going to delete before running the rm command.
For more details see the Bash man page on filename expansion.
If you want to delete all files whose names match a particular form, a wildcard (glob pattern) is the most straightforward solution. Some examples:
$ rm -f abc.log.* # Remove them all
$ rm -f abc.log.2012* # Remove all logs from 2012
$ rm -f abc.log.2012-0[123]* # Remove all files from the first quarter of 2012
Regular expressions are more powerful than wildcards; you can feed the output of grep to rm -f. For example, if some of the file names start with "abc.log" and some with "ABC.log", grep lets you do a case-insensitive match:
$ rm -f $(ls | grep -i '^abc\.log\.')
This will cause problems if any of the file names contain funny characters, including spaces. Be careful.
When I do this, I run the ls | grep ... command first and check that it produces the output I want -- especially if I'm using rm -f:
$ ls | grep -i '^abc\.log\.'
(check that the list is correct)
$ rm -f $(!!)
where !! expands to the previous command. Or I can type up-arrow or Ctrl-P and edit the previous line to add the rm -f command.
This assumes you're using the bash shell. Some other shells, particularly csh and tcsh and some older sh-derived shells, may not support the $(...) syntax. You can use the equivalent backtick syntax:
$ rm -f `ls | grep -i '^abc\.log\.'`
The $(...) syntax is easier to read, and if you're really ambitious it can be nested.
Finally, if the subset of files you want to delete can't be easily expressed with a regular expression, a trick I often use is to list the files to a temporary text file, then edit it:
$ ls > list
$ vi list # Use your favorite text editor
I can then edit the list file manually, leaving only the files I want to remove, and then:
$ rm -f $(<list)
or
$ rm -f `cat list`
(Again, this assumes none of the file names contain funny characters, particularly spaces.)
Or, when editing the list file, I can add rm -f to the beginning of each line and then:
$ . ./list
or
$ source ./list
Editing the file is also an opportunity to add quotes where necessary, for example changing rm -f foo bar to rm -f 'foo bar' .
Just use multiline selection in sublime to combine all of the files into a single line and add a space between each file name and then add rm at the beginning of the list. This is mostly useful when there isn't a pattern in the filenames you want to delete.
[$]> rm abc.log.2012-03-14 abc.log.2012-03-27 abc.log.2012-03-28 abc.log.2012-03-29 abc.log.2012-03-30 abc.log.2012-04-02 abc.log.2012-04-04 abc.log.2012-04-05 abc.log.2012-04-09 abc.log.2012-04-10
A wild card would work nicely for this, although to be safe it would be best to make the use of the wild card as minimal as possible, so something along the lines of this:
rm -rf abc.log.2012-*
Although from the looks of it, are those just single files? The recursive option should not be necessary if none of those items are directories, so best to not use that, just for safety.
I am not a linux guru, but I believe you want to pipe your list of output files to xargs rm -rf. I have used something like this in the past with good results. Test on a sample directory first!
EDIT - I might have misunderstood, based on the other answers that are appearing. If you can use wildcards, great. I assumed that your original list that you displayed was generated by a program to give you your "selection", so I thought piping to xargs would be the way to go.
if you want to delete all files that belong to a directory at once.
For example:
your Directory name is "log" and "log" directory include abc.log.2012-03-14, abc.log.2012-03-15,... etc files. You have to be above the log directory and:
rm -rf /log/*
Related
I have many directories of backup starting with "backup_".
I want to keep only the two last created folders.
I did this command to show the last two created:
ls -1 -t -d */ | head -2
The problem is i don't know how to exclude the result of that command from remove command (rm -rf | ...).
I know grep -v only works with strings.
In general, xargs is the tool you want to use to pass a generated list of names as arguments to a command. In your case, you just need to invert the head -2 to a command that prints everything except the first 2 lines. eg:
cmd-to-generate-file-list | sed -e 1,2d | xargs rm
The sed will delete the first two lines, and xargs will call rm with each line of output as an argument. Note that it is not generally safe to use ls to generate the file list, but that is a different issue entirely.
A zsh specific approach:
setopt extended_glob # Turn on extended globbing if it's not already enabled
dirs=( backup_*(#q/om) ) # Match only directories, sorted by modification time - newest first
rm -rf "${dirs[#]:2}" # Delete all but the first two elements of that array of directory names
See the documentation for more on zsh glob qualifiers like the above uses. They can make things with filenames that are tedious or difficult to do in other shell dialects trivial.
In Linux I use InfluxDB which can make a backup of the database for archival purposes. Each backup comprises a series of files with the same prefix "/tank/Backups/var/Influxdb/20191225T235655Z." and different extensions.
I wanted to write a bash script which first deletes the oldest existing backups, then creates a new one (here I paste only the removal):
ls -tp /tank/Backups/var/Influxdb/* | grep -v '/$' | sed -E 's/\..+//' | \
sort -ru | sed 's/$/.*/' | tail -n +4 | xargs -d '\n' -r rm --
However, when I run the script as "sudo", I get
rm: cannot remove '/tank/Backups/var/Influxdb/20191225T235655Z.*': No such file or directory
When I run the quoted script, except the latest part, I get:
/tank/Backups/var/Influxdb/20190930T215357Z.*
/tank/Backups/var/Influxdb/20190930T215352Z.*
which is correct. Also, if I manually write
sudo /tank/Backups/var/Influxdb/20190930T215357Z.*
the command succeeds.
Why is the script reporting an error?
I'm using Ubuntu 18.04 and the folder "/tank" is a ZFS volume.
Better do :
find /tank/Backups/var/Influxdb/* -mtime +5 -delete
to remove files older than 5 days.
Then, you can run the next command
Explaining the Error
This answer is only here to explain the error and give a deeper understanding of what is happening. If you are simply looking for an elegant solution search for other answers.
When I run the quoted script, except the latest part, I get:
/tank/Backups/var/Influxdb/20190930T215357Z.*
/tank/Backups/var/Influxdb/20190930T215352Z.*
which is correct
The listed strings are not what you want. When you pass these paths to rm it sees them just as literal strings, that is, two files whose names end with a literal *. Since you don't have such files you get an error.
When you type rm * manually into your console bash (not rm!) does globbing. bash searches files and replaces the * with the list of found files. Only after that bash executes rm foundFile1 foundFile2 .... rm never sees the *.
Strings inside a pipeline are not processed by bash, but by the commands in the pipeline, in your case rm. rm does not glob.
You could run bash inside your pipeline and let it expand the * you inserted earlier. To this end, replace the last command in your pipeline with xargs -r bash -c 'rm -- $*' --. However, note that your paths are not quoted here. If there are spaces or literal * in your filenames the command will break. This is necessary for globbing as quoted "*" are not expanded by bash.
To quote your files you have to insert the * glob inside the bash command:
ls -tp /tank/Backups/var/Influxdb/* | grep -v '/$' | sed -E 's/\..+//' |
sort -ru | tail -n +4 | xargs -d\\n -L1 -r bash -c 'rm -- "$0."*'
Above command is only a simple fix for your command. It is neither elegant nor very robust. Using tools like find is strongly recommended.
I have a number of files with the following naming:
name1.name2.s01.ep01.RANDOMWORD.mp4
name1.name2.s01.ep02.RANDOMWORD.mp4
name1.name2.s01.ep03.RANDOMWORD.mp4
I need to remove everything between the last . and ep# from the file names and only have name1.name2.s01.ep01.mp4 (sometimes the extension can be different)
name1.name2.s01.ep01.mp4
name1.name2.s01.ep02.mp4
name1.name2.s01.ep03.mp4
This is a simpler version of #Jesse's [answer]
for file in /path/to/base_folder/* #Globbing to get the files
do
epno=${file#*.ep}
mv "$file" "${file%.ep*}."ep${epno%%.*}".${file##*.}"
#For the renaming part,see the note below
done
Note : Didn't get a grab of shell parameter expansion yet ? Check [ this ].
Using Linux string manipulation (refer: http://www.tldp.org/LDP/abs/html/string-manipulation.html) you could achieve like so:
You need to do per file-extension type.
for file in <directory>/*
do
name=${file}
firstchar="${name:0:1}"
extension=${name##${firstchar}*.}
lastchar=$(echo ${name} | tail -c 2)
strip1=${name%.*$lastchar}
lastchar=$(echo ${strip1} | tail -c 2)
strip2=${strip1%.*$lastchar}
mv $name "${strip2}.${extension}"
done
You can use rename (you may need to install it). But it works like sed on filenames.
As an example
$ for i in `seq 3`; do touch "name1.name2.s01.ep0$i.RANDOMWORD.txt"; done
$ ls -l
name1.name2.s01.ep01.RANDOMWORD.txt
name1.name2.s01.ep02.RANDOMWORD.txt
name1.name2.s01.ep03.RANDOMWORD.txt
$ rename 's/(name1.name2.s01.ep\d{2})\..*(.txt)$/$1$2/' name1.name2.s01.ep0*
$ ls -l
name1.name2.s01.ep01.txt
name1.name2.s01.ep02.txt
name1.name2.s01.ep03.txt
Where this expression matches your filenames, and using two capture groups so that the $1$2 in the replacement operation are the parts outside the "RANDOMWORD"
(name1.name2.s01.ep\d{2})\..*(.txt)$
I have a directory full of sub-directories that look like this:
Track_0000111
Track_0004444
Track_0022222
Track_0333333
Track_5555555
I would like to remove certain directories if they are contained within a list in the file "RemoveFromTop6000_reformatted.txt"
The contents of the text file look like this:
Track_0000111
Track_0022222
Track_0333333
I tried to write a small script to handle this, but it does not seem to work:
#!/bin/bash
for file in cat RemoveFromTop6000_reformatted.txt; do
rm -rfv $file
done
Unfortunately this simply removes the text files, rather than the directories. Any tips?
Thanks!
You forgot backquotes around your call to cat. Without them, rm will simply delete the files cat (which probably doesn't exist, but you might not notice because you're using rm -f) and RemoveFromTop6000_reformatted.txt
Try this:
#!/bin/bash
for file in `cat RemoveFromTop6000_reformatted.txt`; do
rm -rv "$file"
done
or, more simply,
rm -rv `cat $file`
(but this will only work if the directory names don't contain whitespace).
No need to for, for something like this you can do a while read ...; do ... done < file just like this:
#!/bin/bash
while read file
rm -rfv "$file"
done < RemoveFromTop6000_reformatted.txt
you can try below command,
Command:
sed 's/^/"/g' sample.txt | sed 's/$/"/g' | xargs rm -rfv
Description:
Command will remove files as well as directories mentioned in "sample.txt".
NOTE:
In your case,make sure that "RemoveFromTop6000_reformatted.txt"
contains only directories name.
Command will also work if the directories name contains whitespace.
I have a long text file with list of file masks I want to delete
Example:
/tmp/aaa.jpg
/var/www1/*
/var/www/qwerty.php
I need delete them. Tried rm `cat 1.txt` and it says the list is too long.
Found this command, but when I check folders from the list, some of them still have files
xargs rm <1.txt Manual rm call removes files from such folders, so no issue with permissions.
This is not very efficient, but will work if you need glob patterns (as in /var/www/*)
for f in $(cat 1.txt) ; do
rm "$f"
done
If you don't have any patterns and are sure your paths in the file do not contain whitespaces or other weird things, you can use xargs like so:
xargs rm < 1.txt
Assuming that the list of files is in the file 1.txt, then do:
xargs rm -r <1.txt
The -r option causes recursion into any directories named in 1.txt.
If any files are read-only, use the -f option to force the deletion:
xargs rm -rf <1.txt
Be cautious with input to any tool that does programmatic deletions. Make certain that the files named in the input file are really to be deleted. Be especially careful about seemingly simple typos. For example, if you enter a space between a file and its suffix, it will appear to be two separate file names:
file .txt
is actually two separate files: file and .txt.
This may not seem so dangerous, but if the typo is something like this:
myoldfiles *
Then instead of deleting all files that begin with myoldfiles, you'll end up deleting myoldfiles and all non-dot-files and directories in the current directory. Probably not what you wanted.
Use this:
while IFS= read -r file ; do rm -- "$file" ; done < delete.list
If you need glob expansion you can omit quoting $file:
IFS=""
while read -r file ; do rm -- $file ; done < delete.list
But be warned that file names can contain "problematic" content and I would use the unquoted version. Imagine this pattern in the file
*
*/*
*/*/*
This would delete quite a lot from the current directory! I would encourage you to prepare the delete list in a way that glob patterns aren't required anymore, and then use quoting like in my first example.
You could use '\n' for define the new line character as delimiter.
xargs -d '\n' rm < 1.txt
Be careful with the -rf because it can delete what you don't want to if the 1.txt contains paths with spaces. That's why the new line delimiter a bit safer.
On BSD systems, you could use -0 option to use new line characters as delimiter like this:
xargs -0 rm < 1.txt
xargs -I{} sh -c 'rm "{}"' < 1.txt should do what you want. Be careful with this command as one incorrect entry in that file could cause a lot of trouble.
This answer was edited after #tdavies pointed out that the original did not do shell expansion.
You can use this one-liner:
cat 1.txt | xargs echo rm | sh
Which does shell expansion but executes rm the minimum number of times.
Just to provide an another way, you can also simply use the following command
$ cat to_remove
/tmp/file1
/tmp/file2
/tmp/file3
$ rm $( cat to_remove )
In this particular case, due to the dangers cited in other answers, I would
Edit in e.g. Vim and :%s/\s/\\\0/g, escaping all space characters with a backslash.
Then :%s/^/rm -rf /, prepending the command. With -r you don't have to worry to have directories listed after the files contained therein, and with -f it won't complain due to missing files or duplicate entries.
Run all the commands: $ source 1.txt
cat 1.txt | xargs rm -f | bash Run the command will do the following for files only.
cat 1.txt | xargs rm -rf | bash Run the command will do the following recursive behaviour.
Here's another looping example. This one also contains an 'if-statement' as an example of checking to see if the entry is a 'file' (or a 'directory' for example):
for f in $(cat 1.txt); do if [ -f $f ]; then rm $f; fi; done
Here you can use set of folders from deletelist.txt while avoiding some patterns as well
foreach f (cat deletelist.txt)
rm -rf ls | egrep -v "needthisfile|*.cpp|*.h"
end
This will allow file names to have spaces (reproducible example).
# Select files of interest, here, only text files for ex.
find -type f -exec file {} \; > findresult.txt
grep ": ASCII text$" findresult.txt > textfiles.txt
# leave only the path to the file removing suffix and prefix
sed -i -e 's/:.*$//' textfiles.txt
sed -i -e 's/\.\///' textfiles.txt
#write a script that deletes the files in textfiles.txt
IFS_backup=$IFS
IFS=$(echo "\n\b")
for f in $(cat textfiles.txt);
do
rm "$f";
done
IFS=$IFS_backup
# save script as "some.sh" and run: sh some.sh
In case somebody prefers sed and removing without wildcard expansion:
sed -e "s/^\(.*\)$/rm -f -- \'\1\'/" deletelist.txt | /bin/sh
Reminder: use absolute pathnames in the file or make sure you are in the right directory.
And for completeness the same with awk:
awk '{printf "rm -f -- '\''%s'\''\n",$1}' deletelist.txt | /bin/sh
Wildcard expansion will work if the single quotes are remove, but this is dangerous in case the filename contains spaces. This would need to add quotes around the wildcards.