I have a set of files that come in pairs:
/var/log/messages-20111001
/var/log/messages-20111001.hash
I've had several of these rotate away and now I'm left with a ton of /var/log/messages-201110xx.hash files with no associated log. I'd like to clean up the mess, but I'm uncertain how to remove a file that isn't part of a "pair". I can use bash or zsh (or any LSB tool, really). I need to remove all the .hash files that don't have an associated log.
Example
/var/log/messages-20111001.hash
/var/log/messages-20111002.hash
/var/log/messages-20111003.hash
/var/log/messages-20111004.hash
/var/log/messages-20111005
/var/log/messages-20111005.hash
/var/log/messages-20111006
/var/log/messages-20111006.hash
Should be reduced to:
/var/log/messages-20111005
/var/log/messages-20111005.hash
/var/log/messages-20111006
/var/log/messages-20111006.hash
for file in *.hash; do test -f "${file%.hash}" || rm -- "$file"; done
Something like this?
for f in /var/log/messages-????????.hash ; do
[[ -e "${f%.hash}" ]] || rm "$f"
done
Related
I currently have a long list of files, which look somewhat like this:
Gmc_W_GCtl_E_Erz_Aue_Dl_281_heart_xerton
Gmc_W_GCtl_E_Erz_Aue_Dl_254_toe_taixwon
Gmc_W_GCtl_E_Erz_Homersdorf_Dl_201_head_xaubadan
Gmc_W_GCtl_E_Erz_Homersdorf_Dl_262_bone_bainan
Gmc_W_GCtl_E_Thur_Peuschen_Dl_261_blood_blodan
Gmc_W_GCtl_E_Thur_Peuschen_Dl_281_heart_xerton
The naming pattern all follow the same order, where I'm mainly seeking to group the files based on the part with "Aue", "Homersdorf", "Peuschen", and so forth (there are many others down the list), with the position of these keywords being always the same (e.g. they are all followed by Dl; they are all after the fifth underscore...etc.).
All the files are in the same folder, and I am trying to move these files into subfolders based on these keywords in bash, but I'm not quite certain how. Any help on this would be appreciated, thanks!
I am guessing you want something like this:
$ find . -type f | awk -F_ '{system("mkdir -p "$5"/"$6";mv "$0" "$5"/"$6)}'
This will move say Gmc_W_GCtl_E_Erz_Aue_Dl_281_heart_xerton into /Erz/Aue/Gmc_W_GCtl_E_Erz_Aue_Dl_281_heart_xerton.
Using the bash shell with a for loop.
#!/usr/bin/env bash
shopt -s nullglob
for file in Gmc*; do
[[ -d $file ]] && continue
IFS=_ read -ra dir <<< "$file"
echo mkdir -pv "${dir[4]}/${dir[5]}" || exit
echo mv -v "$file" "${dir[4]}/${dir[5]}" || exit
done
Place the script inside the directory in question make it executable and execute it.
Remove the echo's so it create the directories and move the files.
So in bash, if I want I can simply do(where foo is a list of paths to files):
for i in `cat foo`; do ls -lah $i; done
I have a list of files I need to search for. My thought is; why not run them through a for loop with mlocate? I could do:
for i in `cat foo`; do locate $i; done
...but is that the best way to do what I'm trying to do?
Find is SLOW and takes forever when there are millions of files and directories whereas mlocate is super quick.
If files.txt contains a list of absolute paths with a newline line terminator you can do this to ensure they all exist:
set -o errexit
mapfile -t < files.txt
for path in "${MAPFILE[#]}"
do
[[ -e "$path" ]]
done
You can then expand on this if you want to do certain things with existing/non-existing files:
if [[ -e "$path" ]]
then
…
else
…
fi
If files.txt is so huge that the list does not fit in memory you can use a much slower while read loop:
while read -r -u 9 path
do
[[ -e "$path" ]]
done 9< files.txt
If speed really is of the essence you probably want to do this in a different language, like Java or Rust.
On a technical note, mlocate is fast because it queries a pre-generated list of files on your system, but its database does not stay in sync with the actual filesystem contents automatically. Instead you need to run updatedb to populate the database with the current filesystem contents. This is usually done by a root cron job daily.
In terms of style, backticks are deprecated for $(COMMAND) and Use More Quotes™.
In my /opt/myapp dir I have a remote, automated process that will be dropping files of the form <anything>-<version>.zip, where <anything> could literally be any alphanumeric filename, and where <version> will be a version number. So, examples of what this automated process will be delivering are:
fizz-0.1.0.zip
buzz-1.12.35.zip
foo-1.0.0.zip
bar-3.0.9.RC.zip
etc. Through controls outside the scope of this question, I am guaranteed that only one of these ZIP files will exist under /opt/myapp at any given time. I need to write a Bash shell command that will rename these files and move them to /opt/staging. For the rename, the ZIP files need to have their version dropped. And so /opt/myapp/<anything>-<version>.zip is renamed and moved to /opt/staging/<anything>.zip. Using the examples above:
/opt/myapp/fizz-0.1.0.zip => /opt/staging/fizz.zip
/opt/myapp/buzz-1.12.35.zip => /opt/staging/buzz.zip
/opt/myapp/foo-1.0.0.zip => /opt/staging/foo.zip
/opt/myapp/bar-3.0.9.RC.zip => /opt/staging/bar.zip
The directory move is obvious and easy, but the rename is making me pull my hair out. I need to somehow save off the <anything> and then re-access it later on in the command. The command must be generic and can take no arguments.
My best attempt (which doesn't even come close to working) so far is:
file=*.zip; file=?; mv file /opt/staging
Any ideas on how to do this?
for file in *.zip; do
[[ -e $file ]] || continue # handle zero-match case without nullglob
mv -- "$file" /opt/staging/"${file%-*}.zip"
done
${file%-*} removes everything after the last - in the filename. Thus, we change fizz-0.1.0.zip to fizz, and then add a leading /opt/staging/ and a trailing .zip.
To make this more generic (working with multiple extensions), see the following function (callable as a command; function body could also be put into a script with a #!/bin/bash shebang, if one removed the local declarations):
stage() {
local file ext
for file; do
[[ -e $file ]] || continue
[[ $file = *-*.* ]] || {
printf 'ERROR: Filename %q does not contain a dash and a dot\n' "$file" >&2
continue
}
ext=${file##*.}
mv -- "$file" /opt/staging/"${file%-*}.$ext"
done
}
...with that function defined, you can run:
stage *.zip *.txt
...or any other pattern you so choose.
f=foo-1.3.4.txt
echo ${f%%-*}.${f##*.}
Does anyone know of a way to (via bash) setup a "versioning" copy of a file into another? For example: I am copying file into file.bak. If file.bak exists, I am currently overwriting. What I'd like to do is set it up so that it creates multiple files: file, file.bak, file.bak.1, file.bak.2, etc...
Right now, I'm using:
cp -rf file file.bak
This currently overwrites the file(as expected)
or:
cp --backup=t file1 file2
repeat few times to see the result...
see https://www.gnu.org/software/coreutils/manual/html_node/cp-invocation.html
Simply use a test
[ -e file.bak ] && cp -r file file.bak.$(date +%s) || cp -r file file.bak
This will create a unique backup if file.bak already exists in the form file.bak.1411505497
There are many ways to skin this cat.
Since you're using Linux, it's likely you've got the GNU mv command, which may include a --backup option. You could wrap this in a shell function:
bkp() {
file="$1"
if [ -f "$file" ]; then
/bin/mv -v --backup=numbered "$(mktemp ${file}XXX)" "$file"
#/bin/rm "$file"
fi
}
You can put this in your .bashrc, for example. Then you can use this as follows:
# bkp foo
This will copy foo to numbered backup files. You can uncomment the rm if this is, for example, a log file that you're rotating.
Another option, which is more portable to operating systems that don't use GNU tools (i.e. FreeBSD, OSX) might be something like this quick-and-dirty solution might work:
bkp() {
file="$1"
if [ -f "$file" ]; then
# increment existing files up to 10
for n in {9..1}; do
if [ -f $file.$n ]; then
# remove -v if you want less noise.
mv -v "${file}.$n" "${file}.$[n+1]"
fi
done
# move the original to first backup position
mv "$file" "$file.1"
else
echo "Not found: $file" >&2
fi
}
It suffers in that it won't compact your list of files (and will throw errors) if some numbers are missing, but that's stuff you can add if it's important. You'd use it pretty much the same way, changing the final mv to a cp if you need to keep the original in place.
Final option I'll mention is in comments as well. Since you've said that you're using this solution to back up "system files" (which I assume you mean to be things in /etc/) you should consider using an actual version control system to control your versions of these files.
Many options exist, but I'd recommend RCS for its simplicity and low overhead. Simply install the package, mkdir /etc/RCS to keep your /etc directory clean, read the man pages for rcs, ci, co, rlog, rcsdiff and perhaps rcsintro, and you're good to go. You'll get better control of diffs and history, opportunity for comments, none of the overhead of a repository for a large VCS like SVN or Git. I've been using this on various servers for years, as RCS is still built in to the base system in FreeBSD. :)
I'm on a linux, and a download a lot of funny pictures. Unfortunately, I'm left with a bunch of duplicate names like download (1) and image.jpeg. I would like to change them all to something a bit more helpful.
Is there a way to (preferably using bash) to rename all files to sequential 4 digit numbers with leading zeroes?
Eg:
0001
0002
0003
0004
....
The code snippet provided in the previous answer, is an elegant way of doing it but a typo or a shell incompatibility may cause it not to function properly.
please try the code below instead. It does the same thing but every shortcut has been explicitly written with debugging echo commands in the loop.
counter=1
cd /my/image/directory
for f in $(ls -1)
do
new_filename=$(printf "%04d" ${counter})
echo "renaming ${f} ..to.. ${new_filename}"
mv ${f} ${new_filename}
(( counter=${counter}+1 ))
done
the screen output will be a little chatty. if you have too many files, you might want to add | tee screen.out to the end of the line with done command. So that you can go back and see what happened to which file recorded in the screen.out.
I created my own tool to do this. It also maintains file extensions, which I did not mention, but should probably be included. Here is the code:
#!/bin/sh
dir=$1
cd $dir
echo "Renaming all files in $dir."
COUNTER=1
for i in `ls -1`
do
extension=${i##*.}
mv "$i" "$COUNTER.$extension"
echo "$i ==> $COUNTER.$extension"
COUNTER=$(expr $COUNTER + 1 )
done
It does not (at the time of writing) include the leading zeroes, but it gets the job done.
As long as you don't care which file is renamed to what, it's easy :)
counter=1
for f in *; do
mv "$f" "$( printf "%04d" $((counter++)) )"
done
Trying to rename all files with suffix .bash to suffix .sh in a folder is easily done with
rename .bash .sh *.bash