Append directory name to the end of the files and move them - linux

I am finding some files in a directory using this command:
find /Users/myname -type f
output is:
/Users/myname/test01/logs1/err.log
/Users/myname/test01/logs1/std
/Users/myname/test01/logs2/std
/Users/myname/test02/logs2/velocity.log
/Users/myname/test03/logs3/err.log
/Users/myname/test03/logs3/invalid-arg
I need to move this files to a different directory by appending the test directory name to the end of the files. Like below:
err.log-test01
std-test01
std-test01
velocity.log-test02
err.log-test03
invalid-arg-test03
I am trying with the cut command but not getting the desired output.
find /Users/myname -type f | cut -d'/' -f6,4
plus, I also need to move the files to a different directory. I guess a suitable way could be there using sed command, but I am not proficient with sed. How this can be achieved in an efficient way?

You can let find create the mv command, use sed to modify it and then have it run by the shell:
find /Users/myname -type f -printf "mv %p /other/dir/%f\n" |
sed 's,/\(test[0-9]*\)/\(.*\),/\1/\2-\1,' | sh
This assumes there are no spaces in any argument, otherwise liberally add ' or ". Also run it without the final | sh to see what it actually wants to do. If you need to anchor the test[0-9]* pattern better you can include part of the left or right string to match:
's,myname/\(test[0-9]*\)/\(.*\),myname/\1/\2-\1,'

You can move it from the dst to the dst_dir appending the directory, using awk, and the target name would be awk -F/ '{print $5 "-" $4}'. The full command could be as simple as:
for i in `find . -type f`
do mv $i /dst_dir/`echo $i| awk -F/ '{print $5 "-" $4}' `
done

There are a number of things going on that you may want to use a helper script with find to insure you can validate the existence of the directory to move the files to, etc.. A script might take the form of:
#!/bin/bash
[ -z $1 -o -z $2 ] && { # validate at least 2 arguments
printf "error: insufficient input\n"
exit 1
}
ffn="$1" # full file name provided by find
newdir="$2" # the target directory
# validate existence of 'newdir' or create/exit on failure
[ -d "$newdir" ] || mkdir -p "$newdir"
[ -d "$newdir" ] || { printf "error: uname to create '$newdir'\n"; exit 1; }
tmp="${ffn##*test}" # get the test## number
num="${tmp%%/*}"
fn="${ffn##*/}" # remove existing path from ffn
mv "$ffn" "${newdir}/${fn}-test${num}" # move to new location
exit 0
Save it in a location where it is accessible under a name like myscript and make it executable (e.g. chmod 0755 myscript) You may also choose to put it in a directory within your path. You can then call the script for every file returned by find with:
find /Users/myname -type f -exec ./path/to/myscript '{}' somedir \;
Where somedir is the target directory for the renamed file. Helper scripts generally provide the ability to do required validation that would otherwise not be done in one-liners.

Related

How to output difference of files from two folders and save the output with the same name on different folder

I have two folders which have same file names, but different contents. So, I am trying to generate a script to get the difference and to see what is being changed. I wrote a script below :
folder1="/opt/dir1"
folder2=`ls/opt/dir2`
find "$folder1/" /opt/dir2/ -printf '%P\n' | sort | uniq -d
for item in `ls $folder1`
do
if [[ $item == $folder2 ]]; then
diff -r $item $folder2 >> output.txt
fi
done
I believe this script has to work, but it is not giving any output on output folder.
So the desired output should be in one file . Ex:
cat output.txt
diff -r /opt/folder1/file1 /opt/folder2/file1
1387c1387
< ALL X'25' BY SPACE
---
> ALL X'0A' BY SPACE
diff -r /opt/folder1/file2 /opt/folder2/file2
2591c2591
< ALL X'25' BY SPACE
---
> ALL X'0A' BY SPACE
Any help is appreciated!
Ok. So twofold:
First get the files in one folder. Never use ls. Forget it exists. ls is for nice printing in our console. In scripts, use find.
Then for each file do some command. A simple while read loop.
So:
{
# make find print relative to `/opr/dir1` director
cd /opt/dir1 &&
# Use `%P` so that print without leading `./`
find . -mindepth 1 -type f -print "%P\n"
} |
while IFS= read -r file; do
diff /opt/dir1/"$file" /opt/dir2/"$file" >> output/"$file"
done
Notes:
always quote your variable
Why you shouldn't parse the output of ls(1)

How to move files in Linux based on its name in a folder with a corresponding name?

I would need to move a series of files in certain folders via scripts. The files are of the format xxxx.date.0000 and I have to move them to a folder whose name is the same value given.
For example:
file hello.20190131.0000
in folder 20190131
The ideal would be to be able to create folders even before moving files but it is not a priority because I can create them by hand. I managed to get the value of dates on video with
ls * .0000 | awk -F. '{Print $ 2}'
Does anyone have any suggestions on how to proceed?
The initial awk command provided much of the answer. You just need to do something with the directory name you extract:
A simple option:
ls *.0000 | awk -F. '{printf "mkdir -p '%s'; mv '%s' '%s';",$2,$0,$2}' | sh
This might be more efficient with a large number of files:
ls *.0000 | awk -F. '{print $2}' |\
sort | uniq |\
while read dir; do
mkdir -p "$dir"
mv *."$dir".0000 "$dir"
done
I would do something like this:
ls *.0000 |\
sort |\
while read f; do
foldername="`echo $f | cut -d. -f2`"
echo mkdir +p "$foldername/"
echo mv "$f" "$foldername/"
done
i.e.: For eache of your files, I build the folder name using the cut command with a dot as field separator, and getting the second field (the date in this case); then I create that folder with mkdir -p (the -p flag avoids any warning if the folder should exist already), and finally I move the file to the brand new folder.
You can do that with rename, a.k.a. Perl rename.
Try it on a COPY of your files in a temporary directory.
If you use -p parameter, it will make any necessary directories for you automatically. If you use --dry-run parameter, you can see what it would do without actually doing anything.
rename --dry-run -p 'my #X=split /\./; $_=$X[1] . "/" . $_' hello*
Sample Output
'hello.20190131.0000' would be renamed to '20190131/hello.20190131.0000'
'hello.20190137.0000' would be renamed to '20190137/hello.20190137.0000'
All you need to know is that it passes you the current name of the file in a variable called $_ and it expects you to change that to return the new filename you would like.
So, I split the current name into elements of an array X[] with the dot (period) as the separator:
my #X = split /\./
That gives me the output directory in $X[1]. Now I can set the new filename I want by putting the new directory, a slash and the old filename into $_:
$_=$X[1] . "/" . $_
You could also try this, shorter version:
rename --dry-run -p 's/.*\.(\d+)\..*/$1\/$_/' hello*
On ArchLinux, the package you would use is called perl-rename.
On debian, it is called rename
On macOS, use homebrew like this: brew install rename

How to rename file based on parent and child folder name in bash script

I would like to rename file based on parent/subparent directories name.
For example:
test.xml file located at
/usr/local/data/A/20180101
/usr/local/data/A/20180102
/usr/local/data/B/20180101
how to save test.xml file in /usr/local/data/output as
A_20180101_test.xml
A_20180102_test.xml
b_20180101_test.xml
tried shall script as below but does not help.
#!/usr/bin/env bash
target_dir_path="/usr/local/data/output"
for file in /usr/local/data/*/*/test.xml; do
l1="${file%%/*}"
l2="${file#*/}"
l2="${l2%%/*}"
filename="${file##*/}"
target_file_name="${l1}_${l2}_${filename}"
echo cp "$file" "${target_dir_path}/${target_file_name}"
done
Anything i am doing wrong in this shall script?
You can use the following command to do this operation:
source_folder="usr/local/data/";target_folder="target"; find $source_folder -type f -name test.xml | awk -v targetF=$target_folder 'BEGIN{FS="/"; OFS="_"}{printf $0" "; print targetF"/"$(NF-2),$(NF-1),$NF}' | xargs -n2 cp;
or on several lines for readibility:
source_folder="usr/local/data/";
target_folder="target";
find $source_folder -type f -name test.xml |\
awk -v targetF=$target_folder 'BEGIN{FS="/"; OFS="_"}{printf $0" "; print targetF"/"$(NF-2),$(NF-1),$NF}' |\
xargs -n2 cp;
where
target_folder is your target folder
source_folder is your source folder
the find command will search for all the test.xml named files present under this source folder
then the awk command will receive the target folder as a variable to be able to use it, then in the BEGIN bloc you define the field separator and output field separator, then you just print the initial filename as well as the new one
you use xargs to pass the result output grouped by 2 to the cp command and the trick is done
TESTED:
TODO:
you will just need to set up your source_folder and target_folder variables with what is on your environment and eventually put it in a script and you are good to go!
I've modified your code a little to get it to work. See comments in code
target_dir_path=""/usr/local/data/output"
for file in /usr/local/data/*/*/test.xml; do
tmp=${file%/*/*/*}
curr="${file#"$tmp/"}" # Extract wanted part of the filename
mod=${curr//[\/]/_} # Replace forward slash with underscore
mv "$file" "$target_dir_path$mod" # Move the file
done
if you have perl based rename command
$ for f in tst/*/*/test.xml; do
rename -n 's|.*/([^/]+)/([^/]+)/(test.xml)|./$1_$2_$3|' "$f"
done
rename(tst/A/20180101/test.xml, ./A_20180101_test.xml)
rename(tst/A/20180102/test.xml, ./A_20180102_test.xml)
rename(tst/B/20180101/test.xml, ./B_20180101_test.xml)
-n option is for dry run, remove it after testing
change tst to /usr/local/data and ./ to /usr/local/data/output/ for your usecase
.*/ to ignore file path
([^/]+)/([^/]+)/(test.xml) capture required portions
$1_$2_$3 re-arrange as required

Find out if a directory has any files in it using Bash?

I'm writing a simple Bash script that needs to find out how many files are in a directory. Basically, if the directory contains no files, the script will exit. If it has any, it will do other things.
How can I find out if there are files in a directory using Bash?
Thanks!
List almost (no . or ..) all files and directories and count the lines: ls -A1 /mydir | wc -l
If you'd like only files I think you can use find instead: find /mydir -maxdepth 1 -type f | wc -l
shopt -s nullglob
cd /path/to/dir/
arr=( * )
echo "${#arr[#]}"
for i in "${!arr[#]}"; do echo "${arr[i]}"; done
The other answers have shown you ways to get the number of files. Here is how to use the number in your script.
(This is presented in the context of a function, with $1 being the directory name specified when you run it. To use it on the current directory, just omit that, or you can hardwire a directory name in that location.)
checkdir(){
numfiles=$(ls -A "$1" | wc -l)
if [ $numfiles -gt 0 ]
then
echo YES;
else
echo NO;
fi
}
Note: this will count directories as files. If you would like only files, then replace the ls...wc portion with this line:
ls -AF "$1" | grep -v "/$" | wc -l
I needed the same thing yesterday, and found this (it's near the bottom of the page):
# From http://www.etalabs.net/sh_tricks.html
is_empty () (
cd "$1"
set -- .[!.]* ; test -f "$1" && return 1
set -- ..?* ; test -f "$1" && return 1
set -- * ; test -f "$1" && return 1
return 0 )
It's POSIX-compatible, and uses three patterns to match any file name other than . and .. which are guaranteed to exist in an otherwise empty directory.
The first two lines match all files starting with a single . (of length at least 2) or one or more . (of length at least 3), which covers all hidden files that aren't . or ... The third pattern matches all non-hidden files.
Notice that the body of the function is a (...) expression, not {...}. This forces a subshell, in which it is safe to change the working directory to simplify the patterns.

Script for renaming files with logical

Someone has very kindly help get me started on a mass rename script for renaming PDF files.
As you can see I need to add a bit of logical to stop the below happening - so something like add a unique number to a duplicate file name?
rename 's/^(.{5}).*(\..*)$/$1$2/' *
rename -n 's/^(.{5}).*(\..*)$/$1$2/' *
Annexes 123114345234525.pdf renamed as Annex.pdf
Annexes 123114432452352.pdf renamed as Annex.pdf
Hope this makes sense?
Thanks
for i in *
do
x='' # counter
j="${i:0:2}" # new name
e="${i##*.}" # ext
while [ -e "$j$x" ] # try to find other name
do
((x++)) # inc counter
done
mv "$i" "$j$x" # rename
done
before
$ ls
he.pdf hejjj.pdf hello.pdf wo.pdf workd.pdf world.pdf
after
$ ls
he.pdf he1.pdf he2.pdf wo.pdf wo1.pdf wo2.pdf
This should check whether there will be any duplicates:
rename -n [...] | grep -o ' renamed as .*' | sort | uniq -d
If you get any output of the form renamed as [...], then you have a collision.
Of course, this won't work in a couple corner cases - If your files contain newlines or the literal string renamed as, for example.
As noted in my answer on your previous question:
for f in *.pdf; do
tmp=`echo $f | sed -r 's/^(.{5}).*(\..*)$/$1$2/'`
mv -b ./"$f" ./"$tmp"
done
That will make backups of deleted or overwritten files. A better alternative would be this script:
#!/bin/bash
for f in $*; do
tar -rvf /tmp/backup.tar $f
tmp=`echo $f | sed -r 's/^(.{5}).*(\..*)$/$1$2/'`
i=1
while [ -e tmp ]; do
tmp=`echo $tmp | sed "s/\./-$i/"`
i+=1
done
mv -b ./"$f" ./"$tmp"
done
Run the script like this:
find . -exec thescript '{}' \;
The find command gives you lots of options for specifing which files to run on, works recursively, and passes all the filenames in to the script. The script backs all file up with tar (uncompressed) and then renames them.
This isn't the best script, since it isn't smart enough to avoid the manual loop and check for identical file names.

Resources