Linux - How to zip files per subdirectory separately - linux

I have directory structure like this.
From this I want to create different zip files such as
data-A-A_1-A_11.zip
data-A-A_1-A_12.zip
data-B-B_1-B_11.zip
data-C-C_1-C_11.zip

while read line;
do
echo "zip -r ${line//\//-}.zip $line";
# zip -r "${line//\//-}.zip" "$line"
done <<< "$(find data -maxdepth 3 -mindepth 2 -type d)"
Redirect the result of a find command into a while loop. The find command searches the directory data for directories only, searching 3 directories deep only. In the while loop with use bash expansion to convert all forward slashes to "-" and add ".zip" in such a way that we can build a zip command on each directory. Once you are happy that the zip command looks fine when echoed for each directory, comment in the actual zip command

Related

Bash script to sort files into sub folders based on extension

I have the following structure:
FolderA
Sub1
Sub2
filexx.csv
filexx.doc
FolderB
Sub1
Sub2
fileyy.csv
fileyy.doc
I want to write a script that will move the .csv files into the folder sub1 for each parent directory (Folder A, Folder B and so on) giving me the following structure:
FolderA
Sub1
filexx.csv
Sub2
filexx.doc
FolderB
Sub1
fileyy.csv
Sub2
fileyy.doc
This is what I have till now but I get the error mv: cannot stat *.csv: No such file or directory
for f in */*/*.csv; do
mv -v "$f" */*/Sub1;
done
for f in */*/*.doc; do
mv -v "$f" */*/Sub2;
done
I am new to bash scripting so please forgive me if I have made a very obvious mistake. I know I can do this in Python as well but it will be lengthier which is why I would like a solution using linux commands.
find . -name "*.csv" -type f -execdir mv '{}' Sub1/ \;
Using find, search for all files with the extension .csv and then when we find them, execute a move command from within the directory containing the files, moving the files to directory Sub1
find . -name "*.doc" -type f -execdir mv '{}' Sub2/ \;
Follow the same principle for files with the extension .doc but this time, move the files to Sub2.
I believe you are getting this error because no file matched your wildcard. When it happens, the for loop will give $f the value of the wildcard itself. You are basically trying to move the file *.csv which does not exist.
To prevent this behavior, you can add shopt -s nullglob at the top of your script. When using this, if no file is found, your script won't enter the loop.
My advise is, make sure you run your script from the correct location when using wildcards like this. But maybe what you meant to do by writing */*/*.csv is to recursively match all the csv files. If that's what you intended to do, this is not the right way to do it.
To recursively match all csv/doc/etc files using native bash you can add shopt -s globstar to the top of your script and use **/*.csv as wildcard
#!/bin/bash
shopt -s globstar nullglob
for f in **/*.csv; do
mv "$f" Destination/ # Note that $f is surrounded by "" to handle whitespaces in filenames
done
You could also use the find (1) utility to achieve that. But if you're planning to do more processing on the files than just moving them, a for loop might be cleaner as you won't have to inline everything in the same command.
Side note : "Linux commands" as you say are actually not Linux commands, they are part of the GNU utilities (https://www.gnu.org/gnu/linux-and-gnu.en.html)
If csv files you want to move are in the top directories (from the point of view of the current directory), but not in the subdirectories of them, then simply:
#!/bin/bash
for dir in */; do
mv -v "$dir"*.csv "${dir}Sub1/"
mv -v "$dir"*.doc "${dir}Sub2/"
done
If the files in all subdirectories are wanted to be moved similarly, then:
shopt -s globstar
for file in **/*.csv; do
mv -v "$file" "${file%/*}/Sub1/"
done
for file in **/*.doc; do
mv -v "$file" "${file%/*}/Sub2/"
done
Note that, the directories Sub1 and Sub2 are relative to the directory where csv and doc files reside.

How to randomly distribute the files across 3 folders using Bash script?

I have many subdirectories and files in the folder mydata/files. I want to take files and copy them randomly into 3 folders:
train
test
dev
For example, mydata/files/ss/file1.wav could be copied into train folder:
train
file1.wav
And so on and so forth, until all files from mydata/files are copied.
How can I do it using Bash script?
Steps to solve this:
Need to gather all the files in the directory
Assign directories to a map
Generate random number for each file
Move the file to the corresponding directory
The script:
#!/bin/bash
original_dir=test/
## define 3 directories to copy into
# define an associative array (like a map)
declare -A target_dirs
target_dirs[0]="/path/to/train/"
target_dirs[1]="/path/to/test/"
target_dirs[2]="/path/to/dev/"
# recursively find all the files, and loop through them
find $original_dir -type f | while read -r file ; do
# find a random number 0 - (size of target_dirs - 1)
num=$(($RANDOM % ${#target_dirs[#]}))
# get that index in the associative array
target_dir=${target_dirs[$num]}
# copy the file to that directory
echo "Copying $file to $target_dir"
cp $file $target_dir
done
Things you'll need to change:
Change the destination of the directories to match the path in your system
Add executable priviledges to the file so that you can run it.
chmod 744 copy_script_name
./copy_script_name
Notes:
This script should easily be extendable to any number of directories if needed (just add the new directories, and the script will adjust the random numbers.
If you need to only get the files in the current directory (not recursively), you can add
-maxdepth 1 (see How to list only files and not directories of a directory Bash?).
Was able to leverage previous bash experience plus looking at bash documentation (it's generally pretty good). If you end up writing any scripts, be very careful about spaces
You can create a temp file, echo your destination folder to it, then use the shuf command.
dest=$(mktemp)
echo -e "test\ndev\ntrain" >> $dest
while IFS= read -r file; do
mv "$file" "$(shuf -n1 < $dest)/."
done < <(find mydata/files -type f 2>/dev/null)
rm -f "$dest"

Moving a file and renaming it after the directory which contains it on Bash

I'm trying to learn bash on Linux, just for fun. I thought it would be pretty useful to have a .sh that would group together similar files. For example, let's say we have the directory
/home/docs/
Inside the directory we have /mathdocs/, /codingdocs/, etc.
Inside those sub-directories we have doc.txt, in all of them. Same name for all the files on the subdirectories.
Let's say I want to group them together, and I want to move all the files to /home/allthedocs/ and rename them after the directories they were in. (mathdocs.txt, codingdocs.txt, etc.)
How could I do that?
I've tried to create a script based on the ls and cp commmands, but I don't know how I can take the name of the directories to rename the files in it after I moved them. I guess it has to be some sort of iterative sentence (for X on Y directories) but I don't know how to do it.
You can move and rename your file in one shot with mv, with a loop that grabs all your files through a glob:
#!/bin/bash
dest_dir=/home/allthedocs
cd /home/docs
for file in */doc.txt; do
[[ -f "$file" ]] || continue # skip if not a regular file
dir="${file%/*}" # get the dir name from path
mv "$file" "$dest_dir/$dir.txt"
done
See this post for more info:
Copying files from multiple directories into a single destination directory
Here is a one liner solution that treats whitespaces in filenames, just as #codeforester 's solution does with the glob.
Note that white spaces are treated with the "-print0" option passed to "find", the internal field separator (IFS) in while loop and the wrapping of file3 variable with quotes.
The parameter substitution from file2 into file3 gets rid of the leading "./".
The parameter substition inside the move command turns the path into a filename (run under /home/docs/):
find . -maxdepth 2 -mindepth 2 -print0 | while IFS= read -r -d '' file; \
do file2=$(printf '%s\n' "$file"); file3=${file2#*\/*}; \
mv "$file2" ../allsil/"${file3//\//}"; done

Recursively rename .jpg files in all subdirectories

I am on a Linux system and I am trying to rename all .jpg files in many subdirectories to sequential filenames, so all the jpeg files in each subdirectory are renamed 0001.jpg, 0002.jpg, etc. I have a 'rename' command that works in a single directory:
rename -n 's/.*/sprintf("%04d",$::iter++ +1).".jpg"/e' *.jpg
I am trying to use it like this:
for i in ls -D; do rename -n 's/.*/sprintf("%04d",$::iter++ +1).".jpg"/e' *.jpg; done
but for output I get this:
*.jpg renamed as 0001.jpg
for each subdirectory. What am I doing wrong?
You need to put the command in backticks (or use the $( ... ) bash syntax) in order
to iterate over its output. Also use the $i variable together with the *.jpg file
name pattern, e.g.
for i in `ls -D`
do
rename -n 's/.*/sprintf("%04d",$::iter++ +1).".jpg"/e' $i/*.jpg
done
however, for this scenario you want to iterate over all the subdirectories, and you are
better of using the find command:
for i in `find . -type d`; do rename ...
It seems to me you've forgot to change a current working directory so it should looks like
for i in *; do
[ -d "$i" ] || continue
pushd "$i"
# rename is here
popd
done

Copying files in multiple subdirectories in the Linux command line

Let's say I have the following subdirectories
./a/, ./b/, ./c/, ...
That is, in my current working directory are these subdirectories a/, b/ and c/, and in each of these subdirectories are files. In directory a/ is the file a.in, in directory b/ is the file b.in and so forth.
I now want to copy each .in file to a .out file, that is, a.in to a.out and b.in to b.out, and I want them to reside in the directories they were copied from. So a.out will be found in directory a/.
I've tried various different approaches, such as
find ./ -name '*.in'|cp * *.out
which doesn't work because it thinks *.out is a directory. Also tried
ls -d */ | cd; cp *.in *.out
but it that would list the subdirectories, go into each one of them, but won't let cp do it's work (which still doesn't work)
The
find ./ -name '*.in'
command works fine. Is there a way to pipe arguments to an assignment operator? E.g.
find ./ -name '*.in'| assign filename=|cp filename filename.out
where assign filename= gives filename the value of each .in file. In fact, it would be even better if the assignment could get rid of the .in file extension, then instead of getting a.in.out we would get the preferred a.out
Thank you for your time.
Let the shell help you out:
find . -name '*.in' | while read old; do
new=${old%.in}.out # strips the .in and adds .out
cp "$old" "$new"
done
I just took the find command you said works and let bash read its output one filename at a time. So the bash while loop gets the filenames one at a time, does a little substitution, and a straight copy. Nice and easy (but not tested!).
Try a for loop:
for f in */*.in; do
cp $f ${f%.in}.out;
done
The glob should catch all the files one directory down that have a .in extension. In the cp command, it strips off the .in suffix and then appends a .out (see Variable Mangling in Bash with String Operators)
Alternatively, if you want to recurse into every subdirectory (not just 1 level deep) replace the glob with a find:
for f in $(find . -name '*.in'); do
cp $f ${f%.in}.out;
done
This should do the trick!
for f in `find . -type f -name "*.in"`; do cp $f `echo $f | sed 's/in$/out/g'`; done

Resources