Bash script that creates a directory structure - linux

I've been googling all night trying to find a way to create a script that creates a directory structure. That looks something like this:
/
shared
shared/projects
shared/series
shared/movies
shared/movies/action
You get the point.
The file that the script reads from look like this:
shared backup
shared data
shared projects
shared projcets series
shared projects movies
shared projects movies action
I want to create a script that reads each line in the file and run the following for each line:
If the directory exist, it places itself in the directory and create the structure from there, if
The directory doesn’t exist, create it.
When all entries in the row have been preceded by, go back to original directory and read the next line.
My system is Ubuntu 10.10.
So far I’ve done this, but it doesn’t work.
#!/bin/bash
pwd=$(pwd)
for structure in ${column[*]}
do
if [ $structure ]
then
cd $structure
else
mkdir $structure
fi
done
cd $pwd

You can use mkdir -p shared/projects/movies/action to create the whole tree: it will create shared, then shared/projects, then shared/projects/movies, and shared/projects/movies/action.
So basically you need script that runs mkdir -p $dir where $dir is the leaf directory of your directory tree.

If struct.txt contains the directory structure that you mention, then just run:
sed '/^$/d;s/ /\//g' struct.txt | xargs mkdir -p
sed will remove blank lines and make the remaining lines look like directory paths.
xargs will take each line and pass it as a parameter to mkdir.
mkdir will make the directory and the -p flag will create any parent directories if needed.

mkdir has a flag -p that creates all the parent directories of the directory you're creating if needed. you can just just read each line, turn it into a path (i.e. s/ /\//g) and call mkdir -p $path on each line

For my solution it was important to me:
a) I wanted to be able to edit the directory structure directly in my bash script so that I didn't have to jump back and forth between two files
b) The code for the folders should be as clear as possible without redundancy with the same paths, so that I can change it easily
# Creates the folder structure defined in folder structure section below
function createFolderStructure() {
depth="1"
while (( "$#" )); do
while (( $1 != $depth )); do
cd ..
(( depth-- ))
done
shift
mkdir "$1"
cd "$1"
(( depth++ ))
shift
done
while (( 1 != $depth )); do
cd ..
(( depth-- ))
done
}
# Folder Structure Section
read -r -d '' FOLDERSTRUCTURE << EOM
1 shared
2 projects
3 movies
4 action
2 series
2 backup
EOM
createFolderStructure $FOLDERSTRUCTURE
Git needs files to record directories. So I put a readme file in each directory and extended the script as follows:
# Creates the folder structure defined in folder structure section below
function createFolderStructure() {
depth="1"
while (( "$#" )); do
while (( $1 != $depth )); do
cd ..
(( depth-- ))
done
shift
mkdir "$1"
cd "$1"
(( depth++ ))
shift
shift
out=""
while [[ "$1" != "-" ]]; do
out=$out" ""$1"
shift
done
shift
echo "$out" > README.md
done
while (( 1 != $depth )); do
cd ..
(( depth-- ))
done
}
# If you like you can read in user defined values here and use them as variables in the folder structure section, e.g.
# echo -n "Enter month of films"
# read month
# ...
# 1 shared - Folder for shared stuff -
# 2 $month - Films from month $month -
# 3 projects - Folder for projects -
# ...
# Folder Structure Section
read -r -d '' FOLDERSTRUCTURE << EOM
1 shared - Folder for shared stuff -
2 projects - Folder for projects -
3 movies - Folder for movies -
4 action - Folder for action movies -
2 series - Folder for series -
2 backup - Backup folder -
EOM
createFolderStructure $FOLDERSTRUCTURE

1) Do something like this
find . -type d > folder_list.txt
to create a list of the folders you need to create.
2) Transfer the list to your destination
3) Recreate the structure in your new location:
cat folder_list.txt | xargs mkdir
notice that you don't need '-p' option in this case though it wouldn't hurt too.

I use this script in my .bash_profile that I use for new projects:
alias project_setup="mkdir Sites Documents Applications Website_Graphics Mockups Logos Colors Requirements Wireframes"
If you want to make a nested folder structure you you could do something like:
alias shared_setup="mkdir Shared shared/projects shared/series shared/movies shared/movies/action"

Assuming you wish to create a tree of folders / directories as below:
tmpdir
________|______
| | |
branches tags trunk
|
sources
____|_____
| |
includes docs
Also assuming that you have a variable that mentions the directory names.
DOMAIN_NAME=includes,docs
You may issue below command:
$ eval "mkdir -p tmpdir/{trunk/sources/{${DOMAIN_NAME}},branches,tags}"
Note: use the BASH version that supports curly-braces expansion.

Related

Need to divide 8tb directory into 4 directories

I have a directory called BigDataDirectory which has plenty of files and all of it in total adds up to 8tb.
I am trying to upload to our server and want to make sure I can divide the folder into four parts so I have four folders of about 2tb each.
I tried the split command but it doesn't seem to be working
nohup split -b 2T BigDataDirectory "Directory" &
Could you tell me just a simple way to divide my directory/folder into multiple parts?
If all your files are of a similar size or if the total directory size doesn't need to be very precise, you could do it with these fun commands.
It creates a sub-directory every 250 files and moves the files to a sub-directory. If you have 1000 files, they will be moved to 4 sub-directories named 1, 2, 3 and 4. If you have 1001 files, a 5th subdir will be created for the last file.
cd YOUR_BIG_DIR
n=250 # will change destination dir after 250 files
dir=1; mkdir -p $dir # create first destination sub-directory "1"
ls -1 | while read f; do ((c+=1)); echo mv "$f" $dir/; [ $((c % n)) -eq 0 ] && ((dir+=1)) && mkdir -vp $dir; done
If the output looks like what you want, remove echo from the command to really mv the files.
The same command, explained, and without the "echo" used for testing:
# list all files, 1 per line
ls -1 \
| while read f; do # with each file "$f"
((c+=1)) # increase file counter
mv "$f" $dir/ # move the file to $dir/
# if counter c is a divisor of n, increase the directory number
# and create the new destination directory
[ $((c % n)) -eq 0 ] && ((dir+=1)) && mkdir -v -p $dir
done
If you want sub-directories with a more exact size, then you would need to script something more sophisticated, using stat -c %s to get the size of each file or something similar.

Change folder structure and make files available by using links

(Please see also the minimal example at bottom)
I have the following folder structure:
https://drive.google.com/open?id=1an6x1IRtNjOG6d9D5FlYwsUxmgaEegUY
Each folder shows at the end: the date, month and day:
e.g.
/HRIT/EPI/2004/01/14
Inside for each day, each of this sub-folders contains a lot of files (in total about 10 TB).
All available folders are:
/HRIT/EPI/YYYY/MM/DD/
/HRIT/HRV/YYYY/MM/DD/
/HRIT/VIS006/YYYY/MM/DD/
/HRIT/VIS008/YYYY/MM/DD/
/HRIT/WV_062/YYYY/MM/DD/
/HRIT/WV_073/YYYY/MM/DD/
/HRIT/IR_016/YYYY/MM/DD/
/HRIT/IR_039/YYYY/MM/DD/
/HRIT/IR_087/YYYY/MM/DD/
/HRIT/IR_097/YYYY/MM/DD/
/HRIT/IR_108/YYYY/MM/DD/
/HRIT/IR_120/YYYY/MM/DD/
/HRIT/IR_134/YYYY/MM/DD/
/HRIT/PRO/YYYY/MM/DD/
I would like to change the folder structure to:
/HRIT/YYYY/MM/DD/
All files from EPI, HRV, VIS006, VIS008, WV_062, WV_073, IR_016, IR_039, IR_087,IR_097, IR_108, IR_124, IR_134, PRO should not be copied physically but should be accessible by links in my home folder /home
-> /home/HRIT/YYYY/MM/DD/
Is something like that possible and how?
Here is an minimal example what I in principle have and what I wish as result:
I have:
/HRIT/EPI/2019/01/01/epi_1.txt
/HRIT/EPI/2019/01/01/epi_2.txt
/HRIT/HRV/2019/01/01/hrv_1.txt
/HRIT/HRV/2019/01/01/hrv_2.txt
/HRIT/VIS006/2019/01/01/vis006_1.txt
/HRIT/VIS006/2019/01/01/vis006_2.txt
As result I wish:
/home/HRIT/2019/01/01/epi_1.txt
/home/HRIT/2019/01/01/epi_2.txt
/home/HRIT/2019/01/01/hrv_1.txt
/home/HRIT/2019/01/01/hrv_2.txt
/home/HRIT/2019/01/01/vis006_1.txt
/home/HRIT/2019/01/01/vis006_2.txt
As mentioned above the files should not be copied to this new folder structure, but instead made accessible by links (because in reality I have too many files and not enough space to copy them).
!!! This is extremely simplified, since I have different years, month's and days (please see the link above).
Here's a small script to do this. I've made it just echo by default, you'd need to run it with --real to make it do the linking
This assumes you're running the script from within the /HRIT/ dir, and assumes that the date, then filename are the last part of the hierarchy. If there are further dirs below, this might not work for you.
DRYRUN=true
LINK="ln -s" # You can make this "ln" if you prefer hardlinks
if [[ $1 == '--real' ]]; then
DRYRUN=false
fi
for file in $(find "$PWD" -type f); do
dest=$( \
echo "$file" | awk -F/ '{ year = NF-3; mon=NF-2; day=NF-1; print "/home/HRIT/"$year"/"$mon"/"$day"/"$NF }' \
)
if [[ -f "$dest" ]]; then
echo "Skipping '$file' as destination link already exists at '$dest'"
else
if $DRYRUN; then
echo $LINK "$file" "$dest"
else
$LINK "$file" "$dest"
fi
fi
done
The simplest while read loop:
generate_list_of_files_for_example_with_find |
while IFS= read -r line; do
IFS='/' read -r _ hrit _ rest <<<"$line"
echo mkdir -p /home/"$hrit"/"$(dirname "$rest")"
echo ln -s /home/"$hrit/$rest" "$line"
done
Remove echos to really create links and directories.

'Housekeeping' script to create folder and move files

I have been trying to investigate how to automate a script (shell or other) that periodically (once an hour for example) moves files with a constant naming convention to a folder (which is automatically created if not already there).
The files are like this:
Camera1_01_20171213221830928.jpg
Camera1_01_20171213223142881.mp4
Basically it will be doing 'housekeeping'.
I'm new to shell scripts, and I just can't work out how to create a folder if it is not there (folder called 20171213 for example), then move the relevant files into it?
Any help would be greatly appreciated.
You can use if [ ! -d "$DIRNAME ]" to see if a directory called $DIRNAME exists. Further you can use mkdir to create directories.
Alternatively, you can just use mkdir and ignore any error that a directory already exists. Other than that, there are no negative side effects.
I won't be able to write the script for you because you don't have enough specific information in your post, but here are the tools I think you'll need:
Bash - Bash, for writing your script (Here's another useful reference)
Mkdir - For making a directory if its missing (look at the -p flag).
Cron - For scheduling the hourly execution of your script.
Sed - For creating the new file names from the old ones
The basic code will look something like this:
#!/bin/bash
FILES=/home/joshua/photos/sort-me
for photo in $FILES/*.jpg; do
if [[ -f photo ]]; then
new_photo_location= #<figure this part out based on your needs>
echo "I would move $photo to $new_photo_location"
#mv $photo $new_photo_location
fi
done
I recommend running it with that echo only until you see exactly what you like, then comment that line out and uncommont the mv line.
Finally, to run the script every hour on the hour, your cron entry will look something like this. (Type crontab -e to edit your crontab):
# Min Hour Day Month Day-of-Wk Year Must use absolute path
0 * * * * * /home/joshua/bin/sort-photos.sh
VDIR=$( date +%F )
VFILESFX=Camera*
[ ! -d $VDIR ] && mkdir $VDIR
while true ; do
LTMP=$( ls $VFILESFX )
for i in $LTMP ; do
fuser ${i}
[ $? -ne 0 ] && mv $i $VDIR/;
echo "File ${i} moved to ${VDIR}";
done
sleep 3600
done
Just some explanation about this script...
The first part set the current date into a variable VDIR and the prefix Camera* into VFILESFX.
The conditional command create a directory with the value existent in VDIR if this directory does not exist, after that go to a while command interacting every hour listing all the file with the prefix setted into VFILESFX and moving this files to the directory with the current date
I finally went with a perl script which I could more easily trigger from a cron job:
#!/usr/bin/perl -w
use strict;
use Data::Dumper;
use File::Copy;
main();
sub main
{
my $dir = "/srv/NAS1/Camera1";
opendir(my $fh, $dir) or die("Could not open '$dir' for reading: $!\n");
my #files = readdir($fh);
closedir($fh);
foreach my $file(#files)
{
if(-d $file)
{
next; # skip file if its a folder
}
if($file =~ /Camera1_01_(\d{8})\d{9}\.(jpg|mp4)/)
{
my $date = $1;
$date =~ /(\d{4})(\d{2})(\d{2})/;
my $folder = "$1-$2-$3";
# if the directory doesn't exist
if(!(-e -d "${dir}/${folder}"))
{
mkdir("${dir}/${folder}");
}
move("${dir}/$file","${dir}/${folder}");
}
}
}
Thanks for the contributions.

Create .txt of all files in each subdirectory

I need to create a text file in each subdirectory of all files in the list.
For example, subdirectory1 would contain a list of all of its files as a .txt and subdirectory2 would also contain a list of all of subdirectory2 files as a .txt.
I have tried
#!/bin/bash
for X in "$directory" *
do
if [ -d "$X" ];
then
cd "$X"
files="$(ls)"
echo "$files" >> filesNames.txt
fi
done
However this did not generate anything. I absolutely need it as a shell script because it will be part of a pipeline script, but I cannot seem to get it to work.
Here is the adjusted script giving me the no such file or directory comment. I know that the folder exists and have used it in commands that are run before this command.
#!/bin/bash
#Retrieve the base directory path
baseDir=$(dirname "$ini")
#Retrieve the reference genome path
ref=$(dirname "$genome")
#Create required directory structure
tested="$baseDir/tested"
MarkDups1="$baseDir/MarkDups1"
#don't create if already exists
[[ -d "tested" ]] || mkdir "$tested"
[[ -d "MarkDups1" ]] || mkdir "$MarkDups1"
#create a text file with all sorted and indexed bam files paths
#!/bin/bash
for x in $MarkDups1/*/;
do
(cd "$x"; ls > filesNames.txt)
done
The sequence to iterate over should be "$directory"/*/.
for x in "$directory"/*/; do
(cd "$x"
files=(*)
printf '%s\n' "${files[#]}" > filesNames.txt
)
done

One liner shell script to reorganize files by date

I have a folder with a lot of files named by date like this:
(Some numbers)-date-(somenumbers).filet ype
Where 'date' is listed like:
20150730225001
For July 30, 2015 at 0
10:50 (and 1 second)
Is there a quick way to organize this with the Ubuntu terminal to give a structure like this:
Current folder/year/month/day/hour/ (all files fitting criteria to be placed here)
I have tried looking for a solution to this but most have just confused me since I'm pretty new to Unix scripting.
So with these two files inside foo/
42-20150730225001-473826.filet
98-20141115180001-482157.filet
You can either use this Perl script:
#!/usr/bin/perl
for(#ARGV) {
if (/.*-(\d{4})(\d\d)(\d\d)(\d+)-.*[.]filet/) {
`mkdir -p $1/$2/$3/ && mv $_ $1/$2/$3/$4.filet`;
}
}
or this oneliner:
ls | xargs perl -e 'for(#ARGV){qx{mkdir -p $1/$2/$3/ && mv $_ $1/$2/$3/$4.filet} if /.*-(\d{4})(\d\d)(\d\d)(\d+)-.*[.]filet/}'
The solution presumes you are using bash as your shell on Ubuntu and that you have all of the files you wish to organize in the current working directory. While not a one liner, you can accomplish what you are attempting with a few lines using string indexes to separate the various components of the date into year, month, day and hour relatively easily. Then using mkdir -p to check/create the needed relative path and mv to move the file to the new destination. A couple of compound commands can be added to provide a few sanity checks to insure you have a 14-digit date/time string, etc.
There are a number of ways to approach the separation (e.g. date -d, etc.), but simple string indexes are probably the most efficient. Let me know if you have questions:
A semi-one-liner with continuations
for i in *; do dt=${i%-*}; dt=${dt#*-}; [ ${#dt} -eq 14 ] || continue; \
y=${dt:0:4}; mo=${dt:4:2}; d=${dt:6:2}; h=${dt:8:2}; \
mkdir -p "$y/$mo/$d/$h" && mv "$i" "$y/$mo/$d/$h" || \
printf "error: unable to create: '%s'\n" "$y/$mo/$d/$h"; done
formatted
#!/bin/bash
for i in *; do
dt=${i%-*} ## separate date string from filename
dt=${dt#*-}
[ ${#dt} -eq 14 ] || continue ## validate 14 chars
y=${dt:0:4} ## separate into year, mo, day, hour
mo=${dt:4:2} # (you can add checks for ranges)
d=${dt:6:2}
h=${dt:8:2}
## create new dir & move or throw error
mkdir -p "$y/$mo/$d/$h" && mv "$i" "$y/$mo/$d/$h" || \
printf "error: unable to create: '%s'\n" "$y/$mo/$d/$h"
done

Resources