One liner shell script to reorganize files by date - linux

I have a folder with a lot of files named by date like this:
(Some numbers)-date-(somenumbers).filet ype
Where 'date' is listed like:
20150730225001
For July 30, 2015 at 0
10:50 (and 1 second)
Is there a quick way to organize this with the Ubuntu terminal to give a structure like this:
Current folder/year/month/day/hour/ (all files fitting criteria to be placed here)
I have tried looking for a solution to this but most have just confused me since I'm pretty new to Unix scripting.

So with these two files inside foo/
42-20150730225001-473826.filet
98-20141115180001-482157.filet
You can either use this Perl script:
#!/usr/bin/perl
for(#ARGV) {
if (/.*-(\d{4})(\d\d)(\d\d)(\d+)-.*[.]filet/) {
`mkdir -p $1/$2/$3/ && mv $_ $1/$2/$3/$4.filet`;
}
}
or this oneliner:
ls | xargs perl -e 'for(#ARGV){qx{mkdir -p $1/$2/$3/ && mv $_ $1/$2/$3/$4.filet} if /.*-(\d{4})(\d\d)(\d\d)(\d+)-.*[.]filet/}'

The solution presumes you are using bash as your shell on Ubuntu and that you have all of the files you wish to organize in the current working directory. While not a one liner, you can accomplish what you are attempting with a few lines using string indexes to separate the various components of the date into year, month, day and hour relatively easily. Then using mkdir -p to check/create the needed relative path and mv to move the file to the new destination. A couple of compound commands can be added to provide a few sanity checks to insure you have a 14-digit date/time string, etc.
There are a number of ways to approach the separation (e.g. date -d, etc.), but simple string indexes are probably the most efficient. Let me know if you have questions:
A semi-one-liner with continuations
for i in *; do dt=${i%-*}; dt=${dt#*-}; [ ${#dt} -eq 14 ] || continue; \
y=${dt:0:4}; mo=${dt:4:2}; d=${dt:6:2}; h=${dt:8:2}; \
mkdir -p "$y/$mo/$d/$h" && mv "$i" "$y/$mo/$d/$h" || \
printf "error: unable to create: '%s'\n" "$y/$mo/$d/$h"; done
formatted
#!/bin/bash
for i in *; do
dt=${i%-*} ## separate date string from filename
dt=${dt#*-}
[ ${#dt} -eq 14 ] || continue ## validate 14 chars
y=${dt:0:4} ## separate into year, mo, day, hour
mo=${dt:4:2} # (you can add checks for ranges)
d=${dt:6:2}
h=${dt:8:2}
## create new dir & move or throw error
mkdir -p "$y/$mo/$d/$h" && mv "$i" "$y/$mo/$d/$h" || \
printf "error: unable to create: '%s'\n" "$y/$mo/$d/$h"
done

Related

Change folder structure and make files available by using links

(Please see also the minimal example at bottom)
I have the following folder structure:
https://drive.google.com/open?id=1an6x1IRtNjOG6d9D5FlYwsUxmgaEegUY
Each folder shows at the end: the date, month and day:
e.g.
/HRIT/EPI/2004/01/14
Inside for each day, each of this sub-folders contains a lot of files (in total about 10 TB).
All available folders are:
/HRIT/EPI/YYYY/MM/DD/
/HRIT/HRV/YYYY/MM/DD/
/HRIT/VIS006/YYYY/MM/DD/
/HRIT/VIS008/YYYY/MM/DD/
/HRIT/WV_062/YYYY/MM/DD/
/HRIT/WV_073/YYYY/MM/DD/
/HRIT/IR_016/YYYY/MM/DD/
/HRIT/IR_039/YYYY/MM/DD/
/HRIT/IR_087/YYYY/MM/DD/
/HRIT/IR_097/YYYY/MM/DD/
/HRIT/IR_108/YYYY/MM/DD/
/HRIT/IR_120/YYYY/MM/DD/
/HRIT/IR_134/YYYY/MM/DD/
/HRIT/PRO/YYYY/MM/DD/
I would like to change the folder structure to:
/HRIT/YYYY/MM/DD/
All files from EPI, HRV, VIS006, VIS008, WV_062, WV_073, IR_016, IR_039, IR_087,IR_097, IR_108, IR_124, IR_134, PRO should not be copied physically but should be accessible by links in my home folder /home
-> /home/HRIT/YYYY/MM/DD/
Is something like that possible and how?
Here is an minimal example what I in principle have and what I wish as result:
I have:
/HRIT/EPI/2019/01/01/epi_1.txt
/HRIT/EPI/2019/01/01/epi_2.txt
/HRIT/HRV/2019/01/01/hrv_1.txt
/HRIT/HRV/2019/01/01/hrv_2.txt
/HRIT/VIS006/2019/01/01/vis006_1.txt
/HRIT/VIS006/2019/01/01/vis006_2.txt
As result I wish:
/home/HRIT/2019/01/01/epi_1.txt
/home/HRIT/2019/01/01/epi_2.txt
/home/HRIT/2019/01/01/hrv_1.txt
/home/HRIT/2019/01/01/hrv_2.txt
/home/HRIT/2019/01/01/vis006_1.txt
/home/HRIT/2019/01/01/vis006_2.txt
As mentioned above the files should not be copied to this new folder structure, but instead made accessible by links (because in reality I have too many files and not enough space to copy them).
!!! This is extremely simplified, since I have different years, month's and days (please see the link above).
Here's a small script to do this. I've made it just echo by default, you'd need to run it with --real to make it do the linking
This assumes you're running the script from within the /HRIT/ dir, and assumes that the date, then filename are the last part of the hierarchy. If there are further dirs below, this might not work for you.
DRYRUN=true
LINK="ln -s" # You can make this "ln" if you prefer hardlinks
if [[ $1 == '--real' ]]; then
DRYRUN=false
fi
for file in $(find "$PWD" -type f); do
dest=$( \
echo "$file" | awk -F/ '{ year = NF-3; mon=NF-2; day=NF-1; print "/home/HRIT/"$year"/"$mon"/"$day"/"$NF }' \
)
if [[ -f "$dest" ]]; then
echo "Skipping '$file' as destination link already exists at '$dest'"
else
if $DRYRUN; then
echo $LINK "$file" "$dest"
else
$LINK "$file" "$dest"
fi
fi
done
The simplest while read loop:
generate_list_of_files_for_example_with_find |
while IFS= read -r line; do
IFS='/' read -r _ hrit _ rest <<<"$line"
echo mkdir -p /home/"$hrit"/"$(dirname "$rest")"
echo ln -s /home/"$hrit/$rest" "$line"
done
Remove echos to really create links and directories.

'Housekeeping' script to create folder and move files

I have been trying to investigate how to automate a script (shell or other) that periodically (once an hour for example) moves files with a constant naming convention to a folder (which is automatically created if not already there).
The files are like this:
Camera1_01_20171213221830928.jpg
Camera1_01_20171213223142881.mp4
Basically it will be doing 'housekeeping'.
I'm new to shell scripts, and I just can't work out how to create a folder if it is not there (folder called 20171213 for example), then move the relevant files into it?
Any help would be greatly appreciated.
You can use if [ ! -d "$DIRNAME ]" to see if a directory called $DIRNAME exists. Further you can use mkdir to create directories.
Alternatively, you can just use mkdir and ignore any error that a directory already exists. Other than that, there are no negative side effects.
I won't be able to write the script for you because you don't have enough specific information in your post, but here are the tools I think you'll need:
Bash - Bash, for writing your script (Here's another useful reference)
Mkdir - For making a directory if its missing (look at the -p flag).
Cron - For scheduling the hourly execution of your script.
Sed - For creating the new file names from the old ones
The basic code will look something like this:
#!/bin/bash
FILES=/home/joshua/photos/sort-me
for photo in $FILES/*.jpg; do
if [[ -f photo ]]; then
new_photo_location= #<figure this part out based on your needs>
echo "I would move $photo to $new_photo_location"
#mv $photo $new_photo_location
fi
done
I recommend running it with that echo only until you see exactly what you like, then comment that line out and uncommont the mv line.
Finally, to run the script every hour on the hour, your cron entry will look something like this. (Type crontab -e to edit your crontab):
# Min Hour Day Month Day-of-Wk Year Must use absolute path
0 * * * * * /home/joshua/bin/sort-photos.sh
VDIR=$( date +%F )
VFILESFX=Camera*
[ ! -d $VDIR ] && mkdir $VDIR
while true ; do
LTMP=$( ls $VFILESFX )
for i in $LTMP ; do
fuser ${i}
[ $? -ne 0 ] && mv $i $VDIR/;
echo "File ${i} moved to ${VDIR}";
done
sleep 3600
done
Just some explanation about this script...
The first part set the current date into a variable VDIR and the prefix Camera* into VFILESFX.
The conditional command create a directory with the value existent in VDIR if this directory does not exist, after that go to a while command interacting every hour listing all the file with the prefix setted into VFILESFX and moving this files to the directory with the current date
I finally went with a perl script which I could more easily trigger from a cron job:
#!/usr/bin/perl -w
use strict;
use Data::Dumper;
use File::Copy;
main();
sub main
{
my $dir = "/srv/NAS1/Camera1";
opendir(my $fh, $dir) or die("Could not open '$dir' for reading: $!\n");
my #files = readdir($fh);
closedir($fh);
foreach my $file(#files)
{
if(-d $file)
{
next; # skip file if its a folder
}
if($file =~ /Camera1_01_(\d{8})\d{9}\.(jpg|mp4)/)
{
my $date = $1;
$date =~ /(\d{4})(\d{2})(\d{2})/;
my $folder = "$1-$2-$3";
# if the directory doesn't exist
if(!(-e -d "${dir}/${folder}"))
{
mkdir("${dir}/${folder}");
}
move("${dir}/$file","${dir}/${folder}");
}
}
}
Thanks for the contributions.

How can I batch rename multiple images with their path names and reordered sequences in bash?

My pictures are kept in the folder with the picture-date for folder name, for example the original path and file names:
.../Pics/2016_11_13/wedding/DSC0215.jpg
.../Pics/2016_11_13/afterparty/DSC0234.jpg
.../Pics/2016_11_13/afterparty/DSC0322.jpg
How do I rename the pictures into the format below, with continuous sequences and 4-digit padding?
.../Pics/2016_11_13_wedding.0001.jpg
.../Pics/2016_11_13_afterparty.0002.jpg
.../Pics/2016_11_13_afterparty.0003.jpg
I'm using Bash 4.1, so only mv command is available. Here is what I have now but it's not working
#!/bin/bash
p=0
for i in *.jpg;
do
mv "$i" "$dirname.%03d$p.JPG"
((p++))
done
exit 0
Let say you have something like .../Pics/2016_11_13/wedding/XXXXXX.jpg; then go in directory .../Pics/2016_11_13; from there, you should have a bunch of subdirectories like wedding, afterparty, and so on. Launch this script (disclaimer: I didn't test it):
#!/bin/sh
for subdir in *; do # scan directory
[ ! -d "$subdir" ] && continue; # skip non-directory
prognum=0; # progressive number
for file in $(ls "$dir"); do # scan subdirectory
(( prognum=$prognum+1 )) # increment progressive
newname=$(printf %4.4d $prognum) # format it
newname="$subdir.$newname.jpg" # compose the new name
if [ -f "$newname" ]; then # check to not overwrite anything
echo "error: $newname already exist."
exit
fi
# do the job, move or copy
cp "$subdir/$file" "$newname"
done
done
Please note that I skipped the "date" (2016_11_13) part - I am not sure about it. If you have a single date, then it is easy to add these digits in # compose the new name. If you have several dates, then you can add a nested for for scanning the "date" directories. One more reason I skipped this, is to let you develop something by yourself, something you can be proud of...
Using only mv and bash builtins:
#! /bin/bash
shopt -s globstar
cd Pics
p=1
# recursive glob for .jpg files
for i in **/*.jpg
do
# (date)/(event)/(filename).jpg
if [[ $i =~ (.*)/(.*)/(.*).jpg ]]
then
newname=$(printf "%s_%s.%04d.jpg" "${BASH_REMATCH[#]:1:2}" "$p")
echo mv "$i" "$newname"
((p++))
fi
done
globstar is a bash 4.0 feature, and regex matching is available even in OSX's anitque bash.

Linux: Update directory structure for millions of images which are already in prefix-based folders

This is basically a follow-up to Linux: Move 1 million files into prefix-based created Folders
The original question:
I want to write a shell command to rename all of those images into the
following format:
original: filename.jpg new: /f/i/l/filename.jpg
Now, I want to take all of those files and add an additional level to the directory structure, e.g:
original: /f/i/l/filename.jpg new: /f/i/l/e/filename.jpg
Is this possible to do with command line or bash?
One way to do it is to simply loop over all the directories you already have, and in each bottom-level subdirectory create the new subdirectory and move the files:
for d in ?/?/?/; do (
cd "$d" &&
printf '%.4s\0' * | uniq -z |
xargs -0 bash -c 'for prefix do
s=${prefix:3:1}
mkdir -p "$s" && mv "$prefix"* "$s"
done' _
) done
That probably needs a bit of explanation.
The glob ?/?/?/ matches all directory paths made up of three single-character subdirectories. Because it ends with a /, everything it matches is a directory so there is no need to test.
( cd "$d" && ...; )
executes ... after cd'ing to the appropriate subdirectory. Putting that block inside ( ) causes it to be executed in a subshell, which means the scope of the cd will be restricted to the parenthesized block. That's easier and safer than putting cd .. at the end.
We then collecting the subdirectories first, by finding the unique initial strings of the files:
printf '%.4s\0' * | uniq -z | xargs -0 ...
That extracts the first four letters of each filename, nul-terminating each one, then passes this list to uniq to eliminate duplicates, providing the -z option because the input is nul-terminated, and then passes the list of unique prefixes to xargs, again using -0 to indicate that the list is nul-terminated. xargs executes a command with a list of arguments, issuing the command several times only if necessary to avoid exceeding the command-line limit. (We probably could have avoided the use of xargs but it doesn't cost that much and it's a lot safer.)
The command called with xargs is bash itself; we use the -c option to pass it a command to be executed. That command iterates over its arguments by using the for arg in syntax. Each argument is a unique prefix; we extract the fourth character from the prefix to construct the new subdirectory and then mv all files whose names start with the prefix into the newly created directory.
The _ at the end of the xargs invocation will be passed to bash (as with all the rest of the arguments); bash -c uses the first argument following the command as the $0 argument to the script, which is not part of the command line arguments iterated over by the for arg in syntax. So putting the _ there means that the argument list constructed by xargs will be precisely $1, $2, ... in the execution of the bash command.
Okay, so I've created a very crude solution:
#!/bin/bash
for file1 in *; do
if [[ -d "$file1" ]]; then
cd "$file1"
for file2 in *; do
if [[ -d "$file2" ]]; then
cd "$file2"
for file3 in *; do
if [[ -d "$file3" ]]; then
cd "$file3"
for file4 in *; do
if [[ -f "$file4" ]]; then
echo "mkdir -p ${file4:3:1}/; mv $file4 ${file4:3:1}/;"
mkdir -p ${file4:3:1}/; mv $file4 ${file4:3:1}/;
fi
done
cd ..
fi
done
cd ..
fi
done
cd ..
fi
done
I should warn that this is untested, as my actual structure varies slightly, but I wanted to keep the question/answer consistent with the original question for clarity.
That being said, I'm sure a much more elegant solution exists than this one.

Bash script that creates a directory structure

I've been googling all night trying to find a way to create a script that creates a directory structure. That looks something like this:
/
shared
shared/projects
shared/series
shared/movies
shared/movies/action
You get the point.
The file that the script reads from look like this:
shared backup
shared data
shared projects
shared projcets series
shared projects movies
shared projects movies action
I want to create a script that reads each line in the file and run the following for each line:
If the directory exist, it places itself in the directory and create the structure from there, if
The directory doesn’t exist, create it.
When all entries in the row have been preceded by, go back to original directory and read the next line.
My system is Ubuntu 10.10.
So far I’ve done this, but it doesn’t work.
#!/bin/bash
pwd=$(pwd)
for structure in ${column[*]}
do
if [ $structure ]
then
cd $structure
else
mkdir $structure
fi
done
cd $pwd
You can use mkdir -p shared/projects/movies/action to create the whole tree: it will create shared, then shared/projects, then shared/projects/movies, and shared/projects/movies/action.
So basically you need script that runs mkdir -p $dir where $dir is the leaf directory of your directory tree.
If struct.txt contains the directory structure that you mention, then just run:
sed '/^$/d;s/ /\//g' struct.txt | xargs mkdir -p
sed will remove blank lines and make the remaining lines look like directory paths.
xargs will take each line and pass it as a parameter to mkdir.
mkdir will make the directory and the -p flag will create any parent directories if needed.
mkdir has a flag -p that creates all the parent directories of the directory you're creating if needed. you can just just read each line, turn it into a path (i.e. s/ /\//g) and call mkdir -p $path on each line
For my solution it was important to me:
a) I wanted to be able to edit the directory structure directly in my bash script so that I didn't have to jump back and forth between two files
b) The code for the folders should be as clear as possible without redundancy with the same paths, so that I can change it easily
# Creates the folder structure defined in folder structure section below
function createFolderStructure() {
depth="1"
while (( "$#" )); do
while (( $1 != $depth )); do
cd ..
(( depth-- ))
done
shift
mkdir "$1"
cd "$1"
(( depth++ ))
shift
done
while (( 1 != $depth )); do
cd ..
(( depth-- ))
done
}
# Folder Structure Section
read -r -d '' FOLDERSTRUCTURE << EOM
1 shared
2 projects
3 movies
4 action
2 series
2 backup
EOM
createFolderStructure $FOLDERSTRUCTURE
Git needs files to record directories. So I put a readme file in each directory and extended the script as follows:
# Creates the folder structure defined in folder structure section below
function createFolderStructure() {
depth="1"
while (( "$#" )); do
while (( $1 != $depth )); do
cd ..
(( depth-- ))
done
shift
mkdir "$1"
cd "$1"
(( depth++ ))
shift
shift
out=""
while [[ "$1" != "-" ]]; do
out=$out" ""$1"
shift
done
shift
echo "$out" > README.md
done
while (( 1 != $depth )); do
cd ..
(( depth-- ))
done
}
# If you like you can read in user defined values here and use them as variables in the folder structure section, e.g.
# echo -n "Enter month of films"
# read month
# ...
# 1 shared - Folder for shared stuff -
# 2 $month - Films from month $month -
# 3 projects - Folder for projects -
# ...
# Folder Structure Section
read -r -d '' FOLDERSTRUCTURE << EOM
1 shared - Folder for shared stuff -
2 projects - Folder for projects -
3 movies - Folder for movies -
4 action - Folder for action movies -
2 series - Folder for series -
2 backup - Backup folder -
EOM
createFolderStructure $FOLDERSTRUCTURE
1) Do something like this
find . -type d > folder_list.txt
to create a list of the folders you need to create.
2) Transfer the list to your destination
3) Recreate the structure in your new location:
cat folder_list.txt | xargs mkdir
notice that you don't need '-p' option in this case though it wouldn't hurt too.
I use this script in my .bash_profile that I use for new projects:
alias project_setup="mkdir Sites Documents Applications Website_Graphics Mockups Logos Colors Requirements Wireframes"
If you want to make a nested folder structure you you could do something like:
alias shared_setup="mkdir Shared shared/projects shared/series shared/movies shared/movies/action"
Assuming you wish to create a tree of folders / directories as below:
tmpdir
________|______
| | |
branches tags trunk
|
sources
____|_____
| |
includes docs
Also assuming that you have a variable that mentions the directory names.
DOMAIN_NAME=includes,docs
You may issue below command:
$ eval "mkdir -p tmpdir/{trunk/sources/{${DOMAIN_NAME}},branches,tags}"
Note: use the BASH version that supports curly-braces expansion.

Resources