'Housekeeping' script to create folder and move files - linux

I have been trying to investigate how to automate a script (shell or other) that periodically (once an hour for example) moves files with a constant naming convention to a folder (which is automatically created if not already there).
The files are like this:
Camera1_01_20171213221830928.jpg
Camera1_01_20171213223142881.mp4
Basically it will be doing 'housekeeping'.
I'm new to shell scripts, and I just can't work out how to create a folder if it is not there (folder called 20171213 for example), then move the relevant files into it?
Any help would be greatly appreciated.

You can use if [ ! -d "$DIRNAME ]" to see if a directory called $DIRNAME exists. Further you can use mkdir to create directories.
Alternatively, you can just use mkdir and ignore any error that a directory already exists. Other than that, there are no negative side effects.

I won't be able to write the script for you because you don't have enough specific information in your post, but here are the tools I think you'll need:
Bash - Bash, for writing your script (Here's another useful reference)
Mkdir - For making a directory if its missing (look at the -p flag).
Cron - For scheduling the hourly execution of your script.
Sed - For creating the new file names from the old ones
The basic code will look something like this:
#!/bin/bash
FILES=/home/joshua/photos/sort-me
for photo in $FILES/*.jpg; do
if [[ -f photo ]]; then
new_photo_location= #<figure this part out based on your needs>
echo "I would move $photo to $new_photo_location"
#mv $photo $new_photo_location
fi
done
I recommend running it with that echo only until you see exactly what you like, then comment that line out and uncommont the mv line.
Finally, to run the script every hour on the hour, your cron entry will look something like this. (Type crontab -e to edit your crontab):
# Min Hour Day Month Day-of-Wk Year Must use absolute path
0 * * * * * /home/joshua/bin/sort-photos.sh

VDIR=$( date +%F )
VFILESFX=Camera*
[ ! -d $VDIR ] && mkdir $VDIR
while true ; do
LTMP=$( ls $VFILESFX )
for i in $LTMP ; do
fuser ${i}
[ $? -ne 0 ] && mv $i $VDIR/;
echo "File ${i} moved to ${VDIR}";
done
sleep 3600
done
Just some explanation about this script...
The first part set the current date into a variable VDIR and the prefix Camera* into VFILESFX.
The conditional command create a directory with the value existent in VDIR if this directory does not exist, after that go to a while command interacting every hour listing all the file with the prefix setted into VFILESFX and moving this files to the directory with the current date

I finally went with a perl script which I could more easily trigger from a cron job:
#!/usr/bin/perl -w
use strict;
use Data::Dumper;
use File::Copy;
main();
sub main
{
my $dir = "/srv/NAS1/Camera1";
opendir(my $fh, $dir) or die("Could not open '$dir' for reading: $!\n");
my #files = readdir($fh);
closedir($fh);
foreach my $file(#files)
{
if(-d $file)
{
next; # skip file if its a folder
}
if($file =~ /Camera1_01_(\d{8})\d{9}\.(jpg|mp4)/)
{
my $date = $1;
$date =~ /(\d{4})(\d{2})(\d{2})/;
my $folder = "$1-$2-$3";
# if the directory doesn't exist
if(!(-e -d "${dir}/${folder}"))
{
mkdir("${dir}/${folder}");
}
move("${dir}/$file","${dir}/${folder}");
}
}
}
Thanks for the contributions.

Related

Bash script to iterate contents of directory moving only the files not currently open by other process

I have people uploading files to a directory on my Ubuntu Server.
I need to move those files to the final location (another directory) only when I know these files are fully uploaded.
Here's my script so far:
#!/bin/bash
cd /var/uploaded_by_users
for filename in *; do
lsof $filename
if [ -z $? ]; then
# file has been closed, move it
else
echo "*** File is open. Skipping..."
fi
done
cd -
However it's not working as it says some files are open when that's not true. I supposed $? would have 0 if the file was closed and 1 if it wasn't but I think that's wrong.
I'm not linux expert so I'm looking to know how to implement this simple script that will run on a cron job every 1 minute.
[ -z $? ] checks if $? is of zero length or not. Since $? will never be a null string, your check will always fail and result in else part being executed.
You need to test for numeric zero, as below:
lsof "$filename" >/dev/null; lsof_status=$?
if [ "$lsof_status" -eq 0 ]; then
# file is open, skipping
else
# move it
fi
Or more simply (as Benjamin pointed out):
if lsof "$filename" >/dev/null; then
# file is open, skip
else
# move it
fi
Using negation, we can shorten the if statement (as dimo414 pointed out):
if ! lsof "$filename" >/dev/null; then
# move it
fi
You can shorten it even further, using &&:
for filename in *; do
lsof "$filename" >/dev/null && continue # skip if the file is open
# move the file
done
You may not need to worry about when the write is complete, if you are moving the file to a different location in the same file system. As long as the client is using the same file descriptor to write to the file, you can simply create a new hard link for the upload file, then remove the original link. The client's file descriptor won't be affected by one of the links being removed.
cd /var/uploaded_by_users
for f in *; do
ln "$f" /somewhere/else/"$f"
rm "$f"
done

One liner shell script to reorganize files by date

I have a folder with a lot of files named by date like this:
(Some numbers)-date-(somenumbers).filet ype
Where 'date' is listed like:
20150730225001
For July 30, 2015 at 0
10:50 (and 1 second)
Is there a quick way to organize this with the Ubuntu terminal to give a structure like this:
Current folder/year/month/day/hour/ (all files fitting criteria to be placed here)
I have tried looking for a solution to this but most have just confused me since I'm pretty new to Unix scripting.
So with these two files inside foo/
42-20150730225001-473826.filet
98-20141115180001-482157.filet
You can either use this Perl script:
#!/usr/bin/perl
for(#ARGV) {
if (/.*-(\d{4})(\d\d)(\d\d)(\d+)-.*[.]filet/) {
`mkdir -p $1/$2/$3/ && mv $_ $1/$2/$3/$4.filet`;
}
}
or this oneliner:
ls | xargs perl -e 'for(#ARGV){qx{mkdir -p $1/$2/$3/ && mv $_ $1/$2/$3/$4.filet} if /.*-(\d{4})(\d\d)(\d\d)(\d+)-.*[.]filet/}'
The solution presumes you are using bash as your shell on Ubuntu and that you have all of the files you wish to organize in the current working directory. While not a one liner, you can accomplish what you are attempting with a few lines using string indexes to separate the various components of the date into year, month, day and hour relatively easily. Then using mkdir -p to check/create the needed relative path and mv to move the file to the new destination. A couple of compound commands can be added to provide a few sanity checks to insure you have a 14-digit date/time string, etc.
There are a number of ways to approach the separation (e.g. date -d, etc.), but simple string indexes are probably the most efficient. Let me know if you have questions:
A semi-one-liner with continuations
for i in *; do dt=${i%-*}; dt=${dt#*-}; [ ${#dt} -eq 14 ] || continue; \
y=${dt:0:4}; mo=${dt:4:2}; d=${dt:6:2}; h=${dt:8:2}; \
mkdir -p "$y/$mo/$d/$h" && mv "$i" "$y/$mo/$d/$h" || \
printf "error: unable to create: '%s'\n" "$y/$mo/$d/$h"; done
formatted
#!/bin/bash
for i in *; do
dt=${i%-*} ## separate date string from filename
dt=${dt#*-}
[ ${#dt} -eq 14 ] || continue ## validate 14 chars
y=${dt:0:4} ## separate into year, mo, day, hour
mo=${dt:4:2} # (you can add checks for ranges)
d=${dt:6:2}
h=${dt:8:2}
## create new dir & move or throw error
mkdir -p "$y/$mo/$d/$h" && mv "$i" "$y/$mo/$d/$h" || \
printf "error: unable to create: '%s'\n" "$y/$mo/$d/$h"
done

I have series of file in OLD dir, I want to check if the file exist in NEW dir. If it does not exist, I would like to do few operations on it

For instance... I have some files in the format mmddyy.zip in the OLD directory...
041414.zip
041514.zip
041614.zip
041714.zip
041814.zip(today's file Apr 18 2014)
and I have a NEW dir with 041414.zip 041514.zip in it...
I'm trying to copy all the files from OLD to NEW and do some other operations if that file doesn't exist in NEW dir...
I'm thinking of doing it ' while do' statement, but not sure what what to use in the condition...
Thanks,
Sam.
You probably want to use find, and execute a script that does both the check and the operation.
For instance, script.sh (fill in your own variables):
#!/bin/sh
FNAME="`basename $1`"
NEWDIR="/tmp"
NEWFNAME="$NEWDIR/$FNAME"
if [ ! -f "$NEWFNAME" ]; then
# do stuff to "$1"
# optionally cp "$1" "$NEWFNAME"
fi
Then you run something like cd /old/dir; find -name "*.blah" -exec script.sh "{}" ";"

How can I list the path of the output of this script?

How can I list the path of the output of this script?
This is my command:
(ls -d */ ); echo -n $i; ls -R $i | grep "wp-config.php" ;
This is my current output:
/wp-config.php
It seems you want find the path to a file called "wp-config.php".
Does the following help?
find $PWD -name 'wp-config.php'
Your script is kind of confusing: Why does ls -d */ does not show any output? What's the value of $i? Your problem in fact seems to be that ls -R lists the contents of all subdirectories but doesn't give you full paths for their contents.
Well, find is the best tool for that, but you can simulate it in this case via a script like this:
#!/bin/bash
searchFor=wp-config.php
startDir=${1:-.}
lsSubDir() {
local actDir="$1"
for entry in $(ls "$actDir"); do
if [ -d "$actDir/$entry" ]; then
lsSubDir "$actDir/$entry"
else
[ $entry = $searchFor ] && echo "$actDir/$entry"
fi
done
}
lsSubDir $startDir
Save it in a file like findSimulator, make it executable and call it with the directory where to start searching as parameter.
Be warned: this script is not very efficient, it may stop on large subdirectory-trees because of recursion. I would strongly recommend the solution using find.

Bash script that creates a directory structure

I've been googling all night trying to find a way to create a script that creates a directory structure. That looks something like this:
/
shared
shared/projects
shared/series
shared/movies
shared/movies/action
You get the point.
The file that the script reads from look like this:
shared backup
shared data
shared projects
shared projcets series
shared projects movies
shared projects movies action
I want to create a script that reads each line in the file and run the following for each line:
If the directory exist, it places itself in the directory and create the structure from there, if
The directory doesn’t exist, create it.
When all entries in the row have been preceded by, go back to original directory and read the next line.
My system is Ubuntu 10.10.
So far I’ve done this, but it doesn’t work.
#!/bin/bash
pwd=$(pwd)
for structure in ${column[*]}
do
if [ $structure ]
then
cd $structure
else
mkdir $structure
fi
done
cd $pwd
You can use mkdir -p shared/projects/movies/action to create the whole tree: it will create shared, then shared/projects, then shared/projects/movies, and shared/projects/movies/action.
So basically you need script that runs mkdir -p $dir where $dir is the leaf directory of your directory tree.
If struct.txt contains the directory structure that you mention, then just run:
sed '/^$/d;s/ /\//g' struct.txt | xargs mkdir -p
sed will remove blank lines and make the remaining lines look like directory paths.
xargs will take each line and pass it as a parameter to mkdir.
mkdir will make the directory and the -p flag will create any parent directories if needed.
mkdir has a flag -p that creates all the parent directories of the directory you're creating if needed. you can just just read each line, turn it into a path (i.e. s/ /\//g) and call mkdir -p $path on each line
For my solution it was important to me:
a) I wanted to be able to edit the directory structure directly in my bash script so that I didn't have to jump back and forth between two files
b) The code for the folders should be as clear as possible without redundancy with the same paths, so that I can change it easily
# Creates the folder structure defined in folder structure section below
function createFolderStructure() {
depth="1"
while (( "$#" )); do
while (( $1 != $depth )); do
cd ..
(( depth-- ))
done
shift
mkdir "$1"
cd "$1"
(( depth++ ))
shift
done
while (( 1 != $depth )); do
cd ..
(( depth-- ))
done
}
# Folder Structure Section
read -r -d '' FOLDERSTRUCTURE << EOM
1 shared
2 projects
3 movies
4 action
2 series
2 backup
EOM
createFolderStructure $FOLDERSTRUCTURE
Git needs files to record directories. So I put a readme file in each directory and extended the script as follows:
# Creates the folder structure defined in folder structure section below
function createFolderStructure() {
depth="1"
while (( "$#" )); do
while (( $1 != $depth )); do
cd ..
(( depth-- ))
done
shift
mkdir "$1"
cd "$1"
(( depth++ ))
shift
shift
out=""
while [[ "$1" != "-" ]]; do
out=$out" ""$1"
shift
done
shift
echo "$out" > README.md
done
while (( 1 != $depth )); do
cd ..
(( depth-- ))
done
}
# If you like you can read in user defined values here and use them as variables in the folder structure section, e.g.
# echo -n "Enter month of films"
# read month
# ...
# 1 shared - Folder for shared stuff -
# 2 $month - Films from month $month -
# 3 projects - Folder for projects -
# ...
# Folder Structure Section
read -r -d '' FOLDERSTRUCTURE << EOM
1 shared - Folder for shared stuff -
2 projects - Folder for projects -
3 movies - Folder for movies -
4 action - Folder for action movies -
2 series - Folder for series -
2 backup - Backup folder -
EOM
createFolderStructure $FOLDERSTRUCTURE
1) Do something like this
find . -type d > folder_list.txt
to create a list of the folders you need to create.
2) Transfer the list to your destination
3) Recreate the structure in your new location:
cat folder_list.txt | xargs mkdir
notice that you don't need '-p' option in this case though it wouldn't hurt too.
I use this script in my .bash_profile that I use for new projects:
alias project_setup="mkdir Sites Documents Applications Website_Graphics Mockups Logos Colors Requirements Wireframes"
If you want to make a nested folder structure you you could do something like:
alias shared_setup="mkdir Shared shared/projects shared/series shared/movies shared/movies/action"
Assuming you wish to create a tree of folders / directories as below:
tmpdir
________|______
| | |
branches tags trunk
|
sources
____|_____
| |
includes docs
Also assuming that you have a variable that mentions the directory names.
DOMAIN_NAME=includes,docs
You may issue below command:
$ eval "mkdir -p tmpdir/{trunk/sources/{${DOMAIN_NAME}},branches,tags}"
Note: use the BASH version that supports curly-braces expansion.

Resources