The following script (credit to Romeo Ninov) selects the most recent directory and performs a cp operation:
dir=$(ls -tr1 /var/lib/test|tail -1)
cd /var/lib/test/$dir && cp *.zip /home/bobby/
Please see: How can I use a cronjob when another program makes the commands in the cronjob fail? for the previous question.
I would like to modify this so that the cp only happens if the .zip file is larger than a defined byte size e.g. 28,000 bytes. If the .zip file is smaller, nothing is copied.
As before, this would happen in /var/lib/test/**** (where **** goes from 0000 to FFFF and increments every day).
Thanks!
You can rewrite your script on this way:
dir=$(ls -tr1 /var/lib/test|tail -1)
cd /var/lib/test/$dir
for i in *.zip
do
if [ "$(stat --printf="%s" $i)" -gt 28000 ]
then cp $i /home/bobby
fi
done
Related
I have some csv files in location A like this
abc1.csv,
abc2.csv,
abc3.csv
I have a cron job which runs every 30 mins and in each execution I want to copy only 1 file(which shouldn't be repeated) and placed it in location B
I had though of 2 ways of doing this
1)I will pick the first file in the list of files and copy it to location B and will delete it once copied.Problem with this is I am not sure when the file will get copied completely and if i delete before its completed copied it can be an issue
2)I will have a temp folder.So i will copy the file from location A to location B and also keep it in temp location.In next iteration, when I pick the file from list of files I will compare its existence in the temp file location.If it exists I will move to next file .I think this will be more time consuming etc.
Please suggest if there is any other better way
you can use this bash script for your use case:
source="/path/to/.csv/directory"
dest="/path/to/destination/directory"
cd $source
for file in *.csv
do
if [ ! -f $dest/"$file" ]
then
cp -v $file $dest
break
fi
done
You can ensure you move the already copied file with:
cp abc1.csv destination/ && mv abc1.csv.done
(here you can make your logic to find only *.csv files, and not take into account *.done files.. that have been already processed by your script... or use any suffix you want..
if the cp does not succeed, nothing after that will get executed, so the file will not be moved.
You can also replace mv with rm to delete it:
cp abc1.csv destination/ && rm -f abc1.csv
Further more, you can add to the above commands error messages in case you want to be informed if the cp failed:
cp abc1.csv destination/ && mv abc1.csv.done || echo "copy of file abc1.csv failed"
And get informed via CRON/email output
Finally I took some idea from both the opted solution.Here is the final script
source="/path/to/.csv/directory"
dest="/path/to/destination/directory"
cd $source
for file in *.csv
do
if [ ! -f $dest/"$file" ]
then
cp -v $file $dest || echo "copy of file $file failed"
rm -f $file
break
fi
done
I have a directory called BigDataDirectory which has plenty of files and all of it in total adds up to 8tb.
I am trying to upload to our server and want to make sure I can divide the folder into four parts so I have four folders of about 2tb each.
I tried the split command but it doesn't seem to be working
nohup split -b 2T BigDataDirectory "Directory" &
Could you tell me just a simple way to divide my directory/folder into multiple parts?
If all your files are of a similar size or if the total directory size doesn't need to be very precise, you could do it with these fun commands.
It creates a sub-directory every 250 files and moves the files to a sub-directory. If you have 1000 files, they will be moved to 4 sub-directories named 1, 2, 3 and 4. If you have 1001 files, a 5th subdir will be created for the last file.
cd YOUR_BIG_DIR
n=250 # will change destination dir after 250 files
dir=1; mkdir -p $dir # create first destination sub-directory "1"
ls -1 | while read f; do ((c+=1)); echo mv "$f" $dir/; [ $((c % n)) -eq 0 ] && ((dir+=1)) && mkdir -vp $dir; done
If the output looks like what you want, remove echo from the command to really mv the files.
The same command, explained, and without the "echo" used for testing:
# list all files, 1 per line
ls -1 \
| while read f; do # with each file "$f"
((c+=1)) # increase file counter
mv "$f" $dir/ # move the file to $dir/
# if counter c is a divisor of n, increase the directory number
# and create the new destination directory
[ $((c % n)) -eq 0 ] && ((dir+=1)) && mkdir -v -p $dir
done
If you want sub-directories with a more exact size, then you would need to script something more sophisticated, using stat -c %s to get the size of each file or something similar.
I have people uploading files to a directory on my Ubuntu Server.
I need to move those files to the final location (another directory) only when I know these files are fully uploaded.
Here's my script so far:
#!/bin/bash
cd /var/uploaded_by_users
for filename in *; do
lsof $filename
if [ -z $? ]; then
# file has been closed, move it
else
echo "*** File is open. Skipping..."
fi
done
cd -
However it's not working as it says some files are open when that's not true. I supposed $? would have 0 if the file was closed and 1 if it wasn't but I think that's wrong.
I'm not linux expert so I'm looking to know how to implement this simple script that will run on a cron job every 1 minute.
[ -z $? ] checks if $? is of zero length or not. Since $? will never be a null string, your check will always fail and result in else part being executed.
You need to test for numeric zero, as below:
lsof "$filename" >/dev/null; lsof_status=$?
if [ "$lsof_status" -eq 0 ]; then
# file is open, skipping
else
# move it
fi
Or more simply (as Benjamin pointed out):
if lsof "$filename" >/dev/null; then
# file is open, skip
else
# move it
fi
Using negation, we can shorten the if statement (as dimo414 pointed out):
if ! lsof "$filename" >/dev/null; then
# move it
fi
You can shorten it even further, using &&:
for filename in *; do
lsof "$filename" >/dev/null && continue # skip if the file is open
# move the file
done
You may not need to worry about when the write is complete, if you are moving the file to a different location in the same file system. As long as the client is using the same file descriptor to write to the file, you can simply create a new hard link for the upload file, then remove the original link. The client's file descriptor won't be affected by one of the links being removed.
cd /var/uploaded_by_users
for f in *; do
ln "$f" /somewhere/else/"$f"
rm "$f"
done
I am new to scripting. Currently I have a script that backs up a directory every day to a file server. It deletes the oldest file outside of 14 days. My issue is I need it to count the actual files and delete the 14th oldest one. When going by days, if the file server or host is down for a few days or longer, when back up it will delete a couple days worth of backups or even all of them. Pending down time. I want it to always have 14 days worth of backups.
I tried searching around and could only find solutions related to deleting by dates. Like what I have now.
Thank you for the help/advice!
My code I have, sorry its my first attempt at scripting:
#! /bin/sh
#Check for file. If not found, the connection to the file server is down!
if
[ -f /backup/connection ];
then
echo "File Server is connected!"
#Directory to be backed up.
backup_source="/var/www/html/moin-1.9.7"
#Backup directory.
backup_destination="/backup"
#Current date to name files.
date=`date '+%m%d%y'`
#naming the file.
filename="$date.tgz"
echo "Backing up directory"
#Creating the back up of the backup_source directory and placing it into the backup_destination directory.
tar -cvpzf $backup_destination/$filename $backup_source
echo "Backup Finished!"
#Search for folders older than '+X' days and delete them.
find /backup -type f -ctime +13 -exec rm -rf {} \;
else
echo "File Server is NOT connected! Date:`date '+%m-%d-%y'` Time:`date '+%H:%M:%S'`" > /user/Desktop/error/`date '+%m-%d-%y'`
fi
Something along the lines like this might work:
ls -1t /path/to/directory/ | head -n 14 | tail -n 1
in the ls command, -1 is to list just the filenames (nothing else), -t is to list them in chronological order (newest first). Piping through the head command takes just the first 14 from the output of the ls command, then tail -n 1 takes just the last from that list. This should give the the file that is 14th newest.
Here is another suggestion. The following script simply enumerates the backups. This eases the task of keeping track of the last n backups. If you need to know the actual creation date you can simply check the file metadata, e.g. using stat.
#!/bin/sh
set -e
backup_source='somedir'
backup_destination='backup'
retain=14
filename="backup-$retain.tgz"
check_fileserver() {
nc -z -w 5 file.server.net 80 2>/dev/null || exit 1
}
backup_advance() {
if [ -f "$backup_destination/$filename" ]; then
echo "removing $filename"
rm "$backup_destination/$filename"
fi
for i in $(seq $(($retain)) -1 2); do
file_to="backup-$i.tgz"
file_from="backup-$(($i - 1)).tgz"
if [ -f "$backup_destination/$file_from" ]; then
echo "moving $backup_destination/$file_from to $backup_destination/$file_to"
mv "$backup_destination/$file_from" "$backup_destination/$file_to"
fi
done
}
do_backup() {
tar czf "$backup_destination/backup-1.tgz" "$backup_source"
}
check_fileserver
backup_advance
do_backup
exit 0
I have created a script that goes through specific subdirectories of files and tells me how many files are in each sub-directory that start with s. My problem occurs when it is searching and the sub-directory has failed to be created. For some reason, when the sub-directory that this script is searching for does not exist, it replaces the output with another previously created variable????
I am writing this in bash for linux.
I am looking at the following subdirectories...
participantdirectory/EmotMRI
participantdirectory/EmotMRI/firstfour/
participantdirectory/T1
So, this is the output I should get, when the subdirectory exists and everything is ok. It is the same for all files (if it is correct).
/home/orkney_01/jsiegel/ruth_data/participants/analysis2/1206681446/20090303/14693
16 in firstfour
776 in EmotMRI folder
2 files in T1 folder
For a directory which does not have a subdirectory created, I get this output...
bash: cd: /home/orkney_01/jsiegel/ruth_data/participants/analysis2/2102770508/20090210 /14616/EmotMRI/firstfour/: No such file or directory
/home/orkney_01/jsiegel/ruth_data/participants/analysis2/2102770508/20090210/14616
776 in firstfour
114 in EmotMRI folder
2 files in T1 folder
I think that, because firstfour is a subdirectory of EmotMRI, when firstfour folder hasn't been created, it substitutes the scan numbers in EmotMRI for this answer? The number of scans in EmotMRI (in this instance is correct). Here is my script below. If this is happening, how do I stop it from doing this?
for d in $(cat /home/orkney_01/jsiegel/ruth_data/lists/full_participant_list_location_may20)
do
if [ -d "$d" ]
then
gr="failed"
er="failed"
fr="failed"
cd $d/EmotMRI/firstfour/
gr=$(ls s*| wc -l)
echo " "
echo "$d"
echo "$gr in firstfour"
cd $d/EmotMRI/
er=$(ls s*| wc -l)
echo "$er in EmotMRI folder"
cd $d/T1/
fr=$(ls s*| wc -l)
echo "$fr files in T1 folder"
cd $d/EmotMRI
else
echo "$d is currently not available in directory"
fi
done
cd /home/orkney_01/jsiegel/ruth_data/
echo "Check complete"
I know you will probably have many improvements on this script, I am very new to linux. Thanks for your help,
Currently, you set gr to the output of ls s* | wc -l regardless of whether you successfully change your working directory. When that cd fails, it leaves you in whatever directory you were in previously.
You can combine your cd command into your other commands to set gr:
gr=$(cd $d/EmotMRI/firstfour/ && ls s* | wc -l || echo failed)
This way, if you successfully cd into the subdirectory, gr will be set to the output of the commands after &&. Otherwise, gr will be set to the output of the command after the ||. You can do the same thing with er and fr.
You are getting error messages that you shoud fix. Cd is failing because you are not allowed to change into a non-existent directory. Your shell will just stay in the directory it was in. It looks like you know how to test for directory existence, so you should just do that more to avoid trying to go into non-existent directories.