For my VPS I've created a bash script what will run every three hours by a cronjob for backing-up my VPS. All databases (in this case) will be dumpt and moved to a new folder on a storage-services connected via WebDAV.
It runs perfectly until yesterday. The script gives me the error:
mkdir: cannot create directory ‘/stack/VPS-Backups/Srv1/Database/07-10-2016_12:00’: No such file or directory
The script (where it goes wrong):
#!/bin/bash
DB_BACKUP="/stack/VPS-Backups/Srv1/Database/`date +%d-%m-%Y`_`date +%H:%M`"
# Create the backup directory
mkdir $DB_BACKUP
I checked/procees the following things already:
Re-mount the WebDAV
Check if the directory structure "/stack/VPS-Backups/Srv1/Database" exist
The URL/username/password of the WebDAV doesn't changed
The supplier of the storage storage doesn't changed a thing
What can I do? Thanks for helping! :-)
#!/bin/bash
dirname=$(date '+%Y%m%dT%H%M') # subset of an ISO 8601 date (only missing %S)
cd /stack/VPS-Backups/Srv1/Database || exit
mkdir -- "$dirname"
The notable changes encapsulated here:
Break out the cd from the mkdir, to clarify which of these operations is failing.
Remove characters that aren't guaranteed by POSIX to be valid in filenames (the portable filename character set). In practice, this means the colons need to go.
Also, this follows ISO 8601 guidelines for the date format. This means other software will be able to parse your directory names as dates out-of-the-box. Using YYYYMMDD also prevents confusion between MMDDYYYY and DDMMYYYY, and makes your names' ASCII sort order match their logical sort order, so you can use standard UNIX tools for range selection.
Related
I used logrotate to create a log file that gets rolled over every day at 03:00; so the file that is created has the following format:
userlog.%Y-%m-%d
the script then zips the file as well so the end result is userlog.%Y-%m-%d.gz .... an actual file name is userlog.2015-09-09.gz.
I am writing a shell script that will copy the file created every day and will then unzip it, run a search on the log, extract results, format them, and then email them, and then delete the copied zip file.
I can get everything to work smoothly but I cannot copy the file using the method that it was created in.
I am trying to run the following:
sudo cp userlog.%Y-%d-%m.gz /home/local/temp
but the command does not execute. My understanding is that the current date, month, and year should be substituted into the variable fields in the file name.
If you could please correct my approach and understanding of this concept. Or please let me know if you feel this should work. Or if there is an alternate approach to call a file created in this fashion then kindly advise.
Presently I am having to write the filename into the script manually every morning and I would very much like to escape this hardship.
You need a simple command substitution.
sudo cp userlog.$(date +%Y-%d-%m).gz /home/local/temp
%Y by itself is just a static string; but when passed as an argument to the date command, it specifies the four-digit year for the given date. (Default is today's date.)
Regardless of your problems with this particular command, you should certainly not need to edit the script file every day. A very fundamental feature of scripts is the parametrization of arguments. If your current script looks like
#!/bin/bash
gzgrep -c "failure" /path/to/userlog.2015-09-11.gz
sudo cp /path/to/userlog.2015-09-11.gz /home/local/temp
then it can easily be refactored to read the date as a command-line parameter:
#!/bin/bash
: ${1?"Syntax: $0 yyyy-mm-dd"}
gzgrep -c "failure" /path/to/userlog."$1".gz
sudo cp /path/to/userlog."$1".gz /home/local/temp
If you saved this as userlog somewhere in your PATH, the solution to your original problem would now be simply
userlog $(date +%Y-%m-%d)
which of course could be saved as a script, too, or as a shell function in your personal .bash_profile or similar. But if this is mandatory daily routine, you'll probably prefer to add it as a cron job to run every morning (even if you are away). Cron will send you an email with any output. (Do notice that cron does not necessarily have the same PATH etc as your interactive login shell, though.)
Currently I am trying to run MRI software (TBSS) on imaging files(scan.nii.gz) on the Linux command line.
The scans are all stored in separate folders for different participants and the file names are identical,so:
/home/scans/participant1/scan.nii.gz
/home/scans/participant2/scan.nii.gz
/home/scans/participant3/scan.nii.gz
What this software does is it creates the result of the analysis in the current working directory.Since the scans have the same image name, they get overwritten al the time.
I would like to loop through all the participant folders, make it my working directory and then execute the tbss command, which is simply tbss_1_preproc scan.nii.gz. In this way, the file will be stored in the current working directory,which is the participant directory.
Is there any sensible way of doing this in Linux ?
Thanks so much !
Try it in BASH. The code below is untested, but it should give you a clue
#! /bin/bash
find . -name scan.nii.gz | while read line
do
cd $(dirname "${line}")
tbss_1_preproc $(basename "${line}")
done
Put it in a file and make it executable. Copy it to your scans folder and execute it.
I'm in my first semester of BASH Scripting and I am having some difficulty. I've taken other programming courses like C++ or Java but the syntax of Bash is killing me. I'd love some advice on this problem. I need to do the following:
Extract Today's data from /var/log/secure file
Check to see if I have a directory called 'mylogs'
If I don't - then create one
Check to see if you already have a file matching the current day, month and hour
in the ‘mylogs’ directory.
If you do, echo to the screen “File exists, nothing written to my log”, and
exit. If it doesn’t exist then write today’s data from /var/log/secure to your
‘mylog-month-day-hour’ file. Example (February, 4th at 2pm) output:
mylog-02-04-14
I just need help with the syntax portion of the script.
Thanks - I'd love any websites helping out in BASH as well.
Extract Today's data from /var/log/secure file
You could do this ...
grep "^Feb 24" /var/log/secure
Check to see if I have a directory called 'mylogs' and If I don't - then create one
You can do this ...
test -d mylogs || mkdir mylogs
Check to see if you already have a file matching the current day, month and hour in the ‘mylogs’ directory. (Assuming file names are of the format DDMMHH)
test -e mylogs/`date +%d%m%H` && echo "I already have a file"
If you do, echo to the screen “File exists, nothing written to my log”, and exit. If it doesn’t exist then write today’s data from /var/log/secure to your ‘mylog-month-day-hour’ file. Example (February, 4th at 2pm) output: mylog-02-04-14
Eh you should get the idea by now. You can tackle this one now I think ;) A helpful command to know is man -k <keyword>
First read some bash basics. Then, there are links describing your particular problem.
http://www.cyberciti.biz/faq/linux-howto-unzip-files-in-root-directory/
Check if a directory exists in a shell script
How to use Bash to create a folder if it doesn't already exist?
http://www.electrictoolbox.com/test-file-exists-bash-shell/
http://www.cyberciti.biz/tips/howto-linux-unix-write-to-syslog.html
I need to write a shell script to run as a cron task, or preferably on creation of a file in a certain folder.
I have an incoming and an outgoing folder (they will be used to log mail). There will be files created with codes as follows...
bmo-001-012-dfd-11 for outgoing and 012-dfd-003-11 for incoming. I need to filter the project/client code (012-dfd) and then place it in a folder in the specific project folder.
Project folders are located in /projects and follow the format 012-dfd. I need to create symbolic links inside the incoming or outgoing folders of the projects, that leads to the correct file in the general incoming and outgoing folders.
/incoming/012-dfd-003-11.pdf -> /projects/012-dfd/incoming/012-dfd-003-11.pdf
/outgoing/bmo-001-012-dfd-11.pdf -> /projects/012-dfd/outgoing/bmo-001-012-dfd-11.pdf
So my questions
How would I make my script run when a file is added to either incoming or outgoing folder
Additionally, is there any associated disadvantages with running upon file modification compared with running as cron task every 5 mins
How would I get the filename of recent (since script last run) files
How would I extract the code from the filename
How would I use the code to create a symlink in the desired folder
EDIT: What I ended up doing...
while inotifywait outgoing; do find -L . -type l -delete; ls outgoing | php -R '
if(
preg_match("/^\w{3}-\d{3}-(\d{3}-\w{3})-\d{2}(.+)$/", $argn, $m)
&& $m[1] && (file_exists("projects/$m[1]/outgoing/$argn") != TRUE)
){
`ln -s $(pwd)/outgoing/$argn projects/$m[1]/outgoing/$argn;`;
}
'; done;
This works quite well - cleaning up deleted symlinks also (with find -L . -type l -delete) but I would prefer to do it without the overhead of calling php. I just don't know bash well enough yet.
Some near-answers for your task breakdown:
On linux, use inotify, possibly through one of its command-line tools, or script language bindings.
See above
Assuming the project name can be extracted thinking positionally from your examples (meaning not only does the project name follows a strict 7-character format, but what precedes it in the outgoing file also does):
echo `basename /incoming/012-dfd-003-11.pdf` | cut -c 1-7
012-dfd
echo `basename /outgoing/bmo-001-012-dfd-11.pdf`| cut -c 9-15
012-dfd
mkdir -p /projects/$i/incoming/ creates directory /projects/012-dfd/incoming/ if i = 012-dfd,
ln -s /incoming/foo /projects/$i/incoming/foo creates a symbolic link from the latter argument, to the preexisting, former file /incoming/foo.
How would I make my script run when a file is added to either incoming or outgoing folder
Additionally, is there any associated disadvantages with running upon file modification compared with running as cron task
every 5 mins
If a 5 minutes delay isn't an issue, I would go for the cron job (it's easier and -IMHO- more flexible)
How would I get the filename of recent (since script last run) files
If your script runs every 5 minutes, then you can tell that all the files created in between now (and now - 5 minutes) are newso, using the command ls or find you can list those files.
How would I extract the code from the filename
You can use the sed command
How would I use the code to create a symlink in the desired folder
Once you have the desired file names, you can usen ln -s command to create the symbolic link
I'm using a linux software solution that uses the tar command to backup huge amounts of data.
The command which is hardcoded into the binary which calls the tar is:
/bin/tar --exclude "/backup" --exclude / --ignore-failed-read -cvjf - /pbackup 2>>'/tar_err.log' | split -b 1000m - '/backup/temp/backup.tar.bz2'
There is no chance to change the command, as it is harcoded. It uses bzip2 to compress the data. I experienced a strong performance improvement (up to 60%) when using the parameter --use-compress-prog=pbzip2 which utilizes all CPU cores.
By symlinking the bzip2 from /bin/bzip2 to the pbzip2 binary I tried to trick the software, however when monitoring the process it still uses bzip2 as I tink this is built into tar.
I know it is a tricky question but is there any way to utilize pbzip2 without changing this command that is externally called?
My system is Debian Sequeeze.
Thanks very much!
Danger: ugly solution ahead; backup the binary before proceeding
First of all, check if the hardcoded string is easily accessible: use strings on your binary, and see if it displays the string you said (probably it will be in several pieces, e.g. /bin/tar, --exclude, --ignore-failed-read, ...).
If this succeeds, grab your hex editor of choice, open the binary and look for the hardcoded string; if it's split in several pieces, the one you need is the one containing /bin/tar; overwrite tar with some arbitrary three-letter name, e.g. fkt (fake tar; a quick Google search didn't turn up any result for /usr/bin/fkt, so we should be safe).
The program should now call your /usr/bin/fkt instead of the regular tar.
Now, put in your /bin a script like this:
#!/bin/sh
/bin/tar --use-compress-prog=pbzip2 $*
call it with the name you chose before (fkt) and set the permissions correctly (they should be 755 and owned by root). This script just takes all the parameters it gets and call the real tar, adding in front of them the parameter you need.
Another solution, that I suggested in the comments, may be creating a chroot just for the application, renaming tar to some other name (realtar, maybe?) and calling the script above tar (obviously now you should change the /bin/tar inside the script to /bin/realtar).
If the program is not updated very often and the trick worked at the first try I would probably go with the first solution, setting up and maintaining chroots is not fun.
Why not move /bin/tar to (say) /bin/tar-original
Then create a script /bin/tar to do whatever you want it to do.