I'm in my first semester of BASH Scripting and I am having some difficulty. I've taken other programming courses like C++ or Java but the syntax of Bash is killing me. I'd love some advice on this problem. I need to do the following:
Extract Today's data from /var/log/secure file
Check to see if I have a directory called 'mylogs'
If I don't - then create one
Check to see if you already have a file matching the current day, month and hour
in the ‘mylogs’ directory.
If you do, echo to the screen “File exists, nothing written to my log”, and
exit. If it doesn’t exist then write today’s data from /var/log/secure to your
‘mylog-month-day-hour’ file. Example (February, 4th at 2pm) output:
mylog-02-04-14
I just need help with the syntax portion of the script.
Thanks - I'd love any websites helping out in BASH as well.
Extract Today's data from /var/log/secure file
You could do this ...
grep "^Feb 24" /var/log/secure
Check to see if I have a directory called 'mylogs' and If I don't - then create one
You can do this ...
test -d mylogs || mkdir mylogs
Check to see if you already have a file matching the current day, month and hour in the ‘mylogs’ directory. (Assuming file names are of the format DDMMHH)
test -e mylogs/`date +%d%m%H` && echo "I already have a file"
If you do, echo to the screen “File exists, nothing written to my log”, and exit. If it doesn’t exist then write today’s data from /var/log/secure to your ‘mylog-month-day-hour’ file. Example (February, 4th at 2pm) output: mylog-02-04-14
Eh you should get the idea by now. You can tackle this one now I think ;) A helpful command to know is man -k <keyword>
First read some bash basics. Then, there are links describing your particular problem.
http://www.cyberciti.biz/faq/linux-howto-unzip-files-in-root-directory/
Check if a directory exists in a shell script
How to use Bash to create a folder if it doesn't already exist?
http://www.electrictoolbox.com/test-file-exists-bash-shell/
http://www.cyberciti.biz/tips/howto-linux-unix-write-to-syslog.html
Related
For my VPS I've created a bash script what will run every three hours by a cronjob for backing-up my VPS. All databases (in this case) will be dumpt and moved to a new folder on a storage-services connected via WebDAV.
It runs perfectly until yesterday. The script gives me the error:
mkdir: cannot create directory ‘/stack/VPS-Backups/Srv1/Database/07-10-2016_12:00’: No such file or directory
The script (where it goes wrong):
#!/bin/bash
DB_BACKUP="/stack/VPS-Backups/Srv1/Database/`date +%d-%m-%Y`_`date +%H:%M`"
# Create the backup directory
mkdir $DB_BACKUP
I checked/procees the following things already:
Re-mount the WebDAV
Check if the directory structure "/stack/VPS-Backups/Srv1/Database" exist
The URL/username/password of the WebDAV doesn't changed
The supplier of the storage storage doesn't changed a thing
What can I do? Thanks for helping! :-)
#!/bin/bash
dirname=$(date '+%Y%m%dT%H%M') # subset of an ISO 8601 date (only missing %S)
cd /stack/VPS-Backups/Srv1/Database || exit
mkdir -- "$dirname"
The notable changes encapsulated here:
Break out the cd from the mkdir, to clarify which of these operations is failing.
Remove characters that aren't guaranteed by POSIX to be valid in filenames (the portable filename character set). In practice, this means the colons need to go.
Also, this follows ISO 8601 guidelines for the date format. This means other software will be able to parse your directory names as dates out-of-the-box. Using YYYYMMDD also prevents confusion between MMDDYYYY and DDMMYYYY, and makes your names' ASCII sort order match their logical sort order, so you can use standard UNIX tools for range selection.
I'd like to monitor a directory for new files daily using a linux bash script.
New files are added to the directory every 4 hours or so. So I'd like to at the end of the day process all the files.
By process I mean convert them to an alternative file type then pipe them to another folder once converted.
I've looked at inotify to monitor the directory but can't tell if you can make this a daily thing.
Using inotify I have got this code working in a sample script:
#!/bin/bash
while read line
do
echo "close_write: $line"
done < <(inotifywait -mr -e close_write "/home/tmp/")
This does notify when new files are added and it is immediate.
I was considering using this and keeping track of the new files then processing them at all at once, at the end of the day.
I haven't done this before so I was hoping for some help.
Maybe something other than inotify will work better.
Thanks!
You can use a daily cron job: http://linux.die.net/man/1/crontab
Definitely should look into using a cronjob. Edit your cronfile and put this in:
0 0 * * * /path/to/script.sh
That means run your script at midnight everyday. Then in your script.sh, all you would do is for all the files, "convert them to an alternative file type then pipe them to another folder once converted".
Your cron job (see other answers on this page) should keep a list of the files you have already processed and then use comm -3 processed-list all-list to get the new files.
man comm
Its a better alternative to
awk 'FNR==NR{a[$0];next}!($0 in a)' processed-list all-list
and probably more robust than using find since you record the ones that you have actually processed.
To collect the files by the end of day, just use find:
find $DIR -daystart -mtime -1 -type f
Then as others pointed out, set up a cron job to run your script.
I used logrotate to create a log file that gets rolled over every day at 03:00; so the file that is created has the following format:
userlog.%Y-%m-%d
the script then zips the file as well so the end result is userlog.%Y-%m-%d.gz .... an actual file name is userlog.2015-09-09.gz.
I am writing a shell script that will copy the file created every day and will then unzip it, run a search on the log, extract results, format them, and then email them, and then delete the copied zip file.
I can get everything to work smoothly but I cannot copy the file using the method that it was created in.
I am trying to run the following:
sudo cp userlog.%Y-%d-%m.gz /home/local/temp
but the command does not execute. My understanding is that the current date, month, and year should be substituted into the variable fields in the file name.
If you could please correct my approach and understanding of this concept. Or please let me know if you feel this should work. Or if there is an alternate approach to call a file created in this fashion then kindly advise.
Presently I am having to write the filename into the script manually every morning and I would very much like to escape this hardship.
You need a simple command substitution.
sudo cp userlog.$(date +%Y-%d-%m).gz /home/local/temp
%Y by itself is just a static string; but when passed as an argument to the date command, it specifies the four-digit year for the given date. (Default is today's date.)
Regardless of your problems with this particular command, you should certainly not need to edit the script file every day. A very fundamental feature of scripts is the parametrization of arguments. If your current script looks like
#!/bin/bash
gzgrep -c "failure" /path/to/userlog.2015-09-11.gz
sudo cp /path/to/userlog.2015-09-11.gz /home/local/temp
then it can easily be refactored to read the date as a command-line parameter:
#!/bin/bash
: ${1?"Syntax: $0 yyyy-mm-dd"}
gzgrep -c "failure" /path/to/userlog."$1".gz
sudo cp /path/to/userlog."$1".gz /home/local/temp
If you saved this as userlog somewhere in your PATH, the solution to your original problem would now be simply
userlog $(date +%Y-%m-%d)
which of course could be saved as a script, too, or as a shell function in your personal .bash_profile or similar. But if this is mandatory daily routine, you'll probably prefer to add it as a cron job to run every morning (even if you are away). Cron will send you an email with any output. (Do notice that cron does not necessarily have the same PATH etc as your interactive login shell, though.)
I have a RedHat linux box and I had written a script in the past to move files from one location to another with a specific text in the body of the file.
I typically only write scripts once a year so every year I forget more and more... That being said,
Last year I wrote this script and used it and it worked.
For some reason, I can not get it to work today and I know it's a simple issue and I shouldn't even be asking for help but for some reason I'm just not looking at it correctly today.
Here is the script.
ls -1 /var/text.old | while read file
do
grep -q "to.move" $file && mv $file /var/text.old/TBD
done
I'm listing all the files inside the /var/text.old directory.
I'm reading each file
then I'm grep'ing for "to.move" and holing the results
then I'm moving the resulting found files to the folder /var/text.old/TBD
I am an admin and I have rights to the above files and folders.
I can see the data in each file
I can mv them manually
I have use pwd to grab the correct spelling of the directory.
If anyone can just help me to see what the heck I'm missing here that would really make my day.
Thanks in advance.
UPDATE:
The files I need to move do not have Whitespaces.
The Error I'm getting is as follows:
grep: 9829563.msg: No such file or directory
NOTE: the file "982953.msg" is one of the files I need to move.
Also note: I'm getting this error for every file in the directory that I'm listing.
You didn't post any error, but I'm gonna take a guess and say that you have a filename with a space or special shell character.
Let's say you have 3 files, and ls -1 gives us:
hello
world
hey there
Now, while splits on the value of the special $IFS variable, which is set to <space><tab><newline> by default.
So instead of looping of 3 values like you expect (hello, world, and hey there), you loop over 4 values (hello, world, hey, and there).
To fix this, we can do 2 things:
Set IFS to only a newline:
IFS="
"
ls -1 /var/text.old | while read file
...
In general, I like setting IFS to a newline at the start of the script, since I consider this to be slightly "safer", but opinions on this probably vary.
But much better is to not parse the output of ls, and use for:
for file in /var/text.old/*`; do
This won't fork any external processes (piping to ls to while starts 2), and behaves "less surprising" in other ways. See here for some examples.
The second problem is that you're not quoting $file. You should always quote pathnames with double quoted: "$file" for the same reasons. If $file has a space (or a special shell character, such as *, the meaning of your command changes:
file=hey\ *
mv $file /var/text.old/TBD
Becomes:
mv hey * /var/text.old/TBD
Which is obviously very different from what you intended! What you intended was:
mv "hey *" /var/text.old/TBD
Well, I'm a linux newbie, and I'm having an issue with a simple bash script.
I've got a program that adds to a log file while it's running. Over time that log file gets huge. I'd like to create a startup script which will rename and move the log file before each run, effectively creating separate log files for each run of the program. Here's what I've got so far:
pastebin
DATE=$(date +"%Y%m%d%H%M")
mv server.log logs/$DATE.log
echo program
When run, I see this:
: command not found
program
When I cd to the logs directory and run dir, I see this:
201111211437\r.log\r
What's going on? I'm assuming there's some syntax issue I'm missing, but I can't seem to figure it out.
UPDATE: Thanks to shellter's comment below, I've found the problem to be due to the fact that I'm editing the .sh file in Notepad++ in windows, and then sending via ftp to the server, where I run the file via ssh. After running dos2unix on the file, it works.
New question: How can I save the file correctly in the first place, to avoid having to perform this fix every time I resend the file?
mv server.log logs/$(date -d "today" +"%Y%m%d%H%M").log
The few lines you posted from your script look okay to me. It's probably something a bit deeper.
You need to find which line is giving you this error. Add set -xv to the top of your script. This will print out the line number and the command that's being executed to STDERR. This will help you identify where in your script you're getting this particular error.
BTW, do you have a shebang at the top of your script? When I see something like this, I normally expect its an issue with the Shebang. For example, if you had #! /bin/bash on top, but your bash interpreter is located in /usr/bin/bash, you'll see this error.
EDIT
New question: How can I save the file correctly in the first place, to avoid having to perform this fix every time I resend the file?
Two ways:
Select the Edit->EOL Conversion->Unix Format menu item when you edit a file. Once it has the correct line endings, Notepad++ will keep them.
To make sure all new files have the correct line endings, go to the Settings->Preferences menu item, and pull up the Preferences dialog box. Select the New Document/Default Directory tab. Under New Document and Format, select the Unix radio button. Click the Close button.
A single line method within bash works like this.
[some out put] >$(date "+%Y.%m.%d-%H.%M.%S").ver
will create a file with a timestamp name with ver extension.
A working file listing snap shot to a date stamp file name as follows can show it working.
find . -type f -exec ls -la {} \; | cut -d ' ' -f 6- >$(date "+%Y.%m.%d-%H.%M.%S").ver
Of course
cat somefile.log > $(date "+%Y.%m.%d-%H.%M.%S").ver
or even simpler
ls > $(date "+%Y.%m.%d-%H.%M.%S").ver
I use this command for simple rotate a file:
mv output.log `date +%F`-output.log
In local folder I have 2019-09-25-output.log
Well, it's not a direct answer to your question, but there's a tool in GNU/Linux whose job is to rotate log files on regular basis, keeping old ones zipped up to a certain limit. It's logrotate
You can write your scripts in notepad but just make sure you convert them
using this ->
$ sed -i 's/\r$//' yourscripthere
I use it all they time when I'm working in cygwin and it works. Hope this helps
First, thanks for the answers above! They lead to my solution.
I added this alias to my .bashrc file:
alias now='date +%Y-%m-%d-%H.%M.%S'
Now when I want to put a time stamp on a file such as a build log I can do this:
mvn clean install | tee build-$(now).log
and I get a file name like:
build-2021-02-04-03.12.12.log