Script that calls another script to execute on every file in a directory - linux

There are two directories that contains these files:
First one /usr/local/nagios/etc/hosts
[root#localhost hosts]$ ll
total 12
-rw-rw-r-- 1 apache nagios 1236 Feb 7 10:10 10.80.12.53.cfg
-rw-rw-r-- 1 apache nagios 1064 Feb 27 22:47 10.80.12.62.cfg
-rw-rw-r-- 1 apache nagios 1063 Feb 22 12:02 localhost.cfg
And the second one /usr/local/nagios/etc/services
[root#localhost services]$ ll
total 20
-rw-rw-r-- 1 apache nagios 2183 Feb 27 22:48 10.80.12.62.cfg
-rw-rw-r-- 1 apache nagios 1339 Feb 13 10:47 Check usage _etc.cfg
-rw-rw-r-- 1 apache nagios 7874 Feb 22 11:59 localhost.cfg
And I have a script that goes through file in Hosts directory and paste some lines from that file in the file in the Services directory.
The script is ran like this:
./nagios-contacts.sh /usr/local/nagios/etc/hosts/10.80.12.62.cfg /usr/local/nagios/etc/services/10.80.12.62.cfg
How can I achieve that another script calls my script and goes through every file in the Hosts directory and does its job for the files with the same name in the Service directory?
In my script I´m pulling out contacts from the 10.80.12.62.cfg in the Hosts directory and appending them to the file with the same name in the Service directory.

Don't use ls output as an input to for loop instead use the built-in wild-cards. See why it's not a good idea.
for f in /usr/local/nagios/etc/hosts/*.cfg
do
basef=$(basename "$f")
./nagios-contacts.sh "$f" "/usr/local/nagios/etc/services/${basef}"
done

It sounds like you just need to do some iteration.
echo $(pwd)
for file in $(ls); do ./nagious-contacts.sh $file; done;
So it will loop over all files in the current directory.
You can also modify it as well by doing something more absolute.
abspath=$1
for file in $(ls $abspath); do ./nagious-contacts.sh $abspath/$file; done
which would loop over all files in a set directory, and then pass the abspath/filename into your script.

Related

file zip/tar in linux at specific location

I want to zip a set of directories and files on my centos 8 VM.
There are 3 directories and 1 file which I want to zip in such a way that only env.conf file will move to /etc/env.txt after unzipping it and remaining directories will be unzipped at current location.
Is there any way to achieve this.
drwxr-xr-x. 9 root root 114 Feb 25 12:40 config
-rw-r--r--. 1 root root 340 Feb 25 09:01 env.conf
drwxr-xr-x. 9 root root 4096 Feb 28 05:11 platform
drwxr-xr-x. 2 root root 135 Feb 28 07:49 install
I don't think this is possible. in fact this is considered a vulnerability if you could do that.
Imagine you download a zip file from some website. and after you unzip it in a temp folder. It registers itself as a service by writing a file in /etc somewhere, and gets control over your pc.
Example: zip-slip
You could however create a one-liner that extracts and moves the file wherever you want like this:
unzip <filename> && mv env.conf /etc/env.txt

How to take incremental backup of a large file (100GB) with rsync?

I have a scenario where I have to take an incremental backup every 5 minutes of a large file which is around 100GB on the local machine if the content of the file changes.
Filename: example.txt
Backups: example.txt.00:05, example.txt.00:10, example.txt.00:15 and
so on.
What will be the most optimized way to do this?
If I opt for diff than it will take a lot of time to check the content of the file.
I would prefer doing it with rsync but I am unsure about how it will manage the multiple files.
I figured this out with the man page of rsync.
-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir=DIR make backups into hierarchy based in DIR
--suffix=SUFFIX backup suffix (default ~ w/o --backup-dir)
Script:
#!/bin/bash
while True
do
timestamp=$(date +"%H:%M:%S")
echo $timestamp
rsync -avschz --backup --backup-dir=archive --suffix="-$timestamp" example.txt backup
sleep 300
done
The above script will create an archive directory inside the backup directory and rename the files accordingly.
Output:
imohit:rsync-script ethicalmohit$ ls -l backup/archive/
total 88064
-rw-r--r-- 1 ethicalmohit staff 18874368 Mar 25 03:06 example.txt-03:15:41
-rw-r--r-- 1 ethicalmohit staff 12582912 Mar 25 03:17 example.txt-03:25:42
-rw-r--r-- 1 ethicalmohit staff 13631488 Mar 25 03:25 example.txt-03:30:42

Different timestamps when backup files using cron

I need some help to explain cron for backup files.
I have a shell script to backup logfile running in RHEL 6.7 and Solaris 10. It'll move the logfile to backup directory and gunzip each logfile.
Here the script.
#!/bin/bash
# Defined variable
dirLog=/app/rbt3/prod/cda/logs
dirBackup=/app/rbt3/prod/cda/logs/backup
# Change directory to CDA logfile
cd $dirLog
# Backup mechanism
for file in `ls *.log.*` ; do
#echo "FileSemua -> $file"
echo " Pindahkan file $file ke directory $dirBackup "
/bin/mv $dirLog/$file $dirBackup
echo " start Gzip file [$file]..... "
/bin/gzip $dirBackup/$file
echo " done Gzip file [$file]..... "
done
The script is registered in crontab to run every day at 1:20 AM.
20 1 * * * /app/prod/logs/backupLog.sh
Here the backup files that cron created.
-rw-r--r-- 1 user3 user 36344 Nov 18 11:59 alarm.log.20161117.gz
-rw-r--r-- 1 user3 user 35085 Nov 19 11:59 rsync.log.20161117.gz
-rw-r--r-- 1 user3 user 35018 Nov 20 11:59 trace.log.20161117.gz
As far as I know, when we register the script in cron for a specific time. It'll be running and create the files exactly as the cron said (Please correct me if I'm wrong). But in my case, the time for backup files created is different from the cron. Did I miss something?
Thanks.
When you move file time stamps not change. and you use gzip for compressing which use file time stamps.thats why you see different time stamps
$ ls -l
-rw-r--r-- 1 user3 user 36344 Nov 18 11:59 alarm.log.20161117.gz
-rw-r--r-- 1 user3 user 35085 Nov 19 11:59 rsync.log.20161117.gz
-rw-r--r-- 1 user3 user 35018 Nov 20 11:59 trace.log.20161117.gz
ls -l shows the last update time of your log file

anacron script in cron.daily not running via symlink

What can I do to make this script run daily?
If I manually run the script, it works. I can see that it did what it's supposed to do. (backup files) However, it will not run as a cron.daily script. I've let it go for days without touching it -- and it never runs.
The actual script is here /var/www/myapp/backup.sh
There is a symlink to it here /etc/cron.daily/myapp_backup.sh -> /var/www/myapp/backup.sh
The cron log at /var/log/cron shows anacron running this script:
Aug 19 03:09:01 ip-123-456-78-90 anacron[31537]: Job `cron.daily' started
Aug 19 03:09:01 ip-123-456-78-90 run-parts(/etc/cron.daily)[31545]: starting myapp_backup.sh
Aug 19 03:09:01 ip-123-456-78-90 run-parts(/etc/cron.daily)[31559]: finished myapp_backup.sh
Yet there is no evidence that the script actually did anything.
Here is the security info on these files:
ls -la /var/cron.daily
<snip>
lrwxrwxrwx 1 root root 25 Aug 12 21:18 myapp_backup.sh -> /var/www/myapp/backup.sh
</snip>
ls -la /var/www/myapp
<snip>
drwxr-xr-x 2 root root 4096 Aug 13 13:55 .
drwxr-xr-x 10 root root 4096 Jul 12 01:00 ..
-rwxr-xr-x 1 root root 407 Aug 12 23:37 backup.sh
-rw-r--r-- 1 root root 33 Aug 12 21:13 list.txt
</snip>
The file called list.txt is used by backup.sh.
The script just runs tar to create an archive.
From the cron manpage of a debian/ubuntu system:
the files under these directories have to be pass some sanity checks including the following: be executable, be owned by root, not be writable by group or other and, if symlinks, point to files owned by root. Additionally, the file names must conform to the filename requirements of run-parts: they must be entirely made up of letters, digits and can only contain the special signs underscores ('_') and hyphens ('-'). Any file that does not conform to these requirements will not be executed by run-parts. For example, any file containing dots will be ignored.
So:
file need to be owned by root
if symlink, the source file need to be owned by root
if symlink, the link name should NOT contain dots
I had a similar situation with cron.hourly and awstats processing.
I THINK it is related to SELinux and anacron not having the same powers/permissions as cron.
The ACTUAL solution defeated me (so far).
MY WORKAROUND SOLUTION: Run the job via root's cron entries (crontab -e ) and simply schedule it hourly.

Crontab isn't running

My crontab isn't running and I'm trying to figure out why. I've created a symbolic link within /etc/cron.d to /var/www/mysite.crontab
user#ip-xxxxxxxxxx:/etc/cron.d$ ll
total 20
drwxr-xr-x 2 root root 4096 Apr 11 03:48 ./
drwxr-xr-x 96 root root 4096 Apr 16 00:50 ../
lrwxrwxrwx 1 root root 30 Apr 11 03:47 mysite.crontab -> /var/www/mysite.crontab
-rw-r--r-- 1 root root 124 Feb 27 2012 drupal7
-rw-r--r-- 1 root root 544 Sep 12 2012 php5
-rw-r--r-- 1 root root 102 Apr 2 2012 .placeholder
The actual cron file is...
#Purge old deals
4 1 * * * www-data wget -q -O- http://www.mysite.com/cron/clean > /dev/null 2>&1;
Oddly enough the problem is with the name of the file. You are not permitted to use a . as a part of the name of the file when present in the /etc/cron.d dirctory.
The logic for this is in the database.c file, in the function valid_name. Renaming the file to something like mysite_crontab should fix the issue.
In general, the filename should probably just be a simple name mysite the fact that it's in this directory implies that it's a cron file already.
The file that is being pointed to must be owned by root, this is stated in the man page for the support of the /etc/cron.d directory:
Support for /etc/cron.d is included in the cron daemon itself, which handles this location as the system-wide crontab spool. This directory can contain any file defining tasks following the format used in /etc/crontab, i.e. unlike the user cron spool, these files must provide the username to run the task as in the task definition.
Files in this directory have to be owned by root, do not need to be executable (they are configuration files, just like /etc/crontab) and must conform to the same naming convention as used by run-parts(8): they must consist solely of upper- and lower-case letters, digits, underscores, and hyphens. This means that they cannot contain any dots. If the -l option is specified to cron (this option can be setup through /etc/default/cron, see below), then they must conform to the LSB namespace specification, exactly as in the --lsbsysinit option in run-parts.
The intended purpose of this feature is to allow packages that require finer control of their scheduling than the /etc/cron.{hourly,daily,weekly,monthly} directories to add a crontab file to /etc/cron.d. Such files should be named after the package that supplies them.

Resources