I created a log file by running the iostat command to a text file, and ran the command in the background using nohup.
#nohup iostat -xm 5 > /z/logfile.txt &
Later on, I created a cronjob that runs every ten minutes doing the same as above, after I realized my process was being killed by a reboot.
I've also setup log-rotation as below:
/z/logfile.txt {
size 20M
rotate 0
create 0644 root root
missingok
notifempty
}
Now I have realized that the logfile.txt gets deleted but the iostat command keeps pointing at deleted files as shown by the lsof -n | grep deleted command. There the disk space is not freed.
How can I make sure the files are rotated and thereafter iostat points to the newly created file, freeing up disk space?
Any ideas how to set it up correctly?
One solution would be to write a program that will read from iostat, write to the output file, and accept a signal to reopen the file. For example, if you did: iostat -xm 5 | log-daemon /z/logfile.txt where log-daemon is a simple script like:
#!/bin/bash
echo $$ > /var/run/log-daemon
exec > $1
trap 'exec > $1' SIGHUP
read line
while test $? -le 0; do
echo $line
read line
done
Then add a postrotate clause in the logrotate config to send a HUP to the log-daemon:
postrotate
/usr/bin/kill -HUP $(cat /var/run/log-daemon)
Would pointing your cronjob iostat command at a softlink not work?
ln -s /z/logfile.txt iostat_link.txt
nohup iostat -xm 5 > /z/iostat_link.txt &
I haven't used logrotate before, but I tested this by manually changing the file in the background while I had this running:
#Make the files
touch afile1.txt
ln -s afile1.txt file.txt
#Kick off loop
for i in {1..1000};do echo "running still $i" >> file.txt;sleep 3;done &
[localhost (2017-05-15 20:30:55) IP: 26.176 ~]# cat afile1.txt
running still 7
running still 8
running still 9
#Change the file out from under the loop
mv afile1.txt afile1.txt.backup;touch afile1.txt
[localhost (2017-05-15 20:31:21) IP: 26.176 ~]# cat afile1.txt
running still 15
running still 16
running still 17
Check if file system where log is recording is full. If you have such a case find and kill process or reboot server in worst case.
Related
I have created a simple bash script to start capturing traffic from all interfaces I have in my Linux machine (ubuntu 22), but this script should stop capturing traffic 2 hours after the machine has reboot. Below is my bash script
#!/bin/bash
cd /home/user/
tcpdump -U -i any -s 65535 -w output.pcap &
pid=$(ps -e | pgrep tcpdump)
echo $pid
sleep 7200
kill -2 $pid
The script works fine if I run it, but I need to have it running after every reboot.
Whenever I run the script, it works without problem
user#linux:~$ sudo ./startup.sh
[sudo] password for user:
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 65535 bytes
1202
35 packets captured
35 packets received by filter
0 packets dropped by kernel
but when I set it in the crontab as
#reboot /home/user/startup.sh
it does not start at reboot. I used ps -e | pgrep tcpdump to make sure if the script is running but there is not an output, it seems that it is not starting the script after the reboot. I don't know if I need to have root permissions for that. Also, I checked the file permission, and it has
-rwxrwxr-x 1 user user 142 Nov 4 10:11 startup.sh
Any suggestion on why it is not starting the script at the reboot?
Suggesting to update your script:
#!/bin/bash
source /home/user/.bash_profile
cd /home/user/
tcpdump -U -i any -s 65535 -w output.pcap &
pid=$(pgrep -f tcpdump)
echo $pid
sleep 7200
kill -2 $pid
Suggesting to inspect crontab execution log in /var/log/cron
The problem here was that even though the user has root permission, if an script needs to be run in crontab at #reboot, crontab needs to be modified by root. That was the only way I found to run the script. As long as I am running tcpdump, this will require root permission but crontab will not start it at the boot up if it is not modified by sudo.
I logged in a virtual machine via ssh and I tried to run a script in background, the script is shown below:
#!/bin/bash
APP_NAME=`basename $0`
CFG_FILE=$1
. $CFG_FILE #just some variables
CMD=$2
PID_FILE="$PIDS_DIR/$APP_NAME.pid"
CUR_LOG_DIR=$LOGS_RUNNING
echo $$ > $PID_FILE
#Main script code
#This script shall be called using the following syntax
# $ nohup script_name output_dir &
TIMESTAMP=`date +"%Y%m%d%H%M%S"`
CAP_INTERFACE="eth0"
/usr/sbin/tcpdump -nei $CAP_INTERFACE -s 65535 -w file_result
rm $PID_FILE
The result should be tcpdump running in background, redirecting the command result to file_result.
The script is called with:
nohup $SCRIPT_NAME $CFG_FILE start &
And It is stopped calling the STOP_SCRIPT:
##STOP_SCRIPT
PID_FILE="$PIDS_DIR/$APP_NAME.pid"
if [ -f $PID_FILE ]
then
PID=`cat $PID_FILE`
# send SIGTERM to kill all children of $PID
pkill -TERM -P $PID
fi
When I check the file_result, after running the stop script, It is empty.
What is happening? How can I solve it?
I found this link: https://it.toolbox.com/question/launching-tcpdump-processes-in-background-using-ssh-060614
The author seems to have faced a similar issue. They debate about race conditions, but I didn't understand completely.
I'm not sure what you're trying to accomplish by having the startup script itself continue to run, but here's an approach that I think accomplishes what you're trying to do, namely start tcpdump and have it continue to run immune to hangups via nohup. I've simplified things a bit for illustrative purposes - feel free to add any variables back as you see fit, such as the nohup.out output directory, TIMESTAMP, etc.
Script #1: tcpdump_start.sh
#!/bin/sh
rm -f nohup.out
nohup /usr/sbin/tcpdump -ni eth0 -s 65535 -w file_result.pcap &
# Write tcpdump's PID to a file
echo $! > /var/run/tcpdump.pid
Script #2: tcpdump_stop.sh
#!/bin/sh
if [ -f /var/run/tcpdump.pid ]
then
kill `cat /var/run/tcpdump.pid`
echo tcpdump `cat /var/run/tcpdump.pid` killed.
rm -f /var/run/tcpdump.pid
else
echo tcpdump not running.
fi
To start tcpdump, just run tcpdump_start.sh.
To stop the tcpdump instance started with tcpdump_start.sh, just run tcpdump_stop.sh.
The captured packets will be written to the file_result.pcap file, and yes, it's a pcap file, not a text file, so it helps to name it with the proper file extension. The tcpdump statistics will be written to the nohup.out file when tcpdump is terminated.
I too had faced problems when running tcpdump over an SSH session.
In my case, I was running
sudo nohup tcpdump -w {pcap_dump_file} {filter} > /dev/null 2>&1 &
Where, running this command over Paramiko SSH session as a background process was the problem.
To get around this, I used screen utility of Linux.
screen is an easy to use tool for long-running of processes as a service.
Might be an old post, but this is also relevant. I couldn;t understand why no file was being created only to realise that the file might not be created until a certain amount of data had been captured.
https://github.com/the-tcpdump-group/tcpdump/issues/485
I'm trying to find out what script/user/process is writing to a file .
I have 4 hosts that have the same NFS mounted
I have made a scipt and put it on all of the host and with no success
Can sombody please help with this
The script is running from 5:50 to 6:10 this is the period when the my file gets written to
This is the script that I made :
#!/bin/sh
log=~/file-access.log
check_time_to_run() {
tempTime=$1
if [ $tempTime -gt 555 -a $tempTime -lt 610 ]; then
#Intre intervalul 5:55 si 6:10
lsof /cdpool/Xprint/Liste_Drucker >> $log
else
#In afara intervalului
exit 1
fi
}
while true; do
currTime=`date +%k%M`
check_time_to_run $currTime
sleep 0.1s
done
Don't use a shell script for this at all. Instead, install sysdig, and run:
sysdig 'fd.filename=/cdpool/Xprint/Liste_Drucker'
...leave that open, and whenever anything writes to or reads from that file, an appropriate log message will be printed.
If you want to print both the username and the process name (with arguments) for the job printing to the file, the following format string will do so:
sysdig \
-p '%user.name %proc.name - %evt.dir %evt.type %evt.args' \
'fd.filename=/cdpool/Xprint/Liste_Drucker'
Alright, so I have a .sh file that I run that will launch my server with the certain specifics that I'm looking for. It launches the server through screen into it's own screen. Here's the code for my run.sh file.
#!/bin/bash
# run.sh
# conversion of run.bat to shell script.
echo "Protecting srcds from random crashes"
echo "Now launching Garrys Mod RequiemRP"
sleep 5
screen -A -m -d -S gmserver ./srcds_run -console -game garrysmod +maxplayers 32 +map rp_downtown_v6 -autoupdate
echo "Server initialized. Type screen -x to resume"
Usually I use a batch file to do this, but I'm now using linux for my server hosting. Part of that batch file was if srcds (the server itself) were to crash, the run.bat file would restart the server automatically. I'm looking to do this with my run.sh file, but I'm unsure how to.
Perhaps you could make a service or script that will periodically check if the process is running. This will check if it's on and if it isn't, it will turn it on when executed.
#!/bin/bash
ps cax | grep srcds > /dev/null
if [ $? -eq 0 ]; then
exit
else
bash /path/to/run.sh
fi
I tested the command and it works. For my virtualized debian 9 system.
I have a basic script that outputs various status messages. e.g.
~$ ./myscript.sh
0 of 100
1 of 100
2 of 100
...
I wanted to wrap this in a parent script, in order to run a sequence of child-scripts and send an email upon overall completion, e.g. topscript.sh
#!/bin/bash
START=$(date +%s)
/usr/local/bin/myscript.sh
/usr/local/bin/otherscript.sh
/usr/local/bin/anotherscript.sh
RET=$?
END=$(date +%s)
echo -e "Subject:Task Complete\nBegan on $START and finished at $END and exited with status $RET.\n" | sendmail -v group#mydomain.com
I'm running this like:
~$ topscript.sh >/var/log/topscript.log 2>&1
However, when I run tail -f /var/log/topscript.log to inspect the log I see nothing, even though running top shows myscript.sh is currently being executed, and therefore, presumably outputting status messages.
Why isn't the stdout/stderr from the child scripts being captured in the parent's log? How do I fix this?
EDIT: I'm also running these on a remote machine, connected via ssh using pseudo-tty allocation, e.g. ssh -t user#host. Could the pseudo-tty be interfering?
I just tried your the following: I have three files t1.sh, t2.sh, and t3.sh all with the following content:
#!/bin/bash
for((i=0;i<10;i++)) ; do
echo $i of 9
sleep 1
done
And a script called myscript.sh with the following content:
#!/bin/bash
./t1.sh
./t2.sh
./t3.sh
echo "All Done"
When I run ./myscript.sh > topscript.log 2>&1 and then in another terminal run tail -f topscript.log I see the lines being output just fine in the log file.
Perhaps the things being run in your subscripts use a large output buffer? I know when I've run python scripts before, it has a pretty big output buffer so you don't see any output for a while. Do you actually see the entire output in the email that gets sent out at the end of topscript.sh? Is it just that while the processes run you're not seeing the output?
try
unbuffer topscript.sh >/var/log/topscript.log 2>&1
Note that unbuffer is not always available as a std binary in old-style Unix platforms and may require a search and installation for a package to support it.
I hope this helps.