I'm new to Linux and have inherited keeping our single linux server running. It's our SVN server so it's relatively important.
Turns out the guy who maintained it before me had a cron task to email him when there are too many svnserve processes running, as they seem to be left dangling instead of terminating correctly.
First part of the question is, given that I run
ps -fu cvsuser
and get a list of the processes, how can I kill the ones that have an STIME not today? For example, something like
kill where STIME = Oct30
The other question is, does anyone know how to avoid having these dangling svnserve processes? (Here's the other question.)
Just for the fun of it (GNU bash, version 3.2.39)
ps h -u cvsuser -o pid,start # h - no header, only output pid and start
| grep -v ':' # exclude entries from the last 24 hours
| egrep -o '^\ *[0-9]+' # get the pid (handling possible leading space)
| xargs -i echo kill "{}" # pretend to kill - take out the echo if you're happy
This relies on the following from 'man ps' STANDARD FORMAT SPECIFIERS:
If the process was started less than
24 hours ago, the output format is
"HH:MM:SS", else it is " mmm dd"
(where mmm is a three-letter month
name).
At the risk of suggesting you re-engineer your infrastructure, I've had great results using Apache and mod dav svn instead of svnserve - Apache's httpd is pretty darn bulletproof after the last decade or so of production usage.
for our cvs/svn/git server, rather than using a cronjob that looked for old processes by relying on the output of ps, we used the timeout program.
$ grep server /etc/xinetd.d/svnserve
server = /usr/bin/timeout
server_args = 12h /usr/bin/svnserve -i -R -r /svnroot
replace "12h" with an appropriate timeout for your site (although i can't imagine most people needing more than 12 hours to checkout a single repo)
Off the top of my head, I would do something like this:
ps -fu username | awk '$5 !~ /[0-9]:[0-9]/ { print $2 }' | xargs kill
Since the fifth field of the ps output shows day-old processes with the month/day (e.g. Oct31) and without the time (e.g. 12:32), the regex with awk simply excludes those processes whose fifth field is still a time. I am assuming, possibly wrongly, that ps starts to show the date only for processes that have been running for more than 24 hours.
To identify and kill the processes:
ps h -u csvuser -o pid,lstart | grep 'May 29' | sed 's/^ \+//' |
cut -d ' ' -f 1 | xargs -n 1 kill
The ps command will find all processes owned by csvuser and output the pid and start time:
16324 Thu May 29 04:02:06 2008
22144 Tue Jul 22 04:02:05 2008
11315 Wed Oct 8 04:02:00 2008
The grep command will find the date you are looking for
16324 Thu May 29 04:02:06 2008
The sed command will remove leading spaces for cut,
The cut command will output only the first field:
16324
And the xargs command will run the kill command once for each line passing the pid as the argument. Replace the grep statement as needed to match whatever you need.
As for why the svnserve processes are not exiting properly, I don't know, I haven't seen this on my subversion servers, you probably should open a separate question for this.
Related
When I do
ps -ax|grep myApp
I get the one line with PID and stuff of my app.
Now, I'ld liked to process the whole result of ps -ax (without grep, so, the full output):
Either store it in a variable and grep from it later
Or go through the results in a for loop, e.g. like that:
for a in $(ps -ax)
do
echo $a
done
Unfortunally, this splits with every space, not with newline as |grep does it.
Any ideas, how I can accomplish one or the other (grep from variable or for loop)?
Important: No bash please, only POSIX, so #!/bin/sh
Thanks in advance
Like stated above, while loop can be helpful here.
One more useful thing is --no-headers argument which makes ps skip the header.
Or - even better - specify the exact columns you need to process, like ps -o pid,command --no-header ax
The overall code would look like
processes=`ps --no-headers -o pid,command ax`
echo "$processes" | while read pid command; do
echo "we have process with pid $pid and command line $command"
done
The only downside to this approach is that commands inside while loop will be executed in subshell so if you need to export some var to the parent process you'll have to do it using inter-process communication stuff.
I usually dump the results into temp file created before while loop and read them after the loop is finished.
I found a solution by removing the spaces while executing the command:
result=$(ps -aux|sed 's/ /_/g')
You can also make it more filter friendly by removing duplicated spaces:
result=$(ps -aux| tr -s ' '|sed 's/ /_/g')
I am monitoring the new files created in a folder in linux. Every now and then I issue an "ls -ltr" in it. But I wish there was a program/script that would automatically print it, and only the latest entries. I did a short while loop to list it, but it would repeat the entries that were not new and it would keep my screen rolling up when there were no new files. I've learned about "watch", which does show what I want and refreshes every N seconds, but I don't want a ncurses interface, I'm looking for something like tail:
continuous
shows only the new stuff
prints in my terminal, so I can run it in the background and do other things and see the output every now and then getting mixed with whatever I'm doing :D
Summarizing: get the input, compare to a previous input, output only what is new.
Something that do that doesn't sound like such an odd tool, I can see it being used for other situations also, so I would expect it to already exist, but I couldn't find anything. Suggestions?
You can use the very handy command watch
watch -n 10 "ls -ltr"
And you will get a ls every 10 seconds.
And if you add a tail -10 you will only get the 10 newest.
watch -n 10 "ls -ltr|tail -10"
If you have access to inotifywait (available from the inotify-tools package if you are on Debian/Ubuntu) you could write a script like this:
#!/bin/bash
WATCH=/tmp
inotifywait -q -m -e create --format %f $WATCH | while read event
do
ls -ltr $WATCH/$event
done
This is a one-liner that won't give you the same information that ls does, but it will print out the filename:
inotifywait -q -m -e create --format %w%f /some/directory
This works in cygwin and Linux. Some of the previous solutions which write a file will cause the disk to thrash.
This script does not have that problem:
SIG=1
SIG0=SIG
while [ $SIG != 0 ] ; do
while [ $SIG = $SIG0 ] ; do
SIG=`ls -1 | md5sum | cut -c1-32`
sleep 10
done
SIG0=$SIG
ls -lrt | tail -n 1
done
When I run this command:
ps aux|awk {'print $1,$2,$3,$11'}
I get a listing of the user, PID, CPU% and the actual command.
I want to pipe all those listings into a shell script to calculate the CPU% and if greater than, say 5, then to kill the process via the PID.
I tried piping it to a simple shell script, i.e.
ps aux|awk {'print $1,$2,$3,$11'} | ./myscript
where the content of my script is:
#!/bin/bash
# testing using positional parameters
echo "$1 $2 $3 $4"
But I get a blank output. Any idea how to do this?
Many thanks!
If you use awk, you don't need an additional bash script. Also, it is a good idea to reduce the output of the ps command so you don't have to deal with extra information:
ps acxho user,pid,%cpu,cmd | awk '$3 > 5 {system("echo kill " $2)}'
Explanation
The extra ps flags I use:
c: command only, no extra arguments
h: no header, good for scripting
o: output format. In this case, only output the user, PID, %CPU, and command
The awk command compare the %CPU, which is the third column, with a threshold (5). If it is over the threshold, then issue the system command to kill that process.
Note the echo in the command. Once you are certain the scripts works the way you like, then remove the word echo from the command to execute it for real.
Your script needs to read its input
#!/bin/bash
while read a b c d; do
echo $a $b
done
I think you can get it using xargs command to pass the AWK output to your script as arguments:
ps aux|awk {'print $1,$2,$3,$11'} | xargs ./myscript
Some extra info about xargs: http://en.wikipedia.org/wiki/Xargs
When piping input from one process to another in Linux (or POSIX-compliant systems) the output is not given as arguments to the receiving process. Instead, the standard output of the first process is piped into the standard input of the other process.
Because of this, your script cannot work. $1...$n accesses variables that have been passed as arguments to it. As there are none it won't display anything. Instead, you have to read the standard input into variables with the read command (as pointed out by William).
The pipe '|' redirects the standard output of the left to the standard input of the right. In this case, the output of the ps goes to the input of awk, then the output of awk goes to the stdin of the script.
Therefore your scripts needs to read its STDIN.
#!/bin/bash
read var1 var2 var3 ...
Then you can do whatever you want with those variables.
More info, type in bash: help read
If I well understood your problem, you want to kill every process that exceeds X% of the CPU (using ps aux).
Here is the solution using AWK:
ps aux | grep -v "%CPU" | awk '{if ($3 > XXX) { print "Killing process with PID "$2", called "$4", consuming "$3"% and launched by "$1; system( "kill -9 " $2 );}}' -
Where XXX is your threshold (% of CPU).
It also prints related info to the killed process, if it is not desired just remove the print statement.
You can add some filters like: do not remove root's process...
Try putting myscript in front like this:
./myscript `ps aux|awk {'print $1,$2,$3,$11'}`
I have a series of functionally identical servers provided by my school that run various OS and hardware configurations. For the most part, I can use 5 of these interchangeably. Unfortunately, other students tend to bunch up on some machines and It's a pain to find one that isn't bogged down.
What I want to is ssh into a machine, run the command:
w | wc -l
to get a rough estimate of the load on that server, and use that information to select the least impacted one. A sort of client-side load balancer.
Is there a way to do this or achieve the same result?
I'd put this on your .bashrc file
function choose_host(){
hosts="host1 ... hostn"
for host in $hosts
do
echo $(ssh $host 'w|wc -l') $host
done | sort | head -1 | awk '{print $2}'
}
function ssh_host(){
ssh $(choose_host)
}
choose_host should give you the one you're looking for. This is absolutely overkill but i was feeling playful :D
sort will order the output according to the result of w|wc -l, then head -1 gets the first line and awk will just print the hostname !
You can call ssh_host and should log you automatically.
You can use pdsh command from your desktop which run the specified command on the set of machines you specified and return the results. This way you can find out the one which is least loaded. This will avoid you doing ssh to every single machine and run the w | wc -l.
Yes. See e.g.:
ssh me#host "ls /etc | sort" | wc -l
The part inside "" is done remotely. The part afterwards is local.
When I do
$ ps -ef | grep cron
I get
root 1036 1 0 Jul28 ? 00:00:00 cron
abc 21025 14334 0 19:15 pts/2 00:00:00 grep --color=auto cron
My question is why do I see the second line. From my understanding, ps lists the processes and pipes the list to grep. grep hasn't even started running while ps is listing processes, then how come grep process is listed in the o/p ?
Related second question:
When I do
$ ps -ef | grep [c]ron
I get only
root 1036 1 0 Jul28 ? 00:00:00 cron
What is the difference between first and second grep executions?
When you execute the command:
ps -ef | grep cron
the shell you are using
(...I assume bash in your case, due to the color attribute of grep I think you are running a gnu system like a linux distribution, but it's the same on other unix/shell as well...)
will execute the pipe() call to create a FIFO, then it will fork() (make a running copy of itself). This will create a new child process. This new generated child process will close() its standard output file descriptor (fd 1) and attach the fd 1 to the write side of the pipe created by the father process (the shell where you executed the command). This is possible because the fork() syscall will maintain, for each, a valid open file descriptor (the pipe fd in this case). After doing so it will exec() the first (in your case) ps command found in your PATH environment variable. With the exec() call the process will become the command you executed.
So, you now have the shell process with a child that is, in your case, the ps command with -ef attributes.
At this point, the parent (the shell) fork()s again. This newly generated child process close()s its standard input file descriptor (fd 0) and attaches the fd 0 to the read side of the pipe created by the father process (the shell where you executed the command).
After doing so it will exec() the first (in your case) grep command found in your PATH environment variable.
Now you have the shell process with two children (that are siblings) where the first one is the ps command with -ef attributes and the second one is the grep command with the cron attribute. The read side of the pipe is attached to the STDIN of the grep command and the write side is attached to the STDOUT of the ps command: the standard output of the ps command is attached to the standard input of the grep command.
Since ps is written to send on the standard output info on each running process, while grep is written to get on its standard input something that has to match a given pattern, you'll have the answer to your first question:
the shell runs: ps -ef;
the shell runs: grep cron;
ps sends data (that even contains the string "grep cron") to grep
grep matches its search pattern from the STDIN and it matches the string "grep cron" because of the "cron" attribute you passed in to grep: you are instructing grep to match the "cron" string and it does because "grep cron" is a string returned by ps at the time grep has started its execution.
When you execute:
ps -ef | grep '[c]ron'
the attribute passed instructs grep to match something containing "c" followed by "ron". Like the first example, but in this case it will break the match string returned by ps because:
the shell runs: ps -ef;
the shell runs: grep [c]ron;
ps sends data (that even contains the string grep [c]ron) to grep
grep does not match its search pattern from the stdin because a string containing "c" followed by "ron" it's not found, but it has found a string containing "c" followed by "]ron"
GNU grep does not have any string matching limit, and on some platforms (I think Solaris, HPUX, aix) the limit of the string is given by the "$COLUMN" variable or by the terminal's screen width.
Hopefully this long response clarifies the shell pipe process a bit.
TIP:
ps -ef | grep cron | grep -v grep
The shell constructs your pipeline with a series of fork(), pipe() and exec() calls. Depending on the shell any part of it may be constructed first. So grep may already be running before ps even starts. Or, even if ps starts first it will be writing into a 4k kernel pipe buffer and will eventually block (while printing a line of process output) until grep starts up and begins consuming the data in the pipe. In the latter case if ps is able to start and finish before grep even starts you may not see the grep cron in the output. You may have noticed this non-determinism at play already.
In your command
ps -ef | grep 'cron'
Linux is executing the "grep" command before the ps -ef command. Linux then maps the standard output (STDOUT) of "ps -ef" to the standard input (STDIN) of the grep command.
It does not execute the ps command, store the result in memory, and them pass it to grep. Think about that, why would it? Imagine if you were piping a hundred gigabytes of data?
Edit In regards to your second question:
In grep (and most regular expression engines), you can specify brackets to let it know that you'll accept ANY character in the brackets. So writing [c] means it will accept any charcter, but only c is specified. Similarly, you could do any other combination of characters.
ps aux | grep cron
root 1079 0.0 0.0 18976 1032 ? Ss Mar08 0:00 cron
root 23744 0.0 0.0 14564 900 pts/0 S+ 21:13 0:00 grep --color=auto cron
^ That matches itself, because your own command contains "cron"
ps aux | grep [c]ron
root 1079 0.0 0.0 18976 1032 ? Ss Mar08 0:00 cron
That matches cron, because cron contains a c, and then "ron". It does not match your request though, because your request is [c]ron
You can put whatever you want in the brackets, as long as it contains the c:
ps aux | grep [cbcdefadq]ron
root 1079 0.0 0.0 18976 1032 ? Ss Mar08 0:00 cron
If you remove the C, it won't match though, because "cron", starts with a c:
ps aux | grep [abedf]ron
^ Has no results
Edit 2
To reiterate the point, you can do all sorts of crazy stuff with grep. There's no significance in picking the first character to do this with.
ps aux | grep [c][ro][ro][n]
root 1079 0.0 0.0 18976 1032 ? Ss Mar08 0:00 cron
You wrote: "From my understanding, ps lists the processes and pipes the list to grep. grep hasn't even started running while ps is listing processes".
Your understanding is incorrect.
That is not how a pipeline works. The shell does not run the first command to completion, remember the output of the first command, and then afterwards run the next command using that data as input. No. Instead, both processes execute and their inputs/outputs are connected. As Ben Jackson wrote, there is nothing to particularly guarantee that the processes run at the same time, if they are both very short-lived, and if the kernel can comfortably manage the small amount of data passing through the connection. In that case, it really could happen the way you expect, only by chance. But the conceptual model to keep in mind is that they run in parallel.
If you want official sources, how about the bash man page:
A pipeline is a sequence of one or more commands separated by the character |. The format for a pipeline is:
[time [-p]] [ ! ] command [ | command2 ... ]
The standard output of command is connected via a pipe to the standard input of command2. This connection is
performed before any redirections specified by the command (see REDIRECTION below).
...
Each command in a pipeline is executed as a separate process (i.e., in a subshell).
As for your second question (which is not really related at all, I am sorry to say), you are just describing a feature of how regular expressions work. The regular expression cron matches the string cron. The regular expression [c]ron does not match the string [c]ron. Thus the first grep command will find itself in a process list, but the second one will not.
Your actual question has been answered by others, but I'll offer a tip: If you would like to avoid seeing the grep process listed, you can do it this way:
$ ps -ef | grep [c]ron
pgrep is sometimes better than ps -ef | grep word because it exclude the grep. Try
pgrep -f bash
pgrep -lf bash
$ ps -ef | grep cron
Linux Shell always execute command from right to left. so, before ps -ef execution grep cron already executed that's why o/p show's the command itself.
$ ps -ef | grep [c]ron
But in this u specified grep ron followed by only c. so, o/p is without command line because in command there is [c]ron.