I'm developping a script to search for patterns within scripts executed from CRON on a bunch of remote servers through SSH.
Script on client machine -- SSH --> Remote Servers CRON/Scripts
For now I can't get the correct output.
Script on client machine
#!/bin/bash
server_list=( '172.x.x.x' '172.x.x.y' '172.x.x.z' )
for s in ${server_list[#]}; do
ssh -i /home/user/.ssh/my_key.rsa user#${s} crontab -l | grep -v '^#\|^[[:space:]]*$' | cut -d ' ' -f 6- | awk '{print $1}' | grep -v '^$\|^echo\|^find\|^PATH\|^/usr/bin\|^/bin/' | xargs -0 grep -in 'server.tld\|10.x.x.x'
done
This only gives me the paths of scripts from crontab, not the matched lines and line number plus the first line is prefixed with "grep:" keyword (example below):
grep: /opt/directory/script1.sh
/opt/directory/script2.sh
/opt/directory/script3.sh
/opt/directory/script4.sh
How to get proper output, meaning the script path plus line number plus line of matching pattern?
Remote CRON examples
OO 6 * * * /opt/directory/script1.sh foo
30 6 * * * /opt/directory/script2.sh bar
Remote script content examples
1 ) This will match grep pattern
#!/bin/bash
ping -c 4 server.tld && echo "server.tld ($1)"
2 ) This won't match grep pattern
#!/bin/bash
ping -c 4 8.x.x.x && echo "8.x.x.x ($1)"
Without example input, it's really hard to see what your script is attempting to do. But the cron parsing could almost certainly be simplified tremendously by refactoring all of it into a single Awk script. Here is a quick stab, with obviously no way to test.
#!/bin/sh
# No longer using an array for no good reason, so /bin/sh will work
for s in 172.x.x.x 172.x.x.y 172.x.x.z; do
ssh -i /home/user/.ssh/my_key.rsa "user#${s}" crontab -l |
awk '! /^#|^[[:space:]]*$/ && $6 !~ /^$|^(echo|find|PATH|\/usr\/bin|\/bin\/)/ { print $6 }' |
# no -0; use grep -E and properly quote literal dot
xargs grep -Ein 'server\.tld|10.x.x.x'
done
Your command would not output null-delimited data to xargs so probably the immediate problem was that xargs -0 would receive all the file names as a single file name which obviously does not exist, and you forgot to include the ": file not found" from the end of the error message.
The use of grep -E is a minor hack to enable a more modern regex syntax which is more similar to that in Awk, where you don't have to backslash the "or" pipe etc.
This script, like your original, runs grep on the local system where you run the SSH script. If you want to run the commands on the remote server, you will need to refactor to put the entire pipeline in single quotes or a here document:
for s in 172.x.x.x 172.x.x.y 172.x.x.z; do
ssh -i /home/user/.ssh/my_key.rsa "user#${s}" <<\________HERE
crontab -l |
awk '! /^#|^[[:space:]]*$/ && $6 !~ /^$|^(echo|find|PATH|\/usr\/bin|\/bin\/)/ { print $6 }' |
xargs grep -Ein 'server\.tld|10.x.x.x'
________HERE
done
The refactored script contains enough complexities in the quoting that you probably don't want to pass it as an argument to ssh, which requires you to figure out how to quote strings both locally and remotely. It's easier then to pass it as standard input, which obviously just gets transmitted verbatim.
If you get "Pseudo-terminal will not be allocated because stdin is not a terminal.", try using ssh -t. Sometimes you need to add multiple -t options to completely get rid of this message.
I was trying to extract a process size using the command:
size=`ps -eo vsz,pid | grep $pid | cut -'d' -f1`
However, this appeared to only work on some computers but not all. So on the ones where it wasn't working, I tried:
size=`ps -eo vsz,pid | grep $pid | awk '{print $1}'`
however, now this didn't work on the computers where the first command worked.
What I mean by "working" and "not working" is that sometimes:
echo "|$size|"
Will return something like:
|8762348
9835|
And thus the following returns an arithmetic error:
(( $size > $threshold ))
because of the newline or carriage return characters stored in $size. Is there a way to reliable extract simply the first field across different versions of linux?
First you ask ps to displat info for all processes, next you try to select 1 of them.
Your command had problems on some computers, and you tagged the question with both ksh and linux, so I am not sure what command can be used best in your case:
size=$(ps -q ${pid} -o vsz --no-headers)
# or
size=$(ps -p ${pid} -o vsz | grep -v "VSZ")
The following command below does not succeed.
for i in {1..5} ; do cat /etc/fstab | egrep "(ext3|ext4|xfs)" | awk '{print $2}' | cut -d"/" -f1-$i ; done
It seems that $i is ignored completely. It always returns instead result of
cut -d"/" -f1-
Any idea why it fails?
Thanks in advance!
The command itself is a part of a script that should help me to auto re-arrange fstab lines to match the right mount order (like /test/subfolder must come after /test was mounted and not before).
I tried and it didn't work for zsh shell. BUT I tried it in bash and it does work, so if you are using zsh just run the command with bash and it should work ;)
I am new to bash and writing a script that needs to compare the minor version of the kernel to see if it is greater than or equal to 10, and exit if it is not. Currently I have something like this:
KERNEL=$(uname -r)
declare -i MINOR_VERSION=$(echo $KERNEL | cut -c 3-4)
if [ "10" -gt "$MINOR_VERSION"]; then exit 0; fi
This is bad code, and doesn't work if the minor version is < 10 since I am using cut and depending on it being two digits. What I probably need is something that parses the minor version by using the dots.
Example:
$ uname -r
3.4.0-60-generic
$ MNR_VAR=<awesome bash code, with cut or sed or something>
$ echo $MNR_VAR
4
I have been reading cut and sed documentation but have just been slow picking it up. I would appreciate the help!
TL;DR - looking for a bash command that will extract an int surrounded by the first two dots in a variable. "3.13.0.x" returns '13', "3.2.0.x" returns '2', etc.
EDIT:
Some answers as one liners below for those curious.
uname -r | cut -d '.' -f2
uname -r | awk -F . '{print $2}'
kernel="$(uname -r)" | tmp="${kernel#*.}" | minor="${tmp%%.*}" | echo "$minor"
In pure bash:
#!/bin/bash
ker="$(uname -r)"
minker="${ker#*.}"
minker="${minker%%.*}"
echo "$minker"
"${ker#*.}" is the string after the first match of a . in $ker. Thus
$minker becomes 13.0-generic... from 3.13.0-generic...
"${minker%%.*}" is the string left by cutting all matches (from right) of . and whatever after it, in $minker. Thus $minker becomes 13 from 13.0-generic...
See the Bash Parameter Expansion Manual for more info
Using Bash Regex:
#!/bin/bash
regex='([0-9]+)\.([0-9]+)'
[[ $(uname -r) =~ $regex ]]
echo ${BASH_REMATCH[2]}
The problem is you are using -c to cut. Don't do that.
Use the -f and -d flags instead to control the delimiter and fields to output.
Or use awk -F . '{print $2}' <<< "$(uname -r)".
Or use IFS=. read -r _ minor _rest <<< "$(uname -r)"; echo "$minor" (which has the benefit of not using any external utilities).
The usage of <<< "$(uname -r)" is bash-specific (I believe) but avoids the need for a pipe (|) and the sub-shell that it involves.
Extracting just minor version & comparing it with something is risky, because major number can change too...
I normally prefer padding the numbers with zeros, so that they can be easily compared using simple string compare.
kernel_version=$(uname -r | sed -r 's/([0-9]+)/0000\1/g; s/0*([0-9]{4})/\1/g') # gives 0003.0004.0000-0060-generic
if [[ "$kernel_version" < "0003.0010" ]]; then exit 0; fi
I want to do this:
run a command
capture the output
select a line
select a column of that line
Just as an example, let's say I want to get the command name from a $PID (please note this is just an example, I'm not suggesting this is the easiest way to get a command name from a process id - my real problem is with another command whose output format I can't control).
If I run ps I get:
PID TTY TIME CMD
11383 pts/1 00:00:00 bash
11771 pts/1 00:00:00 ps
Now I do ps | egrep 11383 and get
11383 pts/1 00:00:00 bash
Next step: ps | egrep 11383 | cut -d" " -f 4. Output is:
<absolutely nothing/>
The problem is that cut cuts the output by single spaces, and as ps adds some spaces between the 2nd and 3rd columns to keep some resemblance of a table, cut picks an empty string. Of course, I could use cut to select the 7th and not the 4th field, but how can I know, specially when the output is variable and unknown on beforehand.
One easy way is to add a pass of tr to squeeze any repeated field separators out:
$ ps | egrep 11383 | tr -s ' ' | cut -d ' ' -f 4
I think the simplest way is to use awk. Example:
$ echo "11383 pts/1 00:00:00 bash" | awk '{ print $4; }'
bash
Please note that the tr -s ' ' option will not remove any single leading spaces. If your column is right-aligned (as with ps pid)...
$ ps h -o pid,user -C ssh,sshd | tr -s " "
1543 root
19645 root
19731 root
Then cutting will result in a blank line for some of those fields if it is the first column:
$ <previous command> | cut -d ' ' -f1
19645
19731
Unless you precede it with a space, obviously
$ <command> | sed -e "s/.*/ &/" | tr -s " "
Now, for this particular case of pid numbers (not names), there is a function called pgrep:
$ pgrep ssh
Shell functions
However, in general it is actually still possible to use shell functions in a concise manner, because there is a neat thing about the read command:
$ <command> | while read a b; do echo $a; done
The first parameter to read, a, selects the first column, and if there is more, everything else will be put in b. As a result, you never need more variables than the number of your column +1.
So,
while read a b c d; do echo $c; done
will then output the 3rd column. As indicated in my comment...
A piped read will be executed in an environment that does not pass variables to the calling script.
out=$(ps whatever | { read a b c d; echo $c; })
arr=($(ps whatever | { read a b c d; echo $c $b; }))
echo ${arr[1]} # will output 'b'`
The Array Solution
So we then end up with the answer by #frayser which is to use the shell variable IFS which defaults to a space, to split the string into an array. It only works in Bash though. Dash and Ash do not support it. I have had a really hard time splitting a string into components in a Busybox thing. It is easy enough to get a single component (e.g. using awk) and then to repeat that for every parameter you need. But then you end up repeatedly calling awk on the same line, or repeatedly using a read block with echo on the same line. Which is not efficient or pretty. So you end up splitting using ${name%% *} and so on. Makes you yearn for some Python skills because in fact shell scripting is not a lot of fun anymore if half or more of the features you are accustomed to, are gone. But you can assume that even python would not be installed on such a system, and it wasn't ;-).
try
ps |&
while read -p first second third fourth etc ; do
if [[ $first == '11383' ]]
then
echo got: $fourth
fi
done
Your command
ps | egrep 11383 | cut -d" " -f 4
misses a tr -s to squeeze spaces, as unwind explains in his answer.
However, you maybe want to use awk, since it handles all of these actions in a single command:
ps | awk '/11383/ {print $4}'
This prints the 4th column in those lines containing 11383. If you want this to match 11383 if it appears in the beginning of the line, then you can say ps | awk '/^11383/ {print $4}'.
Using array variables
set $(ps | egrep "^11383 "); echo $4
or
A=( $(ps | egrep "^11383 ") ) ; echo ${A[3]}
Similar to brianegge's awk solution, here is the Perl equivalent:
ps | egrep 11383 | perl -lane 'print $F[3]'
-a enables autosplit mode, which populates the #F array with the column data.
Use -F, if your data is comma-delimited, rather than space-delimited.
Field 3 is printed since Perl starts counting from 0 rather than 1
Getting the correct line (example for line no. 6) is done with head and tail and the correct word (word no. 4) can be captured with awk:
command|head -n 6|tail -n 1|awk '{print $4}'
Instead of doing all these greps and stuff, I'd advise you to use ps capabilities of changing output format.
ps -o cmd= -p 12345
You get the cmmand line of a process with the pid specified and nothing else.
This is POSIX-conformant and may be thus considered portable.
Bash's set will parse all output into position parameters.
For instance, with set $(free -h) command, echo $7 will show "Mem:"