I would like to run the following command via docker -c:
for line in `docker ps | awk '{print $1}' | grep -v CONTAINER`; do docker ps | grep $line | awk '{printf $NF" "}' && echo $(( `cat /sys/fs/cgroup/memory/docker/$line*/memory.usage_in_bytes` / 1024 / 1024 ))MB ; done
How do I have to escape the command when passing it via -c? What symbols do I have to escape?
I tried various versions, including this one, which gives me the error unexpected EOF while looking for matching "
bash -c 'for line in \`docker ps | awk \'{print $1}\' | grep -v CONTAINER\`\; do docker ps | grep $line | awk \'{printf $NF\\" \\"}\' && echo $(( \`cat /sys/fs/cgroup/memory/docker/$line*/memory.usage_in_bytes\` / 1024 / 1024 ))MB \; done
Sidenote: The command gives you the memory consumption for each docker container.
I am guessing you are trying to divide the number of bytes.
In order to do arithmetic calculations you need to use expr.
Try this:
for line in `docker ps | awk '{print $1}' | grep -v CONTAINER`; do docker ps | grep $line | awk '{printf $NF" "}' && echo $(expr $( cat /sys/fs/cgroup/memory/docker/$line*/memory.usage_in_bytes) / 1024 / 1024 )MB ; done
Found the solution:
bash -c 'for line in $(docker ps | awk '"'"'{print $1}'"'"' | grep -v CONTAINER); do docker ps | grep $line | awk '"'"'{printf $NF" "}'"'"' && echo $(( $(cat /sys/fs/cgroup/memory/docker/$line*/memory.usage_in_bytes) / 1024 / 1024 ))MB ; done'
Can someone explain it to me?
IIUC, it's a good use case for Bash printf %q specifier. Save a line to a
file, say /tmp/DOCKER-COMMAND and then format it with printf %q:
$ printf "%q\n" "$(< /tmp/DOCKER-COMMAND)"
for\ line\ in\ \`docker\ ps\ \|\ awk\ \'\{print\ \$1\}\'\ \|\ grep\ -v\ CONTAINER\`\;\ do\ docker\ ps\ \|\ grep\ \$line\ \|\ awk\ \'\{printf\ \$NF\"\ \"\}\'\ \&\&\ echo\ \$\(\(\ \`cat\ /sys/fs/cgroup/memory/docker/\$line\*/memory.usage_in_bytes\`\ /\ 1024\ /\ 1024\ \)\)MB\ \;\ done
You can copy the output line and paste it literally to bash -c:
$ bash -c for\ line\ in\ \`docker\ ps\ \|\ awk\ \'\{print\ \$1\}\'\ \|\ grep\ -v\ CONTAINER\`\;\ do\ docker\ ps\ \|\ grep\ \$line\ \|\ awk\ \'\{printf\ \$NF\"\ \"\}\'\ \&\&\ echo\ \$\(\(\ \`cat\ /sys/fs/cgroup/memory/docker/\$line\*/memory.usage_in_bytes\`\ /\ 1024\ /\ 1024\ \)\)MB\ \;\ done
fervent_gates 2MB
youthful_wozniak 3MB
Does it need to fit in a single line? Using a quoted heredoc gets you out of most quoting trouble:
bash <<'END_BASH'
docker ps | {
read header
while read -ra fields; do
mem=$(< /sys/fs/cgroup/memory/docker/"${fields[0]}"*/memory.usage_in_bytes)
printf "%s %dMB\n" "${fields[-1]}" $((mem / 1024 / 1024))
done
}
END_BASH
Related
I'd like to change my script(s) so that the command output is separated by a comma and is all on one line per host. Any ideas on how I can achieve this:
1st Script:
#!/bin/bash
for i in `cat ~/hostnames.txt`
do
ssh -q $i 'bash -s' < server_info.sh
done
2nd Script (server_info.sh):
#!/bin/bash
echo -n "Name: "
uname -n
echo -n "CPUs: "
cat /proc/cpuinfo* | grep processor | wc -l
echo -n "Memory (kb): "
cat /proc/meminfo | grep MemTotal: | awk '{print $2}'
echo -n "Current Kernel: "
uname -a | awk '{print $3}'
echo -n "IP: "
hostname -i
echo -e
Changing your 1st script:
#!/bin/bash
for i in cat ~/hostnames.txt
do
ssh -q $i 'bash -s' < server_info.sh
done | awk -v RS= '{$1=$1}1'
Note: Your server_info.sh can be a lot more optimized.For example:
cat /proc/meminfo | grep MemTotal: | awk '{print $2}'
could be changed to:
awk '/MemTotal:/{print $2}' /proc/meminfo
I'm monitoring from an actively written to file:
My current solution is:
ws_trans=0
sc_trans=0
tail -F /var/log/file.log | \
while read LINE
echo $LINE | grep -q -e "enterpriseID:"
if [ $? = 0 ]
then
((ws_trans++))
fi
echo $LINE | grep -q -e "sc_ID:"
if [ $? = 0 ]
then
((sc_trans++))
fi
printf "\r WSTRANS: $ws_trans \t\t SCTRANS: $sc_trans"
done
However when attempting to do this with AWK I don't get the output - the $ws_trans and $sc_trans remains 0
ws_trans=0
sc_trans=0
tail -F /var/log/file.log | \
while read LINE
echo $LINE | awk '/enterpriseID:/ {++ws_trans} END {print | ws_trans}'
echo $LINE | awk '/sc_ID:/ {++sc_trans} END {print | sc_trans}'
printf "\r WSTRANS: $ws_trans \t\t SCTRANS: $sc_trans"
done
Attempting to do this to reduce load. I understand that AWK doesn't deal with bash variables, and it can get quite confusing, but the only reference I found is a non tail application of AWK.
How can I assign the AWK Variable to the bash ws_trans and sc_trans? Is there a better solution? (There are other search terms being monitored.)
You need to pass the variables using the option -v, for example:
$ var=0
$ printf %d\\n {1..10} | awk -v awk_var=${var} '{++awk_var} {print awk_var}'
To set the variable "back" you could use declare, for example:
$ declare $(printf %d\\n {1..10} | awk -v awk_var=${var} '{++awk_var} END {print "var=" awk_var}')
$ echo $var
$ 10
Your script could be rewritten like this:
ws_trans=0
sc_trans=0
tail -F /var/log/system.log |
while read LINE
do
declare $(echo $LINE | awk -v ws=${ws_trans} '/enterpriseID:/ {++ws} END {print "ws_trans="ws}')
declare $(echo $LINE | awk -v sc=${sc_trans} '/sc_ID:/ {++sc} END {print "sc_trans="sc}')
printf "\r WSTRANS: $ws_trans \t\t SCTRANS: $sc_trans"
done
I'm using awk to parse /etc/hosts and produce a command which will format MapR for me. It's being done in a bash utility in Chef:
egrep '^[0-9]' /etc/hosts | grep -v 127.0.0.1 \
| awk 'NR==1{ips=$1}
NR>1{ips=ips ", " $1}
$2=="namenode"{nn=$1}
END{ printf "/opt/mapr/server/configure.sh -C %s -Z %s -N mycluster --create-user -D /dev/xvdb\n", ips, nn}' \
| bash
sleep 60
The command above should execute the following command:
/opt/mapr/server/configure.sh -C 10.32.237.251 -Z 10.32.237.251 -N mycluster --create-user -D /dev/xvdb
However, looking into my chef output I see:
==> namenode: Executing awk utility
==> namenode: awk: line 1: runaway string constant "/opt/mapr/ ...
The command never got executed in the MapR node... However when i execute it directly on the terminal it works nicely in the way it's supposed to be. What am I doing wrong?
I'm updating the question to show the complete bash script that executes that utility:
DISK_CONFIG=/home/ubuntu/disk_config
if [ -f $DISK_CONFIG ];
then
echo "File already exists"
else
echo "Executing awk utility\n"
touch $DISK_CONFIG
egrep '^[0-9]' /etc/hosts | grep -v 127.0.0.1 \
| awk 'NR==1{ips=$1}
NR>1{ips=ips ", " $1}
$2=="namenode"{nn=$1}
END{ printf "/opt/mapr/server/configure.sh -C %s -Z %s -N mycluster --create-user -D /dev/xvdb\n", ips, nn}' \
| bash
sleep 60
fi
Assuming you're using HEREDOC syntax in your bash resource:
bash "whatever" do
code <<-EOH
DISK_CONFIG=/tmp/disk_config
if [ -f $DISK_CONFIG ];
then
echo "File already exists"
else
echo "Executing awk utility\n"
touch $DISK_CONFIG
egrep '^[0-9]' /etc/hosts | grep -v 127.0.0.1 \
| awk 'NR==1{ips=$1}
NR>1{ips=ips ", " $1}
$2=="namenode"{nn=$1}
END{ printf "/opt/mapr/server/configure.sh -C %s -Z %s -N mycluster --create-user -D /dev/xvdb\n", ips, nn}' \
| bash
fi
EOH
end
this one leads to your error:
Executing awk utility
awk: line 4: runaway string constant "/opt/mapr/ ...
This is due to the \n in your comand (the one into the awk command is likely to be problematic too)
This resource should do (warning I did replace the DISK_CONFIG path for my tests):
bash "whatever" do
code <<-EOH
DISK_CONFIG=/tmp/disk_config
if [ -f $DISK_CONFIG ];
then
echo "File already exists"
else
echo "Executing awk utility"
touch $DISK_CONFIG
egrep '^[0-9]' /etc/hosts | grep -v 127.0.0.1 \
| awk 'NR==1{ips=$1}
NR>1{ips=ips ", " $1}
$2=="namenode"{nn=$1}
END{ printf "/opt/mapr/server/configure.sh -C %s -Z %s -N mycluster --create-user -D /dev/xvdb", ips, nn}' \
| bash
sleep 60
fi
EOH
end
The reason is that Chef already interpret the \n in the code and so awk see a string never ending (runaway).
As you pipe to bash you can omit the \n as the pipe will end the line.
I get the following error:
> echo "${$(qstat -a | grep kig):0:7}"
-bash: ${$(qstat -a | grep kig):0:7}: bad substitution
I'm trying to take the number before. of
> qstat -a | grep kig
1192530.perceus- kigumen lr_regul pbs.sh 27198 2 16 -- 24:00:00 R 00:32:23
and use it as an argument to qdel in openPBS so that I can delete all process that I started with my login kigumen
so ideally, this should work:
qdel ${$(qstat -a | grep kig):0:7}
so far, only this works:
str=$(qstat -a | grep kig); qdel "${str:0:7}"
but I want a clean one-liner without a temporary variable.
The shell substring construct you're using (:0:7) only works on variables, not command substitution. If you want to do this in a single operation, you'll need to trim the string as part of the pipeline, something like one of these:
echo "$(qstat -a | grep kig | sed 's/[.].*//')"
echo "$(qstat -a | awk -F. '/kig/ {print $1}')"
echo "$(qstat -a | awk '/kig/ {print substr($0, 1, 7)}')"
(Note that the first two print everything before the first ".", while the last prints the first 7 characters.) I don't know that any of them are particularly cleaner, but they do it without a temp variable...
qstat -u palle | cut -f 1 -d "." | xargs qdel
Kills all my jobs... normally I grep out the jobname(s) before cut'ing...
So I use a small script "idlist":
qstat -u palle | grep -E "*.in" | grep -E "$1" | cut -f 1 -d "." | xargs
To see all my "map_..." jobs:
idlist "map_*"
For killing all my "map_...." jobs:
idlist "map_*" | xargs qdel
yet another ways :
foreach m1 in $(qstat -a );do
if [[ $m1 =~ kig ]];then
m2=${m1%.kig}
echo "kig found $m2 "
break
fi
done
For example, if I execute
ps aux | awk '{print $1}' | xargs -I {} echo {}
I want to let the shell sleep for 1 second between each echo.
How can I change my shell command?
You can use the following syntax:
ps aux | awk '{print $1}' | xargs -I % sh -c '{ echo %; sleep 1; }'
Be careful with spaces and semicolons though. After every command in between brackets, semicolon is required (even after the last one).
Replace echo by some shell script named sleepecho containing
#!/bin/sh
sleep 1
echo $*
If your awk supports it:
ps aux | awk '{ system("sleep 1"); print $1 }' | xargs -I {} echo {}q
or skip awk and xargs altogether
ps aux | while read -r user rest;
echo $user
sleep 1;
done