producing a bash command with awk giving runaway string constant - awk - linux

I'm using awk to parse /etc/hosts and produce a command which will format MapR for me. It's being done in a bash utility in Chef:
egrep '^[0-9]' /etc/hosts | grep -v 127.0.0.1 \
| awk 'NR==1{ips=$1}
NR>1{ips=ips ", " $1}
$2=="namenode"{nn=$1}
END{ printf "/opt/mapr/server/configure.sh -C %s -Z %s -N mycluster --create-user -D /dev/xvdb\n", ips, nn}' \
| bash
sleep 60
The command above should execute the following command:
/opt/mapr/server/configure.sh -C 10.32.237.251 -Z 10.32.237.251 -N mycluster --create-user -D /dev/xvdb
However, looking into my chef output I see:
==> namenode: Executing awk utility
==> namenode: awk: line 1: runaway string constant "/opt/mapr/ ...
The command never got executed in the MapR node... However when i execute it directly on the terminal it works nicely in the way it's supposed to be. What am I doing wrong?
I'm updating the question to show the complete bash script that executes that utility:
DISK_CONFIG=/home/ubuntu/disk_config
if [ -f $DISK_CONFIG ];
then
echo "File already exists"
else
echo "Executing awk utility\n"
touch $DISK_CONFIG
egrep '^[0-9]' /etc/hosts | grep -v 127.0.0.1 \
| awk 'NR==1{ips=$1}
NR>1{ips=ips ", " $1}
$2=="namenode"{nn=$1}
END{ printf "/opt/mapr/server/configure.sh -C %s -Z %s -N mycluster --create-user -D /dev/xvdb\n", ips, nn}' \
| bash
sleep 60
fi

Assuming you're using HEREDOC syntax in your bash resource:
bash "whatever" do
code <<-EOH
DISK_CONFIG=/tmp/disk_config
if [ -f $DISK_CONFIG ];
then
echo "File already exists"
else
echo "Executing awk utility\n"
touch $DISK_CONFIG
egrep '^[0-9]' /etc/hosts | grep -v 127.0.0.1 \
| awk 'NR==1{ips=$1}
NR>1{ips=ips ", " $1}
$2=="namenode"{nn=$1}
END{ printf "/opt/mapr/server/configure.sh -C %s -Z %s -N mycluster --create-user -D /dev/xvdb\n", ips, nn}' \
| bash
fi
EOH
end
this one leads to your error:
Executing awk utility
awk: line 4: runaway string constant "/opt/mapr/ ...
This is due to the \n in your comand (the one into the awk command is likely to be problematic too)
This resource should do (warning I did replace the DISK_CONFIG path for my tests):
bash "whatever" do
code <<-EOH
DISK_CONFIG=/tmp/disk_config
if [ -f $DISK_CONFIG ];
then
echo "File already exists"
else
echo "Executing awk utility"
touch $DISK_CONFIG
egrep '^[0-9]' /etc/hosts | grep -v 127.0.0.1 \
| awk 'NR==1{ips=$1}
NR>1{ips=ips ", " $1}
$2=="namenode"{nn=$1}
END{ printf "/opt/mapr/server/configure.sh -C %s -Z %s -N mycluster --create-user -D /dev/xvdb", ips, nn}' \
| bash
sleep 60
fi
EOH
end
The reason is that Chef already interpret the \n in the code and so awk see a string never ending (runaway).
As you pipe to bash you can omit the \n as the pipe will end the line.

Related

escaping special character while executing commands remotely using ssh

When ran commands locally on the remote server outputs would work as expected:
desired_kernel_version="5.4.0-105-generic"
cat /tmp/grb.bkp | grep GRUB_DEFAULT
GRUB_DEFAULT=0
kernel_position=$(awk -F\' '$1=="menuentry " || $1=="submenu " {print i++ " : " $2}; /\tmenuentry / {print "\t" i-1">"j++ " : " $2};' /boot/grub/grub.cfg | grep "${desired_kernel_version}" | grep -v recovery | awk '{ print $1}' | sed 's/ //g')
echo $k_position
1>2
sed -i "s/GRUB_DEFAULT=0/GRUB_DEFAULT=\"${k_position}\"/g" /tmp/grb.bkp
cat /tmp/grb.bkp | grep GRUB_DEFAULT
GRUB_DEFAULT="1>2"
desired output when ran from remote server:
replace 0 of GRUB_DEFAULT value to kernel_position within quotes.
server=abcd
kernel_position=$(ssh -qT $server awk -F\' '$1=="menuentry " || $1=="submenu " {print i++ " : " $2}; /\tmenuentry / {print "\t" i-1">"j++ " : " $2};' /boot/grub/grub.cfg | grep "${desired_kernel_version}" | grep -v recovery | awk '{ print $1}' | sed 's/ //g')
ssh -qT $server "sed -i "s/GRUB_DEFAULT=0/GRUB_DEFAULT=\"${k_position}\"/g" /tmp/grb.bkp"
Suggesting to avoid quoting hell.
Send muli-line command into ssh by writing a script remote-script.sh with all lines.
remote-script.sh
#!/bin/bash
source ~/.bash_profile
$k_position=$1
desired_kernel_version="5.4.0-105-generic"
cat /tmp/grb.bkp | grep GRUB_DEFAULT
GRUB_DEFAULT=0
kernel_position=$(awk -F\' '$1=="menuentry " || $1=="submenu " {print i++ " : " $2}; /\tmenuentry / {print "\t" i-1">"j++ " : " $2};' /boot/grub/grub.cfg | grep "${desired_kernel_version}" | grep -v recovery | awk '{ print $1}' | sed 's/ //g')
echo $k_position
1>2
sed -i "s/GRUB_DEFAULT=0/GRUB_DEFAULT=\"${k_position}\"/g" /tmp/grb.bkp
cat /tmp/grb.bkp | grep GRUB_DEFAULT
GRUB_DEFAULT="1>2"
Give current user execution permissions on remote-script.sh
chmod u+x remote-script.sh
Use scp command to copy remote-script.sh to $server. If possible only once at deploy time.
scp -q remote-script.sh $server:/home/your-user
Use ssh command to run remote-script.sh on remote server. Pass $k_position in command line.
ssh -qT $server "bash -c /home/your-user/remote-script.sh $k_position"
BTW, in computing kernel_position, suggesting to fold all awk, grep, sed commands into a single awk script.

Recover network info for each network card in a for loop and do something with it with bash

I created a script that recovers info from a remote VM and generates a second script that creates a new ifcfg-eth* and replaces it in a local copy of the same VM we are migrating (basically the remote VM is in DHCP and we are migrating them in a new location with a static IP and info that we recover from the original VM):
#! /bin/bash
# Version 1.0
# Last review 27/01/2022
# Author Alessandro
# Define variables
RED='\033[0;31m'
NC='\033[0m' # No Color
NCC="\e[0m"
CYAN='\e[96m'
USER='user'
CWD=$(pwd)
# Checks if the IP exists after the script name
if [ -z "$1" ]; then echo -e "${RED}Where is the IP?${NC}" && exit 1; fi
echo -e "${CYAN}Starting...${NCC}"
# Stores the output of of 'script' in the variable $RAW_INPUT
RAW_INPUT=$(ssh -o StrictHostKeyChecking=no $USER#"$1" <<'SCRIPT'
# Script
HOSTNAMEVM=$(hostname -s)
GW=$(/sbin/ip route | awk '/default/ { print $3 }')
INTERFACE1=$(ip link | awk -F: ' $0 !~"lo|vir|wl|^[^1-2]" {print $2;getline}' | awk '{ gsub (" ", "", $0); print}')
IP1=$(ifconfig "${INTERFACE1}" 2>/dev/null|awk '/inet addr:/ {print $2}'|sed 's/addr://')
SUBNET1=$(/sbin/ifconfig "${INTERFACE1}" | grep Mask | cut -d":" -f4)
INTERFACE2=$(ip link | awk -F: ' $0 !~"lo|vir|wl|^[^3-4]" {print $2;getline}' | awk '{ gsub (" ", "", $0); print}')
IP2=$(ifconfig "${INTERFACE2}" 2>/dev/null|awk '/inet addr:/ {print $2}'|sed 's/addr://')
SUBNET2=$(/sbin/ifconfig "${INTERFACE2}" | grep Mask | cut -d":" -f4)
echo "Hostname:${HOSTNAMEVM}"
echo "GW:${GW}"
echo "Network-card1:${INTERFACE1}"
echo "${INTERFACE1}_IP1:${IP1}"
echo "SUBNET_1:${SUBNET1}"
echo "Network-card2:${INTERFACE2}"
echo "${INTERFACE2}_IP2:${IP2}"
echo "SUBNET_2:${SUBNET2}"
echo "ENDECA:${ENDECA}"
SCRIPT
)
# Generate the variables for the pre_failover_script parsing the RAW_INPUT variable
VM_NAME=$( echo "${RAW_INPUT}" | awk -F"Hostname:" '/Hostname:/{print $2}')
# Verifies if the executed pre failover script already exists
if [ -f "${CWD}"/executed_pre_"${VM_NAME}".sh ]; then echo -e "${RED}WARNING${NC}:The pre failover script for${NC} ${CYAN}${VM_NAME}${NC} has been already executed!" && exit 1; fi
JOB_ID=$("${CWD}"/list_jobs.sh | grep "${VM_NAME}" | awk -F"mnt/" '/mnt/{print $2}')
# Checks if a job exists for the vm name
if [ -z "${JOB_ID}" ]; then
echo -e "${RED}The job ID for${NC} ${VM_NAME} ${RED}doesn't exist.${NC}";
exit 1;
fi
IP_GATEWAY=$(echo "${RAW_INPUT}" | awk -F"GW:" '/GW:/{print $2}')
INTERFACE_1=$( echo "${RAW_INPUT}" | awk -F"card1:" '/card1:/{print $2}')
IP_1=$( echo "${RAW_INPUT}" | awk -F"IP1:" '/IP1:/{print $2}')
SUB_1=$( echo "${RAW_INPUT}" | awk -F"NET_1:" '/NET_1:/{print $2}')
INTERFACE_2=$( echo "${RAW_INPUT}" | awk -F"card2:" '/card2:/{print $2}')
IP_2=$( echo "${RAW_INPUT}" | awk -F"IP2:" '/IP2:/{print $2}')
SUB_2=$( echo "${RAW_INPUT}" | awk -F"NET_2:" '/NET_2:/{print $2}')
# Prints the value of ENDECA in case the Symbolic Link has been removed
ENDECA=$( echo "${RAW_INPUT}" | awk -F"ENDECA:" '/ENDECA:/{print $2}')
# Removing existing pre_failover_script file for the ECL2 instance in current directory
rm -rf "${CWD}"/pre_failover_script_"${VM_NAME}".sh
# Define pre_failover_script variable
PRE_FAILOVER_SCRIPT_LOCATION="${CWD}"/pre_failover_script_${VM_NAME}.sh
# Generate ifcfg-eth* file based on information gathered above
echo "#! /bin/bash
PATH_TO_JOB_ID=/opt/dbtk/mnt/${JOB_ID}
PATH_TO_UDEV_NET_RULES_FILES=/etc/udev/rules.d
PATH_TO_GATEWAY_FILE=/etc/sysconfig/network
CLOUD_INIT_LOCAL_STARTUP_SCRIPT=/etc/rc3.d/S50cloud-init-local
CLOUD_INIT_STARTUP_SCRIPT=/etc/rc3.d/S51cloud-init
CLOUD_CONFIG_STARTUP_SCRIPT=/etc/rc3.d/S52cloud-config
CLOUD_FINAL_STARTUP_SCRIPT=/etc/rc3.d/S53cloud-final
PATH_TO_IFCFG_FILES=/etc/sysconfig/network-scripts
rm -f \$PATH_TO_JOB_ID\$PATH_TO_UDEV_NET_RULES_FILES/70-persistent-net.rules*
echo \"GATEWAY=${IP_GATEWAY}\" >> \$PATH_TO_JOB_ID\$PATH_TO_GATEWAY_FILE
rm -f \$PATH_TO_JOB_ID\$CLOUD_INIT_LOCAL_STARTUP_SCRIPT
rm -f \$PATH_TO_JOB_ID\$CLOUD_INIT_STARTUP_SCRIPT
rm -f \$PATH_TO_JOB_ID\$CLOUD_CONFIG_STARTUP_SCRIPT
rm -f \$PATH_TO_JOB_ID\$CLOUD_FINAL_STARTUP_SCRIPT" > "${PRE_FAILOVER_SCRIPT_LOCATION}"
echo "sed '/^BOOTPROTO/d' \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1} > \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1}.new
{
echo 'NAME=${INTERFACE_1}'
echo 'IPADDR=${IP_1}'
echo 'NETMASK=${SUB_1}'
echo 'BOOTPROTO=static'
} >> \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1}.new
cp \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1} \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1}.ori
mv \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1}.new \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1}
rm \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1}.ori" >> "${PRE_FAILOVER_SCRIPT_LOCATION}"
echo "sed '/^BOOTPROTO/d' \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_2} > \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_2}.new
{
echo 'NAME=${INTERFACE_2}'
echo 'IPADDR=${IP_2}'
echo 'NETMASK=${SUB_2}'
echo 'BOOTPROTO=static'
} >> \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_2}.new
cp \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_2} \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_2}.ori
mv \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_2}.new \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_2}
rm \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_2}.ori" >> "${PRE_FAILOVER_SCRIPT_LOCATION}"
The script works perfectly, problem is that now I'm having multiple VMs with 3 or 1 NIC instead of 2 and I was wondering if there was a way to make a for loop that does the same thing but for each Network card. As you can see now I'm generating 'manually' the ifcfg-eth for each NC based on the info collected before:
{
echo 'NAME=${INTERFACE_1}'
echo 'IPADDR=${IP_1}'
echo 'NETMASK=${SUB_1}'
echo 'BOOTPROTO=static'
} >> \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1}.new
cp \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1} \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1}.ori
mv \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1}.new \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1}
rm \$PATH_TO_JOB_ID\$PATH_TO_IFCFG_FILES/ifcfg-${INTERFACE_1}.ori" >> "${PRE_FAILOVER_SCRIPT_LOCATION}"```
Now it's no longer convenient but I'm not sure if what I'm trying to do is achievable.

How can I escape a command passed to bash via bash -c?

I would like to run the following command via docker -c:
for line in `docker ps | awk '{print $1}' | grep -v CONTAINER`; do docker ps | grep $line | awk '{printf $NF" "}' && echo $(( `cat /sys/fs/cgroup/memory/docker/$line*/memory.usage_in_bytes` / 1024 / 1024 ))MB ; done
How do I have to escape the command when passing it via -c? What symbols do I have to escape?
I tried various versions, including this one, which gives me the error unexpected EOF while looking for matching "
bash -c 'for line in \`docker ps | awk \'{print $1}\' | grep -v CONTAINER\`\; do docker ps | grep $line | awk \'{printf $NF\\" \\"}\' && echo $(( \`cat /sys/fs/cgroup/memory/docker/$line*/memory.usage_in_bytes\` / 1024 / 1024 ))MB \; done
Sidenote: The command gives you the memory consumption for each docker container.
I am guessing you are trying to divide the number of bytes.
In order to do arithmetic calculations you need to use expr.
Try this:
for line in `docker ps | awk '{print $1}' | grep -v CONTAINER`; do docker ps | grep $line | awk '{printf $NF" "}' && echo $(expr $( cat /sys/fs/cgroup/memory/docker/$line*/memory.usage_in_bytes) / 1024 / 1024 )MB ; done
Found the solution:
bash -c 'for line in $(docker ps | awk '"'"'{print $1}'"'"' | grep -v CONTAINER); do docker ps | grep $line | awk '"'"'{printf $NF" "}'"'"' && echo $(( $(cat /sys/fs/cgroup/memory/docker/$line*/memory.usage_in_bytes) / 1024 / 1024 ))MB ; done'
Can someone explain it to me?
IIUC, it's a good use case for Bash printf %q specifier. Save a line to a
file, say /tmp/DOCKER-COMMAND and then format it with printf %q:
$ printf "%q\n" "$(< /tmp/DOCKER-COMMAND)"
for\ line\ in\ \`docker\ ps\ \|\ awk\ \'\{print\ \$1\}\'\ \|\ grep\ -v\ CONTAINER\`\;\ do\ docker\ ps\ \|\ grep\ \$line\ \|\ awk\ \'\{printf\ \$NF\"\ \"\}\'\ \&\&\ echo\ \$\(\(\ \`cat\ /sys/fs/cgroup/memory/docker/\$line\*/memory.usage_in_bytes\`\ /\ 1024\ /\ 1024\ \)\)MB\ \;\ done
You can copy the output line and paste it literally to bash -c:
$ bash -c for\ line\ in\ \`docker\ ps\ \|\ awk\ \'\{print\ \$1\}\'\ \|\ grep\ -v\ CONTAINER\`\;\ do\ docker\ ps\ \|\ grep\ \$line\ \|\ awk\ \'\{printf\ \$NF\"\ \"\}\'\ \&\&\ echo\ \$\(\(\ \`cat\ /sys/fs/cgroup/memory/docker/\$line\*/memory.usage_in_bytes\`\ /\ 1024\ /\ 1024\ \)\)MB\ \;\ done
fervent_gates 2MB
youthful_wozniak 3MB
Does it need to fit in a single line? Using a quoted heredoc gets you out of most quoting trouble:
bash <<'END_BASH'
docker ps | {
read header
while read -ra fields; do
mem=$(< /sys/fs/cgroup/memory/docker/"${fields[0]}"*/memory.usage_in_bytes)
printf "%s %dMB\n" "${fields[-1]}" $((mem / 1024 / 1024))
done
}
END_BASH

How to direct linux bash script output to one line per host

I'd like to change my script(s) so that the command output is separated by a comma and is all on one line per host. Any ideas on how I can achieve this:
1st Script:
#!/bin/bash
for i in `cat ~/hostnames.txt`
do
ssh -q $i 'bash -s' < server_info.sh
done
2nd Script (server_info.sh):
#!/bin/bash
echo -n "Name: "
uname -n
echo -n "CPUs: "
cat /proc/cpuinfo* | grep processor | wc -l
echo -n "Memory (kb): "
cat /proc/meminfo | grep MemTotal: | awk '{print $2}'
echo -n "Current Kernel: "
uname -a | awk '{print $3}'
echo -n "IP: "
hostname -i
echo -e
Changing your 1st script:
#!/bin/bash
for i in cat ~/hostnames.txt
do
ssh -q $i 'bash -s' < server_info.sh
done | awk -v RS= '{$1=$1}1'
Note: Your server_info.sh can be a lot more optimized.For example:
cat /proc/meminfo | grep MemTotal: | awk '{print $2}'
could be changed to:
awk '/MemTotal:/{print $2}' /proc/meminfo

Bash formatting for nice ascii logs

I've written a function to do some logging on different aspects of the files, folder, etc and I 'm going to have this automatically email members of my staff. All this is fine but I wouldn't mind some pointers and help to make the formation nicer - this will get sent out via mailutils as part of the script.
Just looking for some nice spacing tips,etc
function report(){
lsb_release -a
echo "OS: $(uname -s)"
echo "ARCH: $(uname -m)"
echo "VER: $(uname -r)"
echo "Apache running "
ps aux|grep -i apache2
echo "Showing if SSL is open"
apache2ctl -M | grep ssl
echo "Space on local disk"
df -h
echo "Showing permissions for the web folders"
echo $'*** /var/www// ***'
ls -l /var/www | awk '{print $1 " " $3 " " $4 " " $9}'
ls -l /var/www/user1 | awk '{print $1 " " $3 " " $4 " " $9}'
ls -l /var/www/user2 | awk '{print $1 " " $3 " " $4 " " $9}'
echo "Showing network status"
ifconfig eth0
echo " DNS "
tail /etc/resolv.conf
echo "Current workspaces set up on the local server "
grep user2 /var/www/temp/text.txt | grep -E -o '[0-9]+[0-9]'
}
Try piping the "column" command instead of or after using awk or cut.
ls -l | awk '{print $1" "$3" "$4" "$9}' | tail -n +2 | column -t
Check out the difference between "mount" and "mount | column -t" or "df -PH and df -PH | column -t"
If you want to concatenate columns of info with the same number or relational fields you can use process redirection and "paste."
paste -d ' ' <(cat /sys/class/scsi_host/host*/device/fc_host:host*/port_name) \
<(cat /sys/class/scsi_host/host*/device/fc_host:host*/speed) \
<(cat /sys/class/scsi_host/host*/device/fc_host:host*/port_state)
0x218000e05a0001aa 4 Gbit Online
0x218000e05a2001aa 4 Gbit Online

Resources