Python subprocess capture output of bash code in parentheses - python-3.x

I have a shell script that I run in a subprocess.popen() and would like to capture the output of that script and send it line by line into a logger object. The issue is that the output from the subprocess is captured by the parentheses (), and I never see the output. Has anyone here ever seen this happen, and would you be able to guide me on how to best do this?
The code takes a backup on a remote server and stores it locally. The parentheses are needed to encapsulate the ssh-agent in case more than one are open at the same time.
The bash code
( \
eval $(ssh-agent) > /dev/null; \
ssh-add -q /home/borgs/.ssh/borg-client_key; \
echo $borgPassword | \
ssh -p ${port} -A -o StrictHostKeyChecking=no ${user}#${ipAddress} \
"BORG_PASSPHRASE=\$(cat) \
borg \
--rsh 'ssh -p 9270 -o StrictHostKeyChecking=no' \
create \
--verbose \
--filter AME \
--list \
--stats \
--show-rc \
--compression lz4 \
--exclude-caches \
ssh://borgs#${ip_address}/~/${hostname}::$hostname-$backupTime $locations";
kill "${SSH_AGENT_PID}"; \
) >&1
The python code
def takeBackup(server, lg):
print(f"Taking backup of {server['hostname']}")
locationString = " ".join(server['locations'])
print(
"Running create-backup.sh command: create-backup.sh",
f"{server['hostname']} {server['ip-address']} '{locationString}'"
)
process = subprocess.Popen(
[
"./create-backup.sh",
server['hostname'],
server['ip-address'],
server['borgPassword'],
locationString
],
stdin=subprocess.PIPE,
stderr=subprocess.PIPE,
stdout=subprocess.PIPE
)
for line in process.stdout:
line = line.decode().rstrip('\n')
lg.print_info(line)
process.wait()
print(process.returncode)

Related

Proper variable interpolation for $#

#!/bin/bash
TARGET_ENV="$1"
shift
commandid=$(aws ssm send-command \
--document-name "AWS-RunShellScript" \
--targets Key=tag:Name,Values=$TARGET_ENV \
--parameters '{"commands":["su -c \"./'$#'\" - ec2-user"]}' \
--query 'Command.CommandId' \
--output text)
echo $commandid
(ssm_runner.sh)
My ec2 instance have a script called hello_world.sh that prints hello world and echo.sh which accepts parameters and echo it.
The following works
ssm_runner.sh dev hello_world.sh
but this one doesn't
ssm_runner.sh dev echo.sh hello
#!/bin/bash
TARGET_ENV="$1"
shift
# Compose a complete su command which can be safely interpreted with
# `eval` or `bash -c`.
printf -v cmd '%q ' "$#"
su="su -c ./${cmd% } - ec2-user"
# Create JSON using jq.
params=$(jq -c --arg su "$su" '.commands = [$su]' <<< '{}')
# Execute.
commandid=$(aws ssm send-command \
--document-name "AWS-RunShellScript" \
--targets Key=tag:Name,Values="$TARGET_ENV" \
--parameters "$params" \
--query 'Command.CommandId' \
--output text)
echo "$commandid"

Enable or disable a task in shell script

I have a few executable files to execute one by one. e.g.
A_1.sh
B_0.sh
cs_2.sh
c_4.sh
3_d.sh
I can execute all files with the following script, but I can't able to disable one, if I don't need that.
#!/bin/sh
for type in \
A_1 \
B_0 \
cs_2 \
c_4 \
3_d
do
echo executing $type.sh
done
When I am disabling one e.g.
#!/bin/sh
for type in \
A_1 \
B_0 \
# cs_2 \
c_4 \
3_d
do
echo executing $type.sh
done
It is showing syntax error near c_4. How to handle it actually?
The simplest is to use Bash:
#!/bin/bash
arr=(
A_1
B_0
# This is a comment
# cs_2
c_4
3_d
)
for type in "${arr[#]}"; do
echo executing $type.sh
done
You can read from a here document and filter comments:
#!/bin/sh
while read -r -u 3 line; do
# Is a comment?
if grep '^ *#' <<<"$line"; then continue; fi
# Close fd3, on the safe side.
echo executing $type.sh <&3-
done 3<<EOF
A_1
B_0
# This is a comment
# cs_2
c_4
3_d
EOF
Or invert the logic, do a function:
run() {
echo executing $1.sh
}
run A_1
run B_0
# run cs_2
run c_4
run 3_d

How to print the output of a bash terminal if it's being called from a batch script?

I've written a batch script that opens a bash terminal and do the following:
#echo off
SETLOCAL
set BASHPATH="C:\Program Files (x86)\Git\bin\"
set START_DIR=%CD%
set release=%1
echo Release outside cyg: %release%
set PATH=%BASHPATH%;%PATH%
%BASHPATH%\bash -l -c "echo Release inside cyg: $release \ && echo Directory: $START_DIR \ && cd $START_DIR \ && cd .. \ && cd .. \ && cd iv_system4 \ && cd ports \ && cd visualC12 \ && cd Debug \ && ./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua C:\sequences\FCW_APTINA\SECTION_1\$release\VASP_DUN722_20160307_Krk_Krk_113048_092_1.dvl &>../../../../FCW/ObjectDetectionTest/VASP_DUN722_20160307_Krk_Krk_113048_092_1_$release.xml \ && echo First XML generated \ && exit; bash";
%BASHPATH%\bash -l -c "echo Release inside cyg: $release \ && echo Directory: $START_DIR \ && cd $START_DIR \ && cd .. \ && cd .. \ && cd iv_system4 \ && cd ports \ && cd visualC12 \ && cd Debug \ && ./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua C:\sequences\FCW_APTINA\SECTION_1\$release\VASP_DUN722_20160307_Krk_Krk_140604_050_1.dvl &>../../../../FCW/ObjectDetectionTest/VASP_DUN722_20160307_Krk_Krk_140604_050_1_$release.xml \ && echo Second XML generated \ && exit; bash";
%BASHPATH%\bash -l -c "echo Release inside cyg: $release \ && echo Directory: $START_DIR \ && cd $START_DIR \ && cd .. \ && cd .. \ && cd iv_system4 \ && cd ports \ && cd visualC12 \ && cd Debug \ && ./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua C:\sequences\FCW_APTINA\SECTION_1\$release\VASP_DUN722_20160308_Krk_Bra_171516_012_1.dvl &>../../../../FCW/ObjectDetectionTest/VASP_DUN722_20160308_Krk_Bra_171516_012_1_$release.xml \ && echo Third XML generated \ && exit; bash";
%BASHPATH%\bash -l -c "echo Release inside cyg: $release \ && echo Directory: $START_DIR \ && cd $START_DIR \ && cd .. \ && cd .. \ && cd iv_system4 \ && cd ports \ && cd visualC12 \ && cd Debug \ && ./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua C:\sequences\FCW_APTINA\SECTION_1\$release\VASP_DUN722_20160309_Krk_Bra_124926_117_1.dvl &>../../../../FCW/ObjectDetectionTest/VASP_DUN722_20160309_Krk_Bra_124926_117_1_$release.xml \ && echo Fourth XML generated \ && exit; bash";
%BASHPATH%\bash -l -c "echo Release inside cyg: $release \ && echo Directory: $START_DIR \ && cd $START_DIR \ && cd .. \ && cd .. \ && cd iv_system4 \ && cd ports \ && cd visualC12 \ && cd Debug \ && ./iv4_console.exe ../embedded/LUA/analysis/verbose-udp-toxml.lua C:\sequences\FCW_APTINA\SECTION_1\$release\VASP_DUN722_20160317_KRK_KRK_105146_011_1.dvl &>../../../../FCW/ObjectDetectionTest/VASP_DUN722_20160317_KRK_KRK_105146_011_1_$release.xml \ && echo Fifth XML generated \ && exit; bash";
set CYGWINPATH="C:\cygwin64\bin"
set PATH=%CYGWINPATH%;%PATH%
ENDLOCAL
For debugging purposes, I need to print the stdout of the bash terminal. How could I do this?
Opening a command prompt ( and running the script above should print everything in on the screen that any of those programs writes to stdout. If that doesn't happend, then there is something which you didn't tell us.
That said: Mixing BASH and BAT will get you in trouble. They have different opinions about many things and switching between the two will confuse you, especially when you're not an expert in both.
Since you use BASH to invoke an Windows executable, I suggest that you scratch BASH and use PowerShell. PowerShell is much more like BASH, but it understands Windows (i.e. paths with backslashes). Just like in BASH, PowerShell will allow to you to redirect the output of iv4_console to a file or pipe it back to the script for further processing.
All those things can be done with BAT, too, but the many "features" will drive you insane in the process.
Apart from that, there are several problems in your script. The ; bash" at the end of several lines will start a new bash shell. That shell will stop the script and wait for you to type commands.
Avoid cd x && cd y, use cd x/y instead.

Makefile not executing the commands as expected

I'm using the following Makefile, which should check whether the files are in some directory *.rpm, if the files hasn't been found i'm going to execute some command (run a script or make the rpm)
Here's a snippet from my Makefile
include /home/user/workspace/test/exec_recipe.mk
export BUILD_LOGS_DIR = $(CURDIR)/build_logs
.PHONY: my_rpm
libpdutil_rpm:
#echo "Making target 'libpdutil_rpm'. Time: $$(date +%T)"
cd /home/user/workspace/test/build/test && $(call exec_recipe,$(ls /home/user/workspace/test/build/test/rpmbuild/RPMS/x86_64d/*.rpm) || $(./test.sh),test.log,)
#echo "Finished making target 'my_rpm'. Time: $$(date +%T)"
And here's the exec_recipe.mk
SHELL:=/bin/bash
exec_recipe = \
echo "The logs dir is: $$BUILD_LOGS_DIR"; \
log_name="$(2)"; \
echo "The log name is $$log_name"; \
cmd="$(1)"; \
eval "$$cmd" 2>&1 | tee -a "$$BUILD_LOGS_DIR/$$log_name"; rc="$${PIPESTATUS[0]}"; \
if [ $$rc = 0 ]; then \
res="PASS"; \
else \
res="FAIL"; \
fi; \
flock $(SUMMARY) echo "Making target '$#': $$res" >> $(SUMMARY); \
exit $$rc
So the problem is when it's reaching the execution of the exec_recipe.mk it giving me some errors:
/bin/bash: eval: line 0: syntax error near unexpected token||'
/bin/bash: eval: line 0: || '
I'm sure that i'm doing something wrong..
I need the helper make file, because i'm using it for other purposes as well
You have to escape all the dollar signs that you don't want make to interpret. You've forgotten to escape the dollar signs in the arguments to $(call exec_recipe,...):
cd /home/user/workspace/test/build/test && $(call exec_recipe,$$(ls /home/user/workspace/test/build/test/rpmbuild/RPMS/x86_64d/*.rpm) || $$(./test.sh),test.log,)
I'm not sure what the final comma is for but whatever.

Bash script for fetching row from Postgresql

I have a Postgresql function which returns table of JSON strings.
I want to fetch those rows under the Linux platform and run CURL program for each string like this:
curl http://XXX.XX.XX.XX:XXXX/update/json -H 'Content-type:application/json' -d ['||solr_interface.dict_cursor_okpd_json()||']'
Do you have any example how to do this using say bash script?
Thanks in advise.
I've found solution. Maybe it will useful for somebody
#!/bin/bash
if [ $# != 2 ]; then
echo "Please select postgres procedure for data retrieving as first parameter"
echo "and Solr index name (dictionary or company) as second parameter"
exit 1;
fi
PSQL=/usr/bin/psql
DB_USER="xxxxxxxx"
DB_PASSWORD="xxxxxxxx"
DB_HOST="xxx.xxx.xxx.xxx"
DB_PORT="xxxx"
DB_NAME="portal_sources"
export PGPASSWORD=$DB_PASSWORD
$PSQL \
-X \
-h $DB_HOST \
-p $DB_PORT $DB_NAME \
-c "select 'curl http://xxx.xxx.xxx.xxx:xxxx/solr/$2/update/json -H ''Content-type:application/json'' -d ''['||$1||']'''" \
--single-transaction \
--set AUTOCOMMIT=off \
--set ON_ERROR_STOP=on \
--no-align \
-t \
--field-separator ' ' \
--quiet \
| while read command ; do
#echo "$command"
eval $command
done

Resources