pass outfile to another program using bash script - linux

I have the following bash code for a batch run of multiple files to be processed by 3 different programs:
for i in *.txt
do
program1 -in ${i} -out Prog1_out_${i}
program2 -in Prog1_out_${i} -out Prog2_out_${i}
program3 -in Prog2_out_${i} -out Prog3_out_${i}
done
I ran into problem with program 2 not finding the input which is the output from program 1, and of course program 3 did not find the required input.
Can anyone help with suggestions for solving the problem?
Thanks

If the programs produce the output whenever they are successful, you could make them dependent of the previous commands success, like this:
program1 -in ${i} -out Prog1_out_${i} &&
program2 -in Prog1_out_${i} -out Prog2_out_${i} &&
program3 -in Prog2_out_${i} -out Prog3_out_${i}
So if one of the programs fails, the rest of the chain will not be invoked.
However, if the creation of the output has nothing to do with the success of the program, but you just want to check if the files exist, you can add the appropriate check before you call programx, i.e.
if [ -f "${i}" ]
then
progx ...
fi
As you are doing the same thing all the time this could be generalized for all programs (untested):
for i in *.txt
do
mv $i Prog0_out_$i
for program in 0 1 2
do
INFILE=Prog{$program}_out_${$i}
if [ ! -r ${INFILE} ]
then
break
fi
program{$program} -in ${INFILE} -out "Prog{$program}_out_$((i+1))"
done
done

Related

How to manage env variables and secrets?

Boy, I'm tired of this topic. I have gitlab CI, local environments, keychain, keepass, gcp, aws and a whole bunch of other places where some of my env variables stored. Furthermore, Expo apps, for example, can't pull .env files, so I have to write bash scripts to create js files. This hurts my brain.
I want to have a cozy place where I store all my variables and secrets safely per project per environment. I want to share it with my team, CI servers etc. I want to just specify a single key: the environemnt title. And all the variables should be pulled from somewhere. Is there such tool anywhere on the github or internet???
Not sure if this question is suitable for stackoverflow, pls direct me to the right stackexchange forum if it doesn't.
Encrypt files and deployment scripts file by file, push them to repo. Whenever you need to switch environment, just decrypt the files with a the environment key word in name:
Encrypt.sh
# sh encrypt.sh ./config.production.js passwordhere
echo "encrypting $1"
openssl enc -aes-128-cbc -a -salt -pass pass:$2 -in $1 -out $1.enc
rm $1
mv $1.enc $1
echo "done"
decrypt.sh:
# sh decrypt.sh production passwordhere
echo "decrypting $1 environment"
for file in $(find . -not -path "*/node_modules/*" -name "*.$1*")
do
echo "decrypting $file to ${file//.$1}"
openssl enc -aes-128-cbc -a -d -salt -pass pass:$2 -in $file -out "${file//.$1}"
done

Decryption in a loop with openssl?

I'm trying to use openssl to decrypt files. I can successfully encrypt them in a loop, but when I try to change the command to make it decrypt, I get "error reading input file."
for f in /dir1/dir2/*.txt ;
do [ -f $f ] && openssl aes-256-cbc enc -in $f -out $f.enc -k PASSWORD ;
done
!the above works for encrypting
for f in /dir1/dir2/*.txt.enc ;
do [ -f $f ] && openssl enc -d -aes-256-cbc -k PASSWORD -in $f -out $f;
done
"error reading input file"
So I've tried making the input file $f.txt.enc, and $f.enc, and similar fiddling with the outfile. No luck.
I think your encoding line has problem, missing - in front of the aes part ... for reference here are lines that work for me:
openssl enc -aes-256-cbc -in $f -out $f.enc -k PASSWORD
openssl enc -aes-256-cbc -d -in $f.enc -out $f.dec -k PASSWORD

'Missing name for redirect.' error in while running command in CSH environment

I am trying to generate some certificate using below command in CSH environment:
/usr/bin/openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout "selfsigned.key" \
-out "selfsigned.crt" -subj "/C=XX/ST=XX/L=XX/O=XX/OU=XX/CN=Some IP" -extensions SAN \
-config <(cat /etc/ssl/openssl.cnf <(printf "\n[SAN]\nsubjectAltName=DNS:Some DNS,Some IP"))
Getting Missing name for redirect error.
How can I fix this?
Part of your command line is:
… <(cat /etc/ssl/openssl.cnf <(printf "\n[SAN]\nsubjectAltName=DNS:Some DNS,Some IP"))
You're using a Bash-specific notation — process substitution — twice. In the C shell, that simply isn't going to work. The C shell has no clue what you mean (witness the error message).
You'll have to wrap the command in a Bash script and use Bash to execute it. Or rethink the command so as not to use process substitution at all.
One option would be to create a temporary file and use that in the command:
set tmpfile `mktemp`
cat /etc/ssl/openssl.cnf > $tmpfile
printf "\n[SAN]\nsubjectAltName=DNS:Some DNS,Some IP\n" >> $tmpfile
/usr/bin/openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout "selfsigned.key" \
-out "selfsigned.crt" -subj "/C=XX/ST=XX/L=XX/O=XX/OU=XX/CN=Some IP" -extensions SAN \
-config $tmpfile
rm -f $tmpfile
The fact that this might leave the temporary file lying around if interrupted is one of the reasons why the standard recommendation is not to write scripts in the C shell. (See C Shell Programming Considered Harmful or Top Ten Reasons not to use the C shell.) With a POSIX shell, you can ensure that the temporary file is removed unless you brutally kill the script with SIGKILL.

How to direct output of time command to a file in shell script?

This does not append the output of the time command to file.txt
echo $(time openssl genrsa -aes128 -out server.key 1024) &> file.txt
You can use the following:
{time openssl genrsa -aes128 -out server.key 1024} 2>> file.txt

OpenSSL in bash script

I'm trying to make a bash script in linux where some encrypted data is embedded and then retrieved and decrypted with openssl, like this:
cat | openssl des3 -d -a -salt -pass pass:asdf > output.txt <<EOF
U2FsdGVkX1/zN55FdyL5j1nbDVt5vK4V3WLQrnHPoycCJPwWO0ei3PCrrMqPaxUH.....blablablah data
EOF
The only problem with this, that would otherwise work, is that I have to hit enter when the script reaches this position. I have tried changing the way \n are placed, but no luck.
I can't afford to press manually enter for all the files that are going to be embedded like this one!!
Thanks for your help!
A couple of things wrong here:
You shouldn't use both cat | ... and also a here document (<<EOF). Use one or the other.
Your example isn't testable because the example text is not the DES3 encryption of any input.
This example works as expected:
cat ~/.profile | openssl des3 -e -a -salt -pass pass:asdf -out /tmp/output.txt
That is, it writes an encrypted version of ~/.profile, base64 encoded, to file /tmp/output.txt.
Here's a working decryption example with a here document:
openssl des3 -d -a -salt -pass pass:asdf <<EOF
U2FsdGVkX1/03DBd+MpEKId2hUY82cLWpYltYy2zSsg=
EOF
Try this in the safety and comfort of your own home...

Resources