tar (child): lbzip2: Cannot exec: No such file or directory - linux

I am getting the following error when my bash script executes tar -xf
[21:02:51][Step 2/2] tar (child): lbzip2: Cannot exec: No such file or directory
[21:02:51][Step 2/2] tar (child): Error is not recoverable: exiting now
[21:02:51][Step 2/2] tar: Child returned status 2
[21:02:51][Step 2/2] tar: Error is not recoverable: exiting now
[21:02:51][Step 2/2] Process exited with code 2
I did check that I had bzip2 installed (as suggested in this answer):
Not quite sure if I need bzip2-libs:
The weird thing is, when I run the script manually from sh, it runs fine! But when I run it via TeamCity (which is running in Docker), it fails with the above error.
The script is literally the following (omitting some details):
#!/bin/bash
whoami
echo "$USER"
set -e
ScriptFilename=$0
ScriptPath=$(cd $(dirname $0) && pwd)
argn=$#
if [[ $argn != '1' ]]; then
echo "usage: ${ScriptFilename} phpArchivesDirTarFile"
exit
fi
phpArchivesDirTarFile=$1
function ExtractPhpArchives {
tar -C $ScriptPath -xf $phpArchivesDirTarFile
}
echo "ExtractPhpArchives"
ExtractPhpArchives
It results in the following TeamCity output:
Suspecting a permissions problem, I added "whoami" (which prints "root") and echo "$USER" which, a bit surprisingly, prints nothing.
What's more interesting, I have a backup copy or the tar file for which the script runs fine in TeamCity. But when I extracted the tarball contents and archived them again, I was back to the "Cannot exec" problem:
TeamCity settings:
Why am I getting this error and what else can I try to debug it?

Related

Bash repeating my directory and can not call the file

have to use a .sh script to unpack and prep some databases. The code is the following:
#
# Downloads and unzips all required data for AlphaFold.
#
# Usage: bash download_all_data.sh /path/to/download/directory
set -e
DOWNLOAD_DIR="$1"
for f in $(ls ${DOWNLOAD_DIR}/*.tar.gz)
do
tar --extract --verbose --file="${DOWNLOAD_DIR}/${f}" /
--directory="${DOWNLOAD_DIR}/mmseqs_dbs"
rm "${f}"
BASENAME="$(basename {f%%.*})"
DB_NAME="${BASENAME}_db"
OLD_PWD=$(pwd)
cd "${DOWNLOAD_DIR}/mmseqs_dbs"
mmseqs tar2exprofiledb "${BASENAME}" "${DB_NAME}"
mmseqs createindex "${DB_NAME}" "${DOWNLOAD_DIR}/tmp/"
cd "${OLD_PWD}"
done
When I run the code, I got that error:
(openfold_venv) watson#watson:~/pedro/openfold$ sudo bash scripts/prep_mmseqs_dbs.sh data/
tar: data//data//colabfold_envdb_202108.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
I don`t understand why the code repeats my "DOWNLOAD_DIR", the correct should be :
data/colabfold_envdb_202108.tar.gz
and not
data//data//colabfold_envdb_202108.tar.gz
Could anyone help me?
New code:
set -e
DOWNLOAD_DIR="$1"
for f in ${DOWNLOAD_DIR}/*.tar.gz;
do
tar --extract --verbose --file="$f" /
--directory="${DOWNLOAD_DIR}/mmseqs_dbs"
rm "${f}"
BASENAME="$(basename {f%%.*})"
DB_NAME="${BASENAME}_db"
OLD_PWD=$(pwd)
cd "${DOWNLOAD_DIR}/mmseqs_dbs"
mmseqs tar2exprofiledb "${BASENAME}" "${DB_NAME}"
mmseqs createindex "${DB_NAME}" "${DOWNLOAD_DIR}/tmp/"
cd "${OLD_PWD}"
done
To answer your first question: why is it repeating? Because you are repeating it in your code:
for f in ${DOWNLOAD_DIR}/*.tar.gz;
do
tar --extract --verbose --file="${DOWNLOAD_DIR}/$f"
If f is downloads/file.tar.gz then ${DOWNLOAD_DIR}/${f} will resolve to downloads/downloads/file.tar.tgz.
As to your second question: the escape character is the backslash \, not the forward slash. Your multiline command should look like this:
tar --extract --verbose --file="${DOWNLOAD_DIR}/${f}" \
--directory="${DOWNLOAD_DIR}/mmseqs_dbs"

Unable to run shell script in crontab

I am unable to make a script execute successfully from crontab.
When the script is executed manually, it works fine. When added to the crontab it gives errors.
When the script is executed manually as follows it all works fine:
cd /home/admin/git/Repo
./lunchpad2.sh
The script is added to crontab as follows:
sudo crontab -e
30 13 * * * /home/admin/git/Repo/lunchpad2.sh > /home/admin/git/Repo/outcome.err
lunchpad2.sh has 744 permissions set;
The script itself:
#!/bin/bash -p
PATH=$PATH:/home/admin/git/Repo
echo "--> Starting!"
echo "--> Stopping docker"
docker-compose down
echo "--> Switching files"
mv dc_conf_standby.py dc_conf_aboutready.py
mv dc_conf.py dc_conf_standby.py
mv dc_conf_aboutready.py dc_conf.py
echo "--> Building docker"
docker-compose up -d --build
echo "--> Completed!"
The errors that are generated:
/home/admin/git/Repo/lunchpad2.sh: line 7: docker-compose: command not found
mv: cannot stat ‘dc_conf_standby.py’: No such file or directory
mv: cannot stat ‘dc_conf.py’: No such file or directory
mv: cannot stat ‘dc_conf_aboutready.py’: No such file or directory
/home/admin/git/Repo/lunchpad2.sh: line 15: docker-compose: command not found
I see two issues here:
You need to either cd in the script or in the cron job. Cron runs the command in your home directory. You can echo "$PWD" to confirm.
You need to specify docker-compose executable path (Run "which docker-compose" to confirm)
#!/bin/bash -p
cd /home/admin/git/Repo
echo "--> Starting!"
echo "--> Stopping docker"
/usr/bin/docker-compose down
echo "--> Switching files"
mv dc_conf_standby.py dc_conf_aboutready.py
mv dc_conf.py dc_conf_standby.py
mv dc_conf_aboutready.py dc_conf.py
echo "--> Building docker"
/usr/bin/docker-compose up -d --build
echo "--> Completed!"

script to copy, install and execute on multiple hosts

I am trying to copy few files into multiple hosts, install/configure those on each with running specific commands depending on OS type. The IP addresses for each host are read from host.txt file.
It appears when I run the script, it does not execute on the remote hosts. Can someone help identify the issues with this script? Sorry for this basic one, I am quite new into shell scripting.
#!/bin/bash
export AGENT=agent-x86-64-linux-5.8.1.tar.gz
export AGENT_PROPERTIES_NONDMZ=agent.properties.nondmz
export agent_INIT=agent.sh
echo "####Installing hqagent####"
while read host; do
scp $AGENT $AGENT_PROPERTIES_NONDMZ $agent_INIT root#$host:/opt
if ssh -n root#$host '[ "$(awk "/CentOS/{print}" /etc/*release)" ] '
then
cd /opt
tar -xvzf $AGENT
mv -f /opt/agent.properties.nondmz /opt/agent-5.8.1/conf/agent.properties
mkdir /opt/hqagent/
ln -s /opt/agent-5.8.1/ /opt/hqagent/agent-current
useradd hqagent
groupadd hqagent
chown -R hqagent:hqagent /opt/hqagent /opt/agent-5.8.1/
cd /etc/init.d
chmod 755 hqagent.sh
chkconfig --add hqagent.sh
su - hqagent
/opt/agent-5.8.1/bin/hq-agent.sh start
else
cd /opt
tar -xvzf $AGENT
mv -f /opt/agent.properties.nondmz /opt/agent-5.8.1/conf/agent.properties
rm -rf /opt/hqagent.sh
mkdir /opt/hqagent/
ln -s /opt/agent-5.8.1/ /opt/hqagent/agent-current
useradd hqagent
groupadd hqagent
chown -R hqagent:hqagent /opt/hqagent /opt/agent-5.8.1
cd /etc/init.d
ln -s /opt/hqagent/agent-current/bin/hq-agent.sh hqagent.sh
cd /etc/init.d/rc3.d/
ln -s /etc/init.d/hqagent.sh S99hqagent
ln -s /etc/init.d/hqagent.sh K01hqagent
cd ../rc5.d
ln -s /etc/init.d/hqagent.sh S99hqagent
ln -s /etc/init.d/hqagent.sh K01hqagent
chkconfig --add hqagent.sh
su - hqagent
/opt/agent-5.8.1/bin/hq-agent.sh start
fi
done < hosts.txt
error:
tar (child): agent-x86-64-linux-5.8.1.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
mv: cannot stat `/opt/agent.properties.nondmz': No such file or directory
mkdir: cannot create directory `/opt/hqagent/': File exists
ln: creating symbolic link `/opt/hqagent/agent-current': File exists
useradd: user 'hqagent' already exists
groupadd: group 'hqagent' already exists
chown: cannot access `/opt/agent-5.8.1/': No such file or directory
chmod: cannot access `hqagent.sh': No such file or directory
error reading information on service hqagent.sh: No such file or directory
-bash: line 1: 10.145.34.6: command not found
-bash: line 2: 10.145.6.10: command not found
./hq-install.sh: line 29: /opt/agent-5.8.1/bin/hq-agent.sh: No such file or directory
It appears that the problem is that you run this script on the "master" server, but somehow expect the branches of your if-statement to be run on the remote hosts. You need to factor those branches out into their own files, copy them to the remote hosts along with the other files, and in your if-statement, each branch should just be a ssh command to the remote host, triggering the script you copied over.
So your master script would look something like:
#!/bin/bash
export AGENT=agent-x86-64-linux-5.8.1.tar.gz
export AGENT_PROPERTIES_NONDMZ=agent.properties.nondmz
export agent_INIT=agent.sh
# Scripts containing the stuff you want done on the remote hosts
centos_setup=centos_setup.sh
other_setup=other_setup.sh
echo "####Installing hqagent####"
while read host; do
echo " ++ Copying files to $host"
scp $AGENT $AGENT_PROPERTIES_NONDMZ $agent_INIT root#$host:/opt
echo -n " ++ Running remote part on $host "
if ssh -n root#$host '[ "$(awk "/CentOS/{print}" /etc/*release)" ] '
then
echo "(centos)"
scp $centos_setup root#$host:/opt
ssh root#host "/opt/$centos_setup"
else
echo "(generic)"
scp $other_setup root#$host:/opt
ssh root#host "/opt/$other_setup"
fi
done < hosts.txt
The contents of the two auxiliary scrips would be the current contents of the if-branches in your original.

Bash Shell: Cannot use variable $ as a path to run tar

This a really short question. But is there something syntatically wrong with placing a variable $example as an argument for tar in a bash file?
I have the file written as
//only portion that really matters
#!/bin/bash
...
tar -cvpzf $filename $backup_source
//here's the actual code
#!/bin/bash
backup_source="~/momobobo"
backup_dest="~/momobobo_backup/"
dater=`date '+%m-%d-%Y-%H-%M-%S'`
filename="$backup_dest$dater.tgz"
echo “Backing Up your Linux System”
tar -cvpzf $filename $backup_source
echo tar -cvpzf $filename $backup_source
echo “Backup finished”
//and heres the error
“Backing Up your Linux System”
tar: ~/momobobo: Cannot stat: No such file or directory
tar (child): ~/momobobo_backup/07-02-2013-18-34-12.tgz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
tar -cvpzf ~/momobobo_backup/07-02-2013-18-34-12.tgz ~/momobobo
Notice the "echo tar ...". When I copy and paste the output and run it in my terminal there is no problem taring the file. I'm currently running Xubuntu and I already did an update.
~ doesn't expand to your home directory in double quotes.
Just remove the double quotes:
backup_source=~/momobobo
backup_dest=~/momobobo_backup/
In cases where you have things you would want to quote, you can use ~/"momobobo"

How to check if a shell command executed properly or not?

#!/bin/sh
tar=$1
if [ -f $tar ]
then
tar xvf $tar
else
exit 1
fi
... <more code>
I need a conformation that the tar actually happened successfully, otherwise abort the execution of the script.
The tar file extract might not go through because
some problems with tar command
access issues with $tar
Do Linux utilities have some return values? How should I use them here?
Check the $? variable after executing a command. If everything is OK, it should be 0, positive otherwise.
tar xvf $tar
[ $? -ne 0 ] && exit 1
More information here.
This
tar xvf "$tar" || exit 1
or this (if you want to check if file exists yourself)
[ -f "$tar" ] && tar xvf "$tar" || exit 1

Resources