Makefile can't understand comments - linux

If I put comments (# ...) in my Makefile, make gives me an error and quit. If I remove the comments, the makefile works fine.
Makefile:1: *** missing separator. Stop.
Make-version: 3.81
Linux: Ubuntu 9.04
The Makefile:
# Backup Makefile
#
# Create backups from various services and the system itself. This
# script is used to perform single backup tasks or a whole backup
# from the system. For more information about this file and how to
# use it, read the README file in the same directory.
BACKUP_ROOT = /srv/backup
ETC_PATH = /srv/config
SVN_PATH = /srv/svn/
TRAC_PATH = /srv/trac/sysinventory
PR10_PATH = /swsd/project/vmimages/...
PR10_MOUNT_PATH = /tmp/temp_sshfs_pr10
MYSQL_USER = "xxx"
MYSQL_PASSWORD = "xxx"
DATE = `date +%F`
help :
cat README
init-environment :
mkdir -p $(BACKUP_ROOT)
mkdir $(BACKUP_ROOT)/tmp
mkdir -p $(PR10_MOUNT_PATH)
backup : backup-mysql backup-configuration backup-svn backup-trac
upload-to-pr10 : mount-pr10
tar cf $(DATE)-backup-blizzard.tar -C $(BACKUP_ROOT) *.-backup.tar.gz
mv $(BACKUP_ROOT)/*-backup-blizzard.tar $(PR10_MOUNT_PATH)/
umount $(PR10_MOUNT_PATH)
mount-pr10 :
su xxx -d "sshfs -o allow_root xxx#xxx:$(PR10_PATH) $(PR10_MOUNT_PATH)"
fusermount -u $(PR10_MOUNT_PATH)
backup-mysql :
mysqldump --comments --user=$(MYSQL_USER) --password=$(MYSQL_PASSWORD) --all-databases --result-file=$(BACKUP_ROOT)/tmp/mysql_dump.sql
tar czf $(BACKUP_ROOT)/$(DATE)-mysql-backup.tar.gz -C
$(BACKUP_ROOT)/tmp/mysql_dump.sql
backup-configuration :
tar czf $(BACKUP_ROOT)/$(DATE)-configuration-backup.tar.gz $(ETC_PATH)/
backup-svn :
svnadmin dump $(SVN_PATH)/repository > $(BACKUP_ROOT)/tmp/svn_repository.dump
tar czf $(BACKUP_ROOT)/$(DATE)-subversion-backup.tar.gz -C $(BACKUP_ROOT)/tmp/svn_repository.dump
backup-trac :
tar czf $(BACKUP_ROOT)/$(DATE)-trac-backup.tar.gz $(TRAC_PATH)/
clean :
rm -f $(BACKUP_ROOT)/tmp/mysql_dump.sql
rm -f $(BACKUP_ROOT)/tmp/svn_repository.dump
rm -f $(BACKUP_ROOT)/*-backup.tar.gz
rm -f $(BACKUP_ROOT)/*-backup-blizzard.tar

Your Makefile works for me (with spaces replaced by tabs), so it sounds like you have a case of stray non-printing chars.
Try inspecting the output of "cat -vet Makefile". That will show where EOL, TAB and other unseen chars are.
You'll want to see something like this:
# Backup Makefile$
#$
# Create backups from various services and the system itself. This$
# script is used to perform single backup tasks or a whole backup$
# from the system. For more information about this file and how to$
# use it, read the README file in the same directory.$
$
BACKUP_ROOT = /srv/backup$
ETC_PATH = /srv/config$
SVN_PATH = /srv/svn/$
TRAC_PATH = /srv/trac/sysinventory$
PR10_PATH = /swsd/project/vmimages/...$
PR10_MOUNT_PATH = /tmp/temp_sshfs_pr10$
$
MYSQL_USER = "xxx"$
MYSQL_PASSWORD = "xxx"$
$
$
DATE = `date +%F`$
$
help :$
^Icat README$
$
$
init-environment :$
^Imkdir -p $(BACKUP_ROOT)$
^Imkdir $(BACKUP_ROOT)/tmp$
^Imkdir -p $(PR10_MOUNT_PATH)$
$
Make sure all commands are preceeded by "^I".
You could also try to looking for stray chars using something like:
cat -vet Makefile | grep "\^[^I]" --colour=auto

You may have used spaces instead of tabs for your comment.
Please post the makefile so we don't have to guess.

Related

Create Directory, download file and execute command from list of URL

I am working on a Red Hat Linux server. My end goal is to run CRB-BLAST on multiple fasta files and have the results from those in separate directories.
My approach is to download the fasta files using wget then run the CRB-BLAST. I have multiple files and would like to be able to download them each to their own directory (the name perhaps should come from the URL list files), then run the CRB-BLAST.
Example URLs:
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_3370_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_CB_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_13_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_37_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_123_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_195_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_31_chr.v0.1.liftover.CDS.fasta.gz
Ideally, the file name determines the directory name, for example, TC_3370/.
I think there might be a solution with cat URL.txt | mkdir | cd | wget | crb-blast
Currently I just run the commands in line:
mkdir TC_3370
cd TC_3370/
wget url
http://assemblies/Genomes/final_assemblies/10x_meta_assemblies_v1.0/TC_3370_chr.v1.0.maker.CDS.fasta.gz
crb-blast -q TC_3370_chr.v1.0.maker.CDS.fasta.gz -t TCV2_annot_cds.fna -e 1e-20 -h 4 -o rbbh_TC
Try this Shellcheck-clean program:
#! /bin/bash -p
while read -r url; do
file=${url##*/}
dir=${file%%_chr.*}
mkdir -v -- "$dir"
(
cd "./$dir" || exit 1
wget -- "$url"
crb-blast -q "$file" -t TCV2_annot_cds.fna -e 1e-20 -h 4 -o rbbh_TC
)
done <URL.txt
See Removing part of a string (BashFAQ/100 (How do I do string manipulation in bash?)) for an explanation of ${url##*/} etc.
The subshell (( ... )) is used to ensure that the cd doesn't affect the main program.
Another implementation
#!/bin/sh
# Read lines as url as long as it can
while read -r url
do
# Get file name by stripping-out anything up to the last / from the url
file_name=${url##*/}
# Get the destination dir name by stripping anything from the first __chr
dest_dir=${file_name%%_chr*}
# Compose the wget output path
fasta_path="$dest_dir/$file_name"
if
# Successfully created the destination directory AND
mkdir -p -- "$dest_dir" &&
# Successfully downloaded the file
wget --output-file="$fasta_path" --quiet -- "$url"
then
# Process the fasta file into fna
fna_path="$dest_dir/TCV2_annot_cds.fna"
crb-blast -q "$fasta_path" -t "$fna_path" -e 1e-20 -h 4 -o rbbh_TC
else
# Cleanup remove destination directory if any of mkdir or wget failed
rm -fr -- "$dest_dir"
fi
# reading from the URL.txt file for the whole while loop
done < URL.txt
Download files from list is task for -i file option, if you have file named say urls.txt with one URL per line you might simply do
wget -i urls.txt
Note that this will put all files inside current working directory, so if you wish to have them in separate dirs, you would need to move them after wget finish.

Bash repeating my directory and can not call the file

have to use a .sh script to unpack and prep some databases. The code is the following:
#
# Downloads and unzips all required data for AlphaFold.
#
# Usage: bash download_all_data.sh /path/to/download/directory
set -e
DOWNLOAD_DIR="$1"
for f in $(ls ${DOWNLOAD_DIR}/*.tar.gz)
do
tar --extract --verbose --file="${DOWNLOAD_DIR}/${f}" /
--directory="${DOWNLOAD_DIR}/mmseqs_dbs"
rm "${f}"
BASENAME="$(basename {f%%.*})"
DB_NAME="${BASENAME}_db"
OLD_PWD=$(pwd)
cd "${DOWNLOAD_DIR}/mmseqs_dbs"
mmseqs tar2exprofiledb "${BASENAME}" "${DB_NAME}"
mmseqs createindex "${DB_NAME}" "${DOWNLOAD_DIR}/tmp/"
cd "${OLD_PWD}"
done
When I run the code, I got that error:
(openfold_venv) watson#watson:~/pedro/openfold$ sudo bash scripts/prep_mmseqs_dbs.sh data/
tar: data//data//colabfold_envdb_202108.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
I don`t understand why the code repeats my "DOWNLOAD_DIR", the correct should be :
data/colabfold_envdb_202108.tar.gz
and not
data//data//colabfold_envdb_202108.tar.gz
Could anyone help me?
New code:
set -e
DOWNLOAD_DIR="$1"
for f in ${DOWNLOAD_DIR}/*.tar.gz;
do
tar --extract --verbose --file="$f" /
--directory="${DOWNLOAD_DIR}/mmseqs_dbs"
rm "${f}"
BASENAME="$(basename {f%%.*})"
DB_NAME="${BASENAME}_db"
OLD_PWD=$(pwd)
cd "${DOWNLOAD_DIR}/mmseqs_dbs"
mmseqs tar2exprofiledb "${BASENAME}" "${DB_NAME}"
mmseqs createindex "${DB_NAME}" "${DOWNLOAD_DIR}/tmp/"
cd "${OLD_PWD}"
done
To answer your first question: why is it repeating? Because you are repeating it in your code:
for f in ${DOWNLOAD_DIR}/*.tar.gz;
do
tar --extract --verbose --file="${DOWNLOAD_DIR}/$f"
If f is downloads/file.tar.gz then ${DOWNLOAD_DIR}/${f} will resolve to downloads/downloads/file.tar.tgz.
As to your second question: the escape character is the backslash \, not the forward slash. Your multiline command should look like this:
tar --extract --verbose --file="${DOWNLOAD_DIR}/${f}" \
--directory="${DOWNLOAD_DIR}/mmseqs_dbs"

subprocess.check_output(cmd) doesn't execute for tar -C option

I am trying to run the following command using subprocess.check_output(). The command (shown below) works fine if run directly in bash :
tar -C /tmp/models/ -czvf model.tar.gz .
It also runs fine if I don't use the "C" option when run via subprocess.
cmd = ['tar', 'czf', "/tmp/model.tar.gz", "/tmp/models/"]
output = subprocess.check_output(cmd).decode("utf-8").strip() # Works
But when I try to use the -C option with the above tar command, I get an exception which says tar: Must specify one of -c, -r, -t, -u, -x.
cmd = ['tar', 'C', '/tmp/models', 'cf', 'model.tar.gz', '.'] # fails. Other variations of this fail too.
How do I run the above tar command using subprocess correctly?. Thanks.
I am using python3.8
Looks like the dashes - should be specified:
$ tar C whatever czvf thing.tar .
tar: Must specify one of -c, -r, -t, -u, -x
$ tar C whatever -czvf thing.tar .
tar: could not chdir to 'whatever'
So the command should look like this:
cmd = ['tar', '-C', '/tmp/models', '-cf', 'model.tar.gz', '.']

Updating phpMyAdmin blowfish_secret via bash shell script in linux

I am working on a bash script that automatically downloads phpMyAdmin and extracts it. I would like to add one more step to this installer script.
Copy config.sample.inc.php as config.inc.php and update this file's line with a random blowfish secret:
$cfg['blowfish_secret'] = ''; /* YOU MUST FILL IN THIS FOR COOKIE AUTH! */
So, this is what I have I have tried:
#!/bin/bash
wget -O phpMyAdmin-4.5.3.1-english.zip https://files.phpmyadmin.net/phpMyAdmin/4.5.3.1/phpMyAdmin-4.5.3.1-english.zip;
unzip phpMyAdmin-4.5.3.1-english.zip >/dev/null 2>/dev/null;
cd phpMyAdmin-4.5.3.1-english;
mv * ..;
cd ..;
rm -rf phpMyAdmin-4.5.3.1-english;
rm -rf phpMyAdmin-4.5.3.1-english.zip;
randomBlowfishSecret=`openssl rand -base64 32`;
cat config.sample.inc.php | sed -e "s/cfg['blowfish_secret'] = ''/cfg['blowfish_secret'] = '$randomBlowfishSecret'/" > config.inc.php
When this script runs, phpMyAdmin is downloaded and extracted and the file is copied, however it does not appear to be setting the randomBlowfishSecret to $cfg['blowfish_secret'].
Any ideas?
A few points:
You don't have to end your lines with ; – a newline has the same effect.
If you want to redirect both stdout and stderr, you can use &>/dev/null instead of >/dev/null 2>/dev/null, but in the case of unzip, you can just use unzip -q to suppress output (or even -qq, but -q was already silent for me).
Instead of
cd phpMyAdmin-4.5.3.1-english;
mv * ..;
cd ..;
you can just use mv phpMyAdmin-4.5.3.1-english/* .
There are two files starting with ., which aren't moved with your command (unless you have the dotglob shell option set), so you have to move them separately:
mv phpMyAdmin-4.5.3.1-english/.*.yml .
The phpMyAdmin-4.5.3.1-english is now empty, so you can remove it with rmdir instead of rm -rf (which would have let you know that it's not empty yet).
phpMyAdmin-4.5.3.1-english.zip is just a file; no need to recursively delete it, rm -f is enough.
Instead of the deprecated backticks for command substitution, you could use the more modern $():
randomBlowfishSecret=$(openssl rand -base64 32)
The sed can be improved in three ways:
No need for cat. cat file | sed "s/x/y/g" > output (replace all x in file with y, save to output) is equivalent to sed "s/x/y/g" file > output, but the latter doesn't spawn an extra subshell.
Your regular expression
s/cfg['blowfish_secret'] = ''/
is interpreted as "cfg, and the any ONE character from the list between [ and ]", but you want literal [ and ], so they have to be escaped: \[ and \]. In the replacement string, they don't have to be escaped.
The password generated by openssl rand can contain forward slashes, which confuses sed. You can use a different delimiter for sed, for example "s|x|y|" instead of "s/x/y/".
All of these are cosmetic, except the last two sed bullet points: those can break the script. Well, and the missing hidden files might be annoying, too.
Cleaned up version that works for me:
#!/bin/bash
wget -O phpMyAdmin-4.5.3.1-english.zip https://files.phpmyadmin.net/phpMyAdmin/4.5.3.1/phpMyAdmin-4.5.3.1-english.zip
unzip -q phpMyAdmin-4.5.3.1-english.zip
mv phpMyAdmin-4.5.3.1-english/* .
mv phpMyAdmin-4.5.3.1-english/.*.yml .
rmdir phpMyAdmin-4.5.3.1-english
rm -f phpMyAdmin-4.5.3.1-english.zip
randomBlowfishSecret=$(openssl rand -base64 32)
sed -e "s|cfg\['blowfish_secret'\] = ''|cfg['blowfish_secret'] = '$randomBlowfishSecret'|" config.sample.inc.php > config.inc.php

How should I sed files on the Yocto-generated rootfs?

I would like to find a way to run sed scripts on files within a Yocto-generated OS from a .bbappend file. My OS has a read-only rootfs, which seems to stop any possibility of a post-installation script. Specifically, I need to make these changes to /etc/default/ssh (as run after booting the generated OS):
sed -i 's/var\/run/etc/' /etc/default/ssh
sed -i 's/_readonly//' /etc/default/ssh
Here's my openssh_7.1p1.bbappend which I've created in an attempt to solve these problems:
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += " \
file://ssh_host_dsa_key.pub \
file://ssh_host_rsa_key.pub \
...
"
do_install_append() {
sed -i 's/var\/run/etc/' ${D}${sysconfdir}/default/ssh
sed -i 's/_readonly//' ${D}${sysconfdir}/default/ssh
# these lines work fine
install -m 0755 ${WORKDIR}/ssh_host_dsa_key ${D}/etc/ssh
install -m 0755 ${WORKDIR}/ssh_host_dsa_key.pub ${D}/etc/ssh
...
}
FILES_${PN} += "${sysconfdir}/default/ssh"
#these lines work
FILES_${PN} += "${sysconfdir}/ssh/ssh_host_dsa_key"
FILES_${PN} += "${sysconfdir}/ssh/ssh_host_dsa_key.pub"
...
BitBake fails during the execution of do_install_append() with this error:
sed: can't read ${TMPDIR}/work/x86-poky-linux/openssh/image/etc/default/ssh: No such file or directory
(where TMPDIR is my actual tmp directory) Obviously this file doesn't exist because the proper copy is created in a separate MULTIMACH_TARGET_SYS directory (i.e. not x86-poky-linux) by image.bbclass.
Is this sort of thing possible to do from within a .bbappend file (or some other compartmentalized way)? I have found a way to do it within a .inc file with ROOTFS_POSTPROCESS_COMMAND within my core-image, but this method is leading to a poor organizational structure.
Maybe moving the operations to a post-installation function in your .bbappend is appropriate for your circumstance?
pkg_postinst_${PN} () {
#!/bin/sh
if [ x"$D" = "x" ]; then
sed -i 's/var\/run/etc/' /etc/default/ssh
sed -i 's/_readonly//' /etc/default/ssh
else
exit 1
fi
}
The function above will cause the sed operations to execute on first boot, rather than during the build.
Include the file you're trying to sed in your SRC_URI variable.

Resources