I would like to find a way to run sed scripts on files within a Yocto-generated OS from a .bbappend file. My OS has a read-only rootfs, which seems to stop any possibility of a post-installation script. Specifically, I need to make these changes to /etc/default/ssh (as run after booting the generated OS):
sed -i 's/var\/run/etc/' /etc/default/ssh
sed -i 's/_readonly//' /etc/default/ssh
Here's my openssh_7.1p1.bbappend which I've created in an attempt to solve these problems:
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += " \
file://ssh_host_dsa_key.pub \
file://ssh_host_rsa_key.pub \
...
"
do_install_append() {
sed -i 's/var\/run/etc/' ${D}${sysconfdir}/default/ssh
sed -i 's/_readonly//' ${D}${sysconfdir}/default/ssh
# these lines work fine
install -m 0755 ${WORKDIR}/ssh_host_dsa_key ${D}/etc/ssh
install -m 0755 ${WORKDIR}/ssh_host_dsa_key.pub ${D}/etc/ssh
...
}
FILES_${PN} += "${sysconfdir}/default/ssh"
#these lines work
FILES_${PN} += "${sysconfdir}/ssh/ssh_host_dsa_key"
FILES_${PN} += "${sysconfdir}/ssh/ssh_host_dsa_key.pub"
...
BitBake fails during the execution of do_install_append() with this error:
sed: can't read ${TMPDIR}/work/x86-poky-linux/openssh/image/etc/default/ssh: No such file or directory
(where TMPDIR is my actual tmp directory) Obviously this file doesn't exist because the proper copy is created in a separate MULTIMACH_TARGET_SYS directory (i.e. not x86-poky-linux) by image.bbclass.
Is this sort of thing possible to do from within a .bbappend file (or some other compartmentalized way)? I have found a way to do it within a .inc file with ROOTFS_POSTPROCESS_COMMAND within my core-image, but this method is leading to a poor organizational structure.
Maybe moving the operations to a post-installation function in your .bbappend is appropriate for your circumstance?
pkg_postinst_${PN} () {
#!/bin/sh
if [ x"$D" = "x" ]; then
sed -i 's/var\/run/etc/' /etc/default/ssh
sed -i 's/_readonly//' /etc/default/ssh
else
exit 1
fi
}
The function above will cause the sed operations to execute on first boot, rather than during the build.
Include the file you're trying to sed in your SRC_URI variable.
Related
I am working on a Red Hat Linux server. My end goal is to run CRB-BLAST on multiple fasta files and have the results from those in separate directories.
My approach is to download the fasta files using wget then run the CRB-BLAST. I have multiple files and would like to be able to download them each to their own directory (the name perhaps should come from the URL list files), then run the CRB-BLAST.
Example URLs:
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_3370_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_CB_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_13_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_37_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_123_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_195_chr.v0.1.liftover.CDS.fasta.gz
http://assemblies/Genomes/final_assemblies/10x_assemblies_v0.1/TC_31_chr.v0.1.liftover.CDS.fasta.gz
Ideally, the file name determines the directory name, for example, TC_3370/.
I think there might be a solution with cat URL.txt | mkdir | cd | wget | crb-blast
Currently I just run the commands in line:
mkdir TC_3370
cd TC_3370/
wget url
http://assemblies/Genomes/final_assemblies/10x_meta_assemblies_v1.0/TC_3370_chr.v1.0.maker.CDS.fasta.gz
crb-blast -q TC_3370_chr.v1.0.maker.CDS.fasta.gz -t TCV2_annot_cds.fna -e 1e-20 -h 4 -o rbbh_TC
Try this Shellcheck-clean program:
#! /bin/bash -p
while read -r url; do
file=${url##*/}
dir=${file%%_chr.*}
mkdir -v -- "$dir"
(
cd "./$dir" || exit 1
wget -- "$url"
crb-blast -q "$file" -t TCV2_annot_cds.fna -e 1e-20 -h 4 -o rbbh_TC
)
done <URL.txt
See Removing part of a string (BashFAQ/100 (How do I do string manipulation in bash?)) for an explanation of ${url##*/} etc.
The subshell (( ... )) is used to ensure that the cd doesn't affect the main program.
Another implementation
#!/bin/sh
# Read lines as url as long as it can
while read -r url
do
# Get file name by stripping-out anything up to the last / from the url
file_name=${url##*/}
# Get the destination dir name by stripping anything from the first __chr
dest_dir=${file_name%%_chr*}
# Compose the wget output path
fasta_path="$dest_dir/$file_name"
if
# Successfully created the destination directory AND
mkdir -p -- "$dest_dir" &&
# Successfully downloaded the file
wget --output-file="$fasta_path" --quiet -- "$url"
then
# Process the fasta file into fna
fna_path="$dest_dir/TCV2_annot_cds.fna"
crb-blast -q "$fasta_path" -t "$fna_path" -e 1e-20 -h 4 -o rbbh_TC
else
# Cleanup remove destination directory if any of mkdir or wget failed
rm -fr -- "$dest_dir"
fi
# reading from the URL.txt file for the whole while loop
done < URL.txt
Download files from list is task for -i file option, if you have file named say urls.txt with one URL per line you might simply do
wget -i urls.txt
Note that this will put all files inside current working directory, so if you wish to have them in separate dirs, you would need to move them after wget finish.
I'd like to execute the following command for several files in same repository in linux:
../../../../../openSMILE-2.1.0/SMILExtract -C ../../../../../openSMILE-2.1.0/config/IS13_ComParE.conf -I inputfilename.wav -D outputfilename.csv
there are several files (named 1.wav, 2.wav, 3.wav) in the directory and if I execute
../../../../../openSMILE-2.1.0/SMILExtract -C ../../../../../openSMILE-2.1.0/config/IS13_ComParE.conf -nologfile 1 -noconsoleoutput 1 -I 1.wav -D 1.csv
it outputs 1.csv.
How can I create 1.csv, 2.csv, 3.csv, .. by executing just one single command in linux? (or do I have to make .sh file?)
It's probably cleaner to put the following to a script, but you can type it directly into the bash command line as well:
#! /bin/bash
for file in *.wav ; do
prefix=${file%.wav} # Remove from the right.
../../../../../openSMILE-2.1.0/SMILExtract \
-C ../../../../../openSMILE-2.1.0/config/IS13_ComParE.conf \
-I "$file" -D "$prefix".csv
done
I am working on a bash script that automatically downloads phpMyAdmin and extracts it. I would like to add one more step to this installer script.
Copy config.sample.inc.php as config.inc.php and update this file's line with a random blowfish secret:
$cfg['blowfish_secret'] = ''; /* YOU MUST FILL IN THIS FOR COOKIE AUTH! */
So, this is what I have I have tried:
#!/bin/bash
wget -O phpMyAdmin-4.5.3.1-english.zip https://files.phpmyadmin.net/phpMyAdmin/4.5.3.1/phpMyAdmin-4.5.3.1-english.zip;
unzip phpMyAdmin-4.5.3.1-english.zip >/dev/null 2>/dev/null;
cd phpMyAdmin-4.5.3.1-english;
mv * ..;
cd ..;
rm -rf phpMyAdmin-4.5.3.1-english;
rm -rf phpMyAdmin-4.5.3.1-english.zip;
randomBlowfishSecret=`openssl rand -base64 32`;
cat config.sample.inc.php | sed -e "s/cfg['blowfish_secret'] = ''/cfg['blowfish_secret'] = '$randomBlowfishSecret'/" > config.inc.php
When this script runs, phpMyAdmin is downloaded and extracted and the file is copied, however it does not appear to be setting the randomBlowfishSecret to $cfg['blowfish_secret'].
Any ideas?
A few points:
You don't have to end your lines with ; – a newline has the same effect.
If you want to redirect both stdout and stderr, you can use &>/dev/null instead of >/dev/null 2>/dev/null, but in the case of unzip, you can just use unzip -q to suppress output (or even -qq, but -q was already silent for me).
Instead of
cd phpMyAdmin-4.5.3.1-english;
mv * ..;
cd ..;
you can just use mv phpMyAdmin-4.5.3.1-english/* .
There are two files starting with ., which aren't moved with your command (unless you have the dotglob shell option set), so you have to move them separately:
mv phpMyAdmin-4.5.3.1-english/.*.yml .
The phpMyAdmin-4.5.3.1-english is now empty, so you can remove it with rmdir instead of rm -rf (which would have let you know that it's not empty yet).
phpMyAdmin-4.5.3.1-english.zip is just a file; no need to recursively delete it, rm -f is enough.
Instead of the deprecated backticks for command substitution, you could use the more modern $():
randomBlowfishSecret=$(openssl rand -base64 32)
The sed can be improved in three ways:
No need for cat. cat file | sed "s/x/y/g" > output (replace all x in file with y, save to output) is equivalent to sed "s/x/y/g" file > output, but the latter doesn't spawn an extra subshell.
Your regular expression
s/cfg['blowfish_secret'] = ''/
is interpreted as "cfg, and the any ONE character from the list between [ and ]", but you want literal [ and ], so they have to be escaped: \[ and \]. In the replacement string, they don't have to be escaped.
The password generated by openssl rand can contain forward slashes, which confuses sed. You can use a different delimiter for sed, for example "s|x|y|" instead of "s/x/y/".
All of these are cosmetic, except the last two sed bullet points: those can break the script. Well, and the missing hidden files might be annoying, too.
Cleaned up version that works for me:
#!/bin/bash
wget -O phpMyAdmin-4.5.3.1-english.zip https://files.phpmyadmin.net/phpMyAdmin/4.5.3.1/phpMyAdmin-4.5.3.1-english.zip
unzip -q phpMyAdmin-4.5.3.1-english.zip
mv phpMyAdmin-4.5.3.1-english/* .
mv phpMyAdmin-4.5.3.1-english/.*.yml .
rmdir phpMyAdmin-4.5.3.1-english
rm -f phpMyAdmin-4.5.3.1-english.zip
randomBlowfishSecret=$(openssl rand -base64 32)
sed -e "s|cfg\['blowfish_secret'\] = ''|cfg['blowfish_secret'] = '$randomBlowfishSecret'|" config.sample.inc.php > config.inc.php
I need to touch a file with an absolute file name such as: /opt/test/test.txt, but I'm not sure if there is /opt/test existed on the system. So the code should similar with this:
if (-d '/opt/test') {
touch '/opt/test/test.txt';
} else {
mkdir -p '/opt/test';
touch '/opt/test/test.txt'
}
Is there any better way to simplify the code? I hope there is some system commands that can do the same job with only one line.
mkdir B && touch B/myfile.txt
Alternatively, create a function:
mkfile() {
mkdir -p $( dirname "$1") && touch "$1"
}
Execute it with 1 arguments: filepath. Saying:
mkfile B/C/D/myfile.txt
would create the file myfile.txt in the directory B/C/D.
In a shell script, you can simply do:
mkdir -p /opt/test && touch /opt/test/test.txt
mkdir -p will not fail (and won't do anything) if the directory already exists.
In perl, use make_path from the File::Path module, then create the file however you want. make_path also doesn't do anything if the directory exists already, so no need to check yourself.
In perl, using one of my favorite module: Path::Tiny.
path("/opt/test/test.txt")->touchpath;
From the doc:
Combines mkpath and touch. Creates the parent directory if it doesn't
exist, before touching the file.
I like typing very little, so I put this command into a named fn in my .profile, but I used this formulation for years before I did it:
mkdir -p dirname/sub/dir && touch $_/filename.ext
The variable $_ stores the last argument to the previous command. Pretty handy to know about overall.
I defined a touchp in my ~/.bash_aliases:
function touchp() {
/bin/mkdir -p "$(dirname "$1")/" && /usr/bin/touch "$1"
}
It silently creates the structure above the file if not present, and is perfectly safe to use when passed a single filename without any directory in front of it.
Perl from command line,
perl -MFile::Basename -MFile::Path=make_path -e'
make_path(dirname($_)), open(F, ">>", $_) for pop;
' /opt/test/test.txt
I have this shell function in my .zshalias file:
function touch-safe {
for f in "$#"; do
[ -d $f:h ] || mkdir -p $f:h && command touch $f
done
}
alias touch=touch-safe
If either the test or the mkdir command fail, no touch command is invoked.
Bring Python to command line.
i.e. Use pyp
cat filepaths.txt | pyp "'mkdir -p '+s[0:-1]|s+'; touch '+o" | sh
The Pyed Piper", or pyp, is a linux command line text manipulation tool similar to awk or sed, but which uses standard python string and list methods as well as custom functions evolved to generate fast results in an intense production environment.
If I put comments (# ...) in my Makefile, make gives me an error and quit. If I remove the comments, the makefile works fine.
Makefile:1: *** missing separator. Stop.
Make-version: 3.81
Linux: Ubuntu 9.04
The Makefile:
# Backup Makefile
#
# Create backups from various services and the system itself. This
# script is used to perform single backup tasks or a whole backup
# from the system. For more information about this file and how to
# use it, read the README file in the same directory.
BACKUP_ROOT = /srv/backup
ETC_PATH = /srv/config
SVN_PATH = /srv/svn/
TRAC_PATH = /srv/trac/sysinventory
PR10_PATH = /swsd/project/vmimages/...
PR10_MOUNT_PATH = /tmp/temp_sshfs_pr10
MYSQL_USER = "xxx"
MYSQL_PASSWORD = "xxx"
DATE = `date +%F`
help :
cat README
init-environment :
mkdir -p $(BACKUP_ROOT)
mkdir $(BACKUP_ROOT)/tmp
mkdir -p $(PR10_MOUNT_PATH)
backup : backup-mysql backup-configuration backup-svn backup-trac
upload-to-pr10 : mount-pr10
tar cf $(DATE)-backup-blizzard.tar -C $(BACKUP_ROOT) *.-backup.tar.gz
mv $(BACKUP_ROOT)/*-backup-blizzard.tar $(PR10_MOUNT_PATH)/
umount $(PR10_MOUNT_PATH)
mount-pr10 :
su xxx -d "sshfs -o allow_root xxx#xxx:$(PR10_PATH) $(PR10_MOUNT_PATH)"
fusermount -u $(PR10_MOUNT_PATH)
backup-mysql :
mysqldump --comments --user=$(MYSQL_USER) --password=$(MYSQL_PASSWORD) --all-databases --result-file=$(BACKUP_ROOT)/tmp/mysql_dump.sql
tar czf $(BACKUP_ROOT)/$(DATE)-mysql-backup.tar.gz -C
$(BACKUP_ROOT)/tmp/mysql_dump.sql
backup-configuration :
tar czf $(BACKUP_ROOT)/$(DATE)-configuration-backup.tar.gz $(ETC_PATH)/
backup-svn :
svnadmin dump $(SVN_PATH)/repository > $(BACKUP_ROOT)/tmp/svn_repository.dump
tar czf $(BACKUP_ROOT)/$(DATE)-subversion-backup.tar.gz -C $(BACKUP_ROOT)/tmp/svn_repository.dump
backup-trac :
tar czf $(BACKUP_ROOT)/$(DATE)-trac-backup.tar.gz $(TRAC_PATH)/
clean :
rm -f $(BACKUP_ROOT)/tmp/mysql_dump.sql
rm -f $(BACKUP_ROOT)/tmp/svn_repository.dump
rm -f $(BACKUP_ROOT)/*-backup.tar.gz
rm -f $(BACKUP_ROOT)/*-backup-blizzard.tar
Your Makefile works for me (with spaces replaced by tabs), so it sounds like you have a case of stray non-printing chars.
Try inspecting the output of "cat -vet Makefile". That will show where EOL, TAB and other unseen chars are.
You'll want to see something like this:
# Backup Makefile$
#$
# Create backups from various services and the system itself. This$
# script is used to perform single backup tasks or a whole backup$
# from the system. For more information about this file and how to$
# use it, read the README file in the same directory.$
$
BACKUP_ROOT = /srv/backup$
ETC_PATH = /srv/config$
SVN_PATH = /srv/svn/$
TRAC_PATH = /srv/trac/sysinventory$
PR10_PATH = /swsd/project/vmimages/...$
PR10_MOUNT_PATH = /tmp/temp_sshfs_pr10$
$
MYSQL_USER = "xxx"$
MYSQL_PASSWORD = "xxx"$
$
$
DATE = `date +%F`$
$
help :$
^Icat README$
$
$
init-environment :$
^Imkdir -p $(BACKUP_ROOT)$
^Imkdir $(BACKUP_ROOT)/tmp$
^Imkdir -p $(PR10_MOUNT_PATH)$
$
Make sure all commands are preceeded by "^I".
You could also try to looking for stray chars using something like:
cat -vet Makefile | grep "\^[^I]" --colour=auto
You may have used spaces instead of tabs for your comment.
Please post the makefile so we don't have to guess.