how to redirect the output file of a bash script to a desired folder - linux

Hello I'm running this bash script here and the command "bwa index .." will generate several files, and this command doesn't have the "-o" flag for specifying output path. I want to redirect these files into a new folder. I've searched online and all the answers I came across are redirecting files generated to a specific file name, but how to generally just redirect the output files into a new folder? Thanks very much!
#!/bin/bash
### qsub file.name to run from anywhere
### first step of BMW, making indexed fasta file
#PBS -N bwa_index
#PBS -S /bin/bash
#PBS -l walltime=24:00:00
#PBS -l nodes=1:ppn=8
#PBS -l mem=64gb
#PBS -o /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/bwa_index.out
#PBS -e /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/bwa_index.err
date
module load gcc/6.2.0
module load bwa/0.7.17
bwa index -p mm10_bwa_idx -a bwtsw /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/mm10.fa
date
echo END

You can change to the target directory and then run the bwa command. Using pushd and popd will ensure that rest of the script runs in correct directory.
pushd some_directory
bwa index -p mm10_bwa_idx -a bwtsw /gpfs/data/mcnerney-lab/liuweihan/chip_seq/becker_lab/mm10.fa
popd

First
You can define a variable and use it in your script.
Second
You can pass arguments to your script and capture them via $1, $2, $3 and so on.
Third if you wanted to be fully dynamic by using option ( e.g. --output | -o ) you can use a simple parse as bellow:
#!/bin/bash
################################################################################
# main flags, both longs and shorts
################################################################################
ARGS=`getopt -o "o::" -l "output:" -- "$#"`
eval set -- "$ARGS"
declare -A _output;
_output['flag']=0;
_output['path']=0;
################################################################################
# extract options and their arguments into variables.
################################################################################
while true ; do
case "$1" in
-o | --output )
_output['flag']=1;
_output['path']=$2;
shift 2;
;;
--)
shift;
break;
;;
* )
echo "Internal error!" ; exit 1
;;
esac
done
echo "output: ${_output['path']}";
And usage is super simple, you can use --output <YOUR-PATH> or -o <YOUR-PATH>
# test
./cli.sh --output /path/to/dir
output: /path/to/dir
./cli.sh -o /path/to/dir
output: /path/to/dir

Related

Increment the title of files output by a command in a shell script

I made this simple bash script to take a full-screen screenshot and save it to the pictures folder
#!/usr/bin/bash
xfce4-screenshooter -f -s /home/rgcodes/Pictures/Screenshot_$scrshotcount.png
let "scrshotcount++"
...which runs into a problem. scrshotcount is a global variable I defined in /etc/environment to be incremented every time the script runs. However, the script fails to increment the variable globally, and causes the script to just overwrite the previous screenshot. Searches on Google and Stack Overflow revealed that the problem isn't straightforward at all (something about child shells being unable to change variables for parents), and finding some other method would be better.
Here's my question. How do we append numbers (in ascending order) to the screenshots the script throws out so that they are saved just like those taken on Windows?(Windows auto-suffixes matching filenames, rather than overwriting them, so all Screenshots have the same name 'Screenshot' and the number of times the screenshot command has been used.)
I am using #erikMD's method as a temporary stop-gap for now.
In addition to the excellent advice about using a date instead of a counter, here's a way to use a counter :/
dir=$HOME/Pictures
# find the current maximum value
current_max=$(
find "$dir" -name Screenshot_\*.png -print0 \
| sort -z -V \
| tail -z -n 1
)
if [[ ! $current_max =~ _([0-9]+)\.png ]]; then
echo "can't find the screenshot with the maximum counter value" >&2
exit 1
fi
# increment it
counter=$(( 1 + ${BASH_REMATCH[1]} ))
# and use it
xfce4-screenshooter -f -s "$dir/Screenshot_${counter}.png"
You'll have to manually create the Screenshot_1.png file.
#rgcodes below is a script that will capture screenshots with a numeric count indicator per your original post. (tested it on Ubuntu 20.04)
Script contents:
#!/bin/bash
set -uo pipefail
# add source file and line number to xtrace output
# i.e. when running: bash -x ./your_script_name.sh
export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
capture_screenshot() {
local output_dir="${1:-/tmp/screenshot}"
local img_name="${2:-Screenshot}"
local img_ext="${3:-png}"
# create output directory (if not already existing)
mkdir -p "${output_dir}"
# get the last image in the sorted ls output
local latest_png=$(tail -n 1 \
<(sort -n -t _ -k 2 \
<(ls ${output_dir}/*.${img_ext} 2> /dev/null)))
# use the latest image to determine img_cnt value for next image
local img_cnt=0
if [[ ${latest_png} =~ _([0-9]+)\.png ]]; then
img_cnt=$((1+${BASH_REMATCH[1]}))
elif [[ ${latest_png} =~ ${img_name}.${img_ext} ]] ; then
img_cnt=1
fi
# build path to output image
local img_path="${output_dir}/${img_name}_${img_cnt}.${img_ext}"
# remove count from output image path if count == 0
if [[ "${img_cnt}" -eq "0" ]] ; then
img_path="${output_dir}/${img_name}.${img_ext}"
fi
xfce4-screenshooter -f -s "${img_path}"
}
capture_screenshot "$#"
The uses the following as defaults, but you can change them to meet your requirements.
output directory for screenshots:
/tmp/screenshot
base screenshot image name:
Screenshot
screenshot file extension:
.png
The script will attempt to create the output directory if it does not already exist (subject to user permission for creation). Below is a sample usage.
Prior to initial script execution, the output directory does not exist:
$ ls screenshot
$
Initial execution (directory is created and Screenshot.png created:
$ ./script.sh
$ ls /tmp/screenshot/
Screenshot.png
Subsequent executions:
$ ./script.sh
$ ls /tmp/screenshot/
Screenshot_1.png Screenshot.png
$ ./script.sh
$ ls /tmp/screenshot/
Screenshot_1.png Screenshot_2.png Screenshot.png
Indeed, as suggested by #j_b in the comments, you should definitely give a try to using a timestamp with the command date +"$format".
FTR, the same idea is implemented here in this project of a gnome-screenshot bash wrapper
(disclaimer: I am the author of this repo).
Example command:
date "+%Y-%m-%d_%H-%M-%S"
↓
2021-07-29_19-13-30
So the overall script could just be something like:
#!/usr/bin/env bash
xfce4-screenshooter -f -s "$HOME/Pictures/Screenshot_$(date "+%Y-%m-%d_%H-%M-%S").png"
(Note that I added missing double-quotes, and modified your shebang, as /usr/bin/env bash is more portable than /bin/bash or /usr/bin/bash.)

Bash unable to assign variable

#!/usr/bin/env bash
#SBATCH --partition=standard
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=20
#SBATCH --mem=100G
USEAGE="metascript.sh <wd> <wd1>"
source ~/anaconda2/etc/profile.d/conda.sh
conda activate assembly
wd=$1
wd1=$2
cd $wd
cd $wd1
for f in SRR*/ ; do
[[ -e $f ]] || continue
SRR=${f::-1}
cd ../..
jdid=$(sbatch -J FirstQC_$SRR ./pipelines/preprocessingbowtietrinity/FirstFastqc.sh $wd $wd1 $SRR)
#echo ${jdid[0]}|grep -o '[0-9]\+'
jobid=${jdid[0]}
jobid1=${jobid[0]}|grep -o '[0-9]\+'
#echo $jobid1
Hi all just having issues with my bash scripting, so I can print the line ${jdid[0]}|grep -o '[0-9]+' however when I assign it to a variable it is unable to return anything.
If the idea is to extract just the job ID from the output of sbatch, you can also use sbatch's --parsable argument. See here in the documentation.
jdid=$(sbatch --parsable -J FirstQC_$SRR ./pipelines/preprocessingbowtietrinity/FirstFastqc.sh $wd $wd1 $SRR)
and jdij will only contain the job ID if the cluster is not part of a federation.
jobid1=${jobid[0]}|grep -o '[0-9]\+'
I can print the line ${jdid[0]}|grep -o '[0-9]+' however when I assign it to a variable it is unable to return anything.
In order to assign the output of a pipeline to a variable or insert it into a command line for any other purpose, you have Command Substitution at hand:
jobid1=`echo ${jobid[0]}|grep -o '[0-9]\+'`
Of course, with bash this is better written as:
jobid1=`<<<${jobid[0]} grep -o '[0-9]\+'`
If the issue is printing the line ${jdid[0]}|grep -o '[0-9]+' as your question.
Just put the line in double quotation marks and it will work out.
Here is a little test i made:
jobid1="{jobid[0]}|grep -o '[0-9]\+'"
echo $jobid1
the out put is {jobid[0]}|grep -o '[0-9]\+'

Error when stacking SSH command arguments within a bash script using other scripts as variables

I have a csv file called addresses.csv which looks like this,
node-1,xx.xxx.xx.xx,us-central-a
....
node-9,xxx.xx.xxx.xx,us-east1-a
I have a script below called 0run.sh,
#!/bin/bash
username='user'
persist="bash /home/${username}/Documents/scripts/disk/persistentDisk.sh"
first="bash /home/${username}/Documents/scripts/disk/firstAttach.sh"
while IFS=, read -r int ip <&3; do
if [ "$int" == "node-1" ]; then
--->ssh -i ~/.ssh/key -o StrictHostKeyChecking=no -l ${username} ${ip} "${persist}; ${first}"<---
else
ssh -i ~/.ssh/key -o StrictHostKeyChecking=no -l ${username} ${ip} "${first}"
fi
done 3<addresses.csv
The error occurs in the part of the code where I drew the arrows.
When it runs on node-1, instead of running ..persistentDisk.sh followed by ..firstAttach.sh, it only runs ..persistentDisk.sh and gives me the following error before it runs ..persistentDisk.
bash: /home/user/Documents/scripts/disk/firstAttach.sh: No such file or directory
The rest of the script runs completely fine. The only error occurs at this one part where it misses the 2nd script.
When I run the command like this it runs fine.
ssh -i ~/.ssh/key -o StrictHostKeyChecking=no -l ${username} ${ext} "${first}"
When I run it like this, it runs fine as well.
ssh -i ~/.ssh/key -o StrictHostKeyChecking=no -l user xxx.xx.xxx.xx "bash /home/${username}/Documents/scripts/disk/persistentDisk.sh; bash /home/${username}/Documents/scripts/disk/firstAttach.sh"
When I run the command like with a \ before the ; to escape it like this,
ssh -i ~/.ssh/key -o StrictHostKeyChecking=no -l ${username} ${ext} "${persist}\; ${first}"
I get the following error, and neither scripts run within the node-1 part of the code, but the rest of the code's else loops run fine.
bash: /home/user/Documents/scripts/disk/persistentDisk.sh;: No such file or directory
Why can't I stack the 2 commands within the if statement in the ssh using variables?
If I clearly understand: your real problem consist to leave STDIN free for interaction in target host!
About read and redirection
Try using:
#!/bin/bash
username='user'
persist="bash /home/${username}/Documents/scripts/disk/persistentDisk.sh"
first="bash /home/${username}/Documents/scripts/disk/firstAttach.sh"
while IFS=, read -r -u $list int ip foo; do
if [ "$int" == "node-1" ]; then
echo CMD... $ip, $persist
else
[ "$ip" ] && echo CMD... $ip, $first
fi
done {list}<addresses.csv
Tested, this èroduce:
CMD... xx.xxx.xx.xx, bash /home/user/Documents/scripts/disk/persistentDisk.sh
CMD... xxx.xx.xxx.xx, bash /home/user/Documents/scripts/disk/firstAttach.sh
-u flag to read, tell to use file descriptor ${list} instead of STDIN
foo is some useless variable used to prevent rest of line to be stored in $ip (xx.xxx.xx.xx,us-central-a in this case)
{list}</path/to/filename create a new variable by finding any free file descriptor.
About ssh (and redirection)
You could use:
#!/bin/bash
username='user'
persist="/home/${username}/Documents/scripts/disk/persistentDisk.sh"
first="/home/${username}/Documents/scripts/disk/firstAttach.sh"
while IFS=, read -r -u $list int ip foo; do
[ "$int" = "node-1" ] && cmd=persist || cmd=first
[ "$ip" ] && ssh -i ~/.ssh/key -t -o StrictHostKeyChecking=no \
-l ${username} ${ext} /bin/bash "${!cmd}"
done {list}<addresses.csv
By using this syntax, you will keep STDIN free for script running on target host.

How to develop a Condition to close program only when log file has been updated in Bash Script [duplicate]

I want to run a shell script when a specific file or directory changes.
How can I easily do that?
You may try entr tool to run arbitrary commands when files change. Example for files:
$ ls -d * | entr sh -c 'make && make test'
or:
$ ls *.css *.html | entr reload-browser Firefox
or print Changed! when file file.txt is saved:
$ echo file.txt | entr echo Changed!
For directories use -d, but you've to use it in the loop, e.g.:
while true; do find path/ | entr -d echo Changed; done
or:
while true; do ls path/* | entr -pd echo Changed; done
I use this script to run a build script on changes in a directory tree:
#!/bin/bash -eu
DIRECTORY_TO_OBSERVE="js" # might want to change this
function block_for_change {
inotifywait --recursive \
--event modify,move,create,delete \
$DIRECTORY_TO_OBSERVE
}
BUILD_SCRIPT=build.sh # might want to change this too
function build {
bash $BUILD_SCRIPT
}
build
while block_for_change; do
build
done
Uses inotify-tools. Check inotifywait man page for how to customize what triggers the build.
Use inotify-tools.
The linked Github page has a number of examples; here is one of them.
#!/bin/sh
cwd=$(pwd)
inotifywait -mr \
--timefmt '%d/%m/%y %H:%M' --format '%T %w %f' \
-e close_write /tmp/test |
while read -r date time dir file; do
changed_abs=${dir}${file}
changed_rel=${changed_abs#"$cwd"/}
rsync --progress --relative -vrae 'ssh -p 22' "$changed_rel" \
usernam#example.com:/backup/root/dir && \
echo "At ${time} on ${date}, file $changed_abs was backed up via rsync" >&2
done
How about this script? Uses the 'stat' command to get the access time of a file and runs a command whenever there is a change in the access time (whenever file is accessed).
#!/bin/bash
while true
do
ATIME=`stat -c %Z /path/to/the/file.txt`
if [[ "$ATIME" != "$LTIME" ]]
then
echo "RUN COMMNAD"
LTIME=$ATIME
fi
sleep 5
done
Check out the kernel filesystem monitor daemon
http://freshmeat.net/projects/kfsmd/
Here's a how-to:
http://www.linux.com/archive/feature/124903
As mentioned, inotify-tools is probably the best idea. However, if you're programming for fun, you can try and earn hacker XPs by judicious application of tail -f .
Just for debugging purposes, when I write a shell script and want it to run on save, I use this:
#!/bin/bash
file="$1" # Name of file
command="${*:2}" # Command to run on change (takes rest of line)
t1="$(ls --full-time $file | awk '{ print $7 }')" # Get latest save time
while true
do
t2="$(ls --full-time $file | awk '{ print $7 }')" # Compare to new save time
if [ "$t1" != "$t2" ];then t1="$t2"; $command; fi # If different, run command
sleep 0.5
done
Run it as
run_on_save.sh myfile.sh ./myfile.sh arg1 arg2 arg3
Edit: Above tested on Ubuntu 12.04, for Mac OS, change the ls lines to:
"$(ls -lT $file | awk '{ print $8 }')"
Add the following to ~/.bashrc:
function react() {
if [ -z "$1" -o -z "$2" ]; then
echo "Usage: react <[./]file-to-watch> <[./]action> <to> <take>"
elif ! [ -r "$1" ]; then
echo "Can't react to $1, permission denied"
else
TARGET="$1"; shift
ACTION="$#"
while sleep 1; do
ATIME=$(stat -c %Z "$TARGET")
if [[ "$ATIME" != "${LTIME:-}" ]]; then
LTIME=$ATIME
$ACTION
fi
done
fi
}
Quick solution for fish shell users who wanna track a single file:
while true
set old_hash $hash
set hash (md5sum file_to_watch)
if [ $hash != $old_hash ]
command_to_execute
end
sleep 1
end
replace md5sum with md5 if on macos.
Here's another option: http://fileschanged.sourceforge.net/
See especially "example 4", which "monitors a directory and archives any new or changed files".
inotifywait can satisfy you.
Here is a common sample for it:
inotifywait -m /path -e create -e moved_to -e close_write | # -m is --monitor, -e is --event
while read path action file; do
if [[ "$file" =~ .*rst$ ]]; then # if suffix is '.rst'
echo ${path}${file} ': '${action} # execute your command
echo 'make html'
make html
fi
done
Suppose you want to run rake test every time you modify any ruby file ("*.rb") in app/ and test/ directories.
Just get the most recent modified time of the watched files and check every second if that time has changed.
Script code
t_ref=0; while true; do t_curr=$(find app/ test/ -type f -name "*.rb" -printf "%T+\n" | sort -r | head -n1); if [ $t_ref != $t_curr ]; then t_ref=$t_curr; rake test; fi; sleep 1; done
Benefits
You can run any command or script when the file changes.
It works between any filesystem and virtual machines (shared folders on VirtualBox using Vagrant); so you can use a text editor on your Macbook and run the tests on Ubuntu (virtual box), for example.
Warning
The -printf option works well on Ubuntu, but do not work in MacOS.

Counter to add up number of issues/failures for a summary at the end

I have a makefile like this:
default:
%:
#$(MAKE) -i -C subdir1 $*
#$(MAKE) -i -C subdir2 $*
#$(MAKE) -i -C subdir3 $*
#$(MAKE) -i -C subdir4 $*
#$(MAKE) -i -C subdir5 $*
The basic concept is that I have 5 (or more) sub projects which I will call make on sequentially. I use "-i" flag so that the make can continue to the end and the "-C dir" flag to call make in a sub-directory.
So, lets say that sub project 2 and 5 are failing, then at the end I want to be able to print something like:
3 projects built ok, 2 projects have errors.
So I think I want a counter of some sort, but I have no idea how I can set/increment it on an error. Any ideas?
As every call to $(MAKE) spawns its own subprocess, I can't think of a way to record these numbers easily with an ordinary make variable. You can, however, log the return value of each invocation to a (possibly hidden) file and then grep for your build stats like this:
errLog = .errLog
default:
%:
#$(MAKE) -i -C subdir1 $*; echo $$? > $(errLog)
#$(MAKE) -i -C subdir2 $*; echo $$? >> $(errLog)
#$(MAKE) -i -C subdir3 $*; echo $$? >> $(errLog)
#$(MAKE) -i -C subdir4 $*; echo $$? >> $(errLog)
#$(MAKE) -i -C subdir5 $*; echo $$? >> $(errLog)
#echo "`grep -c '^0' $(errLog)` built ok, `grep -c '^[^0]' $(errLog)` have errors."
Note that the first output redirection must be a single > to overwrite previous return codes in the file, while all others should be two > to not overwrite the file content.

Resources