I have a shell script which needs to be run on paired files (e.g. file1 and file2). For each pair, there should be 5 output files generated in the current directory. How can I run a shell script on multiple paired files in different directories?
To be more specific, the structure looks like the following:
pair1: /x/y/z/1/file1 and x/y/z/1/file2
pair2: /x/y/z/2/file1 and x/y/z/2/file2
I want to run the same shell script on multiple pairs like these two. The output needs to be saved in the current directory, e.g. /x/y/z/1/ for pair1.
Run the script as:
myscript /x/y/z/1/file1 /x/y/z/1/file2 /x/y/z/2/file1 /x/y/z/2/file2
The script contains:
#!/bin/bash
create_five_files_with_pair() {
cp $1 outfile1
cp $2 outfile2
cat $1 $2 > outfile3
cat $2 $1 > outfile4
paste $1 $2 > outfile5
}
while test $# -gt 1
do
cd $(dirname $1)
create_five_files_with_pair $(basename $1) $(basename $2)
shift
shift
done
Related
So guys,
I need your help trying to identify the fastest and the most "fault" tolerant solution to my problem.
I have a shell script which executes some functions, based on a txt file, in which I have a list of files.
The list can contain from 1 file to X files.
What I would like to do is iterate over the content of the file and execute my scripts for only 4 items out of the file.
Once the functions have been executed for these 4 files, go over to the next 4 .... and keep on doing so until all the files from the list have been "processed".
My code so far is as follows.
#!/bin/bash
number_of_files_in_folder=$(cat list.txt | wc -l)
max_number_of_files_to_process=4
Translated_files=/home/german_translated_files/
while IFS= read -r files
do
while [[ $number_of_files_in_folder -gt 0 ]]; do
i=1
while [[ $i -le $max_number_of_files_to_process ]]; do
my_first_function "$files" & # I execute my translation function for each file, as it can only perform 1 file per execution
find /home/german_translator/ -name '*.logs' -exec mv {} $Translated_files \; # As there will be several files generated, I have them copied to another folder
sed -i "/$files/d" list.txt # We remove the processed file from within our list.txt file.
my_second_function # Without parameters as it will process all the files copied at step 2.
done
# here, I want to have all the files processed and don't stop after the first iteration
done
done < list.txt
Unfortunately, as I am not quite good at shell scripting, I do not know how to structure it so that it won't waste any resources and mostly, to make sure that it "processes" everything from that file.
Do you have any advice on how to achieve what I am trying to achieve?
only 4 items out of the file. Once the functions have been executed for these 4 files, go over to the next 4
Seems to be quite easy with xargs.
your_function() {
echo "Do something with $1 $2 $3 $4"
}
export -f your_function
xargs -d '\n' -n 4 bash -c 'your_function "$#"' _ < list.txt
xargs -d '\n' for each line
-n 4 take for arguments
bash .... - run this command with 4 arguments
_ - the syntax is bash -c <script> $0 $1 $2 etc..., see man bash.
"$#" - forward arguments
export -f your_function - export your function to environment so child bash can pick it up.
I execute my translation function for each file
So you execute your translation function for each file, not for each 4 files. If the "translation function" is really for each file with no inter-file state, consider rather executing 4 processes in parallel with same code and just xargs -P 4.
If you have GNU Parallel it looks something like this:
doit() {
my_first_function "$1"
my_first_function "$2"
my_first_function "$3"
my_first_function "$4"
my_second_function "$1" "$2" "$3" "$4"
}
export -f doit
cat list.txt | parallel -n4 doit
Hello this is purely practicing my cronjob
Here is my bash script below :
#!/bin/bash
grep $1 $2
rc=$?
if [ $rc != 0 ]
then
echo "specified string $1 not present in $2"
else
echo "specified string $1 is present in the file $2"
fi
# number of lines of in a file
wc -l $2 | awk '{print $1}'
Here is my crontab below :
20 16 * * * /home/mbarrett/Desktop/ ./find_string.sh sam text_string.file > /var/log/backupstring.log 2>&1
Any advice to what I may be doing wrong?
You have the Desktop directory at the beginning of your command, which will try to execute it as a program, which won't work. If you want to run the script from that directory, you need to execute the cd command to change to it.
20 16 * * * cd /home/mbarrett/Desktop/; ./find_string.sh sam text_string.file > /var/log/backupstring.log 2>&1
Also, get in the habit of always quoting your variables in the script. Your script as written won't work if the pattern or filename contains whitespace.
for file in $*
head -n 1 $file | while read folder
do
mkdir $directory $folder
done
Hello guys, I'm having problem with my script. What I want to do is: read first line from my specifed file and create new directories in my specifed directory from words that i have taken from that file.
I'm getting errors like this:
./scriptas: line 2: syntax error near unexpected token `head'
./scriptas: line 2: `head -n 1 $file | while read folder'
And my second question: how do I add a second variable from command line (putty) $directory ?
Example i have file with text:
one two three
five seven nine eleven
okey
i need script to take the first line and create directories "one" "two" "three"
You have to put do before the command in a for/while cycle.
Your code should look like something like this:
#!/bin/bash
files=$*
for file in $files
do
head -n1 "$file" | while read dname
do
mkdir $dname
done
done
as for other variables, the simple syntax is a number behind the $ sign.
so you could do
files="$1"
directory="$2"
and then run the script as
./script.sh "file1.txt file2.txt file3.txt" dir2
More complex solutions include getopts and such....
Updated the script. You can use it in this way:
script.sh "one.txt two.txt three.txt" destdir
#! /bin/bash
for files in $1
do
for i in $(head -n 1 $files)
do
if [ -z $2 ]
then
mkdir $i
else
mkdir $2/$i -p
fi
done
done
I want to rename files I have downloaded from the following script:
exec < input_list.txt
while read line
do
get $line
wget ftp://hgdownload.cse.ucsc.edu/goldenPath/hg19/encodeDCC/$2/$4
# Rename $4
mv $4 $1"_"$3".bam"
done
The input file (input_list.txt) is tab delimited and contains four columns. The first, $1= name, $2= wget address, $3= factor and $4 is the file name.
A549 wgEncodeBroadHistone H2azDex100nm wgEncodeBroadHistoneA549H2azDex100nmAlnRep1.bam
I want to rename $4 (the file that has been downloaded) to a shorter file name that only includes the corresponding $1 and $3 terms. For example, wgEncodeBroadHistoneA549H2azDex100nmAlnRep1.bam
becomes A549_H2azDex100nm.bam
I've played around with " but I keep getting error messages for the mv command and that $4 is a bad variable name. Any suggestions would be greatly appreciated.
You don't need to rename the file if you use wget's -O option:
#!/bin/bash
[ -n "$BASH_VERSION" ] || {
echo "You need Bash to run this script."
exit 1
}
while IFS=$'\t' read -a INPUT; do
wget -O "${INPUT[0]}_${INPUT[2]}.bam" "ftp://hgdownload.cse.ucsc.edu/goldenPath/hg19/encodeDCC/${INPUT[1]}/${INPUT[3]}"
done < input_list.txt
Make sure you save the file in UNIX file format like script.sh and run bash script.sh.
#!/bin/sh
#My script
echo "Are you sure you want to reorganize your files?"
echo "Type y or Y to continue. Anything else will stop the process"
read response
if [ "$response" = "y" ] || [ "$response" = "Y" ]; then
mkdir video
mkdir audio
mkdir text
mv -v *.txt text >> log.txt
mv -v *.wmv video >> log.txt
mv -v *.mov video >> log.txt
mv -v *.mpg video >> log.txt
mv -v *.mp3 audio >> log.txt
mv -v *.wma audio >> log.txt
echo "Yay, it worked!"
else
echo "Nothing happened."
fi
I wrote the script above to organize files into subfolders. For instance the music files will go into an audio folder. Now I would like to take a step further and make it more global.I would like to allow the script to accept a command line argument, which is the folder that contains the unorganized files. This should allow the script to be located and run from anywhere in the file system, and accept any folder of unorganized files.
Example:
organizefiles.sh mystuff/media // subfolders would go inside "media"
the folder media contains all of the media files.
Thank you!
A portion of your script could use the first positional parameter like this:
if [ -d $1 ]
then
mkdir video
mkdir audio
mkdir text
mv -v $1/*.txt text >> log.txt
mv -v $1/*.wmv video >> log.txt
mv -v $1/*.mov video >> log.txt
mv -v $1/*.mpg video >> log.txt
mv -v $1/*.mp3 audio >> log.txt
mv -v $1/*.wma audio >> log.txt
else
echo "The destination directory does not exist"
exit 1
fi
You can refer to the command line parameters as $1, $2, etc. The first one is $1. Here's a good description of how to pass arguments to a script: http://docsrv.sco.com:507/en/OSUserG/_Passing_to_shell_script.html
Scripts has access to arguments on the command line via some variables like this:
$1, $2, ..., $n - refers to first, second up to n arguments.
Example: Typing myscript.sh foo will set foo to the $1 variable.
Bash arguments are fairly straightforward, using a $# format. So for example, you could access the first argument of the command line from your script with $1
In your script, you could do something like so:
if [ -z $1 ]
then
dir = $1
else
dir = './'
fi
Then just add the new $dir variable to the paths in your mv commands. I recommend checking out Bash By Example from IBM. A great article series to teach you Bash.
Note that there may be a petter better to do what I suggested but I am nowhere near an expert in Bash. :-)
here's a simple system. you can use case/esac instead of if/else for neatness. also, rearranged the mv commands a bit
#!/bin/bash
dir=$1
cd $dir
while true
do
echo "Are you sure you want to reorganize your files?"
printf "Type y or Y to continue. Anything else will stop the process: "
read response
case "$response" in
y|Y )
mv -v *.txt text >> log.txt
for vid in "*.mov" "*.wmv" "*.mpg" "*.wma"
do
mv $vid video >> log.txt
done
echo "yay"
break;;
*) echo "Invalid choice";;
esac
done