When trying to do a simple script that tars the files and moves them to another directory i'am having problems implementing the recursive loop.
#!/bin/bash
for directorio in $1; do #my thought process: start for loop goes into dir
for file in $directorio; do #enter for loop for files
fwe="${file%.*}" #file_without_extension
tar -czPf "$fwe.tar.bz2" "$(pwd)" && mv "$fwe.tar.bz2" /home/cunha/teste
done
done
the script seems to be doing nothing ...
when the script is called like: ./script.sh /home/blablabla
How could i get this fixed?
You can better follow below option. How will it work? First it will list all the directories in short_list.txt. In while loop it will read each directory name and zip it in location /home/cunha/teste
find $1 -type d -name "*" > short_list.txt
cdr=$(pwd)
while read line
do
cd $line
base=$(basename $PWD)
cd ..
tar -cf /home/cunha/teste/$base.tar.gz $base
done < short_list.txt
cd $cdr
rm -rf short_list.txt
Related
I have ~10,000 directories. Most of them have a similarly named text file.
I would like to take these .txt files and copy them to a folder in the main directory, ALL_RESULTS. How can i accomplish this? What I have below
for d in *_directories/; do
#go into directory
cd "$d"
#check if file exists using wildcard, then copy it into ALL_RESULTS and print the name of
#directory out
if ls *SCZ_PGC3_GWAS.sumstats.gz*.txt 1> /dev/null 2>&1; then
cp *SCZ_PGC3_GWAS.sumstats.gz*.txt ../ALL_RESULTS && echo "$d"
#if file does not exist, print the name of the directory we're
#in
else
echo "$d"
echo "files do not exist"
cd ..
fi
done
I keep getting errors saying the directories themselves don't exist. What am I doing wrong?
All relative paths are interpreted relative to the directory you are in (the "current working directory"). So, imagine, you cd into the first directory. So now you are in that directory. Then you loop executes, and you try to cd into the second directory. But that directory is no longer then, you need to go "up" and then cd into the directory. That is the reason the directory does not exists - you have to go "up" a directory for each directory you cd into.
So you need to cd .. on the end of your loop to go back to the directory you started from.
I have ~10,000 directories. ... I would like to take these .txt files and move them to a folder in the main directory, ALL_RESULTS
If you don't need to output anything, just use find for that with a proper regex. Doing ls and cd and a loop will be very slow. Something along:
find . -maxdepth 2 -type f -regex '\./.*_directories/.*SCZ_PGC3_GWAS.sumstats.gz.*\.txt' -exec cp {} ALL_RESULTS \;
You can also add -v to cp to see what it copies.
You miss
shopt -s nullglob
and don't parse ls output :
#!/bin/bash
shopt -s nullglob
for d in *_directories/; do
# check if file exists using wildcard, then copy it into ALL_RESULTS and print
# the name of directory
files=$( $d/*SCZ_PGC3_GWAS.sumstats.gz*.txt )
if [[ ${files[#]} ]]; then
cp "${files[#]}" ALL_RESULTS && echo "$d"
#if file does not exist, print the name of the directory we're
#in
else
echo "$d"
echo "files do not exist"
fi
done
I am using Centos on a HPC to run my code. I typically have a folder that contains a run_calc File, which is what I want to run as:
qsub run_calc
Now I want to write a script "submit_all.sh" that submits all run_calc files in all the subfolders in their current directory and not from the from a parent folder where I runt the submit_all.sh script.
I found similar questions posted here Solution and here Solution2
that seems to be a partial answer to this question. I am not confident just submitting scripts until I found a solution which is why I ask:
In the second link I found this solution:
for i in {1..1000}; do
cd "$i"
qsub submit.sh
cd ..
done
were "i" was a list of folders with the names 1-100. Is it somehow possible to use find to create a list of all the subdirectories and path it to the for loop? How would i deal with subsubdirectories? Would I be able to change the cd .. statement such that I always go back to the parent folder directly in that case?
I fond this solution here: Solution
#!/bin/sh
for i in `find /var/www -type d -maxdepth 1 -mindepth 1`; do
cd $i
# do something here
done
But I do not understand what is going on? Is it possible to change the above script to the only dive into folders containing a run_calc File and also include subsubdirectries?
Thank you in advance
Assuming that you are using bash as your shell:
$ cat ./test.sh
#!/bin/bash
IFS=$'\n'
while read -r fname ;
do
pushd $(dirname "${fname}") > /dev/null
qsub run_calc
popd > /dev/null
done < <(find . -type f -name 'run_calc')
find . -type f -name 'run_calc' finds all paths to file run_calc inside the current directory and its subdirectories. This is input for the while loop.
pushd, popd are bash specific, and adds in or pops out of directory stack.
for d in `find . -type d`
do ( cd "$d"
if test ! -f run_calc; then continue; fi
qsub run_calc
) done
( commnds ) execute commands in a separate process and effect of cd does not "leak".
I have a problem with find command in shell script.
My script finds the archives then unpacks some logs from it and tar the results move it level up and remove file that were created in process.
And there is a problem it will do all operation even when script didn't find archive or logs as a result it creating empty tar. with name:
date_patern_patern.tar
i was trying fix it using if-statement or && and || operators but i can't handle it could you please direct me how to do it ?
I have modified the script:
#!/bin/bash
printf " -----\nSerch date: $1 \n"
printf "Serch patern: $2 \n"
printf "serch patern: $3 \n -----\n\n"
printf "Serch archives:\n"
mkdir /cc/xx/logs/aaa/envelop/tmp
find archives/ -name "$1_*_messages.tar.gz" -exec cp {} tmp \;
ls -l tmp/$1_*_messages.tar.gz || exit 1
cd tmp
tar -xf $1_*_messages.tar.gz --wildcards --no-anchored "*$2_$3*"
printf "Find envelop:\n"
ls -l alsb-message/ || exit 1
mv alsb-message $1_$2_$3
tar -cvf $1_$2_$3.tar $1_$2_$3
mv $1_$2_$3.tar ..
rm *.gz
rm -R $1_$2_$3
cd ..
rm -r tmp
there is another problem i want my script to stop when it serch and doesn't find patern in example when schript is serching:
./serch_script.sh 20151110 patern2 patern3
and it doesn't find either parern2 or patern3 i want it to stop but it do again empyt .tar i tried to do like Prasanna told but it didn't work in this case,
Please replace following line
ls -l tmp/
with following line
ls -l tmp/$1_*_messages.tar.gz || exit 1
Basically with this code, you are running exit 1 if there are no files in tmp directory matching your criteria.
Let's say I have a bunch of *.tar.gz files located in a hierarchy of folders. What would be a good way to find those files, and then execute multiple commands on it.
I know if I just need to execute one command on the target file, I can use something like this:
$ find . -name "*.tar.gz" -exec tar xvzf {} \;
But what if I need to execute multiple commands on the target file? Must I write a bash script here, or is there any simpler way?
Samples of commands that need to be executed a A.tar.gz file:
$ tar xvzf A.tar.gz # assume it untars to folder logs
$ mv logs logs_A
$ rm A.tar.gz
Here's what works for me (thanks to Etan Reisner suggestions)
#!/bin/bash # the target folder (to search for tar.gz files) is parsed from command line
find $1 -name "*.tar.gz" -print0 | while IFS= read -r -d '' file; do # this does the magic of getting each tar.gz file and assign to shell variable `file`
echo $file # then we can do everything with the `file` variable
tar xvzf $file
# mv untar_folder $file.suffix # untar_folder is the name of folder after untar
rm $file
done
As suggested, the array way is unsafe if file name contained space(s), and also doesn't seem to work properly in this case.
Writing a shell script is probably easiest. Take a look at sh for loops. You could use the output of a find command in an array, and then loop over that array to perform a set of commands on each element.
For example,
arr=( $(find . -name "*.tar.gz" -print0) )
for i in "${arr[#]}"; do
# $i now holds each of the filenames output by find
tar xvzf $i
mv $i $i.suffix
rm $i
# etc., etc.
done
in my script I receive as parameter $1 a .tgz file and I have to filter the size of its elements in a temporal directory, create a new one and rewrite the original.
If $1 is "~/Directory/File.tgz", I need to know hot to get to "~/Directory" so I can work with it.
This is my code:
dtemp=`mktemp -d ./tmpdirXXX`
cp $1 $dtemp #Copy
cd $dtemp
comprimido=`find ./ -name "*.tgz"`
tar xzvf $comprimido
rm $comprimido
for archivo in *
do
Tarchivo=`du -b "$archivo" | cut -f1`
if test 70192 -lt $Tarchivo
then
echo "$archivo es mayor de 8KB"
rm -r $archivo
fi
done
tar czvf $1 $dtemp
rm -r $dtemp
The last two lines don't work, it says that the file or directory doesn't exist.
Thanks for your help!
Your last two lines don't work because you cd'd to the temp directory, but never cd'd back.