I created a script to download file from URL and I want to download it in the specific directory but the problem is when its time in downloading it will not put to the directory given and also when extracting the file is in the given directory.
diskspace=$(df -h /var/ | sed '1d' | awk '{print $5}' | cut -d'%' -f1)
bundle=$(awk -F = '{print $2}' config.txt)
allowed=10
if [ "${diskspace}" -gt "${allowed}" ]; then
cd `/var/`
wget $bundle
else
echo "Not enough space to download the bundle"
echo $output
exit
fi
while true; do
for f in *.tar.gz; do
case $f in '*.tar.gz') exit 0;; esac
tar zxf "$f"
rm -v "$f"
done
done
Can Someone help me to this problem ? The thing that I want to happen is to download the file in the given directory and also extract it there. Help is greatly appreciated.
Related
I have a script that queries the Twitter API for several queries, and then writes the raw data to a file with the query in the name, plus a timestamp. I'd like to have a script that, given the list of query strings (regexs?) and for all files in a folder, if one of the query strings is a substring in that file, move it to a specific folder. Right now I have just a script with just a few dozen mv commands, but I'd like a simpler and more maintainable version. Here's an example of what I'm doing now:
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*femin*/home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*patriarchy* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*yesallwomen* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*womanpower* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
I would use a for loop:
for i in femin patriarchy yesallwomen womanpower; do
mv /home/nick/TwitterSearchToDatabase/queries_for_amita/*$i* /home/nick/TwitterSearchToDatabase/queries_for_amita/feminism
done
That way the list is in the first line so it is easy to amend.
I would isolate data (the words to be moved to feminism) and code.
When you have more keywords (feminism and so), you can make files with keywords and check these keywordfiles for the files you are considering to move.
With ${fromdir} where the files come from, ${todir} where you want them and ${keyfiledir} with the keywords, you get something like
for keyfile in ${keyfiledir}/*; do
key="${keyfile##*/}"
find $from -type f | sed 's#.*/##' | while read -r file; do
echo "${file}" | grep -q -f "${keyfiledir}"/"${key}" && mv "${from}"/"${file}" "${to}"/"${key}"
done
done
How does that work? I tested the solution above with the following script.
from=fromdir
to=todir
keyfiledir=keyfiledir
rm -rf ${from} ${to} ${keyfiledir}
mkdir ${from} ${to} ${keyfiledir}
mkdir ${to}/feminism ${to}/so
touch ${from}/yesallwomen ${from}/women ${from}/some_femin ${from}/"help move"
cat <<# > ${keyfiledir}/feminism
femin
patriarchy
yesallwomen
womanpower
#
touch ${from}/yesallwomen ${from}/women ${from}/some_femin
cat <<# > ${keyfiledir}/so
stack
exchange
help
#
test ! -d "${from}" && echo " Wrong dir ${from}" && exit 1
test ! -d "${to}" && echo " Wrong dir ${to}" && exit 1
test ! -d "${keyfiledir}" && echo " Wrong dir ${keyfiledir}" && exit 1
for keyfile in ${keyfiledir}/*; do
key="${keyfile##*/}"
find $from -type f | sed 's#.*/##' | while read -r file; do
echo "${file}" | grep -q -f "${keyfiledir}"/"${key}" && mv "${from}"/"${file}" "${to}"/"${key}"
done
done
echo "Not moved"
ls ${from}
echo "Moved"
ls -R ${to}
A simple combination of mv and egrep should suffice. egrep can take a pattern list from a file (and then you get to use full regexp syntax, not just glob syntax.) Make sure to exclude the name of the target folder.
cd /home/nick/TwitterSearchToDatabase/queries_for_amita
mv $(ls | egrep -f patterns.txt | grep -v '^feminism$') feminism
I created a script here and the script is all about checking the disk space if the disk space had enough space then it will download the file needed after that it will extract the file 5 times and it must not extract the last tar.gz file . But when I executed the script I got a problem here. For the extract side it will just extract 1 time but I already out it 5 times.
#!/bin/bash
outout=$(df -h)
file=$(awk -F = '{print $2}' config.txt)
filename=$(echo "$bundle" | awk -F / '{print $11}')
diskspace=$(df -h /var/ | sed '1d' | awk '{print $5}' | cut -d'%' -f1)
allowed=10
address=$(awk -F = '{print $2}' newconfig.txt)
if [ "${diskspace}" -gt "${allowed}" ]; then
# cd "/var/"
wget $bundle
else
echo "Not enough space to download the bundle"
echo $output
exit
fi
i=0
for tarfile in *.tar.gz
do
$(($i++))
[ $i = 5 ] && break
tar -xf "$tarfile"
done
For the config.txt
https://www.google.com.ph/files.tar.gz
files.tar.gz is composed of 5 tar inside
files1.tar.gz -> files2.tar.gz - > files3.tar.gz -> files4.tar.gz - > files5.tar.gz
What should I do to extract it five times until files5.tar.gz will display and it must not be extracted.
Help is greatly appreciated.
Since I couldn't figure out the exact code you wan't,
I would give a example code how to use while loop instead
i=0
tar xf files.tar.gz
while [ $i -lt 4 ];do
((i++))
tar xf files${i}.tar.gz
done
Or use for to do the same thing.
for i in "" {1..4};do
tar xf files${i}.tar.gz
done
These code executes following command
tar xf files.tar.gz
tar xf files1.tar.gz
tar xf files2.tar.gz
tar xf files3.tar.gz
tar xf files4.tar.gz
please see my code below : The code above below shows that first checking the disk space available in the specific path and it must have the greater space than the allotted space and after checking the disk space it will download the file through using the "wget" and after that it will extract all tar.gz files. As I execute my script, error's occur and it displayed : blah.blah.blah.tar.gz: Scheme missing.
#!/bin/bash
bundle=$(awk -F = '{print $2}' config.txt)
bundlename=$(echo "$bundle" | awk -F / '{print $11}')
diskspace=$(df -h /dev/shm | sed '1d' | awk '{print $5}' | cut -d'%' -f1)
allowed=0
if [ "${diskspace}" -gt "${allowed}" ]; then
wget -A "$bundle"
for file in *.tar.gz; do
gunzip -c "$file" | tar xf -
done
rm -vf "$file"
else
echo "Not enough space to download the bundle"
exit
fi
My question here. What does error means and can you correct my codes if you feel it's wrong ? thank you for the help.
This is my first ever bash script and I am trying to iron out all of the creases and make the script run nicely. The script is to archive all of the specified .jpg files that it finds in multiple directories on a HDD/Flash drive. There are files with the same name but different content so I have used an Md5 sum to hash them.
I am getting the directory does not exist error in Geany but it runs fine from command bar missing out two of the images. I have tried everything I can think of to fix it. Is it messy code that is doing this?
#!/bin/sh
if [ ! -d "$1" ]; then
echo Directory "$1" cannot be found. Please try again.
exit
fi
if [ $# -eq 1 ]; then
echo "usage: Phar image_path archive_path"
exit
fi
if [ -d "$2" ]; then
echo "archive exists"
else
echo "the directory 'archive' does't exist. Creating directory 'archive'."
mkdir -p ~/archive
fi
find $1 -iname "IMG_[0-9][0-9][0-9][0-9].JPG" | cat > list.txt
[ -f ~/my-documents/md5.txt ] && rm md5.txt || break
while read line;
do md5sum $line | xargs >> md5.txt
done < list.txt
sort -k 1,1 -u md5.txt | cat > uniquemd5.txt
cut -d " " -f 2- uniquemd5.txt > uniquelist.txt
sort uniquelist.txt -r -o uniquelist.txt
for line in $(cat uniquelist.txt)
do
file=$(basename $line) path="$2/file"
if [ ! -f $path ];
then
cp $line $2
else
cp $line $path.JPG
fi
done
You haven't guarded against spaces in the folder and file names everywhere.
For instance:
cp $line $2
should be:
cp "$line" "$2"
You should start by eliminating these spaces as a source to your error by evaluating each variable you are referencing and adding ""'s.
If you still get the error please provide us with the arguments used and which directory that does not exist.
I run bash scripts from time to time on my servers, I am trying to write a script that monitors log folders and compress log files if folder exceeds defined capacity. I know there are better ways of doing what I am currently trying to do, your suggestions are more than welcome. The script below is throwing an error "unexpected end of file" .Below is my script.
dir_base=$1
size_ok=5000000
cd $dir_base
curr_size=du -s -D | awk '{print $1}' | sed 's/%//g' zipname=archivedate +%Y%m%d
if (( $curr_size > $size_ok ))
then
echo "Compressing and archiving files, Logs folder has grown above 5G"
echo "oldest to newest selected."
targfiles=( `ls -1rt` )
echo "rocess files."
for tfile in ${targfiles[#]}
do
let `du -s -D | awk '{print $1}' | sed 's/%//g' | tail -1`
if [ $curr_size -lt $size_ok ];
then
echo "$size_ok has been reached. Stopping processes"
break
else if [ $curr_size -gt $size_ok ];
then
zip -r $zipname $tfile
rm -f $tfile
echo "Added ' $tfile ' to archive'date +%Y%m%d`'.zip and removed"
else [ $curr_size -le $size_ok ];
echo "files in $dir_base are less than 5G, not archiving"
fi
Look into logrotate. Here is an example of putting it to use.
With what you give us, you lack a "done" to end the for loop and a "fi" to end the main if. Please reformat your code and You will get more precise answers ...
EDIT :
Looking at your reformatted script, it is as said : The "unexpected end of file" comes from the fact you have not closed your "for" loop neither your "if"
As it seems that you mimick the logrotate behaviour, check it as suggested by #Hank...
my2c
My du -s -D does not show % sign. So you can just do.
curr_size=$(du -s -D)
set -- $curr_size
curr_size=$1
saves you a few overheads instead of du -s -D | awk '{print $1}' | sed 's/%//g.
If it does show % sign, you can get rid of it like this
du -s -D | awk '{print $1+0}'. No need to use sed.
Use $() syntax instead of backticks whenever possible
For targfiles=(ls -1rt) , you can omit the -1. So it can be
targfiles=( $(ls -rt) )
Use quotes around your variables whenever possible. eg "$zipname" , "$tfile"