How to make my script continue mirroring where it left off? - linux

Question

I'm creating a script to download and mirror a site, URLs are taken from a .txt file. The script is supposed to run daily for a few hours, so I need to get it to continue mirroring where it left off.
Here is the script:
# Created by Salik Sadruddin Merani
# email: ssm14293#gmail.com
# site: http://www.dragotech-innovations.tk
clear
echo ' Created by: Salik Sadruddin Merani'
echo ' email: ssm14293#gmail.com'
echo ' site: http://www.dragotech-innovations.tk'
echo
echo ' Info:'
echo ' This script will use the URLs provided in the File "urls.txt"'
echo ' Info: Logs will be saved in logfile.txt'
echo ' URLs are taken from the urls.txt file'
#
url=`< ./urls.txt`
useragent='Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:25.0) Gecko/20100101 Firefox/25.0'
echo ' Mozilla Firefox User agent will be used'
cred='log=abc#123.org&pwd=abc123&wp-submit=Log In&redirect_to=http://abc#123.org/wp-admin/&testcookie=1'
echo ' Loaded Credentails'
echo ' Logging In'
wget --save-cookies cookies.txt --post-data ${cred} --keep-session-cookies http://members.ebenpagan.com/wp-login.php --delete-after
OIFS=$IFS
IFS=','
arr2=$url
for x in $arr2
do
echo ' Loading Cookies'
wget --spider --load-cookies cookies.txt --keep-session-cookies --mirror --convert-links --page-requisites ${x} -U ${useragent} -np --adjust-extension --continue -e robots=no --span-hosts --no-parent -o log-file-$x.txt
done
IFS=$OIFS
Problems with the script:
The script is not referencing its links correctly by making it referable to the file in the parent directory, please tell me about that.
The script is not resuming after being aborted even with the --continue option.

Answer 1

A smarter way to solve the problem is, work with two .txt files, let's affectionately call them "to_mirror.txt" and "mirrored.txt". Keep each URL on a single line. Declare in your script a variable of value 0, for example total_mirrored=0, it will be very important in our code. Therefore, every time the wget command is executed and, consequently, the site is mirrored, increment the value of the "total_mirrored" variable by +1.
Upon exiting the loop, "total_mirrored" will have any integer value.
Then you must extract the lines from "to_mirror.txt" in the range: first line up to "total_mirrored"; then attach this to "mirrored.txt".
After that delete the range from the file "to_mirror.txt".
In this case the sed command can help you, see my example:
sed -n "1,$total_mirrored p" to_mirror.txt >> mirrored.txt && sed -i "1,$total_mirrored d" to_mirror.txt
You can learn a lot about the sed command by running man sed in your terminal, so I won't explain here what each option does as it's redundant.
But know that:
>> appends the existing file, or creates a file if the file of the mentioned name is not present in the directory. && A && B — run B only if A succeeded.

Answer 2

The --continue flag in wget will attempt to resume the downloading of a single file in the current directory. Please refer to the man page of wget for more info. It is quite detailed.
What you need is resuming the mirroring/downloading from where the script previously left off.
So, its more of a modification of script than some setting in wget. I can suggest a way to do that, but mind you, you can use a different approach as well.
Modify the urls.txt file to have one URL per line. Then refer this pseudocode:
get the URL from the file
if (URL ends with a token #DONE), continue
else, wget command
append a token #DONE to the end of the URL in the file
This way, you will know which URL to continue from, the next time you run the script. All URLs that have a "#DONE" at the end will be skipped, and the rest will be downloaded.