make my autodownloading shell script better - linux

Question

So I want to download multiple files from rapidshare. This what I currently have. I created a cookie by running-
wget \
--save-cookies ~/.cookies/rapidshare \
--post-data "login=USERNAME&password=PASSWORD" \
--no-check-certificate \
-O - \
https://ssl.rapidshare.com/cgi-bin/premiumzone.cgi \
> /dev/null
and now I have a shell script which I run which looks like this-
#!/bin/bash
wget -c --load-cookies ~/.cookies/rapidshare http://rapidshare.com/files/219920856/file1.rar
wget -c --load-cookies ~/.cookies/rapidshare http://rapidshare.com/files/393839302/file2.rar
wget -c --load-cookies ~/.cookies/rapidshare http://rapidshare.com/files/398293204/file3.rar
....
I want two things-
The shell script needs to read the files to download from a file.
The shell script should download anywhere from 2 - 8 files at a time.
Thanks!

Answer 1

When you want parallel jobs, think make.
#!/usr/bin/make -f
login:
wget -qO/dev/null \
--save-cookies ~/.cookies/rapidshare \
--post-data "login=USERNAME&password=PASSWORD" \
--no-check-certificate \
https://ssl.rapidshare.com/cgi-bin/premiumzone.cgi
$(MAKEFILES):
%: login
wget -ca$(addsuffix .log,$(notdir $#)) \
--load-cookies ~/.cookies/rapidshare $#
#echo "Downloaded $# (log in $(addsuffix .log,$(notdir $#)))"
Save this as rsget somewhere in $PATH (make sure you use tabs and not spaces for indentation), give it chmod +x, and run
rsget -kj8 \
http://rapidshare.com/files/219920856/file1.rar \
http://rapidshare.com/files/393839302/file2.rar \
http://rapidshare.com/files/398293204/file3.rar \
...
This will log in, then wget each target. -j8 tells make to run up to 8 jobs in parallel, and -k means "keep going even if a target returned failure".
Edit
Tested with GNU Make 3.79 and 3.81.

Answer 2

Try this. I think it should do what you want:
#! /bin/bash
MAX_CONCURRENT=8
URL_BASE="http://rapidshare.com/files/"
cookie_file=~/.cookies/rapidshare
# do your login thing here...
[ -n "$1" -a -f "$1" ] || { echo "please provide a file containing the stuff to download"; exit 1; }
inputfile=$1
count=0
while read x; do
if [ $count -ge $MAX_CONCURRENT ]; then
count=0
wait
fi
{ wget -c --load-cookies "$cookie_file" "${URL_BASE}$x" && echo "Downloaded $x"; } &
count=$((count + 1))
done < $inputfile