breaking the parameters of shell script into several causes errors - linux

Question

I have a shell script using a phantom js function
The Phantom JS function crashes when it is run on too many urls apparently, I tried to rewrite it to only do part at a time but I get an error where the $url variable does not change so it always copies down the same url though it appears to write down the actual size of the site.
If I could be helped through that error that would be glorious.
#!/bin/bash
echo "Importing URLs..."
file=sizeurls.csv
url=""
while IFS= read -r line
do
url+=" "
url+=$line
done < "$file"
echo "Gathering page sizes..."
phantomjs yslow.js --info basic --format plain $url | grep 'url\|size' > temp.txt
echo "Formatting data..."
sed -i 's/size://g' temp.txt
sed -i 's/url://g' temp.txt
paste - - -d, < temp.txt > pagesize.csv
echo "Done!"
version that is supposed to do part at a time, it may be mortally screwed because I just messed with it a little bit an d I am not sure I returned it to the previous state
#!/bin/bash
echo "Importing URLs..."
file=sizeurls.csv
url=""
i=0;
while IFS= read -r line
do
while [ $i -le 10 ] #10 at a time i < 10
do
url+=" "
url+=$line
i=$((i+1));
done < "$file"
phantomjs yslow.js --info basic --format plain $url | grep 'url\|size' >> temp.txt
#echo "Formatting data..."
sed -i 's/size://g' temp.txt
sed -i 's/url://g' temp.txt
paste - - -d, < temp.txt >> pagesize.csv
done < "$file"
i = 0
echo "Done!"

Answer 1

Why not simply do one at a time?
#!/bin/bash
echo "Importing URLs..."
file=sizeurls.csv
echo "Gathering page sizes..."
while IFS= read -r url
do
phantomjs yslow.js --info basic --format plain $url | grep 'url\|size'
done < "$file" > temp.txt
echo "Formatting data..."
sed -i -e 's/size://g' -e 's/url://g' temp.txt
paste - - -d, < temp.txt > pagesize.csv
echo "Done!"

Answer 2

This may give you some ideas (untested, but looks about right). The processing is moved to a function where it is called every 10th URL, and again at the end if there are any leftover URLs.
#!/bin/bash
echo "Importing URLs..."
file=sizeurls.csv
rm pagesize.csv
ProcessURLs () {
echo "Gathering page sizes..."
phantomjs yslow.js --info basic --format plain $# | grep 'url\|size' > temp.txt
echo "Formatting data..."
sed -i 's/size://g' temp.txt
sed -i 's/url://g' temp.txt
paste - - -d, < temp.txt >> pagesize.csv
}
url=""
count=0
while IFS= read -r line
do
url+="$line$ "
(( count++ ))
# Procss URLs in 10-URL chunks
if [[ $count -ge 10 ]] ; then
ProcessURLs $url
url=''
count=0
fi
done < "$file"
# Handle any remaining URLs
[ -n "$url" ] && ProcessURLs $url
echo "Done!"