How to generate sequential dates/times in a shell script? - linux

Question

I would like to download a bunch of data sets from 1981 to 2000 (20 years). Those are in every 10 minutes. I was trying to write a script which will call all times and download the data. But I am unable to complete it. I can't check the leap years and days in each month. My script is:
#!/bin/sh
for yr in {1981..2000};do
for mm in 01 02 03 04 05 06 07 08 09 10 11 12;do
for dd in {1..31};do
if [[ $dd -le 9 ]];then nn=0$dd;else nn=$dd;fi
for tt in 00 10 20 30 40 50; do
echo wget www.xyz.com/$yy/$mm/$nn/$tt.txt
done;
done;
done;
done
How can I fix the problems of leap years, and days in the month generally?

Answer 1

You seem to have left out the hours.
Assuming you have GNU date, you can deal with it by using the date calculations. Do you have to worry about switches between winter and summer (standard and daylight saving) time? If so, there'll be some entertainment to be had with gaps of an hour in the spring and a period in the fall when the raw date/time values repeat.
$ /opt/gnu/bin/date -d '1981-01-01 00:00:00' +'%s %Y-%m-%d %H:%M:%S'
347184000 1981-01-01 00:00:00
$ /opt/gnu/bin/date -d '2000-12-31 23:50:00' +'%s %Y-%m-%d %H:%M:%S'
978335400 2000-12-31 23:50:00
$
That gives you start and end times in Unix timestamp notation (and in the US/Pacific time zone — adjust to suit your needs). You could then use a loop such as:
now=347184000
end=978335400
while [ "$now" -le "$end" ]
do
url=$(date -d "#$now" +'www.example.com/%y/%m/%d/%H/%M.txt')
echo wget "$url"
now=$(($now + 600))
done
There are multiple ways of writing that. I've assumed that there's a directory of hourly files, and within that the 10-minute files, but you can tweak the format to suit your requirements. The use of # in the -d is crucial.
You might prefer to use a scripting language such as Perl or Python instead of repeatedly invoking date as shown.
Note that you have a vast number of files to collect. With about 31 million seconds per year, and 600 seconds per 10 minute interval, you're looking at over 50,000 files per year for 20 years, or 1 million files in total. The target (victim) web site might not be happy with you running that flat out. You'd probably need to pace the retrieval operations — check their terms and conditions.

Answer 2

This is how it can be (please note that this leap year calculation is good until 2100 only):
#!/bin/sh
for yr in {1981..2000};do
for mm in 1 2 3 4 5 6 7 8 9 10 11 12;do
for dd in {1..31};do
if [[ $dd -eq 31 ]] && ( [[ $mm -eq 4 ]] || [[ $mm -eq 6 ]] || [[ $mm -eq 9 ]] || [[ $mm -eq 11 ]] )
then
continue
elif ( [[ $dd -gt 28 ]] && [[ $mm -eq 2 ]] && [[ $(( $yr % 4 )) -ne 0 ]] ) || ([[ $dd -gt 29 ]] && [[ $mm -eq 2 ]] )
then
continue
fi
if [[ $mm -le 9 ]];then mon=0$mm;else mon=$mm;fi
if [[ $dd -le 9 ]];then nn=0$dd;else nn=$dd;fi
for tt in 00 10 20 30 40 50; do
echo wget www.xyz.com/$yy/$mon/$nn/$tt.txt
done;
done;
done;
done

Answer 3

I would use something to figure out the leap years etc for me ie date. The following might give a hint on how to do this.
They way you're using wget means it's going to create a bunch of files with
"10.txt.1"
"10.txt.2"
"10.txt.3"
"10.txt.4"
"10.txt.5"
This might be fine but if you want to put these in a directory on their own or to name the file as something else
#!/bin/bash
#Jan 01 1980
COUNTER=347155200
while [ $COUNTER -lt 978263999 ]; do
year=`date -r $COUNTER +"%y"`;
month=`date -r $COUNTER +"%m"`;
day=`date -r $COUNTER +"%d"`;
hour=`date -r $COUNTER +"%H"`;
min=`date -r $COUNTER +"%M"`;
let COUNTER=COUNTER+600
url="www.xyz.com/$year/$month/$day/$hour/$min.txt";
dir="$year/$month/$day/$hour";
file="$year/$month/$day/$hour/$min.txt"
mkdir -p $dir;
wget "$url" $file;
#Post process files here...
done