HTTP requests in wget taking most of the time - linux

Question

I'm retrieving large amount of data via wget, with the following command:
wget --save-cookies ~/.urs_cookies --load-cookies ~/.urs_cookies --keep-session-cookies --content-disposition -i links.dat
My problem is that links.dat contains thousands of links. The files are relatively small (100kb). So it takes 0.2s to download the file, and 5s to await for HTTP request response. So it ends up taking 14h to download my whole data, most of the time spent waiting for the requests.
URL transformed to HTTPS due to an HSTS policy
--2017-02-15 18:01:37-- https://goldsmr4.gesdisc.eosdis.nasa.gov/daac-bin/OTF/HTTP_services.cgi?FILENAME=%2Fdata%2FMERRA2%2FM2I1NXASM.5.12.4%2F1980%2F01%2FMERRA2_100.inst1_2d_asm_Nx.19800102.nc4&FORMAT=bmM0Lw&BBOX=43%2C1.5%2C45%2C3.5&LABEL=MERRA2_100.inst1_2d_asm_Nx.19800102.SUB.nc4&FLAGS=&SHORTNAME=M2I1NXASM&SERVICE=SUBSET_MERRA2&LAYERS=&VERSION=1.02&VARIABLES=t10m%2Ct2m%2Cu50m%2Cv50m
Connecting to goldsmr4.gesdisc.eosdis.nasa.gov (goldsmr4.gesdisc.eosdis.nasa.gov)|198.118.197.95|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 50223 (49K) [application/octet-stream]
Saving to: ‘MERRA2_100.inst1_2d_asm_Nx.19800102.SUB.nc4.1’
This might be a really noob question, but it seems really counter productive that this is working this way. I have really little knowledge on what is happening behind the scenes, but I just wanted to be sure I'm not doing anything wrong and that the process can indeed be faster.
If details helps, I'm downloading MERRA-2 data for specific nodes.
Thanks!

Answer 1

Wget will re-use an existing connection for multiple requests to the same server, potentially saving you the time required to establish and tear down the socket.
You can do this by providing multiple URLs on the command line. For example, to download 100 per batch:
#!/usr/bin/env bash
wget_opts=(
--save-cookies ~/.urs_cookies
--load-cookies ~/.urs_cookies
--keep-session-cookies
--content-disposition
)
manyurls=()
while read url; do
manyurls+=( "$url" )
if [ ${#manyurls[#]} -eq 100 ]; then
wget "${wget_opts[#]}" "${manyurls[#]}"
manyurls=()
fi
done < links.dat
if [ ${#manyurls[#]} -gt 0 ]; then
wget "${wget_opts[#]}" "${manyurls[#]}"
fi
Note that I haven't tested this. It may work. If it doesn't, tell me your error and I'll try to debug.
So ... that's "connection re-use" or "keepalive". The other thing that would speed up your download is HTTP Pipelining, which basically allows a second request to be sent before the first response has been received. wget does not support this, and curl supports it in its library, but not the command-line tool.
I don't have a ready-made tool to suggest that supports HTTP pipelining. (Besides which, tool recommendations are off-topic.) You can see how pipelining works in this SO answer. If you feel like writing something in a language of your choice that supports libcurl, I'm sure any difficulties you come across make for another an interesting additional StackOverflow question.