[Bash]Read URL from txt file - linux

Question

I have a text file and I would like to read URLs from this text file one by one and check if they expire within 30 days or more.
Exame URLS:
example01.example.com:8080
example02.example.com:8002
example03.example.com:8003
https://example04.example.com:8111
...
I have a Bash script which checks an SSL cert's expiry date:
#!/bin/bash
TARGET="example01.example.com:8080";
DAYS=30;
echo "checking if $TARGET expires in less than $DAYS days";
expirationdate=$(date -d "$(: | openssl s_client -connect $TARGET -servername $TARGET 2>/dev/null \
| openssl x509 -text \
| grep 'Not After' \
|awk '{print $4,$5,$7}')" '+%s');
time=$(($(date +%s) + (86400*$DAYS)));
if [ $time -gt $expirationdate ]; then
echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d #$expirationdate '+%Y-%m-%d')" ;
echo $TARGET on $(date -d #$expirationdate '+%Y-%m-%d') > expireEndpoint.txt;
else
echo "OK - Certificate expires on $(date -d #$expirationdate '+%Y-%m-%d')";
fi;
Some of the URLs are duplicates too.

Answer 1

modified the above script to fix the openssl error
> unreachableOrInsecure.txt
> certLessThan30Days.txt
> certMoreThan30Days.txt
input="url.txt"
while IFS= read -r line
do
if [ -z "$line" ]; then
continue # ignore if line is empty
fi
protocol=`echo $line | awk -F: '{print $1}'`
temp=(${line//:/ })
PORT=`echo ${temp[2]} | awk -F/ '{print $1}'`
temp=`echo $line | awk -F/ '{print $3}'`
TARGET=`echo $temp | awk -F: '{print $1}'`
DAYS=30;
# 1. Check if it is uses secure HTTP
if [ $protocol == "https" ]; then
echo "$TARGET is secure: uses HTTPS"
else
echo "$TARGET uses HTTP: insecure" >> unreachableOrInsecure.txt
echo "$TARGET uses HTTP: insecure"
fi
# 2. check if server is reachable
if curl --output /dev/null --silent --head --fail "$TARGET"; then
echo "$TARGET is reachable "
else
echo "$TARGET is unreachable " >> unreachableOrInsecure.txt
echo "============" >> unreachableOrInsecure.txt
echo "$TARGET is unreachable "
echo "============"
continue # do not continue anymore, check the next target
fi
# 3. check if cert is reachable, timeout after 3 secs, don't keep waiting for openssl s_client to return
echo "checking if $TARGET:$PORT expires in less than $DAYS days";
expirationdate=$(timeout 3 openssl s_client -servername $TARGET -connect $TARGET:$PORT </dev/null 2>/dev/null | \
openssl x509 -noout -dates 2>/dev/null | \
awk -F= '/^notAfter/ { print $2; exit }')
expire_epoch=$(date +%s -d "$expirationdate")
epoch_warning=$(($DAYS*86400)) #look for 30 days
today_epoch="$(date +%s)"
timeleft=`expr $expire_epoch - $today_epoch`
if [[ $timeleft -le $epoch_warning ]]; then
echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d #$expire_epoch)" ;
echo "Cert expires for $TARGET on $(date -d #$expire_epoch '+%Y-%m-%d')" >> certLessThan30Days.txt;
echo "============" >> certLessThan30Days.txt
else
echo "OK - Certificate expires on $(date -d #$expire_epoch)";
echo "Cert expires for $TARGET on $(date -d #$expire_epoch '+%Y-%m-%d')" >> certMoreThan30Days.txt;
echo "============" >> certMoreThan30Days.txt
fi;
echo "============"
done < "$input"
Output now looks like this
msn.com is secure: uses HTTPS
msn.com is reachable
checking if msn.com:443 expires in less than 30 days
OK - Certificate expires on Thursday 07 April 2022 02:18:44 AM IST
============
google.com is secure: uses HTTPS
google.com is reachable
checking if google.com:443 expires in less than 30 days
OK - Certificate expires on Wednesday 23 September 2020 02:13:12 AM IST
============
example.com uses HTTP: insecure
example.com is reachable
checking if example.com:80 expires in less than 30 days
KO - Certificate for example.com expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.google.ie is secure: uses HTTPS
www.google.ie is reachable
checking if www.google.ie: expires in less than 30 days
KO - Certificate for www.google.ie expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.facebook.com is secure: uses HTTPS
www.facebook.com is reachable
checking if www.facebook.com: expires in less than 30 days
KO - Certificate for www.facebook.com expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.kooba.ie is secure: uses HTTPS
www.kooba.ie is reachable
checking if www.kooba.ie: expires in less than 30 days
KO - Certificate for www.kooba.ie expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============

Answer 2

slightly hacky but I came up with this quick and dirty solution ... here is my take
EDIT:fixed a small bug
> unreachableOrInsecure.txt
> certLessThan30Days.txt
> certMoreThan30Days.txt
input="url.txt"
while IFS= read -r line
do
if [ -z "$line" ]; then
continue # ignore if line is empty
fi
protocol=`echo $line | awk -F: '{print $1}'`
temp=(${line//:/ })
PORT=`echo ${temp[2]} | awk -F/ '{print $1}'`
temp=`echo $line | awk -F/ '{print $3}'`
TARGET=`echo $temp | awk -F: '{print $1}'`
DAYS=30;
# 1. Check if it is uses secure HTTP
if [ $protocol == "https" ]; then
echo "$TARGET is secure: uses HTTPS"
else
echo "$TARGET uses HTTP: insecure" >> unreachableOrInsecure.txt
echo "$TARGET uses HTTP: insecure"
fi
# 2. check if server is reachable
if curl --output /dev/null --silent --head --fail "$TARGET"; then
echo "$TARGET is reachable "
else
echo "$TARGET is unreachable " >> unreachableOrInsecure.txt
echo "============" >> unreachableOrInsecure.txt
echo "$TARGET is unreachable "
echo "============"
continue # do not continue anymore, check the next target
fi
# 3. check if cert is reachable, timeout after 3 secs, don't keep waiting for openssl s_client to return
echo "checking if $TARGET:$PORT expires in less than $DAYS days";
expirationdate=$(date -d "$(: | timeout 3 openssl s_client -connect $TARGET:$PORT -servername $TARGET 2>/dev/null \
| openssl x509 -text \
| grep 'Not After' \
|awk '{print $4,$5,$7}')" '+%s');
time=$(($(date +%s) + (86400*$DAYS)));
if [ $time -gt $expirationdate ]; then
echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d #$expirationdate '+%Y-%m-%d')" ;
echo $TARGET on $(date -d #$expirationdate '+%Y-%m-%d') >> certLessThan30Days.txt;
echo "============" >> certLessThan30Days.txt
else
echo "OK - Certificate expires on $(date -d #$expirationdate '+%Y-%m-%d')";
echo $TARGET on $(date -d #$expirationdate '+%Y-%m-%d') >> certMoreThan30Days.txt;
echo "============" >> certMoreThan30Days.txt
fi;
echo "============"
done < "$input"
EDIT: adding the url.txt and the output
Here is my url.txt
https://msn.com:443/services/f?a
https://google.com:443
http://example.com:80/c/ca/s
https://www.google.ie/
https://www.facebook.com/
https://www.kooba.ie/
output of the script
msn.com is secure: uses HTTPS
msn.com is reachable
checking if msn.com:443 expires in less than 30 days
OK - Certificate expires on 2022-04-06
============
google.com is secure: uses HTTPS
google.com is reachable
checking if google.com:443 expires in less than 30 days
OK - Certificate expires on 2020-09-22
============
example.com uses HTTP: insecure
example.com is reachable
checking if example.com:80 expires in less than 30 days
unable to load certificate
140024794944832:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for example.com expires in less than 30 days, on 2020-07-23
============
www.google.ie is secure: uses HTTPS
www.google.ie is reachable
checking if www.google.ie: expires in less than 30 days
unable to load certificate
139738534053184:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.google.ie expires in less than 30 days, on 2020-07-23
============
www.facebook.com is secure: uses HTTPS
www.facebook.com is reachable
checking if www.facebook.com: expires in less than 30 days
unable to load certificate
139623076582720:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.facebook.com expires in less than 30 days, on 2020-07-23
============
www.kooba.ie is secure: uses HTTPS
www.kooba.ie is reachable
checking if www.kooba.ie: expires in less than 30 days
unable to load certificate
139831347127616:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.kooba.ie expires in less than 30 days, on 2020-07-23
============