Submitting Base64 CSR to a Microsoft CA (via cURL) - iis

I have written a bash script to automate IIS7 Certificate generation as per this ServerFault link.
I would like to automate sending the Code Signing Request (CSR) to an internal Microsoft Certification Authority (MS CA) via cURL, the following code is promising and is successfully submitting the CSR to MS CA:
$ curl -k -u '<Domain>\<Username>':<Password> --ntlm
'https://<InternalMSCA>/certsrv/certfnsh.asp'
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
-H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.5'
-H 'Connection: keep-alive'
-H 'Host: <InternalMSCA>'
-H 'Referer: https://<InternalMSCA>/certsrv/certrqxt.asp'
-H 'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko'
-H 'Content-Type: application/x-www-form-urlencoded'
--data 'Mode=newreq&CertRequest=-----BEGIN+CERTIFICATE+REQUEST-----%0D%0AMIIDBjCCAe4CAQAwaDELMAkGA1UEBhMCQVUxDDAKBgNVBAgTA05TVzEPMA0GA1UE%0D%0ABxMGU3lkbmV5MQwwCgYDVQQKEwNZdW0xDjAMBgNVBAsTBVl1bUlTMRwwGgYDVQQD%0D%0AExN0ZXN0LmF1LmludC50Z3IubmV0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB%0D%0ACgKCAQEAygZvKhfs0mw4tModevTxOIz7eYYM%2B1axNv8FqoNyKr7xtqSbOMiNzf8R3rZ%0D%0A4cTcu5nv7oC7GHPMhnF7AdsO4XexwnKfnCkofECGkO6O4oTmRfUPLa38nV1%2BmytB%0D%0AlrQAl272jQdM9LSxTYW0OR9qO4mjAH1tvLF3IcC1OKOh6UNubdRFfE7dEXWnk%2BSF%0D%0AM8tgl0t3SFsRxrZL3vkgL%2B%2FEmvdOKXeoIey%2F7UMNeWRcwTkS1mw30HjvitJdQGZi%0D%0AgYJ6ldXrrITVKe9QXvVTxSl9NfzPHYp4yf%2FZvAJQmGLZ16aQo0PBeEfjkgkrcY5j%0D%0AMnVI2Q8yC%2BW9Bg%3D%3D%0D%0A-----END+CERTIFICATE+REQUEST-----&CertAttrib=CertificateTemplate%3A*WebServer%0D%0AUserAgent%3AMozilla%2F5.0+%28Windows+NT+6.3%3B+WOW64%3B+Trident%2F7.0%3B+rv%3A11.0%29+like+Gecko%0D%0A&FriendlyType=Saved-Request+Certificate+%287%2F7%2F2015%2C+3%3A46%3A39+PM%29&ThumbPrint=&TargetStoreFlags=0&SaveCert=yes'
| firefox "data:text/html;base64,$(base64 -w 0 <&0)"
I am interested in replaying this request after modifying it:
Decode --data (OK)
Modify --data (OK)
Re-encode... (Not OK)
Encoded:
Mode=newreq&CertRequest=-----BEGIN+CERTIFICATE+REQUEST-----%0D%0AMIIDBjCCAe4CAQAwaDELMAkGA1UEBhMCQVUxDDAKBgNVBAgTA05TVzEPMA0GA1UE%0D%0ABxMGU3lkbmV5MQwwCgYDVQQKEwNZdW0xDjAMBgNVBAsTBVl1bUlTMRwwGgYDVQQD%0D%0AExN0ZXN0LmF1LmludC50Z3IubmV0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB%0D%0ACgKCAQEAygZvKhfs0mw4tModevTxOIz7eYYM%2B1axNv8FqoNyKr7xtqSbOMiNzf8R3rZ%0D%0A4cTcu5nv7oC7GHPMhnF7AdsO4XexwnKfnCkofECGkO6O4oTmRfUPLa38nV1%2BmytB%0D%0AlrQAl272jQdM9LSxTYW0OR9qO4mjAH1tvLF3IcC1OKOh6UNubdRFfE7dEXWnk%2BSF%0D%0AM8tgl0t3SFsRxrZL3vkgL%2B%2FEmvdOKXeoIey%2F7UMNeWRcwTkS1mw30HjvitJdQGZi%0D%0AgYJ6ldXrrITVKe9QXvVTxSl9NfzPHYp4yf%2FZvAJQmGLZ16aQo0PBeEfjkgkrcY5j%0D%0AMnVI2Q8yC%2BW9Bg%3D%3D%0D%0A-----END+CERTIFICATE+REQUEST-----&CertAttrib=CertificateTemplate%3A*WebServer%0D%0AUserAgent%3AMozilla%2F5.0+%28Windows+NT+6.3%3B+WOW64%3B+Trident%2F7.0%3B+rv%3A11.0%29+like+Gecko%0D%0A&FriendlyType=Saved-Request+Certificate+%287%2F7%2F2015%2C+3%3A46%3A39+PM%29&ThumbPrint=&TargetStoreFlags=0&SaveCert=yes
Decoded:
Mode=newreq&CertRequest=-----BEGIN CERTIFICATE REQUEST-----
MIIDBjCCAe4CAQAwaDELMAkGA1UEBhMCQVUxDDAKBgNVBAgTA05TVzEPMA0GA1UE
BxMGU3lkbmV5MQwwCgYDVQQKEwNZdW0xDjAMBgNVBAsTBVl1bUlTMRwwGgYDVQQD
ExN0ZXN0LmF1LmludC50Z3IubmV0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEAygZvKhfs0mw4tModevTxOIz7eYYM+1axNv8FqoNyKr7xtqSbOMiNzf8R3rZ
4cTcu5nv7oC7GHPMhnF7AdsO4XexwnKfnCkofECGkO6O4oTmRfUPLa38nV1+mytB
lrQAl272jQdM9LSxTYW0OR9qO4mjAH1tvLF3IcC1OKOh6UNubdRFfE7dEXWnk+SF
M8tgl0t3SFsRxrZL3vkgL+/EmvdOKXeoIey/7UMNeWRcwTkS1mw30HjvitJdQGZi
gYJ6ldXrrITVKe9QXvVTxSl9NfzPHYp4yf/ZvAJQmGLZ16aQo0PBeEfjkgkrcY5j
MnVI2Q8yC+W9Bg==
-----END CERTIFICATE REQUEST-----&CertAttrib=CertificateTemplate:*WebServer
UserAgent:Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0)
like Gecko &FriendlyType=Saved-Request Certificate (7/7/2015, 3:46:39
PM)&ThumbPrint=&TargetStoreFlags=0&SaveCert=yes
Re-encoded: (URLEncode1, URLEncode2, URLEncode3 ):
Mode%3Dnewreq%26CertRequest%3D-----BEGIN+CERTIFICATE+REQUEST-----+MIIDBjCCAe4CAQAwaDELMAkGA1UEBhMCQVUxDDAKBgNVBAgTA05TVzEPMA0GA1UE+BxMGU3lkbmV5MQwwCgYDVQQKEwNZdW0xDjAMBgNVBAsTBVl1bUlTMRwwGgYDVQQD+ExN0ZXN0LmF1LmludC50Z3IubmV0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB+CgKCAQEAygZvKhfs0mw4tModevTxOIz7eYYM%2B1axNv8FqoNyKr7xtqSbOMiNzf8R3rZ+4cTcu5nv7oC7GHPMhnF7AdsO4XexwnKfnCkofECGkO6O4oTmRfUPLa38nV1%2BmytB+lrQAl272jQdM9LSxTYW0OR9qO4mjAH1tvLF3IcC1OKOh6UNubdRFfE7dEXWnk%2BSF+M8tgl0t3SFsRxrZL3vkgL%2B%2FEmvdOKXeoIey%2F7UMNeWRcwTkS1mw30HjvitJdQGZi+gYJ6ldXrrITVKe9QXvVTxSl9NfzPHYp4yf%2FZvAJQmGLZ16aQo0PBeEfjkgkrcY5j+MnVI2Q8yC%2BW9Bg%3D%3D+-----END+CERTIFICATE+REQUEST-----%26CertAttrib%3DCertificateTemplate%3A%2AWebServer+UserAgent%3AMozilla%2F5.0+%28Windows+NT+6.3%3B+WOW64%3B+Trident%2F7.0%3B+rv%3A11.0%29+like+Gecko+%26FriendlyType%3DSaved-Request+Certificate+%287%2F7%2F2015%2C+3%3A46%3A39+PM%29%26ThumbPrint%3D%26TargetStoreFlags%3D0%26SaveCert%3Dyes
The three websites linked above (in Re-encoded) all fail to re-encode properly.
The tricky part is that "=" and "&" should not be encoded.
URL Encode Simple:
CR LF %0D%0A (Not %)
Space + (Not %20)
- - (Not %2D)
& & (Not %26)
= = (Not %3D)
+ (in CSR) %2B
\ (in CSR) %2F
( %28
) %29
I could specifically use sed for this example, but I would like to know if there is a way to know what encoding the server is expecting, and encode in the proper charset automatically. Is this possible ?

I solved in linux on bash and curl:
#!/bin/sh
# tested on SUSE Linux 12 SP1
# $1 - CN Object name
# $2 - username
# $3 - password
MSCA='HOSTNAME' # Internal Microsoft Certification Authority
Username=$2
Password=$3
function show_usage()
{
echo "Scrip for retrive certificate from MS SubCA"
echo "Usage: $0 <CN> [domain\\\\username] [password]"
echo " "
echo "Example: $0 example.com workgroup\\\\foo bar"
exit 0
}
if [ -z "$1" ]
then
show_usage
exit 0
fi
if [ -z "$2" ]
then
Username="workgroup\\foo"
Password="bar"
fi
echo -e "\e[32m1. Generate private key...\e[0m"
openssl req -new -nodes -out $1.pem -keyout $1.key -subj "/C=RU/ST=State/L=City/O=Org/CN=$1/emailAddress=postmaster#example.com"
CERT=`cat $1.pem | tr -d '\n\r'`
DATA="Mode=newreq&CertRequest=${CERT}&C&TargetStoreFlags=0&SaveCert=yes"
CERT=`echo ${CERT} | sed 's/+/%2B/g'`
CERT=`echo ${CERT} | tr -s ' ' '+'`
CERTATTRIB="CertificateTemplate:Server%0D%0A"
echo -e "\e[32m2. Request cert...\e[0m"
OUTPUTLINK=`curl -k -u "${Username}":${Password} --ntlm \
"https://${MSCA}/certsrv/certfnsh.asp" \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Connection: keep-alive' \
-H "Host: ${MSCA}" \
-H "Referer: https://${MSCA}/certsrv/certrqxt.asp" \
-H 'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko' \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data "Mode=newreq&CertRequest=${CERT}&CertAttrib=${CERTATTRIB}&TargetStoreFlags=0&SaveCert=yes&ThumbPrint=" | grep -A 1 'function handleGetCert() {' | tail -n 1 | cut -d '"' -f 2`
CERTLINK="https://${MSCA}/certsrv/${OUTPUTLINK}"
echo -e "\e[32m3. Retrive cert: $CERTLINK\e[0m"
curl -k -u "${Username}":${Password} --ntlm $CERTLINK \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Connection: keep-alive' \
-H "Host: ${MSCA}" \
-H "Referer: https://${MSCA}/certsrv/certrqxt.asp" \
-H 'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko' \
-H 'Content-Type: application/x-www-form-urlencoded' > $1.crt
echo -e "\e[32m4. Verifying cert for $1\e[0m"
openssl verify -verbose $1.crt
if [ "0" -eq "$?" ] ;
then
echo -e "\e[32mWell done. Have a nice day.\e[0m"
exit 0
else
echo -e "\e[31;47mError code: $?. Stopping.\e[0m"
exit 1
fi

Here's the bash script on Windows (CYGWIN):
#!/bin/bash
#############################################################################################
# AUTHOR: FlORIAN BIDABE #
# #
# VERSION 2.0 RELEASE DATE February 07, 2018 #
# This script helps you with generating SSL material from an internal Micorosft CA #
# 1) Define your variables and CA Bundle in this script (between < and >) #
# 2) Run the script #
# #
# Process: #
# 1- Generate or import CSR #
# 2- Submit CSR and specify additional Subject Alternate Name (SAN) #
# 3- Collect certificate from your CA (Certificate Authority) ==> MANUAL #
# 4- Generate SSL material and format #
# #
# #
# Tested on: #
# Certificate Authority: Windows Server 2008 R2 / 2012 #
# Client: Windows 8.1 and Windows 10 with cygwin (cURL, OpenSSL, clip) #
#############################################################################################
#_____________________________________________________________________________________________
######################################## Variables ########################################
# Internal Env Settings
MSCA='server.domain.tld' # Internal Microsoft Certification Authority FQDN
CertTplt='WebServer' # Internal Cert Template Name
UA='Mozilla%2F5.0+%28Windows+NT+6.3%3B+WOW64%3B+Trident%2F7.0%3B+rv%3A11.0%29+like+Gecko'
Domain='localdomain' # Used for signing both hostname and FQDN
Username=""$Domain"\\userid" # Required for certificate submission
#Password='WH4t3v3r' # Can be commented to be interactive
# Email Settings (Unauthenticated in my case)
mailserver="mailserver.com" # Mailserver FQDN
mailport=25 # Might need SSL if 465, 587
to="ServiceDesk#company.com"
cc="who.ever#company.com"
bcc="yourself#company.com,fortrackingpurpose#company.com"
MailTemplate="
Hello,
Please create and assign a ticket to track this certificate request from our market.
The Request ID has been attached in the email (HTTP Response)
Date: `date "+%Y-%m-%d %H:%M"`
Issuer: "$Username"
Information Systems
Phone number
Company Pty Ltd
Address and contact details"
# SaveIn=~/Desktop/Certs
SaveIn=~/Certificates/NewRequests # Save the file in Team's OneDrive folder mapped with ln using cygwin
FileMgr=explorer # File Manager
# OpenSSL CFG settings for CSR (Code Signing Request) submission
Country='AU'
State='NSW'
City='Sydney'
Company='Company'
UrOrg='Information Systems'
# Internal Base64 Root and Intermediate CAs (Used for creating PEM and PKCS12 bundles)
IntRoot=`echo '
-----BEGIN CERTIFICATE-----
<Intermediate>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<Root>
-----END CERTIFICATE-----'`
#_____________________________________________________________________________________________
######################################## Requirements ########################################
# OpenSSL
type openssl > /dev/null 2>&1 || {
echo "Cannot find OpensSSL, it is required to generate certificates. Aborting..." 1>&2
exit 1
}
# cURL
type curl > /dev/null 2>&1 || {
echo "Cannot find cURL, it is required to submit certificates. Aborting..." 1>&2
exit 1
}
#_____________________________________________________________________________________________
######################################## Optional ############################################
# Clip
type clip > /dev/null 2>&1 || {
echo -e "Cannot find clip ! it is required to save the CSR into your clipboard.\n Attempting to install it in System32..." 1>&2
cd 'C:\Windows\system32'; curl -L -O "https://www.dropbox.com/s/cvkxeak0j0wtjj0/clip.exe"
}
# GNU Email
type email > /dev/null 2>&1 || {
echo -e "Cannot find GNU email ! it is required to send an email to notify a security administrator and issue the certificate." 1>&2
}
# Internet Explorer
if [ -f '/cygdrive/c/Program\ Files/Internet\ Explorer/iexplore.exe' ]; then iexplore='/cygdrive/c/Program\ Files/Internet\ Explorer/iexplore.exe'
else iexplore=$(sed 's| |\\ |g' <<< "$(find /cygdrive/ -name "iexplore.exe" -exec sh -c 'printf "%s\n" "$1"; kill "$PPID"' bash {} \;)")
fi
#_____________________________________________________________________________________________
######################################## Functions ########################################
gencsr() {
# Generate Config File (CFG) for Code Signing Request (CSR)
echo "`date "+%Y-%m-%d %H:%M:%S"` - User Option: 1) Generate CSR and Private Key" >> $LOGS
echo "`date "+%Y-%m-%d %H:%M:%S"` - Parsing Config File (CFG)" >> $LOGS
# Set additional SAN (for CFG)
local n=1 #Enter Loop
local SAN
SAN="subjectAltName = DNS:"$Hostname", DNS: "$Hostname.$Domain""
while (( n > 0 && n < 4 )); do
echo -e "\n\n\nDo you want to set an additional Subject Alternate Name (Config File) ? (No)"
echo -e "Current SAN:\n"$SAN""
echo -e "Select your choice and press [ENTER]\n\t[1] Add an IP address\n\t[2] Add an hostname\n\t[3] Reset SAN to default\n\t[*] Continue"
read -p "Option number : " n
case $n in
1) # Add Extra IP for SAN
while [[ -z ${IP+x} || $? != 0 ]]; do
read -p "What is the server's IP address: " IP
[[ "$IP" =~ ^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$ ]]
if [ $? != 0 ]; then echo "This IP address ("$IP") does not look quite right! Please try again..."; fi
[[ "$IP" =~ ^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$ ]]
done
SAN+=", IP:"$IP", DNS:"$IP""; unset IP
;;
2) # Add extra DNS name to SAN
while [[ -z ${extraSAN+x} || $? != 0 ]]; do
read -p "Specify a Fully Qualified Domain Name for the extra SAN : " extraSAN
[[ "$extraSAN" =~ ^[A-Za-z0-9.-]+$ ]]
if [ $? != 0 ]; then echo "This syntax is incorrect! Please try again..."; fi
[[ "$extraSAN" =~ ^[A-Za-z0-9.-]+$ ]]
done
SAN+=", DNS:"$extraSAN""; unset extraSAN
;;
3) SAN="subjectAltName = DNS:"$Hostname", DNS:"$Hostname.$Domain"" ;;
*) n=4 ;; #Quit loop
esac
done
echo "`date "+%Y-%m-%d %H:%M:%S"` - Subject Alternate Name (CFG): "$SAN"" >> $LOGS
echo "
[ req ]
default_md = sha512
default_bits = 2048
default_keyfile = "$Hostname"_pk8.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
input_password = password
output_password = password
[ v3_req ]
basicConstraints = CA:false
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
"$SAN"
[ req_distinguished_name ]
countryName = "$Country"
stateOrProvinceName = "$State"
localityName = "$City"
0.organizationName = "$Company"
organizationalUnitName = "$UrOrg"
commonName = "$Hostname.$Domain"" > "$Hostname".cfg
echo "`date "+%Y-%m-%d %H:%M:%S"` - Config File (CFG) parsed ! Located at `pwd`/"$Hostname".cfg" >> $LOGS
# Generate CSR and private key (PKCS8) & convert Private Key (PKCS8 to PKCS1)
echo -e "\n\nGenerating Code Signing Request (CSR) and Private Key (PKCS#8)..."
echo "`date "+%Y-%m-%d %H:%M:%S"` - Generating Code Signing Request (CSR) and Private Key (PKCS#8): "$Hostname".csr and "$Hostname"_pk8.key" >> $LOGS
openssl req -out "$Hostname".csr -new -nodes -config "$Hostname".cfg > /dev/null 2>&1
echo "Generating private key (PKCS#1)..."
echo "`date "+%Y-%m-%d %H:%M:%S"` - Generating Private Key (PKCS#1): "$Hostname"_pk1.key" >> $LOGS
openssl rsa -in "$Hostname"_pk8.key -out "$Hostname"_pk1.key > /dev/null 2>&1
if [ $? != 0 ]; then
echo "An error has occured ! Exiting..., Please consult the logs"
echo "`date "+%Y-%m-%d %H:%M:%S"` - Error on generating CSR or Private Keys" >> $LOGS
exit 1
fi
}
importcsr() {
# Importing Code Signing Request (CSR)
echo "`date "+%Y-%m-%d %H:%M:%S"` - User Option: 2) Import CSR" >> $LOGS
local n
printf "\033c"
echo -e "This function automates IIS7 certificate generation for "$Company $UrOrg"
\tServer name:\t"$Hostname"\n\tFQDN:\t\t"$Hostname"."$Domain"\n"
echo "Importing Code Signing Request..."
#Verify CSR
# If CSR is not Base 64
openssl req -text -noout -verify -in *.csr > /dev/null 2>&1
while [ $? != 0 ]; do
# Check if there are multiple csr files
while [ $(find -name "*.csr" | wc -l) != 1 ]; do
echo -e "\nError, $(find -name "*.csr" | wc -l) CSR(s) found ! One CSR is required..."
echo "Please import your CSR in "$SaveIn" and make sure the extension is *.csr"
echo "`date "+%Y-%m-%d %H:%M:%S"` - WARNING: There should be one CSR only in "$SaveIn"" >> $LOGS
$FileMgr . 2> /dev/null
read -p "Press any key to continue...";
done
openssl req -text -noout -verify -inform DER -in *.csr > /dev/null 2>&1
if [ $? == 0 ]; then
echo -e "\n\nThis Code Signing Request is not a Base64 request !\nConverting DER request to Base64... Success !"
mv *.csr "$Hostname".dcsr
openssl req -out "$Hostname".csr -outform PEM -inform DER -in *.dcsr
echo "`date "+%Y-%m-%d %H:%M:%S"` - DER CSR detected, converting to Base64... Success !" >> $LOGS
echo "`date "+%Y-%m-%d %H:%M:%S"` - DER CSR: "$Hostname".dcsr\tBase64 CSR: "$Hostname".csr" >> $LOGS
else
openssl req -text -noout -verify -in *.csr > /dev/null 2>&1
if [ $? != 0 ]; then
echo -e "Your CSR file is not valid or is corrupted!\nPlease import your CSR in "$SaveIn"..."
echo "`date "+%Y-%m-%d %H:%M:%S"` - ERROR: This CSR is invalid, it is neither a DER or Base64 CSR" >> $LOGS
$FileMgr . 2> /dev/null
read -p "Press any key to continue..."; fi
fi
openssl req -text -noout -verify -in *.csr > /dev/null 2>&1
done
# Optional: Converting a Base64 CSR to DER
if [ ! -f *.dcsr ]; then
openssl req -outform DER -inform PEM -in *.csr -out "$Hostname".dcsr > /dev/null 2>&1
if [ $? == 0 ]; then
echo "`date "+%Y-%m-%d %H:%M:%S"` - Base64 CSR detected, converting to DER... Success !" >> $LOGS
echo "`date "+%Y-%m-%d %H:%M:%S"` - DER CSR: "$Hostname".dcsr\tBase64 CSR: "$Hostname".csr" >> $LOGS
fi
fi
}
urlencode() {
local data
if [[ $# != 1 ]]; then return 1; fi
data="$(curl -s -o /dev/null -w %{url_effective} --get --data-urlencode "$1" "")"
if [[ $? != 3 ]]; then return 2; fi
echo "${data##/?}"; return 0
}
getcert() {
######################### 3- Get Certificate ########################
echo -e "\n\n`date "+%Y-%m-%d %H:%M:%S"` - Step 3: Getting the Certifiate" >> $LOGS
printf "\033c"
echo -e "This function automates IIS7 certificate generation for "$Company $UrOrg"
\tServer name:\t"$Hostname"\n\tFQDN:\t\t"$Hostname"."$Domain"\n"
echo -e "Open \"Certificate Authority\" in a Management Console (MMC) and connect to "$MSCA"\nVerify that your certificate request is in "Pending Requests".\nIssue the Certificate (Right Click, All Tasks, Issue)\nNavigate to "Issue Certificates", order by Request ID (Descending) and export it (Open / Details / Copy To File) 'Base-64 Encoded X.509' to "$SaveIn".\nThe file must have a *.cer extension\n"
read -p "Press any keys when the certificate (*.cer) has been place in "$SaveIn""
#Verify Certificate
openssl x509 -text -noout -in "$Hostname".cer > /dev/null 2>&1
while [ $? != 0 ]; do
# Verify that there is only one certificate
while [ $(find -name "*.cer" | wc -l) == 0 ]; do
echo "Please import certificate (*.cer) in "$SaveIn""
if [ -z ${Manual+x} ]; then $FileMgr . 2> /dev/null
else
#If the certificate has been uploaded using a browser, it can be retrieved using the browser
if [ -z ${iexplore+x} ]; then echo "Open "https://"$MSCA"/certsrv/certckpn.asp""
else eval $iexplore "https://"$MSCA"/certsrv/certckpn.asp"; fi
fi
read -p "Press any key to continue..."; done
while [ $(find -name "*.cer" | wc -l) != 1 ]; do
echo "Error, $(find -name "*.cer" | wc -l) certificates found in "$SaveIn"! Please clean it up !"
$FileMgr . 2> /dev/null
read -p "Press any key to continue..."
done
# Verify Certificate Integrity and format
mv *.cer "$Hostname".cer
echo -e "`date "+%Y-%m-%d %H:%M:%S"` - Certificate found at `pwd`/"$Hostname".cer" >> $LOGS
openssl x509 -text -noout -in *.cer > /dev/null 2>&1
if [ $? != 0 ]; then
openssl x509 -inform der -text -noout -in *.cer > /dev/null 2>&1 # Test if DER
if [ $? == 0 ]; then # Convert DER to Base64
mv *.cer "$Hostname".der
openssl x509 -inform der -in "$Hostname".der -out "$Hostname".cer > /dev/null 2>&1
echo "`date "+%Y-%m-%d %H:%M:%S"` - DER certificate detected, converting to Base64... Success !" >> $LOGS
echo "`date "+%Y-%m-%d %H:%M:%S"` - DER certificate: "$Hostname".der\tBase64 certificate: "$Hostname".cer" >> $LOGS
else
echo -e "This certificate is invalid or corrupted!\nPlease import it again in "$SaveIn"..."
echo "`date "+%Y-%m-%d %H:%M:%S"` - ERROR: The certificate is invalid, it is neither a DER or Base64 certificate" >> $LOGS
read -p "Press any key to continue..."
fi
openssl x509 -text -noout -in *.cer > /dev/null 2>&1
fi
done
# Optional: Converting a Base64 CSR to DER
if [ ! -f *.der ]; then
openssl x509 -outform der -in "$Hostname".cer -out "$Hostname".der > /dev/null 2>&1
if [ $? == 0 ]; then
echo "`date "+%Y-%m-%d %H:%M:%S"` - Base64 Certificate detected, converting to DER... Success !" >> $LOGS
echo "`date "+%Y-%m-%d %H:%M:%S"` - DER Certificate: "$Hostname".dcsr\tBase64 Certificate: "$Hostname".csr" >> $LOGS
fi
fi
###################### 4- Generating SSL material #########################
# Creating PEM certificate chain
echo -e "`date "+%Y-%m-%d %H:%M:%S"` - Step 4 (Final): Generating SSL material" >> $LOGS
if [ -f ""$Hostname"_pk1.key" ]; then
cat "$Hostname"_pk1.key > ""$Hostname".pem"
cat *.cer >> ""$Hostname".pem"
echo -e "`date "+%Y-%m-%d %H:%M:%S"` - A PEM has been generated containing the Private Key and entire certificate chain: Public Key for "$Hostname" and CA Bundle (intermediate and root certificates) " >> $LOGS
else
cat *.cer > ""$Hostname".pem"
echo -e "`date "+%Y-%m-%d %H:%M:%S"` - A PEM has been generated containing the entire certificate chain: Public Key for "$Hostname" and CA Bundle (intermediate and root certificates)" >> $LOGS
echo -e "`date "+%Y-%m-%d %H:%M:%S"` - As the CSR was imported, no private key can be included in the PEM container" >> $LOGS
fi
echo "$IntRoot" >> ""$Hostname".pem"
sed -i '/^$/d' "$Hostname".pem"" # Delete empty lines
# Converting PEM certificate chain to PKCS#12 (.pfx)"
cat *.pfx 2> /dev/null #Enter Loop
while [ $? != 0 ]; do
if [ -f "$Hostname"_pk1.key ]; then openssl pkcs12 -export -out ""$Hostname".pfx" -in ""$Hostname".pem"
else openssl pkcs12 -export -nokeys -out ""$Hostname".pfx" -in ""$Hostname".pem"
fi
done
echo -e "`date "+%Y-%m-%d %H:%M:%S"` - A PKCS12 (.pfx, .p12) has been generated from the PEM" >> $LOGS
echo -e "`date "+%Y-%m-%d %H:%M:%S"` - Ending gracefully :)" >> $LOGS
mv ../"$Hostname" ../../INTERNAL/
cd ../../INTERNAL/"$Hostname"
$FileMgr . 2> /dev/null
exit 0
}
#_____________________________________________________________________________________________
######################################## GUI ########################################
printf "\033c"
echo -e "This function automates IIS7 certificate generation for "$Company $UrOrg""
# Set Hostname and IP address
Hostname="$1"; [[ "$Hostname" =~ ^[-A-Za-z0-9]+$ ]]
while [ $? != 0 ]; do
read -p "Specify the server hostname (Not FQDN !): " Hostname
[[ "$Hostname" =~ ^[-A-Za-z0-9]+$ ]]
if [ $? != 0 ]; then echo "This hostname syntax is incorrect, try again !"; fi
[[ "$Hostname" =~ ^[-A-Za-z0-9]+$ ]]
done
LOGS=""$Hostname".logs"
# Set destination folder for SSL material
SaveIn+="/"$Hostname"";
if [ -d "$SaveIn" ]; then
echo "A folder named "$Hostname" already exists, Start over (delete existing materials) or quit ?"
echo -e "Select your choice and press [ENTER]\n\t[1] Start Over (Delete existing content)\n\t[2] Resume (Certificate Generation)\n\t[*] Quit"
read -p "Option number : " n
case $n in
1) rm -R "$SaveIn" > /dev/null 2>&1; mkdir -p "$SaveIn" > /dev/null 2>&1; cd "$SaveIn" ;;
2) cd "$SaveIn"; getcert "$#" ;;
*) echo "Aborting..."; exit 0 ;;
esac
else mkdir -p "$SaveIn"; cd "$SaveIn"
fi
###LOGGING GUI###
echo "`date "+%Y-%m-%d %H:%M:%S"` - Starting... Path: `pwd`" > $LOGS
echo "`date "+%Y-%m-%d %H:%M:%S"` - OpenSSL Version: `openssl version`" >> $LOGS
echo "`date "+%Y-%m-%d %H:%M:%S"` - cURL Version: `head -n 1 <(curl --version)`" >> $LOGS
echo "`date "+%Y-%m-%d %H:%M:%S"` - Server name: "$Hostname" FQDN: "$Hostname"."$Domain"" >> $LOGS
######################### 1- Get CSR ###############################
echo -e "\n\n`date "+%Y-%m-%d %H:%M:%S"` - Step 1: Code Signing Request" >> $LOGS
printf "\033c"
echo -e "This function automates IIS7 certificate generation for "$Company $UrOrg"
\tServer name:\t"$Hostname"\n\tFQDN:\t\t"$Hostname"."$Domain"\n"
echo -e "\nCode Signing Request (CSR):\n\tYou can generate a CSR and Private key or import a CSR (generated by an appliance and downloaded by you).
\tPlease note that importing a CSR means that the private key remains on the appliance or vendor's site.
\tSelect your choice and press [ENTER]\n\t[1] Generate CSR and Private Key\n\t[2] Import CSR\n\t[*] Quit"
read -p "Option number : " n
case $n in
1) gencsr "$#" ;;
2) importcsr "$#" ;;
*) echo "`date "+%Y-%m-%d %H:%M:%S"` - User Option: Quit" >> $LOGS; echo "Aborting..."; exit 0 ;;
esac
######################### 2- Submit CSR ############################
echo -e "\n\n`date "+%Y-%m-%d %H:%M:%S"` - Step 2: Submitting CSR" >> $LOGS
# Capture Attempt: Session ID cookie
echo "`date "+%Y-%m-%d %H:%M:%S"` - Capturing Session ID cookie from "$MSCA"" >> $LOGS
echo 'Capturing ASP Session ID (Cookie)...'
if [ -z "$Password" ]; then echo "What is the password for $Username ?: "; read -s Password; fi
RE=': ([^;]*);' #Regex to capture ASP Session ID from cookie string
while read l; do [[ $l =~ $RE ]] && AspSession="${BASH_REMATCH[1]}"; done <<<"$(grep "Cookie" <<< "$(curl --silent -Iku "$Username":"$Password" --ntlm https://"$MSCA"/certsrv/certrqxt.asp)")"
# If fail capturing cookie ==> Manual (Browser-Mode)
if [ -z "$AspSession" ]; then
echo "`date "+%Y-%m-%d %H:%M:%S"` - ERROR: Cannot capture Session ID cookie, failover to browser-mode..." >> $LOGS
echo "WARNING: Cannot capture Session ID cookie for "$MSCA", failover to browser-mode...\nPlease verify your credentials to connect to $MSCA\n\n"
echo "Paste CSR directly in internal CA web interface"
echo -e "\tConfirm the Subject Alternate Name field before submission !\n\tNote that the CSR may already include SAN(s) !
Current Subject: `openssl req -in *.csr -noout -text | grep "Subject:"`
Current SAN: `openssl req -in *.csr -noout -text | grep "DNS:"`"
clip <<< "$(cat *.csr 2> /dev/null)" ; Manual=1
echo -e "Please upload your Code Signing Request to your Internal Certificate Authority ("$MSCA") :"
if [ -z ${iexplore+x} ]; then
echo "Open "https://"$MSCA"/certsrv/certrqxt.asp" in a browser"
else
eval $iexplore "https://"$MSCA"/certsrv/certrqxt.asp" &
echo "Press any key to continue..." ; read
fi
fi
# If Session ID cookie sucessfully captured ==> Automatic (cURL-Mode)
if [ -z ${Manual+x} ]; then
echo "ASP cookie captured !"
# Set additional SAN (for cURL)
echo -e "\n\nConfirm the Subject Alternate Name before submission:
Current Common Name: `openssl req -in *.csr -noout -text | grep "Subject:"`
Current SAN: `openssl req -in *.csr -noout -text | grep "DNS:"`"
echo -e "\nDo you want to add a Subject Alternate Name (No) ?\nSelect your choice and press [ENTER]\n\t[1] Yes\n\t[*] No\n"
read -p "Option number : " n
case $n in
1) unset n; n=1 #Enter Loop
unset SAN; SAN="san%3Adns%3D"$Hostname"%26dns%3D"$Hostname.$Domain""
while (( n > 0 && n < 4 )); do
echo -e "\n\n\nDo you want to set an additional Subject Alternate Name ? (No)"
echo -e "Current SAN (URL Encoded): "$SAN""
echo -e "Select your choice and press [ENTER]\n\t[1] Add an IP address\n\t[2] Add an hostname\n\t[3] Reset SAN to default\n\t[*] Continue"
read -p "Option number : " n
case $n in
1) # Add Extra IP for SAN
while [[ -z ${IP+x} || $? != 0 ]]; do
read -p "What is the server's IP address: " IP
[[ "$IP" =~ ^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$ ]]
if [ $? != 0 ]; then echo "This IP address ("$IP") does not look quite right! Please try again..."; fi
[[ "$IP" =~ ^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])$ ]]
done
SAN+="%26dns%3D"$IP""; unset IP
;;
2) # Add extra DNS name to SAN
while [[ -z ${extraSAN+x} || $? != 0 ]]; do
read -p "Specify a Fully Qualified Domain Name (FQDN) for the extra SAN: " extraSAN
[[ "$extraSAN" =~ ^[A-Za-z0-9.-]+$ ]]
if [ $? != 0 ]; then echo "This syntax is incorrect! Please try again..."; fi
[[ "$extraSAN" =~ ^[A-Za-z0-9.-]+$ ]]
done
SAN+="%26dns%3D"$extraSAN""; unset extraSAN
;;
3) SAN="san%3Adns%3D"$Hostname"%26dns%3D"$Hostname.$Domain"" ;;
*) n=4 ; SAN+='%0D%0A' ;; #Quit loop
esac
done
;;
*) ;;
esac
CertFormat=$(sed 's| |+|g' <<< $(sed 's|+|%2B|g' <<< $(sed 's|/|%2F|g' <<< $(sed ':a;N;$!ba;s/\n/%0D%0A/g' *.csr))))
Date=$(sed 's|%20|+|g' <<< $(urlencode "`date '+%m/%d/%Y,%r'`"))
cURLData="Mode=newreq&CertRequest="$CertFormat"&CertAttrib="$SAN"CertificateTemplate%3A"$CertTplt"%0D%0AUserAgent="$UA"%0D%0A&FriendlyType=Saved-Request+Certificate+%28"$Date"%29&ThumbPrint=&TargetStoreFlags=0&SaveCert=yes"
echo "`date "+%Y-%m-%d %H:%M:%S"` - Generating and encoding cURL POST data..." >> $LOGS
echo -e "Injecting crafted POST request to Internal CA using cURL and NTLM authentication...\n"
echo "`date "+%Y-%m-%d %H:%M:%S"` - Injecting crafted POST request to Internal CA using cURL and NTLM authentication..." >> $LOGS
InjectCmd="curl --silent -i -ku '$Username':'$Password' --ntlm '"https://"$MSCA"/certsrv/certfnsh.asp"' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Connection: keep-alive' -H 'Cookie: "$AspSession"' \
-H 'Host: "$MSCA"' \
-H 'Referer: https://"$MSCA"/certsrv/certrqxt.asp' \
-H 'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko' \
-H 'Content-Type: application/x-www-form-urlencoded' --data '"$cURLData"'"
InjectCmdLog=`echo $InjectCmd | sed "s|"$Password"|<password>|g"`
echo "`date "+%Y-%m-%d %H:%M:%S"` - Command: "$InjectCmdLog"" >> $LOGS
echo "`date "+%Y-%m-%d %H:%M:%S"` - BEGIN HTTP REPLY: Consult "$Hostname".html" >> $LOGS
eval "$InjectCmd" &> "$Hostname.html"
echo "`date "+%Y-%m-%d %H:%M:%S"` - END HTTP REPLY" >> $LOGS
if [ $? != 0 ] || grep -q 'Access Denied' "$Hostname.html" || grep -q 'Denied by Policy Module' "$Hostname.html"; then
echo -e "Injection seems to have gone wrong! Please verify if the request is missing in the Certificate Authority Snap-In on "$MSCA""
echo -e "Consult Log file for analysis of the cURL query: it might be malformed!"
echo -e "Log file location: `pwd`/"$LOGS""
echo "`date "+%Y-%m-%d %H:%M:%S"` - Injection has failed !" >> $LOGS
exit 1
fi
fi
email "$to" -cc "$cc" -bcc "$bcc" -a "$Hostname".html -s "Certificate Request: Please issue $Hostname.$Domain certificate" -r $mailserver -p $mailport <<< "$MailTemplate"
echo "An email has been sent to the US (You are in CC) ! Once approved, please connect to "$MSCA" to retrieve your certificate using the Certificate Authority via mmc.exe"
echo "Once retrieved, open again this utility, enter the same hostname ("$Hostname") and resume operations: this will generates cryptographic material bundles (PEM, #PKCS12... etc.)"
echo "Please take notes of password you set to access the private key on the PKCS12 material"

I solved it in Java. this is the class:
package com.qequipe.dsiglib.p10;
import java.io.BufferedInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.math.BigInteger;
import java.net.Authenticator;
import java.net.HttpURLConnection;
import java.net.PasswordAuthentication;
import java.net.URL;
import java.net.URLEncoder;
import java.nio.charset.Charset;
import java.nio.charset.StandardCharsets;
import java.security.KeyPair;
import java.security.KeyPairGenerator;
import java.util.Base64;
import java.util.Date;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLSession;
import javax.security.auth.x500.X500Principal;
import org.spongycastle.operator.ContentSigner;
import org.spongycastle.operator.OperatorCreationException;
import org.spongycastle.operator.jcajce.JcaContentSignerBuilder;
import org.spongycastle.pkcs.PKCS10CertificationRequest;
import org.spongycastle.pkcs.PKCS10CertificationRequestBuilder;
import org.spongycastle.pkcs.jcajce.JcaPKCS10CertificationRequestBuilder;
import com.qequipe.dsiglib.util.FileUtil;
public class P10Generator {
public static byte[] generate(KeyPair pair) throws OperatorCreationException, IOException
{
//KeyPair pair = generateKeyPair();
PKCS10CertificationRequestBuilder p10Builder = new JcaPKCS10CertificationRequestBuilder(
new X500Principal("CN=Requested Test Certificate"), pair.getPublic());
JcaContentSignerBuilder csBuilder = new JcaContentSignerBuilder("SHA256withRSA");
ContentSigner signer = csBuilder.build(pair.getPrivate());
PKCS10CertificationRequest csr = p10Builder.build(signer);
return csr.getEncoded();
}
public static void main(String[] args)
{
try
{
KeyPairGenerator gen = KeyPairGenerator.getInstance("RSA");
gen.initialize(2048);
KeyPair pair = gen.generateKeyPair();
byte[] pkcs10 = generate(pair);
String base64pkcs10 = new String(Base64.getEncoder().encode(pkcs10));
System.out.println(new BigInteger(1, pkcs10).toString(16));
System.out.println(base64pkcs10);
Date date = new Date();
Authenticator.setDefault(new Authenticator() {
#Override
public PasswordAuthentication getPasswordAuthentication() {
System.out.println("Scheme:" + getRequestingScheme());
return new PasswordAuthentication("\\<username>", "password".toCharArray());
}
});
String request = "Mode=newreq&CertRequest=" + URLEncoder.encode(base64pkcs10) +
"&CertAttrib=" + URLEncoder.encode("CertificateTemplate:User\r\nUserAgent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36\r\n") +
"&FriendlyType=" + URLEncoder.encode("Saved-Request Certificate (" + (date.getDay()) + "/" + (date.getMonth() + 1) + "/" + date.getYear() + ", " + date.getHours() + ":" + date.getMinutes() + ":" + date.getSeconds() + ")") +
"&ThumbPrint=&TargetStoreFlags=0&SaveCert=yes";
byte[] cert = sendRequest("http://<yourcaurl>/certsrv/certfnsh.asp", request, "username", "password");
System.out.println(new BigInteger(1, cert).toString(16));
}
catch(Exception ex)
{
ex.printStackTrace();
}
}
public static byte[] sendRequest(String urlString, String content, String username, String password) throws IOException {
HttpURLConnection con;
ByteArrayOutputStream out = new ByteArrayOutputStream();
URL url = new URL(urlString);
con = (HttpURLConnection) url.openConnection();
//String protocol = url.getProtocol();
con.setInstanceFollowRedirects(false);
con.setRequestMethod("POST");
//String encoded = Base64.getEncoder().encodeToString((username+":"+password).getBytes(StandardCharsets.UTF_8)); //Java 8
//con.setRequestProperty("Authorization", "NTML "+encoded);
con.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
con.setRequestProperty("Accept-Encoding","gzip, deflate");
con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");
con.setRequestProperty("Connection","keep-alive");
con.setRequestProperty("Host", url.getHost());
con.setRequestProperty("Referer", url.getHost() + "/certsrv/certrqxt.asp");
con.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko");
con.setRequestProperty("Content-Type","application/x-www-form-urlencoded");
con.setRequestProperty("Content-Length", "" + content.length());
con.setDoOutput(true);
OutputStream outs = con.getOutputStream();
outs.write(content.getBytes());
outs.flush();
BufferedInputStream bis = new BufferedInputStream(con.getInputStream());
int length;
while ((length = bis.read()) != -1) {
out.write(length);
}
out.close();
System.out.println(out);
int respCode;
System.out.println("RESP code = " + (respCode = con.getResponseCode()));
//System.out.println("RESPONSE = \n" + out);
if(con.getResponseCode() < HttpURLConnection.HTTP_BAD_REQUEST)
{
con.disconnect();
Pattern p = Pattern.compile("certnew.cer\\?ReqID=(\\d+)&");//. represents single character.
Matcher m = p.matcher(new String(out.toByteArray(), Charset.forName("UTF-8")));
if(m.find())
{
String reqid = m.group(1);
System.out.println(reqid);
URL certurl = new URL(url.getProtocol() +"://" + url.getHost() + "/certsrv/certnew.cer?ReqID=" + reqid + "&Enc=bin");
InputStream ins = certurl.openStream();
ByteArrayOutputStream bouts = new ByteArrayOutputStream();
FileUtil.copy(ins, bouts);
ins.close();
return bouts.toByteArray();
}
else
{
return null;
}
}
else
{
con.disconnect();
return null;
}
}
}

This is a combination of bits I've borrowed from forum posts including this one, and some stuff I've done myself. It has some stuff to accommodate Aruba Instant APs and the certificate format it accepts. Specifically tailored for the needs in my lab.
The answers here are great, and #SkazochNik J nailed it providing the final piece (automating the csr submission and retrieving the cert). #SkazochNik if you happen to see this, the last thing I'm looking for is pulling the CA cert from the CA via curl/bash. If you happen to know the format there, that would be awesome...
What I built (so far) in case this is useful:
#!/usr/bin/env bash
MSCA='dc1.myDomain.com' # Internal Microsoft Certification Authority
Username=<domain\\user>
Password=<pass>
echo "-- Certificate csr and key generator --"
echo "-- all certificates use myDomain.com domain --"
echo " "
echo "Is this certificate for a wireless controller/vc [y/n]"
read _isWlan
echo "provide all hostnames to be used as the CN first then additional SAN values (only the hostname)"
read _host1 _host2 _host3 _host4 _host5
echo "provide all IP addresses desired as SAN values"
read _ip1 _ip2 _ip3 _ip4 _ip5
if [ ${_isWlan,,} == 'y' ] || [ ${_isWlan,,} == 'yes' ]; then _isWlan=true; fi
if [ -z "$_host2" ]; then _dns2=false; else _dns2=true; fi
if [ -z "$_host3" ]; then _dns3=false; else _dns3=true; fi
if [ -z "$_host4" ]; then _dns4=false; else _dns4=true; fi
if [ -z "$_host5" ]; then _dns5=false; else _dns5=true; fi
if [ -z "$_ip2" ]; then _ip2in=false; else _ip2in=true; fi
if [ -z "$_ip3" ]; then _ip3in=false; else _ip3in=true; fi
if [ -z "$_ip4" ]; then _ip4in=false; else _ip4in=true; fi
if [ -z "$_ip5" ]; then _ip5in=false; else _ip5in=true; fi
echo "[req]" > $_host1.cnf
echo "default_bits = 2048" >> $_host1.cnf
echo "prompt = no" >> $_host1.cnf
echo "default_md = sha256" >> $_host1.cnf
echo "req_extensions = req_ext" >> $_host1.cnf
echo "distinguished_name = dn" >> $_host1.cnf
echo " " >> $_host1.cnf
echo "[ dn ]" >> $_host1.cnf
echo "C=US" >> $_host1.cnf
echo "ST=Indiana" >> $_host1.cnf
echo "L=Noblesville" >> $_host1.cnf
echo "O=WadeLab" >> $_host1.cnf
echo "OU=Engineering" >> $_host1.cnf
echo "emailAddress=wade1#hpe.com" >> $_host1.cnf
if $_isWlan; then
echo "CN = securelogin.myDomain.com" >> $_host1.cnf
else
echo "CN = "$_host1".myDomain.com" >> $_host1.cnf
fi
echo " " >> $_host1.cnf
echo "[ req_ext ]" >> $_host1.cnf
echo "subjectAltName = #alt_names" >> $_host1.cnf
echo " " >> $_host1.cnf
echo "[ alt_names ]" >> $_host1.cnf
# it would be prettier to populate an array from the input and loop through them, but it would take me longer
# to refresh my knowledge for the bash way to do that then it did to copy paste all this
if $_isWlan; then
echo "DNS.1 = securelogin.myDomain.com" >> $_host1.cnf
echo "DNS.2 = "$_host1".myDomain.com" >> $_host1.cnf
echo "DNS.3 = "$_host1 >> $_host1.cnf
if $_dns2; then
echo "DNS.4 = "$_host2".myDomain.com" >> $_host1.cnf
fi
if $_dns3; then
echo "DNS.5 = "$_host3".myDomain.com" >> $_host1.cnf
fi
if $_dns4; then
echo "DNS.6 = "$_host4".myDomain.com" >> $_host1.cnf
fi
if $_dns5; then
echo "DNS.7 = "$_host5".myDomain.com" >> $_host1.cnf
fi
else
echo "DNS.1 = "$_host1".myDomain.com" >> $_host1.cnf
echo "DNS.2 = "$_host1 >> $_host1.cnf
if $_dns2; then
echo "DNS.3 = "$_host2".myDomain.com" >> $_host1.cnf
fi
if $_dns3; then
echo "DNS.4 = "$_host3".myDomain.com" >> $_host1.cnf
fi
if $_dns4; then
echo "DNS.5 = "$_host4".myDomain.com" >> $_host1.cnf
fi
if $_dns5; then
echo "DNS.6 = "$_host5".myDomain.com" >> $_host1.cnf
fi
fi
echo "IP.1 = "$_ip1 >> $_host1.cnf
if $_ip2in; then
echo "IP.2 = "$_ip2 >> $_host1.cnf
fi
if $_ip3in; then
echo "IP.3 = "$_ip3 >> $_host1.cnf
fi
if $_ip4in; then
echo "IP.4 = "$_ip4 >> $_host1.cnf
fi
if $_ip5in; then
echo "IP.5 = "$_ip5 >> $_host1.cnf
fi
echo
echo "1. Creating Certificate Request..."
openssl req -nodes -newkey rsa:2048 -keyout $_host1.key -out $_host1.csr -config $_host1.cnf
#echo "--- csr and key creation complete ---"
#echo " "
#echo "---------------------------"
#echo "-----Below is your CSR-----"
#echo "---------------------------"
#cat $_host1.csr
#echo
#echo
# >> Start Send csr to CA and retrieve certificate
CERT=`cat $_host1.csr | tr -d '\n\r'`
DATA="Mode=newreq&CertRequest=${CERT}&C&TargetStoreFlags=0&SaveCert=yes"
CERT=`echo ${CERT} | sed 's/+/%2B/g'`
CERT=`echo ${CERT} | tr -s ' ' '+'`
CERTATTRIB="CertificateTemplate:WebServerv2Exportable%0D%0A"
echo "2. Request cert from "$MSCA"..."
OUTPUTLINK=`curl -k -u "${Username}":${Password} --ntlm \
"https://${MSCA}/certsrv/certfnsh.asp" \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Connection: keep-alive' \
-H "Host: ${MSCA}" \
-H "Referer: https://${MSCA}/certsrv/certrqxt.asp" \
-H 'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko' \
-H 'Content-Type: application/x-www-form-urlencoded' \
--data "Mode=newreq&CertRequest=${CERT}&CertAttrib=${CERTATTRIB}&TargetStoreFlags=0&SaveCert=yes&ThumbPrint=" | grep -A 1 'function handleGetCert() {' | tail -n 1 | cut -d '"' -f 2`
CERTLINK="https://${MSCA}/certsrv/${OUTPUTLINK}"
echo "3. Retrive cert: "$CERTLINK"..."
curl -k -u "${Username}":${Password} --ntlm $CERTLINK \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Connection: keep-alive' \
-H "Host: ${MSCA}" \
-H "Referer: https://${MSCA}/certsrv/certrqxt.asp" \
-H 'User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko' \
-H 'Content-Type: application/x-www-form-urlencoded' > $_host1.cer
echo "4. Verifying cert for "$_host1"..."
openssl verify -CAfile ca.cer -verbose $_host1.cer
if [ "0" -eq "$?" ] ;
then
echo "Certificate Validation Completed Succesfully"
echo
#exit 0
else
echo "Error code: "$?". Stopping."
exit 1
fi
#/END send csr to CA and retrieve certificate
if $_isWlan; then
echo "Do you need the certficate combined with the CA and key for Aruba IAP? [y/n]"
read _isIap
fi
if [ ${_isIap,,} == 'y' ] || [ ${_isIap,,} == 'yes' ]; then
echo "-------------------------------------------------------"
echo "----- Combining Server CA and Key file for VC -----"
echo "-------------------------------------------------------"
echo
# echo "-----Please Paste in the resulting PEM certificate-----"
# echo "----- then press [CTRL+D] to continue -----"
# echo "-------------------------------------------------------"
# _svrCert=$(</dev/stdin)
# echo "$_svrCert" > tmp.cer
# no error check right now for existing of CA.cer
mv $_host1.cer tmp.cer
cat ca.cer >> tmp.cer
cat $_host1.key >> tmp.cer
cat tmp.cer | sed '/^[[:space:]]*$/d' > $_host1.cer ##Remove blank lines
rm -f tmp.cer
echo "--------------------------------------------------"
echo "-----Cert Creation Complete. Combined cert ------"
echo "----- is in this directory "$_host1".cer ------"
echo "--------------------------------------------------"
else
echo "--------------------------------------------------"
echo "--------------------- DONE ---------------------"
echo "----- "$_host1".cer is in this directory ------"
echo "--------------------------------------------------"
fi
echo
echo Outputting Certificate Contents to Display...
sleep 4
clear
cat $_host1.cer

Related

Trying to use this bash script to login to a remote ftp from ssh and delete files older than N days old

I am trying to use the following bash script to login to a remote ftp and delete files older than N days old. Script says it is working and does not give an error - but files are not being deleted. What am I missing? Or is there a better way to do this? Keep in mind this is only a remote FTP and not SSH so I can NOT use the mtime function is why I am trying to do this. Can anyone help?
The usage is all commands - here is what I am using via ssh to run the script
./ftprem.sh -s ftp.server.com -u myusername -p mypassword -f /directory -d 3
#!/bin/bash
PROGNAME=$(basename $0)
OUTFILE="/tmp/ftplist.$RANDOM.txt"
CMDFILE="/tmp/ftpcmd.$RANDOM.txt"
ndays=14
print_usage() {
echo ""
echo "$PROGNAME - Delete files older than N days from an FTP server"
echo ""
echo "Usage: $PROGNAME -s -u -p -f (-d)"
echo ""
echo " -s FTP Server name"
echo " -u User Name"
echo " -p Password"
echo " -f Folder"
echo " -d Number of Days (Default: $ndays)"
echo " -h Show this page"
echo ""
echo "Usage: $PROGNAME -h"
echo ""
exit
}
# Parse parameters
options=':hs:u:p:f:d:'
while getopts $options flag
do
case $flag in
s)
FTPSITE=$OPTARG
;;
u)
FTPUSER=$OPTARG
;;
p)
FTPPASS=$OPTARG
;;
f)
FTPDIR=$OPTARG
;;
d)
ndays=$OPTARG
;;
h)
print_usage
;;
\?)
echo "Invalid option: -$OPTARG" >&2
exit 1
;;
:)
echo "Option -$OPTARG requires an argument." >&2
exit 1
;;
esac
done
shift $(($OPTIND - 1))
if [[ -z "$FTPSITE" || -z "$FTPUSER" || -z "$FTPPASS" || -z "$FTPDIR" ]];
then
echo "ERROR: Missing parameters"
print_usage
fi
# work out our cutoff date
TDATE=`date --date="$ndays days ago" +%Y%m%d`
echo FTP Site: $FTPSITE
echo FTP User: $FTPUSER
echo FTP Password: $FTPPASS
echo FTP Folder: $FTPDIR
echo Removing files older than $TDATE
# get directory listing from remote source
ftp -i -n $FTPSITE <<EOMYF > /dev/null
user $FTPUSER $FTPPASS
binary
cd $FTPDIR
ls -l $OUTFILE
quit
EOMYF
if [ -f "$OUTFILE" ]
then
# Load the listing file into an array
lista=($(<$OUTFILE))
# Create the FTP command file to delete the files
echo "user $FTPUSER $FTPPASS" > $CMDFILE
echo "binary" >> $CMDFILE
echo "cd $FTPDIR" >> $CMDFILE
COUNT=0
# loop over our files
for ((FNO=0; FNO<${#lista[#]}; FNO+=9));do
# month (element 5), day (element 6) and filename (element 8)
FMM=${lista[`expr $FNO+5`]}
FDD=${lista[`expr $FNO+6`]}
FYY=${lista[`expr $FNO+7`]}
if [[ $FYY == *\:* ]]
then
FDATE=`date -d "$FMM $FDD" +'%Y%m%d'`
else
FDATE=`date -d "$FMM $FDD $FYY" +'%Y%m%d'`
fi
# echo $FDATE
# check the date stamp
if [[ $FDATE -lt $TDATE ]];
then
echo "Deleting ${lista[`expr $FNO+8`]}"
echo "delete ${lista[`expr $FNO+8`]}" >> $CMDFILE
COUNT=$[$COUNT + 1]
fi
done
echo "quit" >> $CMDFILE
if [[ $COUNT -gt 0 ]];
then
cat $CMDFILE | tr -d "\r" > $CMDFILE
ftp -i -n $FTPSITE < $CMDFILE > /dev/null
else
echo "Nothing to delete"
fi
rm -f $OUTFILE $CMDFILE
fi
If this helps your debugging...
In the# Parse parameter section of the script, the options variable your have just before the case block has value options=':hs:u:p:f:d:' instead of options=':h:s:u:p:f:d:'
I thought i should point that out.

How to mute signalfx detector alerts using cli with the help of bash script

I have created a simple bash script that help you to mute alert within a few second without going into the console.
prerequisite:
jq needs to be installed on local
you should know some portion of your detector.
ex- If I want to mute alert with for detector name "prod-test memory utilization" then while mute alert via script I will simply type prod-test
Token should be created on signalfx so that we can authenticate from local
Ref: https://docs.splunk.com/Observability/alerts-detectors-notifications/mute-notifications.html#nav-Mute-alert-notifications
#!/bin/bash
echo -e "Please enter SFx token to authenticate\n"
read TOKEN
echo -e " \n"
echo -e "Please copy-paste any one muting option from below. \n a) Mute_Indefinitely \n b) Mute_By_Duration \n"
read MUTE_OPTION
echo -e " \n"
## this is comman name from detector in my case it is prod-test and prod-test2"
echo -e "Please copy-paste the server name which you wish to mute.
1. prod-test
2. prod-test2\n"
read SERVER_NAME
echo "Fatching detectors ids, Please wait for few seconds..."
DETECTOR_ID=$(curl -s -X GET "https://<your sfx url>/v2/detector?limit=2000" -H "Content-Type: application/json" -H "X-SF-TOKEN: "$TOKEN"" | jq -r '.results[] | select(.name | contains("'"$SERVER_NAME"'")).id')
############### MUTE BY DURATION FUNCTION ###############
mute_by_duration () {
START_TIME=`date +%s%N | cut -b1-13`
END_TIME=`date -d "+$min minutes" +%s%N | cut -b1-13`
curl -X "POST" "https://<your sfx url>/v2/alertmuting" \
-H 'X-SF-TOKEN: '$TOKEN'' \
-H 'Content-Type: application/json' \
-d $'{
"startTime": "'"$START_TIME"'",
"stopTime": "'"$END_TIME"'",
"filters": [
{
"property": "sf_detectorId",
"propertyValue": "'"$i"'"
}
]
}'
if [ $? -eq 0 ]; then
echo -e "\nDetector has been muted\n"
fi
}
############# MUTE ALERT INDEFINITELY ################
mute_indefinitely () {
curl -X "POST" "https://<your sfx url>/v2/alertmuting" \
-H 'X-SF-TOKEN: '$TOKEN'' \
-H 'Content-Type: application/json' \
-d $'{
"startTime": "",
"stopTime": "",
"filters": [
{
"property": "sf_detectorId",
"propertyValue": "'"$i"'"
}
]
}'
if [ $? -eq 0 ]; then
echo -e "\nDetector has been muted\n"
fi
}
#################### MUTING OPTION ###############
muting_rule() {
case "$MUTE_OPTION" in
"Mute_Indefinitely") echo -e "\n***** You have selected Mute_Indefinitely option, don't forget to unmute later *****\n";
for i in $DETECTOR_ID; do mute_indefinitely; done
;;
"Mute_By_Duration") echo -e "\n***** You have selected Mute_By_Duration option, This Alert will auto unmute after $min minutes *****\n";
for i in $DETECTOR_ID; do mute_by_duration; done
;;
*) echo "Please select correct option"
esac
}
############# SELECTION BASED ON MUTING OPTION #######################
if [ "$MUTE_OPTION" == Mute_By_Duration ]; then
echo -e "How much minutes you want to mute alert from current time? \n Example: Type "30" to mute for 30 mins \n"
read min
muting_rule
elif [ "$MUTE_OPTION" == Mute_Indefinitely ]; then
muting_rule
else
echo "Invalid Option"
fi

[Bash]Read URL from txt file

I have a text file and I would like to read URLs from this text file one by one and check if they expire within 30 days or more.
Exame URLS:
example01.example.com:8080
example02.example.com:8002
example03.example.com:8003
https://example04.example.com:8111
...
I have a Bash script which checks an SSL cert's expiry date:
#!/bin/bash
TARGET="example01.example.com:8080";
DAYS=30;
echo "checking if $TARGET expires in less than $DAYS days";
expirationdate=$(date -d "$(: | openssl s_client -connect $TARGET -servername $TARGET 2>/dev/null \
| openssl x509 -text \
| grep 'Not After' \
|awk '{print $4,$5,$7}')" '+%s');
time=$(($(date +%s) + (86400*$DAYS)));
if [ $time -gt $expirationdate ]; then
echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d #$expirationdate '+%Y-%m-%d')" ;
echo $TARGET on $(date -d #$expirationdate '+%Y-%m-%d') > expireEndpoint.txt;
else
echo "OK - Certificate expires on $(date -d #$expirationdate '+%Y-%m-%d')";
fi;
Some of the URLs are duplicates too.
modified the above script to fix the openssl error
> unreachableOrInsecure.txt
> certLessThan30Days.txt
> certMoreThan30Days.txt
input="url.txt"
while IFS= read -r line
do
if [ -z "$line" ]; then
continue # ignore if line is empty
fi
protocol=`echo $line | awk -F: '{print $1}'`
temp=(${line//:/ })
PORT=`echo ${temp[2]} | awk -F/ '{print $1}'`
temp=`echo $line | awk -F/ '{print $3}'`
TARGET=`echo $temp | awk -F: '{print $1}'`
DAYS=30;
# 1. Check if it is uses secure HTTP
if [ $protocol == "https" ]; then
echo "$TARGET is secure: uses HTTPS"
else
echo "$TARGET uses HTTP: insecure" >> unreachableOrInsecure.txt
echo "$TARGET uses HTTP: insecure"
fi
# 2. check if server is reachable
if curl --output /dev/null --silent --head --fail "$TARGET"; then
echo "$TARGET is reachable "
else
echo "$TARGET is unreachable " >> unreachableOrInsecure.txt
echo "============" >> unreachableOrInsecure.txt
echo "$TARGET is unreachable "
echo "============"
continue # do not continue anymore, check the next target
fi
# 3. check if cert is reachable, timeout after 3 secs, don't keep waiting for openssl s_client to return
echo "checking if $TARGET:$PORT expires in less than $DAYS days";
expirationdate=$(timeout 3 openssl s_client -servername $TARGET -connect $TARGET:$PORT </dev/null 2>/dev/null | \
openssl x509 -noout -dates 2>/dev/null | \
awk -F= '/^notAfter/ { print $2; exit }')
expire_epoch=$(date +%s -d "$expirationdate")
epoch_warning=$(($DAYS*86400)) #look for 30 days
today_epoch="$(date +%s)"
timeleft=`expr $expire_epoch - $today_epoch`
if [[ $timeleft -le $epoch_warning ]]; then
echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d #$expire_epoch)" ;
echo "Cert expires for $TARGET on $(date -d #$expire_epoch '+%Y-%m-%d')" >> certLessThan30Days.txt;
echo "============" >> certLessThan30Days.txt
else
echo "OK - Certificate expires on $(date -d #$expire_epoch)";
echo "Cert expires for $TARGET on $(date -d #$expire_epoch '+%Y-%m-%d')" >> certMoreThan30Days.txt;
echo "============" >> certMoreThan30Days.txt
fi;
echo "============"
done < "$input"
Output now looks like this
msn.com is secure: uses HTTPS
msn.com is reachable
checking if msn.com:443 expires in less than 30 days
OK - Certificate expires on Thursday 07 April 2022 02:18:44 AM IST
============
google.com is secure: uses HTTPS
google.com is reachable
checking if google.com:443 expires in less than 30 days
OK - Certificate expires on Wednesday 23 September 2020 02:13:12 AM IST
============
example.com uses HTTP: insecure
example.com is reachable
checking if example.com:80 expires in less than 30 days
KO - Certificate for example.com expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.google.ie is secure: uses HTTPS
www.google.ie is reachable
checking if www.google.ie: expires in less than 30 days
KO - Certificate for www.google.ie expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.facebook.com is secure: uses HTTPS
www.facebook.com is reachable
checking if www.facebook.com: expires in less than 30 days
KO - Certificate for www.facebook.com expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
www.kooba.ie is secure: uses HTTPS
www.kooba.ie is reachable
checking if www.kooba.ie: expires in less than 30 days
KO - Certificate for www.kooba.ie expires in less than 30 days, on Friday 24 July 2020 12:00:00 AM IST
============
slightly hacky but I came up with this quick and dirty solution ... here is my take
EDIT:fixed a small bug
> unreachableOrInsecure.txt
> certLessThan30Days.txt
> certMoreThan30Days.txt
input="url.txt"
while IFS= read -r line
do
if [ -z "$line" ]; then
continue # ignore if line is empty
fi
protocol=`echo $line | awk -F: '{print $1}'`
temp=(${line//:/ })
PORT=`echo ${temp[2]} | awk -F/ '{print $1}'`
temp=`echo $line | awk -F/ '{print $3}'`
TARGET=`echo $temp | awk -F: '{print $1}'`
DAYS=30;
# 1. Check if it is uses secure HTTP
if [ $protocol == "https" ]; then
echo "$TARGET is secure: uses HTTPS"
else
echo "$TARGET uses HTTP: insecure" >> unreachableOrInsecure.txt
echo "$TARGET uses HTTP: insecure"
fi
# 2. check if server is reachable
if curl --output /dev/null --silent --head --fail "$TARGET"; then
echo "$TARGET is reachable "
else
echo "$TARGET is unreachable " >> unreachableOrInsecure.txt
echo "============" >> unreachableOrInsecure.txt
echo "$TARGET is unreachable "
echo "============"
continue # do not continue anymore, check the next target
fi
# 3. check if cert is reachable, timeout after 3 secs, don't keep waiting for openssl s_client to return
echo "checking if $TARGET:$PORT expires in less than $DAYS days";
expirationdate=$(date -d "$(: | timeout 3 openssl s_client -connect $TARGET:$PORT -servername $TARGET 2>/dev/null \
| openssl x509 -text \
| grep 'Not After' \
|awk '{print $4,$5,$7}')" '+%s');
time=$(($(date +%s) + (86400*$DAYS)));
if [ $time -gt $expirationdate ]; then
echo "KO - Certificate for $TARGET expires in less than $DAYS days, on $(date -d #$expirationdate '+%Y-%m-%d')" ;
echo $TARGET on $(date -d #$expirationdate '+%Y-%m-%d') >> certLessThan30Days.txt;
echo "============" >> certLessThan30Days.txt
else
echo "OK - Certificate expires on $(date -d #$expirationdate '+%Y-%m-%d')";
echo $TARGET on $(date -d #$expirationdate '+%Y-%m-%d') >> certMoreThan30Days.txt;
echo "============" >> certMoreThan30Days.txt
fi;
echo "============"
done < "$input"
EDIT: adding the url.txt and the output
Here is my url.txt
https://msn.com:443/services/f?a
https://google.com:443
http://example.com:80/c/ca/s
https://www.google.ie/
https://www.facebook.com/
https://www.kooba.ie/
output of the script
msn.com is secure: uses HTTPS
msn.com is reachable
checking if msn.com:443 expires in less than 30 days
OK - Certificate expires on 2022-04-06
============
google.com is secure: uses HTTPS
google.com is reachable
checking if google.com:443 expires in less than 30 days
OK - Certificate expires on 2020-09-22
============
example.com uses HTTP: insecure
example.com is reachable
checking if example.com:80 expires in less than 30 days
unable to load certificate
140024794944832:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for example.com expires in less than 30 days, on 2020-07-23
============
www.google.ie is secure: uses HTTPS
www.google.ie is reachable
checking if www.google.ie: expires in less than 30 days
unable to load certificate
139738534053184:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.google.ie expires in less than 30 days, on 2020-07-23
============
www.facebook.com is secure: uses HTTPS
www.facebook.com is reachable
checking if www.facebook.com: expires in less than 30 days
unable to load certificate
139623076582720:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.facebook.com expires in less than 30 days, on 2020-07-23
============
www.kooba.ie is secure: uses HTTPS
www.kooba.ie is reachable
checking if www.kooba.ie: expires in less than 30 days
unable to load certificate
139831347127616:error:0909006C:PEM routines:get_name:no start line:../crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
KO - Certificate for www.kooba.ie expires in less than 30 days, on 2020-07-23
============

Using a for loop in a bash scrip to iterate over extentions held in a csv file to download data from NASA

I am currently trying to download data from (https://ladsweb.modaps.eosdis.nasa.gov/search/order). They provide a bash file (https://ladsweb.modaps.eosdis.nasa.gov/tools-and-services/data-download-scripts/) which I need to try and edit. I have a .csv file with all of the file extensions I need to download (specifically I need all VNP46A1 data for China from 2015 until now.
In pseudo-code form I would like to add the following:
FOR url_path IN url_list:
recurse "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5000/VNP46A1/“+”$url_path"+”.h5" “your_directory”+”$url_path" "TOKEN_HERE"
I need to edit this bash to iterate over the files in the csv and download them into a folder for use later.
The bash file is as follows:
#!/bin/bash
function usage {
echo "Usage:"
echo " $0 [options]"
echo ""
echo "Description:"
echo " This script will recursively download all files if they don't exist"
echo " from a LAADS URL and stores them to the specified path"
echo ""
echo "Options:"
echo " -s|--source [URL] Recursively download files at [URL]"
echo " -d|--destination [path] Store directory structure to [path]"
echo " -t|--token [token] Use app token [token] to authenticate"
echo ""
echo "Dependencies:"
echo " Requires 'jq' which is available as a standalone executable from"
echo " https://stedolan.github.io/jq/download/"
}
function recurse {
local src=$1
local dest=$2
local token=$3
echo "Querying ${src}.json"
for dir in $(curl -s -H "Authorization: Bearer ${token}" ${src}.json | jq '.[] | select(.size==0) | .name' | tr -d '"')
do
echo "Creating ${dest}/${dir}"
mkdir -p "${dest}/${dir}"
echo "Recursing ${src}/${dir}/ for ${dest}/${dir}"
recurse "${src}/${dir}/" "${dest}/${dir}"
done
for file in $(curl -s -H "Authorization: Bearer ${token}" ${src}.json | jq '.[] | select(.size!=0) | .name' | tr -d '"')
do
if [ ! -f ${dest}/${file} ]
then
echo "Downloading $file to ${dest}"
# replace '-s' with '-#' below for download progress bars
curl -s -H "Authorization: Bearer ${token}" ${src}/${file} -o ${dest}/${file}
else
echo "Skipping $file ..."
fi
done
}
POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
-s|--source)
src="$2"
shift # past argument
shift # past value
;;
-d|--destination)
dest="$2"
shift # past argument
shift # past value
;;
-t|--token)
token="$2"
shift # past argument
shift # past value
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
if [ -z ${src+x} ]
then
echo "Source is not specified"
usage
exit 1
fi
if [ -z ${dest+x} ]
then
echo "Destination is not specified"
usage
exit 1
fi
if [ -z ${token+x} ]
then
echo "Token is not specified"
usage
exit 1
fi
recurse "$src" "$dest" "$token"
and a shortened (for testing purposes) csv file is given as:
/archive/allData/5000/VNP46A1/2015/001/VNP46A1.A2015001.h30v05.001.2019135185504.h5
/archive/allData/5000/VNP46A1/2015/002/VNP46A1.A2015002.h30v05.001.2019136091632.h5
/archive/allData/5000/VNP46A1/2015/003/VNP46A1.A2015003.h30v05.001.2019136075625.h5
/archive/allData/5000/VNP46A1/2015/004/VNP46A1.A2015004.h30v05.001.2019136081706.h5
/archive/allData/5000/VNP46A1/2015/005/VNP46A1.A2015005.h30v05.001.2019136084155.h5
/archive/allData/5000/VNP46A1/2015/006/VNP46A1.A2015006.h30v05.001.2019136084128.h5
/archive/allData/5000/VNP46A1/2015/007/VNP46A1.A2015007.h30v05.001.2019136085336.h5
/archive/allData/5000/VNP46A1/2015/008/VNP46A1.A2015008.h30v05.001.2019136103147.h5
/archive/allData/5000/VNP46A1/2015/009/VNP46A1.A2015009.h30v05.001.2019136100110.h5
Any help or suggestions will be much appreciated.
Kind regards
You want to create a bash script that will loop over each line to download the data that you need, using the script provided by NASA.
For example say the below script was a file called save-data.sh:
#!/bin/bash
while read p; do
./laads-data-download.sh -s $P -d "destination" -t "token"
echo "$p"
done <paths.txt
Example tree structure:
nasa-satellite-data
├── laads-data-download.sh
├── paths.txt
└── save-data.sh

How to start download from CGI in BASH?

I have this CGI script, but how do I start a download in the browser?
#!/bin/bash
echo "Content-Type: text/plain"
echo
CUR=$(ls -t /var/www/html/certs/*$AUTHENTICATE_UID* | head -1)
if [ "$CUR" != "" ]; then
# start download in the browser
fi
Typically, you would emit the Content-Disposition header:
echo "Content-Type: text/plain"
CUR=$(ls -t /var/www/html/certs/*$AUTHENTICATE_UID* | head -1)
if [ "$CUR" != "" ]; then
echo 'Content-Disposition: attachment; filename="yourFile.txt"'
fi
echo # End of headers.

Resources