I have a text file already that contain project id and project_names. But a request has been made to call together both group and project together in the same curl.
How can I call group and project id together and how would I modify the text file (config.txt) that contain the projectid and projectnames to contain the groupid?
config.txt
324566 projectnames1 projecttypes1
333456 projectnames2 projecttypes2
IN_FILE='./config.txt'
if [ -f "$IN_FILE" ]; then
echo "$IN_FILE exists and the operation can continue"
while read -ra LINE
do
a="${LINE[0]}"; b="${LINE[1]}"; c="${LINE[2]}"
echo "$a"
echo "$b"
echo "$c"
projectid=$a
projectnames=$b
projecttypes=$c
mkdir ./archive_$projectid-$projectnames
sudo chmod -R 755 /Users/names/git/archive_$projectid-$projectnames
curlfile=$(curl -s -I --header "PRIVATE-TOKEN:xxxxxxx" "https://gitlab.com/api/v4/projects/$projectid/variables" | grep HTTP/ | awk {'print $2'})
echo "$curlfile"
### This will check if status response is between 200 -299 inclusive
if (( $curlfile >= 200 && $curlfile <= 300 )) ; then
echo "$curlfile exists and the operation can start processing"
curl --header "PRIVATE-TOKEN:xxxxxxxxx" "https://gitlab.com/api/v4/projects/$projectid/variables" | python3 -m json.tool > $projectid-$projectnames.json
# echo "https://gitlab.com/api/v4/projects/$projectid/variables"
Related
I am currently trying to download data from (https://ladsweb.modaps.eosdis.nasa.gov/search/order). They provide a bash file (https://ladsweb.modaps.eosdis.nasa.gov/tools-and-services/data-download-scripts/) which I need to try and edit. I have a .csv file with all of the file extensions I need to download (specifically I need all VNP46A1 data for China from 2015 until now.
In pseudo-code form I would like to add the following:
FOR url_path IN url_list:
recurse "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5000/VNP46A1/“+”$url_path"+”.h5" “your_directory”+”$url_path" "TOKEN_HERE"
I need to edit this bash to iterate over the files in the csv and download them into a folder for use later.
The bash file is as follows:
#!/bin/bash
function usage {
echo "Usage:"
echo " $0 [options]"
echo ""
echo "Description:"
echo " This script will recursively download all files if they don't exist"
echo " from a LAADS URL and stores them to the specified path"
echo ""
echo "Options:"
echo " -s|--source [URL] Recursively download files at [URL]"
echo " -d|--destination [path] Store directory structure to [path]"
echo " -t|--token [token] Use app token [token] to authenticate"
echo ""
echo "Dependencies:"
echo " Requires 'jq' which is available as a standalone executable from"
echo " https://stedolan.github.io/jq/download/"
}
function recurse {
local src=$1
local dest=$2
local token=$3
echo "Querying ${src}.json"
for dir in $(curl -s -H "Authorization: Bearer ${token}" ${src}.json | jq '.[] | select(.size==0) | .name' | tr -d '"')
do
echo "Creating ${dest}/${dir}"
mkdir -p "${dest}/${dir}"
echo "Recursing ${src}/${dir}/ for ${dest}/${dir}"
recurse "${src}/${dir}/" "${dest}/${dir}"
done
for file in $(curl -s -H "Authorization: Bearer ${token}" ${src}.json | jq '.[] | select(.size!=0) | .name' | tr -d '"')
do
if [ ! -f ${dest}/${file} ]
then
echo "Downloading $file to ${dest}"
# replace '-s' with '-#' below for download progress bars
curl -s -H "Authorization: Bearer ${token}" ${src}/${file} -o ${dest}/${file}
else
echo "Skipping $file ..."
fi
done
}
POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
-s|--source)
src="$2"
shift # past argument
shift # past value
;;
-d|--destination)
dest="$2"
shift # past argument
shift # past value
;;
-t|--token)
token="$2"
shift # past argument
shift # past value
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
if [ -z ${src+x} ]
then
echo "Source is not specified"
usage
exit 1
fi
if [ -z ${dest+x} ]
then
echo "Destination is not specified"
usage
exit 1
fi
if [ -z ${token+x} ]
then
echo "Token is not specified"
usage
exit 1
fi
recurse "$src" "$dest" "$token"
and a shortened (for testing purposes) csv file is given as:
/archive/allData/5000/VNP46A1/2015/001/VNP46A1.A2015001.h30v05.001.2019135185504.h5
/archive/allData/5000/VNP46A1/2015/002/VNP46A1.A2015002.h30v05.001.2019136091632.h5
/archive/allData/5000/VNP46A1/2015/003/VNP46A1.A2015003.h30v05.001.2019136075625.h5
/archive/allData/5000/VNP46A1/2015/004/VNP46A1.A2015004.h30v05.001.2019136081706.h5
/archive/allData/5000/VNP46A1/2015/005/VNP46A1.A2015005.h30v05.001.2019136084155.h5
/archive/allData/5000/VNP46A1/2015/006/VNP46A1.A2015006.h30v05.001.2019136084128.h5
/archive/allData/5000/VNP46A1/2015/007/VNP46A1.A2015007.h30v05.001.2019136085336.h5
/archive/allData/5000/VNP46A1/2015/008/VNP46A1.A2015008.h30v05.001.2019136103147.h5
/archive/allData/5000/VNP46A1/2015/009/VNP46A1.A2015009.h30v05.001.2019136100110.h5
Any help or suggestions will be much appreciated.
Kind regards
You want to create a bash script that will loop over each line to download the data that you need, using the script provided by NASA.
For example say the below script was a file called save-data.sh:
#!/bin/bash
while read p; do
./laads-data-download.sh -s $P -d "destination" -t "token"
echo "$p"
done <paths.txt
Example tree structure:
nasa-satellite-data
├── laads-data-download.sh
├── paths.txt
└── save-data.sh
I run a bash script, and looping as much line in text file. to cURL the site listed in the txt file.
here is my script :
SECRET_KEY='zuhahaha'
FILE_NAME=""
case "$1" in
"sma")
FILE_NAME="sma.txt"
;;
"smk")
FILE_NAME="smk.txt"
;;
"smp")
FILE_NAME="smp.txt"
;;
"sd")
FILE_NAME="sd.txt"
;;
*)
echo "not in case !"
;;
esac
function save_log()
{
printf '%s\n' \
"Header Code : $1" \
"Executed at : $(date)" \
"Response Body : $2" \
"====================================================================================================="$'\r\n\n' >> output.log
}
while IFS= read -r line;
do
HTTP_RESPONSE=$(curl -L -s -w "HTTPSTATUS:%{http_code}\\n" -H "X-Gitlab-Event: Push Hook" -H 'X-Gitlab-Token: '$SECRET_KEY --insecure $line 2>&1) &
HTTP_BODY=$(echo $HTTP_RESPONSE | sed -e 's/HTTPSTATUS\:.*//g') &
HTTP_STATUS=$(echo $HTTP_RESPONSE | tr -d '\n' | sed -e 's/.*HTTPSTATUS://') &
save_log "$HTTP_STATUS" "$HTTP_BODY" &
done < $FILE_NAME
how i can run threading or make the loop fast in bash ?
You should be able to do this relatively easily. Don't try to background each command, but instead put the body of your while loop into a subshell and background that. That way, your commands (which clearly depend on each other) run sequentially, but all the lines in the file can be process in parallel.
while IFS= read -r line;
do
(
HTTP_RESPONSE=$(curl -L -s -w "HTTPSTATUS:%{http_code}\\n" -H "X-Gitlab-Event: Push Hook" -H 'X-Gitlab-Token: '$SECRET_KEY --insecure $line 2>&1)
HTTP_BODY=$(echo $HTTP_RESPONSE | sed -e 's/HTTPSTATUS\:.*//g')
HTTP_STATUS=$(echo $HTTP_RESPONSE | tr -d '\n' | sed -e 's/.*HTTPSTATUS://')
save_log "$HTTP_STATUS" "$HTTP_BODY" ) &
done < $FILE_NAME
My favourite was to do this is generate a file that lists all the commands you wish to perform. If you have a script that performs your operations create a file like:
$ cat commands.txt
echo 1
echo 2
echo $[12+3]
....
For example this could be hundreds of commands long.
To execute each line in parallel, use the parallel command with, say, at most 3 jobs running in parallel at any time.
$ cat commands.txt | parallel -j
1
2
15
For your curl example you could generate thousands of curl commands, execute them say 30 in parallel at any one time.
I have a shell Unix running every hour (crontab on CentOS 7).
Inside that shell, a loop read and proceed treatment for all new files find in a defined folder.
At the end of each files's treatment a CURL command is send with some parameters, for example :
curl https://aaaaaa.com/website -d param1=value1 -d param2=value2 ....
Each time the shell is run by crontab, the 1st CURL is correctly converted to a true URL and received by Apache/Tomcat, but all the others are bad. In fact the 2nd and the following CURLs seem not converted in the correct format like
https://aaaaaa.com/website?param1=value1¶m2=value2
but they are sent like
https://aaaaaa.com/website -d param1=value1 -d param2=value2
So the website is unable to treat the parameters properly.
Why the 1st command is correctly converted to a correct URL format and not the following ?
EDIT - EDIT
The part of shell :
#!/bin/bash
...
#======================================================
# FUNCTIONS
#======================================================
UpdateStatus () {
CMD_CURL="${URL_WEBSITE} -d client=CLIENT -d site=TEST -d produit=MEDIASFILES -d action=update"
CMD_CURL="${CMD_CURL} -d codecmd=UPDATE_MEDIA_STATUS"
CMD_CURL="${CMD_CURL} -d idmedia=$4"
CMD_CURL="${CMD_CURL} -d idbatch=$3"
CMD_CURL="${CMD_CURL} -d statusmedia=$2"
if [[ ! -z "$5" ]]; then
CMD_CURL="${CMD_CURL} -d filename=$5"
fi
echo " ${CMD_CURL}" >> $1
CURL_RESULT=`curl -k ${CMD_CURL}`
CURL_RESULT=`echo ${CURL_RESULT} | tr -d ' '`
echo " Result CURL = ${CURL_RESULT}" >> $1
if [ "${CURL_RESULT}" = "OK" ]; then
return 0
fi
return 1
}
#======================================================
# MAIN PROGRAM
#======================================================
echo "----- Batch in progress : `date '+%d/%m/%y - %H:%M:%S'` -----"
for file in $( ls ${DIR_FACTORY_BATCHFILES}/*.batch )
do
...
old_IFS=$IFS
while IFS=';' read <&3 F_STATUS F_FILEIN F_TYPE F_CODE F_ID F_IDPARENT F_TAGID3 F_PROF F_YEARMEDIA F_DATECOURS F_TIMEBEGINCOURS F_LANG || [[ -n "$F_STATUS $F_FILEIN $F_TYPE $F_CODE $F_ID $F_IDPARENT $F_TAGID3 $F_PROF $F_YEARMEDIA $F_DATECOURS $F_TIMEBEGINCOURS $F_LANG" && $F_STATUS ]];
do
...
UpdateStatus ${LOG_FILENAME} ${STATUS_ERROR} ${F_ID} ${F_IDPARENT}
...
done 3< $file
IFS=$Old_IFS
...
done
You need to provide the "-d" flags and values before the URL so:
curl -d param1=value1 -d param2=value2 https://aaaaaa.com/website
Moreover, this command is going to send the parameters/values as POST parameters, not query parameters. You can use the "-G" flag, possibly combined with "--url-encode" to send as query parameters, see:
https://unix.stackexchange.com/questions/86729/any-way-to-encode-the-url-in-curl-command
when using sed -e to update some parameters of a config file and pipe it to | tee (to write the updated content into the file), this randomly breaks and causes the file to be invalid (size 0).
In Summary, this code is used for updating parameters:
# based on the provided linenumber, add some comments, add the new value, delete old line
sed -e "$lineNr a # comments" -e "$lineNr a $newValue" -e "$lineNr d" $myFile | sudo tee $myFile
I set up an script which calls this update command 100 times.
In a Ubuntu VM (Parallels Desktop) on a shared Directory with OSX this
behaviour occurs up to 50 times
In a Ubuntu VM (Parallels Desktop) on the
Ubuntu partition this behaviour occurs up to 40 times
On a native System (IntelNUC with Ubuntu) this behaviour occurs up to 15 times
Can someone explain why this is happening?
Here is a fully functional script where you can run the experiment as well. (All necessary files are generated by the script, so you can simply copy/paste it into a bashscriptfile and run it)
#!/bin/bash
# main function at bottom
#====================
#===HELPER METHOD====
#====================
# This method updates parameters with a new value. The replacement is performed linewise.
doUpdateParameterInFile()
{
local valueOfInterest="$1"
local newValue="$2"
local filePath="$3"
# stores all matching linenumbers
local listOfLines=""
# stores the linenumber which is going to be replaced
local lineToReplace=""
# find value of interest in all non-commented lines and store related lineNumber
lineToReplace=$( grep -nr "^[^#]*$valueOfInterest" $filePath | sed -n 's/^\([0-9]*\)[:].*/\1/p' )
# Update parameters
# replace the matching line with the desired value
oldValue=$( sed -n "$lineToReplace p" $filePath )
sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" $filePath | sudo tee $filePath >/dev/null
# Sanity check to make sure file did not get corrupted by updating parameters
if [[ ! -s $filePath ]] ; then
echo "[ERROR]: While updating file it turned invalid."
return 31
fi
}
#===============================
#=== Actual Update Function ====
#===============================
main_script()
{
echo -n "Update Parameter1 ..."
doUpdateParameterInFile "Parameter1" "Parameter1 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 33 ; fi
echo -n "Update Parameter2 ..."
doUpdateParameterInFile "Parameter2" "Parameter2=90" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 34 ; fi
echo -n "Update Parameter3 ..."
doUpdateParameterInFile "Parameter3" "Parameter3 YES" "config.txt"
if [[ "$?" == "0" ]] ; then echo "[ OK ]" ; else echo "[FAIL]"; return 35 ; fi
}
#=================
#=== Main Loop ===
#=================
#generate file config.txt
printf "# Configfile with 3 Parameters\n#[Parameter1]\n#only takes YES or NO\nParameter1 NO \n\n#[Parameter2]\n#Parameter2 takes numbers\nParameter2 = 100 \n\n#[Parameter3]\n#Parameter3 takes YES or NO \nParameter3 YES\n" > config.txt
cp config.txt config.txt.bkup
# Start the experiment and let it run 100 times
cnt=0
failSum=0
while [[ $cnt != "100" ]] ; do
echo "==========run: $cnt; fails: $failSum======="
main_script
if [[ $? != "0" ]] ; then cp config.txt.bkup config.txt ; failSum=$(($failSum+1)) ; fi
cnt=$((cnt+1))
sleep 0.5
done
regards
DonPromillo
The problem is that you're using tee to overwrite $filepath at the same time as sed is trying to read from it. If tee truncates it first then sed gets an empty file and you end up with a 0 length file at the other end.
If you have GNU sed you can use the -i flag to have sed modify the file in place (other versions support -i but require an argument to it). If your sed doesn't support it you can have it write to a temp file and move it back to the original name like
tmpname=$(mktemp)
sed -e "$lineToReplace a # $(date '+%Y-%m-%d %H:%M:%S'): replaced: $oldValue with: $newValue" -e "$lineToReplace a $newValue" -e "$lineToReplace d" "$filePath" > "$tmpname"
sudo mv "$tmpname" "$filePath"
or if you want to preserve the original permissions you could do
sudo sh -c "cat '$tmpname' > '$filePath'"
rm "$tmpname"
or use your tee approach like
sudo tee "$filePath" >/dev/null <"$tmpname"
rm "$tmpname"
I have written a shell script which SSH to a remote host and does some processing. The code that executes remotely has to to use the local variables which are read from the properties file. My code is as below. The below code is not executed properly. Its giving an error that
-printf: unknown primary or command.
Please help me with this.
Note: datadir, username and ftphostname are defined in properties file.
. config.properties
ssh $username#$ftphostname << EOF
filelist=;
filelist=($(find "$datadir" -type f -printf "%T# %p\n"| sort -n | head -5 | cut -f2- -d" "));
filecount=\${#filelist[#]};
while [ \${#filelist[#]} -gt 0 ]; do
checkCount=;
filesSize=$(wc -c \${filelist[#]}|tail -n 1 | cut -d " " -f1) ;
if [ "\$filesSize" == "\$fileSizeStored" ]; then
fileSizeStored=0;
printf "\n*********** \$(date) ************* " >> /home/chisan/logs/joblogs.log;
echo "Moved below files" >> /home/joblogs.log;
for i in "\${filelist[#]}"
do
# echo "file is \$i"
checkCount=0;
mv \$i /home/outputdirectory/;
if [ $? -eq 0 ]; then
echo "File Moved to the server: \$i" >> /home/joblogs.log;
else
echo "Error: Failed to move file: \$i" >> /home/joblogs.log;
fi
done
filelist=($(find "$datadir" -type f -printf '%T# %p\n' | sort -n | head -5 | cut -f2- -d" "));
else
((checkCount+=1));
sleep 4;
fileSizeStored=\$filesSize;
fi
done
EOF
But this one works
#ssh to remote system and sort the files and fetch the files which are copied first(based on modification time)
ssh -o StrictHostKeyChecking=no user#server 'filelist=($(find /home/data -type f - printf "%T# %p\n" | sort -n | head -5 | cut -f2- -d" "));
# filelist array variable holds the file names which have the oldest modification date.
#check the directory until it has atleast one file.
while [ ${#filelist[#]} -gt 0 ]; do
filesSize=$(wc -c "${filelist[#]}"|tail -n 1 | cut -d " " -f1) ;
#filesSize contains the total size of the files that are in the filelist array.
if [ -e "$HOME/.storeFilesSize" ]; then
fileSizeStored=$(cat "$HOME/.storeFilesSize");
if [ "$filesSize" == "$fileSizeStored" ]; then
echo "Moved below files" >> /home/joblogs.log;
for i in "${filelist[#]}"
do
mv "$i" /home/dmpdata1 &>/dev/null;
if [ $? -eq 0 ]; then
echo "File Moved to the server: $i" >>/home/joblogs.log;
else
echo "Error: Failed to move file: $i" >>/home/joblogs.log;
fi
done
filelist=($(find /home/data -type f -printf "%T# %p\n" | sort -n | head -5 | cut -f2- -d" "));
else
sleep 4;
echo "$filesSize" > "$HOME/.storeFilesSize";
fi
else
echo "creating new file";
echo "$filesSize" > "$HOME/.storeFilesSize";
fi
done'
I will not answer directly (ie, not with your specific needs and actions), but give a generic possibility and how to use local and remote variables :
Your master script should create a "specific script", locally.
And then copy it over and run it remotely (with additionnal arguments if needed)
Generic example of Master script :
#local Master script: This script creates a local script,
# and then copy it to remotehost and start it
#Some local variables will be defined here.
#They can be used below, and will be replaced by their value locally
localvar1="...."
localvar2="...."
#now we create the script
cat > /tmp/localscript_to_be_copied_to_remote.sh <<EOF
#remote_script
for i in ..... ; do
something ;
somethingelse
done
......
.....
EOF
#in the above, each time you used "$localvar1" or "$localvar2", the script
# /tmp/localscript_to_be_copied_to_remote.sh will instead have their values,
# as the local shell will replace them on the fly during the cat > ... <<EOF .
# if you want to have some remotevariable "as is" (and not as their local value) in the script,
# write them as "\$remotevariable" there, instead of "$remotevariable", so the local shell
# won't interpret them during the 'cat', and the script will receive "$remotevariable"
# as is, instead of its local value.
#then you copy the script:
scp -p /tmp/localscript_to_be_copied_to_remote.sh user#remotehost:/some/dir/name.sh
#and you run it:
# UNCOMMENT the line below ONLY when /tmp/localscript_to_be_copied_to_remote.sh is correct!
# ssh user#remotehost "/some/dir/name.sh" #+ maybe some parameters as well
#end of local Master script.
You then run "local Master script" and have it create the tmp file locally (which you can check to make sure it is supposed to be like this on the remote host), and then copy it remotely and execute it.
Specific example of master script :
#!/bin/bash
local1="/tmp /var /usr /home" # this will be the default name of the dirs (on the remote host)
# that the script will print the size of (+ any additionnal parameters)
cat > /tmp/printsizes.bash <<EOF
#!/bin/bash
for dir in $local1 "\$#" ; do
du -ks "\$dir"
done
EOF
scp -p /tmp/printsizes.bash user#remotehost:/tmp/print_dir_sizes.bash
ssh user#remotehost "/tmp/print_dir_sizes.bash /etc /root"
This (weird...) example will create a LOCAL script containing:
#!/bin/bash
for dir in /tmp /var /usr /home "$#" ; do
du -ks "$dir"
done
And will execute it with:
ssh user#remotehost "/tmp/print_dir_sizes.bash /etc /root"
so it will do remotely:
for dir in /tmp /var /usr /home /etc /root ; do
du -ks "$dir"
done
I hope it helps to see how to use local and remote variables...