I have a shell Unix running every hour (crontab on CentOS 7).
Inside that shell, a loop read and proceed treatment for all new files find in a defined folder.
At the end of each files's treatment a CURL command is send with some parameters, for example :
curl https://aaaaaa.com/website -d param1=value1 -d param2=value2 ....
Each time the shell is run by crontab, the 1st CURL is correctly converted to a true URL and received by Apache/Tomcat, but all the others are bad. In fact the 2nd and the following CURLs seem not converted in the correct format like
https://aaaaaa.com/website?param1=value1¶m2=value2
but they are sent like
https://aaaaaa.com/website -d param1=value1 -d param2=value2
So the website is unable to treat the parameters properly.
Why the 1st command is correctly converted to a correct URL format and not the following ?
EDIT - EDIT
The part of shell :
#!/bin/bash
...
#======================================================
# FUNCTIONS
#======================================================
UpdateStatus () {
CMD_CURL="${URL_WEBSITE} -d client=CLIENT -d site=TEST -d produit=MEDIASFILES -d action=update"
CMD_CURL="${CMD_CURL} -d codecmd=UPDATE_MEDIA_STATUS"
CMD_CURL="${CMD_CURL} -d idmedia=$4"
CMD_CURL="${CMD_CURL} -d idbatch=$3"
CMD_CURL="${CMD_CURL} -d statusmedia=$2"
if [[ ! -z "$5" ]]; then
CMD_CURL="${CMD_CURL} -d filename=$5"
fi
echo " ${CMD_CURL}" >> $1
CURL_RESULT=`curl -k ${CMD_CURL}`
CURL_RESULT=`echo ${CURL_RESULT} | tr -d ' '`
echo " Result CURL = ${CURL_RESULT}" >> $1
if [ "${CURL_RESULT}" = "OK" ]; then
return 0
fi
return 1
}
#======================================================
# MAIN PROGRAM
#======================================================
echo "----- Batch in progress : `date '+%d/%m/%y - %H:%M:%S'` -----"
for file in $( ls ${DIR_FACTORY_BATCHFILES}/*.batch )
do
...
old_IFS=$IFS
while IFS=';' read <&3 F_STATUS F_FILEIN F_TYPE F_CODE F_ID F_IDPARENT F_TAGID3 F_PROF F_YEARMEDIA F_DATECOURS F_TIMEBEGINCOURS F_LANG || [[ -n "$F_STATUS $F_FILEIN $F_TYPE $F_CODE $F_ID $F_IDPARENT $F_TAGID3 $F_PROF $F_YEARMEDIA $F_DATECOURS $F_TIMEBEGINCOURS $F_LANG" && $F_STATUS ]];
do
...
UpdateStatus ${LOG_FILENAME} ${STATUS_ERROR} ${F_ID} ${F_IDPARENT}
...
done 3< $file
IFS=$Old_IFS
...
done
You need to provide the "-d" flags and values before the URL so:
curl -d param1=value1 -d param2=value2 https://aaaaaa.com/website
Moreover, this command is going to send the parameters/values as POST parameters, not query parameters. You can use the "-G" flag, possibly combined with "--url-encode" to send as query parameters, see:
https://unix.stackexchange.com/questions/86729/any-way-to-encode-the-url-in-curl-command
Related
I have a text file already that contain project id and project_names. But a request has been made to call together both group and project together in the same curl.
How can I call group and project id together and how would I modify the text file (config.txt) that contain the projectid and projectnames to contain the groupid?
config.txt
324566 projectnames1 projecttypes1
333456 projectnames2 projecttypes2
IN_FILE='./config.txt'
if [ -f "$IN_FILE" ]; then
echo "$IN_FILE exists and the operation can continue"
while read -ra LINE
do
a="${LINE[0]}"; b="${LINE[1]}"; c="${LINE[2]}"
echo "$a"
echo "$b"
echo "$c"
projectid=$a
projectnames=$b
projecttypes=$c
mkdir ./archive_$projectid-$projectnames
sudo chmod -R 755 /Users/names/git/archive_$projectid-$projectnames
curlfile=$(curl -s -I --header "PRIVATE-TOKEN:xxxxxxx" "https://gitlab.com/api/v4/projects/$projectid/variables" | grep HTTP/ | awk {'print $2'})
echo "$curlfile"
### This will check if status response is between 200 -299 inclusive
if (( $curlfile >= 200 && $curlfile <= 300 )) ; then
echo "$curlfile exists and the operation can start processing"
curl --header "PRIVATE-TOKEN:xxxxxxxxx" "https://gitlab.com/api/v4/projects/$projectid/variables" | python3 -m json.tool > $projectid-$projectnames.json
# echo "https://gitlab.com/api/v4/projects/$projectid/variables"
I used this bash script here to download only short parts of long yt videos:
#!/bin/bash
#taken from https://unix.stackexchange.com/a/388148/48971
if [ $# -lt 4 ]; then
echo "Usage: $0 <youtube's URL> <HH:mm:ss.milisecs from time> <HH:mm:ss.milisecs to time> <output_file_name>"
echo "e.g.:"
echo "$0 https://www.youtube.com/watch?v=T1n5gXIPyws 00:00:25 00:00:42 intro.mp4"
exit 1
fi
echo "processing..."
from=$(date "+%s" -d "UTC 01/01/1970 $2")
to=$(date "+%s" -d "UTC 01/01/1970 $3")
from_pre=$(($from - 30))
if [ $from_pre -lt 0 ]
then
from_pre=0
fi
from_pre_command_print=$(date -u "+%T" -d #$from_pre)
from_command_print=$(date -u "+%T" -d #$(($from - $from_pre)))$(grep -o "\..*" <<< $2)
to_command_print=$(date -u "+%T" -d #$(($to - $from_pre)))$(grep -o "\..*" <<< $3)
command="ffmpeg "
for uri in $(youtube-dl -g $1)
do
command+="-ss $from_pre_command_print -i $uri "
done
command+="-ss $from_command_print -to $to_command_print $4"
echo "downloading with the following command:"
echo "$command"
$command
but the problem is its only precise to the full second.
I want to use it to download a bunch of very short clips (only one word long mostly) those are often below 1 second long.
I tryed to fix this by using milliseconds in the date.
But then i found out bash can only subtract with integers.
Here is what i tryed:
from=$(date "+%s.%3N" -d "UTC 01/01/1970 $2")
to=$(date "+%s.%3N" -d "UTC 01/01/1970 $3")
function diff {
diff="$(echo $from - 5 | bc)"
echo $diff
}
from_pre=$diff
echo $diff
but the workaround using bc didnt work because later in the script its again throwing errors because bash dosnt know what to do with non integers.
a sample command would look like this:
sh download_youtube.sh https://www.youtube.com/watch?v=dH3auOKyxio 00:06:28.230 00:06:28.740 clip004.mp4
and this works...if the the timeframe is more than 1 second.
Sadly i am out of knowlage here how to make it more precise.
This all is part of a project to make automated compilations of specific word from a youtube channel more on that here:
https://github.com/moeC137/video-recutter
I am very thankfull for every help
edit:
after a bit try an error i found out that this seams to work for me:
#!/bin/bash
#taken from https://unix.stackexchange.com/a/388148/48971
if [ $# -lt 4 ]; then
echo "Usage: $0 <youtube's URL> <HH:mm:ss.milisecs from time> <HH:mm:ss.milisecs to time> <output_file_name>"
echo "e.g.:"
echo "$0 https://www.youtube.com/watch?v=T1n5gXIPyws 00:00:25 00:00:42 intro.mp4"
exit 1
fi
echo "processing..."
from=$(date "+%s" -d "UTC 01/01/1970 $2")
to=$(date "+%s" -d "UTC 01/01/1970 $3")
from_pre=$(($from - 20))
#to_post=$(($to + 20))
if [ $from_pre -lt 0 ]
then
from_pre=0
fi
from_pre_command_print=$(date -u "+%T" -d #$from_pre)
from_command_print=$(date -u "+%T" -d #$(($from - $from_pre)))$(grep -o "\..*" <<< $2)
to_command_print=$(date -u "+%T" -d #$(($to - $from_pre)))$(grep -o "\..*" <<< $3)
#to_post_command_print=$(date -u "+%T.%3N" -d #$to_post)
command="ffmpeg "
for uri in $(youtube-dl -g $1)
do
command+="-ss $from_pre_command_print -i $uri "
done
command+="-ss $from_command_print -to $to_command_print $4"
echo "downloading with the following command:"
echo "$command"
$command
But i starting to think the orginal script actually works with milleseconds.
Because this:
sh download_youtube.sh https://www.youtube.com/watch?v=dH3auOKyxio 00:06:28.230 00:06:28.740 clip004.mp4
works, but when try to combine it with the script from here How to repeat a command for every line in a textfile with given arguments from the textfile (Bash?) it dosent use the milliseconds anymore.
Solved
The orginal Script worked. It just didnt like the "," in the timestamps, i replaced them with "." and now its working.
solved by replacing "," in timestamps with "."
I am currently trying to download data from (https://ladsweb.modaps.eosdis.nasa.gov/search/order). They provide a bash file (https://ladsweb.modaps.eosdis.nasa.gov/tools-and-services/data-download-scripts/) which I need to try and edit. I have a .csv file with all of the file extensions I need to download (specifically I need all VNP46A1 data for China from 2015 until now.
In pseudo-code form I would like to add the following:
FOR url_path IN url_list:
recurse "https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5000/VNP46A1/“+”$url_path"+”.h5" “your_directory”+”$url_path" "TOKEN_HERE"
I need to edit this bash to iterate over the files in the csv and download them into a folder for use later.
The bash file is as follows:
#!/bin/bash
function usage {
echo "Usage:"
echo " $0 [options]"
echo ""
echo "Description:"
echo " This script will recursively download all files if they don't exist"
echo " from a LAADS URL and stores them to the specified path"
echo ""
echo "Options:"
echo " -s|--source [URL] Recursively download files at [URL]"
echo " -d|--destination [path] Store directory structure to [path]"
echo " -t|--token [token] Use app token [token] to authenticate"
echo ""
echo "Dependencies:"
echo " Requires 'jq' which is available as a standalone executable from"
echo " https://stedolan.github.io/jq/download/"
}
function recurse {
local src=$1
local dest=$2
local token=$3
echo "Querying ${src}.json"
for dir in $(curl -s -H "Authorization: Bearer ${token}" ${src}.json | jq '.[] | select(.size==0) | .name' | tr -d '"')
do
echo "Creating ${dest}/${dir}"
mkdir -p "${dest}/${dir}"
echo "Recursing ${src}/${dir}/ for ${dest}/${dir}"
recurse "${src}/${dir}/" "${dest}/${dir}"
done
for file in $(curl -s -H "Authorization: Bearer ${token}" ${src}.json | jq '.[] | select(.size!=0) | .name' | tr -d '"')
do
if [ ! -f ${dest}/${file} ]
then
echo "Downloading $file to ${dest}"
# replace '-s' with '-#' below for download progress bars
curl -s -H "Authorization: Bearer ${token}" ${src}/${file} -o ${dest}/${file}
else
echo "Skipping $file ..."
fi
done
}
POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
-s|--source)
src="$2"
shift # past argument
shift # past value
;;
-d|--destination)
dest="$2"
shift # past argument
shift # past value
;;
-t|--token)
token="$2"
shift # past argument
shift # past value
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
if [ -z ${src+x} ]
then
echo "Source is not specified"
usage
exit 1
fi
if [ -z ${dest+x} ]
then
echo "Destination is not specified"
usage
exit 1
fi
if [ -z ${token+x} ]
then
echo "Token is not specified"
usage
exit 1
fi
recurse "$src" "$dest" "$token"
and a shortened (for testing purposes) csv file is given as:
/archive/allData/5000/VNP46A1/2015/001/VNP46A1.A2015001.h30v05.001.2019135185504.h5
/archive/allData/5000/VNP46A1/2015/002/VNP46A1.A2015002.h30v05.001.2019136091632.h5
/archive/allData/5000/VNP46A1/2015/003/VNP46A1.A2015003.h30v05.001.2019136075625.h5
/archive/allData/5000/VNP46A1/2015/004/VNP46A1.A2015004.h30v05.001.2019136081706.h5
/archive/allData/5000/VNP46A1/2015/005/VNP46A1.A2015005.h30v05.001.2019136084155.h5
/archive/allData/5000/VNP46A1/2015/006/VNP46A1.A2015006.h30v05.001.2019136084128.h5
/archive/allData/5000/VNP46A1/2015/007/VNP46A1.A2015007.h30v05.001.2019136085336.h5
/archive/allData/5000/VNP46A1/2015/008/VNP46A1.A2015008.h30v05.001.2019136103147.h5
/archive/allData/5000/VNP46A1/2015/009/VNP46A1.A2015009.h30v05.001.2019136100110.h5
Any help or suggestions will be much appreciated.
Kind regards
You want to create a bash script that will loop over each line to download the data that you need, using the script provided by NASA.
For example say the below script was a file called save-data.sh:
#!/bin/bash
while read p; do
./laads-data-download.sh -s $P -d "destination" -t "token"
echo "$p"
done <paths.txt
Example tree structure:
nasa-satellite-data
├── laads-data-download.sh
├── paths.txt
└── save-data.sh
I have just trying to write a script which just controls about the response contains "connected" or not
#!/bin/bash
cat control.txt | while read link // control.txt contains http and https urls
do
if [[ $(wget --spider -S $link 2>&1 | grep "connected") =~~ *"connected"* ]];
then echo "OK";
else echo "FAIL";
fi
done
Output:
sh -x portcontrol.sh
portcontrol.sh[2]: Syntax error at line 4 : `=~' is not expected.
If I read your script correctly, you're retrieving the page, but ignoring its contents, and all you want is to see whether wget shows the string 'connected'.
If that is so, your code can be simplified as follows:
if wget --spider -S $link 2>&1 | grep "connected" > /dev/null
then
echo "OK";
else
echo "FAIL";
fi
You don't need to capture wget's output and run a regexp search on it; grep already returns 0 (success) or 1 (not found) when searching for the string you gave.
That return code can be used directly to control the if.
The output of grep is redirected to /dev/null as to not show up on the screen or script output.
If you simply want to see if the connection request succeeded, and the wget output is of the form:
Connecting to <hostname>|<ip_addr>|:<port>... connected.
it should be sufficient to just do:
if [[ $(wget --spider -S $link 2>&1 | grep -c " connected\.") -gt 0 ]];
then echo "OK";
else echo "FAIL";
fi
Checking exit code works too, but it depends on what your requirements really are.
I run a bash script, and looping as much line in text file. to cURL the site listed in the txt file.
here is my script :
SECRET_KEY='zuhahaha'
FILE_NAME=""
case "$1" in
"sma")
FILE_NAME="sma.txt"
;;
"smk")
FILE_NAME="smk.txt"
;;
"smp")
FILE_NAME="smp.txt"
;;
"sd")
FILE_NAME="sd.txt"
;;
*)
echo "not in case !"
;;
esac
function save_log()
{
printf '%s\n' \
"Header Code : $1" \
"Executed at : $(date)" \
"Response Body : $2" \
"====================================================================================================="$'\r\n\n' >> output.log
}
while IFS= read -r line;
do
HTTP_RESPONSE=$(curl -L -s -w "HTTPSTATUS:%{http_code}\\n" -H "X-Gitlab-Event: Push Hook" -H 'X-Gitlab-Token: '$SECRET_KEY --insecure $line 2>&1) &
HTTP_BODY=$(echo $HTTP_RESPONSE | sed -e 's/HTTPSTATUS\:.*//g') &
HTTP_STATUS=$(echo $HTTP_RESPONSE | tr -d '\n' | sed -e 's/.*HTTPSTATUS://') &
save_log "$HTTP_STATUS" "$HTTP_BODY" &
done < $FILE_NAME
how i can run threading or make the loop fast in bash ?
You should be able to do this relatively easily. Don't try to background each command, but instead put the body of your while loop into a subshell and background that. That way, your commands (which clearly depend on each other) run sequentially, but all the lines in the file can be process in parallel.
while IFS= read -r line;
do
(
HTTP_RESPONSE=$(curl -L -s -w "HTTPSTATUS:%{http_code}\\n" -H "X-Gitlab-Event: Push Hook" -H 'X-Gitlab-Token: '$SECRET_KEY --insecure $line 2>&1)
HTTP_BODY=$(echo $HTTP_RESPONSE | sed -e 's/HTTPSTATUS\:.*//g')
HTTP_STATUS=$(echo $HTTP_RESPONSE | tr -d '\n' | sed -e 's/.*HTTPSTATUS://')
save_log "$HTTP_STATUS" "$HTTP_BODY" ) &
done < $FILE_NAME
My favourite was to do this is generate a file that lists all the commands you wish to perform. If you have a script that performs your operations create a file like:
$ cat commands.txt
echo 1
echo 2
echo $[12+3]
....
For example this could be hundreds of commands long.
To execute each line in parallel, use the parallel command with, say, at most 3 jobs running in parallel at any time.
$ cat commands.txt | parallel -j
1
2
15
For your curl example you could generate thousands of curl commands, execute them say 30 in parallel at any one time.