I have an pages.txt file with 100 URLs inside. I want to check them one by one and fail on the first problem. This is what I'm doing:
cat pages.txt | xargs -n 1 curl --silent \
--output /dev/null --write-out '%{url_effective}: %{http_code}\n'; echo $?
Exit code is 1, but I see it only when the entire file is done. How to stop earlier, on the first problem?
General method
xargs -n 1 sh -c '<your_command> $0 || exit 255' < input
Specific case
xargs -n 1 sh -c 'curl --silent --output /dev/null \
--write-out "%{url_effective}: %{http_code}\n" $0 || exit 255' < pages.txt
Explanation
For every URL in pages.txt, executes sh -c 'curl ... $0 || exit 255' one by one (-n 1) forcing to exit with 255 if the command fails.
From man xargs:
If any invocation of the command exits with a status of 255, xargs will stop immediately without reading any further input. An error message is issued on stderr when this happens.
I haven't found a way to do what you ask for with xargs, but a loop with read might be what you are looking for.
while read URL; do
curl --silent \
--output /dev/null --write-out '%{url_effective}: %{http_code}\n' $URL;
RET=$?;
echo $RET;
if [ $RET -ne 0 ]; then break; fi
done < pages.txt
Related
I have just trying to write a script which just controls about the response contains "connected" or not
#!/bin/bash
cat control.txt | while read link // control.txt contains http and https urls
do
if [[ $(wget --spider -S $link 2>&1 | grep "connected") =~~ *"connected"* ]];
then echo "OK";
else echo "FAIL";
fi
done
Output:
sh -x portcontrol.sh
portcontrol.sh[2]: Syntax error at line 4 : `=~' is not expected.
If I read your script correctly, you're retrieving the page, but ignoring its contents, and all you want is to see whether wget shows the string 'connected'.
If that is so, your code can be simplified as follows:
if wget --spider -S $link 2>&1 | grep "connected" > /dev/null
then
echo "OK";
else
echo "FAIL";
fi
You don't need to capture wget's output and run a regexp search on it; grep already returns 0 (success) or 1 (not found) when searching for the string you gave.
That return code can be used directly to control the if.
The output of grep is redirected to /dev/null as to not show up on the screen or script output.
If you simply want to see if the connection request succeeded, and the wget output is of the form:
Connecting to <hostname>|<ip_addr>|:<port>... connected.
it should be sufficient to just do:
if [[ $(wget --spider -S $link 2>&1 | grep -c " connected\.") -gt 0 ]];
then echo "OK";
else echo "FAIL";
fi
Checking exit code works too, but it depends on what your requirements really are.
I run a bash script, and looping as much line in text file. to cURL the site listed in the txt file.
here is my script :
SECRET_KEY='zuhahaha'
FILE_NAME=""
case "$1" in
"sma")
FILE_NAME="sma.txt"
;;
"smk")
FILE_NAME="smk.txt"
;;
"smp")
FILE_NAME="smp.txt"
;;
"sd")
FILE_NAME="sd.txt"
;;
*)
echo "not in case !"
;;
esac
function save_log()
{
printf '%s\n' \
"Header Code : $1" \
"Executed at : $(date)" \
"Response Body : $2" \
"====================================================================================================="$'\r\n\n' >> output.log
}
while IFS= read -r line;
do
HTTP_RESPONSE=$(curl -L -s -w "HTTPSTATUS:%{http_code}\\n" -H "X-Gitlab-Event: Push Hook" -H 'X-Gitlab-Token: '$SECRET_KEY --insecure $line 2>&1) &
HTTP_BODY=$(echo $HTTP_RESPONSE | sed -e 's/HTTPSTATUS\:.*//g') &
HTTP_STATUS=$(echo $HTTP_RESPONSE | tr -d '\n' | sed -e 's/.*HTTPSTATUS://') &
save_log "$HTTP_STATUS" "$HTTP_BODY" &
done < $FILE_NAME
how i can run threading or make the loop fast in bash ?
You should be able to do this relatively easily. Don't try to background each command, but instead put the body of your while loop into a subshell and background that. That way, your commands (which clearly depend on each other) run sequentially, but all the lines in the file can be process in parallel.
while IFS= read -r line;
do
(
HTTP_RESPONSE=$(curl -L -s -w "HTTPSTATUS:%{http_code}\\n" -H "X-Gitlab-Event: Push Hook" -H 'X-Gitlab-Token: '$SECRET_KEY --insecure $line 2>&1)
HTTP_BODY=$(echo $HTTP_RESPONSE | sed -e 's/HTTPSTATUS\:.*//g')
HTTP_STATUS=$(echo $HTTP_RESPONSE | tr -d '\n' | sed -e 's/.*HTTPSTATUS://')
save_log "$HTTP_STATUS" "$HTTP_BODY" ) &
done < $FILE_NAME
My favourite was to do this is generate a file that lists all the commands you wish to perform. If you have a script that performs your operations create a file like:
$ cat commands.txt
echo 1
echo 2
echo $[12+3]
....
For example this could be hundreds of commands long.
To execute each line in parallel, use the parallel command with, say, at most 3 jobs running in parallel at any time.
$ cat commands.txt | parallel -j
1
2
15
For your curl example you could generate thousands of curl commands, execute them say 30 in parallel at any one time.
I try to solve this problem with the sh and not the bash.
All i want is a if statement that check some regex AND something else. Normally in bash this would be an easy job but with the sh i only find solutions online that wont work for me
First thing i want to check:
if echo "$1"| grep -E -q '^(\-t|\-\-test)$';
Than i want to check:
if echo "$#"| grep -E -q '^(1|2)$';
Combined:
if [ \(echo "$1"| grep -E -q '^(\-h|\-\-help)$'\) -a \(echo "$#"| grep -E -q '^(1|\2)$'\) ];
ERROR:
grep: grep: ./analysehtml.sh: 41: [: missing ]
Invalid back reference
(echo: No such file or directory
grep: 1: No such file or directory
I also try many diffrent combinations with this brackets but non of them worked for me. Maybe someone can help me here :)
logical and between commands is &&
if
echo "$1"| grep -E -q '^(\-h|\-\-help)$' &&
echo "$#"| grep -E -q '^(1|\2)$';
By default the exit status of a pipe is the exit status of last command.
set -o pipefail the exit status is fail if if any command of pipe has a fail exit status.
when only the exit status of the last command of a sequence must be checked
if { command11; command12;} && { command21; command22;};
However to check parameters there is no need to launch another process grep with a pipe there's an overhead.
Consider using following constructs work with any POSIX sh.
if { [ "$1" = -h ] || [ "$1" = --help ];} &&
{ [ $# -eq 1 ] || [ $# -eq 2 ];};
EDIT: Following are not POSIX but may work with many shell
if [[ $1 = -h || $1 = --help ]] && [[ $# = 1 || $# = 2 ]];
Works also with bash with set -o posix
Perhaps for your particular case, pattern matching might be better:
if [[ $1 =~ ^(\-h|\-\-help)$ && $# =~ ^(1|\2)$ ]]; then
The problem with your command is that the part within test or [ command is expression, not commands list.
So when you run [ echo 'hello' ] or [ \( echo 'hello' \) ] complains error in spite of sh or Bash. Refer to the classic test usage: The classic test command
And the syntax of if is:
if list; then list; fi
So you can just combine command with && operator in if statements:
if echo "$1"| grep -E -q '^(\-h|\-\-help)$' && echo "$#"| grep -E -q '^(1|\2)$';
I have a webservice provided at http://localhost/test/testweb
I want to write a script to check if webservice is up with curl
If there a curl parameter given, returns 200 OK ok true false so that I can use it is if-else block in linux script
curl -sL -w "%{http_code}\\n" "http://www.google.com/" -o /dev/null
-s = Silent cURL's output
-L = Follow redirects
-w = Custom output format
-o = Redirects the HTML output to /dev/null
Example:
[~]$ curl -sL -w "%{http_code}\\n" "http://www.google.com/" -o /dev/null
200
I would probably remove the \\n if I were to capture the output.
I use:
curl -f -s -I "http://example.com" &>/dev/null && echo OK || echo FAIL
-f --fail Fail silently (no output at all) on HTTP errors
-s --silent Silent mode
-I --head Show document info only
Note:
depending on needs you can also remove the "-I" because in some cases you need to do a GET and not a HEAD
Same as #burhan-khalid, but added --connect-timeout 3 and --max-time 5.
test_command='curl -sL \
-w "%{http_code}\\n" \
"http://www.google.com:8080/" \
-o /dev/null \
--connect-timeout 3 \
--max-time 5'
if [ $(test_command) == "200" ] ;
then
echo "OK" ;
else
echo "KO" ;
fi
That will check the headers via wget 2>&1pipes the stderr to stdout
grep filters
-O /dev/null just throws the content of the page
if [ "\`wget http://example.org/ -O /dev/null -S --quiet 2>&1 | grep '200 OK'\`" != "" ];
then
echo Hello;
fi;
I know not curl, but still a solution
I needed a better answer to this, so I wrote the script below.
The fakePhrase is used to detect ISP "Search Assist" adware HTTP resposnes.
#!/bin/bash
fakePhrase="verizon"
siteList=(
'http://google.com'
'https://google.com'
'http://wikipedia.org'
'https://wikipedia.org'
'http://cantgettherefromhere'
'http://searchassist.verizon.com'
)
exitStatus=0
function isUp {
http=`curl -sL -w "%{http_code}" "$1" -o temp_isUp`
fakeResponse=`cat temp_isUp | grep $fakePhrase`
if [ -n "$fakeResponse" ]; then
http=$fakePhrase
fi
case $http in
[2]*)
;;
[3]*)
echo 'Redirect'
;;
[4]*)
exitStatus=4
echo "$1 is DENIED with ${http}"
;;
[5]*)
exitStatus=5
echo "$1 is ERROR with ${http}"
;;
*)
exitStatus=6
echo "$1 is NO RESPONSE with ${http}"
;;
esac
}
for var in "${siteList[#]}"
do
isUp $var
done
if [ "$exitStatus" -eq "0" ]; then
echo 'All up'
fi
rm temp_isUp
exit $exitStatus
Use this:
curl -o $CURL_OUTPUT -s -w %{http_code}\\n%{time_total}\\n $URL > $TMP_FILE 2>&1
cat $TMP_FILE
Writing a simple bash script to do some checks for me in the morning:
One part is pulling down some html files and making sure that they exist.
The other part is ensuring some local files are present, and emailing if they are not.
The problem I am facing is that I am receiving a "syntax error: unexpected end of file" and I can't really understand why it i occuring.
Here is a simplified version of the code:
for myHost in $HOSTS
do
result=$(wget -T $TIMEOUT -t 1 $myHost -O /dev/null -o /dev/stdout)
result2=$(echo $result | grep "awaiting response")
connected=$(echo $result | grep "404");
if [ "$connected" != "" ]; then
for myEMAIL in $EMAIL
do
echo -e "$(date) - $myHost is down! \n This is an automated message." | mailx -r "box.email.com" -s "$SUBJECT" $myEMAIL
done
fi
done
numoffiles=`find . -maxdepth 1 -mtime -1 | grep -i .html | wc -l`
if [ "$numoffiles" -ne 5 ]; then
FILE=$(find . -maxdepth 1 -mtime -1 -print| grep -i .html)
mailx -r "box.email.com" -s "FILE MISSIN" "$EMAIL" << EOF
EOF
fi
from using sh -x I can see that it gets to assigning the number of reports to the var "numoffiles", but then it just believes that is the end of the file. has anyone got any suggestions?
There should not be any space before the end of heredoc label:
EOF
^^^
Change it to
EOF