here i am trying to check if wget -s returns HTTP/1.1 200 OK using shell script.
i am using this wget command to get the http status from url
#!/bin/sh
#
URL="http://www.example.com"
wget -S $URL
if i get it returns HTTP/1.1 200 OK, then it should exit else run scr.sh script.
How can i do this?
Here I'm using exit code of silent wget command:
wget -q --spider $URL
if [ $? -ne 0 ] ; then
echo "do something"
fi
In case if response is an error it runs some command.
Using wget:
SHOULD_EXIT=$(wget --server-response --content-on-error=off ${URL} | awk -F':' '$1 ~ / Status$/ { print $2 ~ /200 OK/ }')
${SHOULD_EXIT} will be 1 for 200 OK and 0 otherwise.
Related
I am writing a shell script, I need to download 6 files from internet, In the script I have done it as
curl a
curl b
curl c
and so on
It works, but sometimes curl: (7) couldn't connect to host for some files in the middle of script for example, It will successfully download a but miss the file b with the above error and will download the file c. I would like to catch this error so that my code will execute with the successful download of all the files.
You can use a loop:
while true; do
curl --fail b && break
done
The loop won't break until b is downloaded. You can make it a retry function which you can call if a download fails on the first try:
retry(){
while true; do
curl --fail "$1" && break ||
echo "Download failed for url: $1
retrying..."
done
}
Then do this:
curl --fail a || retry a
curl --fail b || retry b
curl --fail c || retry c
If you just want to silent the error messages then:
curl a 2>/dev/null
curl b 2>/dev/null
...
Or if you want to just detect the error then:
if ! curl --fail a; then
echo "Failed"
fi
or, a one liner:
curl --fail a || echo Failed
If you want to exit after a failure and also show your own message:
curl --fail a 2>/dev/null || { echo failed; exit 1; }
You could chain them with &&...
curl --fail a && curl --fail b && curl --fail c...
Update: as #nwk pointed out below, we need to add --fail to make curl fail on bad http codes.
Put set -e at the beginning of the shell script and use curl --fail. E.g.,
#!/bin/sh
set -e
curl --fail http://example.com/a
curl --fail http://example.com/b
curl --fail http://example.com/c
The set -e will make the script stop with an error on the first unsuccessful command (one with an exit status ≠ zero).
You can check the status of execution of the last command by looking at the $? shell variable.
Execute command below to check the status of the last command anything other 0 shall indicate an error.
echo $?
curl a
if [ "$?" -gt 0 ]
then
echo "Error downloading file a. Exiting"
exit
fi
curl b
if [ "$?" -gt 0 ]
then
echo "Error downloading file b. Exiting"
exit
fi
...
A simple modified form after #andlrc suggestion:
if ! curl a ; then echo "Got error downloading a"; fi
if ! curl b ; then echo "Got error downloading b"; fi
if ! curl c ; then echo "Got error downloading c"; fi
for file2get in a b c d;
do
do :;
until curl --fail $file2get;
done
Or add an iterator counter to prevent endless looping
I need to get exit code of ftp execution. My command line is:
wget -N ftp://server:pass#server/path/
Using:
if [ $? -ne 0 ]; then
will check wget execution.
not tested:
wget [wget options] 2>&1 | grep -i "failed\|error"
I'm struggling with a problem in linux bash.
I want a script to execute a command
curl -s --head http://myurl/ | head -n 1
and if the result of the command contains 200 it executes another command.
Else it is echoing something.
What i have now:
CURLCHECK=curl -s --head http://myurl | head -n 1
if [[ $($CURLCHECK) =~ "200" ]]
then
echo "good"
else
echo "bad"
fi
The script prints:
HTTP/1.1 200 OK
bad
I tried many ways but none of them seems to work.
Can someone help me?
I would do something like this:
if curl -s --head http://myurl | head -n 1 | grep "200" >/dev/null 2>&1; then
echo good
else
echo bad
fi
You need to actually capture the output from the curl command:
CURLCHECK=$(curl -s --head http://myurl | head -n 1)
I'm surprised you're not getting a "-s: command not found" error
You can use this curl command with -w "%{http_code}" to just get http status code:
[[ $(curl -s -w "%{http_code}" -A "Chrome" -L "http://myurl/" -o /dev/null) == 200 ]] &&
echo "good" || echo "bad"
using wget
if wget -O /dev/null your_url 2>&1 | grep -F HTTP >/dev/null 2>&1 ;then echo good;else echo bad; fi
I have to check the status of 200 http URLs and find out which of these are broken links. The links are present in a simple text file (say URL.txt present in my ~ folder). I am using Ubuntu 14.04 and I am a Linux newbie. But I understand the bash shell is very powerful and could help me achieve what I want.
My exact requirement would be to read the text file which has the list of URLs and automatically check if the links are working and write the response to a new file with the URLs and their corresponding status (working/broken).
I created a file "checkurls.sh" and placed it in my home directory where the urls.txt file is also located. I gave execute privileges to the file using
$chmod +x checkurls.sh
The contents of checkurls.sh is given below:
#!/bin/bash
while read url
do
urlstatus=$(curl -o /dev/null --silent --head --write-out '%{http_code}' "$url" )
echo "$url $urlstatus" >> urlstatus.txt
done < $1
Finally, I executed it from command line using the following -
$./checkurls.sh urls.txt
Voila! It works.
#!/bin/bash
while read -ru 4 LINE; do
read -r REP < <(exec curl -IsS "$LINE" 2>&1)
echo "$LINE: $REP"
done 4< "$1"
Usage:
bash script.sh urls-list.txt
Sample:
http://not-exist.com/abc.html
https://kernel.org/nothing.html
http://kernel.org/index.html
https://kernel.org/index.html
Output:
http://not-exist.com/abc.html: curl: (6) Couldn't resolve host 'not-exist.com'
https://kernel.org/nothing.html: HTTP/1.1 404 Not Found
http://kernel.org/index.html: HTTP/1.1 301 Moved Permanently
https://kernel.org/index.html: HTTP/1.1 200 OK
For everything, read the Bash Manual. See man curl, help, man bash as well.
What about to add some parallelism to the accepted solution. Lets modify the script chkurl.sh to be little easier to read and to handle just one request at a time:
#!/bin/bash
URL=${1?Pass URL as parameter!}
curl -o /dev/null --silent --head --write-out "$URL %{http_code} %{redirect_url}\n" "$URL"
And now you check your list using:
cat URL.txt | xargs -P 4 -L1 ./chkurl.sh
This could finish the job up to 4 times faster.
Herewith my full script that checks URLs listed in a file passed as an argument e.g. 'checkurls.sh listofurls.txt'.
What it does:
check url using curl and return HTTP status code
send email notifications when url returns other code than 200
create a temporary lock file for failed urls (file naming could be improved)
send email notification when url becoms available again
remove lock file once url becomes available to avoid further notifications
log events to a file and handle increasing log file size (AKA log
rotation, uncomment echo if code 200 logging required)
Code:
#!/bin/sh
EMAIL=" your#email.com"
DATENOW=`date +%Y%m%d-%H%M%S`
LOG_FILE="checkurls.log"
c=0
while read url
do
((c++))
LOCK_FILE="checkurls$c.lock"
urlstatus=$(/usr/bin/curl -H 'Cache-Control: no-cache' -o /dev/null --silent --head --write-out '%{http_code}' "$url" )
if [ "$urlstatus" = "200" ]
then
#echo "$DATENOW OK $urlstatus connection->$url" >> $LOG_FILE
[ -e $LOCK_FILE ] && /bin/rm -f -- $LOCK_FILE > /dev/null && /bin/mail -s "NOTIFICATION URL OK: $url" $EMAIL <<< 'The URL is back online'
else
echo "$DATENOW FAIL $urlstatus connection->$url" >> $LOG_FILE
if [ -e $LOCK_FILE ]
then
#no action - awaiting URL to be fixed
:
else
/bin/mail -s "NOTIFICATION URL DOWN: $url" $EMAIL <<< 'Failed to reach or URL problem'
/bin/touch $LOCK_FILE
fi
fi
done < $1
# REMOVE LOG FILE IF LARGER THAN 100MB
# alow up to 2000 lines average
maxsize=120000
size=$(/usr/bin/du -k "$LOG_FILE" | /bin/cut -f 1)
if [ $size -ge $maxsize ]; then
/bin/rm -f -- $LOG_FILE > /dev/null
echo "$DATENOW LOG file [$LOG_FILE] has been recreated" > $LOG_FILE
else
#do nothing
:
fi
Please note that changing order of listed urls in text file will affect any existing lock files (remove all .lock files to avoid confusion). It would be improved by using url as file name but certain characters such as : # / ? & would have to be handled for operating system.
I recently released deadlink, a command-line tool for finding broken links in files. Install with
pip install deadlink
and use as
deadlink check /path/to/file/or/directory
or
deadlink replace-redirects /path/to/file/or/directory
The latter will replace permanent redirects (301) in the specified files.
Example output:
if your input file contains one url per line you can use a script to read each line, then try to ping the url, if ping success then the url is valid
#!/bin/bash
INPUT="Urls.txt"
OUTPUT="result.txt"
while read line ;
do
if ping -c 1 $line &> /dev/null
then
echo "$line valid" >> $OUTPUT
else
echo "$line not valid " >> $OUTPUT
fi
done < $INPUT
exit
ping options :
-c count
Stop after sending count ECHO_REQUEST packets. With deadline option, ping waits for count ECHO_REPLY packets, until the timeout expires.
you can use this option as well to limit waiting time
-W timeout
Time to wait for a response, in seconds. The option affects only timeout in absense
of any responses, otherwise ping waits for two RTTs.
curl -s -I --http2 http://$1 >> fullscan_curl.txt | cut -d: -f1 fullscan_curl.txt | cat fullscan_curl.txt | grep HTTP >> fullscan_httpstatus.txt
its work me
I wrote the following script:
#!/bin/bash
if [ $# -ne 1 ];then
echo "Usage: ./script <input-file>"
exit 1
fi
while read user pass; do
curl -iL --data-urlencode user="$user" --data-urlencode password="$pass" http://foo.com/signin 1>/dev/null 2>&1
if [ $? -eq 0 ];then
echo "ok"
elif [ $? -ne 0 ]; then
echo "failed"
fi
done < $1
question:
Whenever I run with even wrong user & pass the result is ok for me...
How can I be sure that my parameters are correct or not?
Thanks
It’s because you are getting output from your curl command. Typing that command with random user/pass gets this:
$ curl -iL --data-urlencode user=BLAHBLAH --data-urlencode password=BLAH http://foo.com/signin
HTTP/1.1 301 Moved Permanently
Server: nginx/1.0.5
Date: Wed, 19 Feb 2014 05:53:36 GMT
Content-Type: text/html
Content-Length: 184
Connection: keep-alive
Location: http://www.foo.com/signin
. . .
. . .
<body>
<!-- This file lives in public/500.html -->
<div class="dialog">
<h1>We're sorry, but something went wrong.</h1>
</div>
</body>
</html>
Hence,
$ echo $?
0
But modify the URL to garbage:
$ curl -iL --data-urlencode user=BLAHBLAH --data-urlencode password=BLAH http://foof.com/signin
curl: (6) Could not resolve host: foof.com
$ echo $?
6
Even when the login fails, the HTTP server will still return a page with an error message. As curl is able to retrieve this page it will finish successfully.
In order to get curl to fail on server errors, you need the --fail parameter. Although this may not be fail-safe according to the curl man page, it is worth a try.
If --fail does not work, you could parse the header in the output of your curl request or have a look at the --write-out parameter.