I am using curl command like following curl -i -X POST -H 'Content-type:application/json' some_url -d '{ "some":"json"}' and getting output like below.
HTTP/1.1 200 OK
Server: openresty
Date: Tue, 03 Apr 2018 14:28:38 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 3925
Connection: keep-alive
X-Powered-By: PHP/5.5.9-1ubuntu4.23
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Content-Type
Access-Control-Allow-Methods: POST
Access-Control-Max-Age: 1000
{
"jsonrpc": "2.0",
"result": [{
"hostid": "10225",
"proxy_hostid": "0",
"host": "sgw-2585-bus",
"status": "0"
}]
}
I just want the json output of curl command for further processing. how can i achieve it.
Note: curl doesn't show json output like this i've formatted json block.
Have you tried the -s or --silent option?
curl man page
You should just need to omit the -i flag
From the curl man page:
-i, --include
Include the HTTP-header in the output. The HTTP-header includes things like server-name, date of the document, HTTP-version and more...
See also -v, --verbose.
Simply remove the -i switch from your command. This code below gives your pure JSON response
curl -s -X POST -H 'Content-type:application/json' some_url -d '{ "some":"json"}'
I do have this script to get some variables hitting a server. The point is that is doing it slow and I have to run multiple times same script to get a decent rate of requests. How can I multiply number of threads with this curl script without needing to run it 4 or 5 times?
Also I would like to make it faster and performed, this is original one
while ! grep "TokenException" output.txt > /dev/null
do
echo -e '\n'$(date +%x_%H:%M:%S:%3N) > output.txt
curl -s -H 'Host: host.com' -H "Cookie: session-token="$SESSION\" -H "x-amz-access-token: $token" -H "x-flex-instance-id: $flex" -H 'Accept: */*' -H 'User-Agent: Dalvik/2.1.0 (Linux; U; Android 7.1.1; Nexus 5X Build/N4F26T) RabbitAndroid/3.0.6778.0' -H 'Accept-Language: en-us' --compressed 'https://hostname.com/GetOffersForProvider?serviceAreaIds=16' >> output.txt
if grep -q "OFFERED" output.txt; then
cat output.txt >> foundb.txt
./getlast.bat
if [ ! -f pageflag.txt ]; then
/usr/bin/php alert.php
echo "paged" > pageflag.txt
fi
sleep 0.05
fi
done
I modified to
while ! grep "TokenException" output.txt > /dev/null
do
echo -e '\n'$(date +%x_%H:%M:%S:%3N) > output.txt
curl -s -H 'Host: host.com' -H "Cookie: session-token="$SESSION\" -H "x-amz-access-token: $token" -H "x-flex-instance-id: $flex" -H 'Accept: */*' -H 'User-Agent: Dalvik/2.1.0 (Linux; U; Android 7.1.1; Nexus 5X Build/N4F26T) RabbitAndroid/3.0.6778.0' -H 'Accept-Language: en-us' --compressed 'https://hostname.com/GetOffersForProvider?serviceAreaIds=16' >> output.txt
if grep -q "OFFERED" output.txt; then
cat output.txt >> foundb.txt
./getlast.bat
/usr/bin/php alert.php
sleep 0.05
fi
done
Some suggestion to run on multi threading and faster even with sleep or another way to pause for milliseconds instead? Main point is that it should execute ./getlast.bat as fast as possible with variables caught from curl, but it has a time difference between data and getlast.bat execution of 2 seconds, too much
curl itself doesn't support multiple connections/threads and/or resume cancelled/stalled operations.
Use aria2 or something similar.
I am using cURL to post a XML file on Linux as follows:
curl -X POST --header "Content-Type: text/xml" -d #test.xml "https://www.example.com"
How can I check the status of the cURL command to see if the file was posted or not?
Thanks for any help.
I'm not sure if I got you but you can basically check the return value of the curl command. The return value of the last command is stored in the variable $?.
Example:
curl -X POST --header "Content-Type: text/xml" -d #test.xml "https://www.example.com"
ret=$? # store return value for later usage in the error message
if [ $ret != 0 ] ; then
echo "POST failed with exit code $ret"
fi
This list of possible error codes can be found at the bottom of the man page. They are very helpful for debugging.
You can get the response status of your operation using -w %{http_code}
curl -s -o out.txt -w %{http_code} http://www.example.com/
In this example -s means silent mode, and -o out.txt means to save the response(usually html) into a file.
For this above command, you'l have output 200 when its success.
I wrote the following script:
#!/bin/bash
if [ $# -ne 1 ];then
echo "Usage: ./script <input-file>"
exit 1
fi
while read user pass; do
curl -iL --data-urlencode user="$user" --data-urlencode password="$pass" http://foo.com/signin 1>/dev/null 2>&1
if [ $? -eq 0 ];then
echo "ok"
elif [ $? -ne 0 ]; then
echo "failed"
fi
done < $1
question:
Whenever I run with even wrong user & pass the result is ok for me...
How can I be sure that my parameters are correct or not?
Thanks
It’s because you are getting output from your curl command. Typing that command with random user/pass gets this:
$ curl -iL --data-urlencode user=BLAHBLAH --data-urlencode password=BLAH http://foo.com/signin
HTTP/1.1 301 Moved Permanently
Server: nginx/1.0.5
Date: Wed, 19 Feb 2014 05:53:36 GMT
Content-Type: text/html
Content-Length: 184
Connection: keep-alive
Location: http://www.foo.com/signin
. . .
. . .
<body>
<!-- This file lives in public/500.html -->
<div class="dialog">
<h1>We're sorry, but something went wrong.</h1>
</div>
</body>
</html>
Hence,
$ echo $?
0
But modify the URL to garbage:
$ curl -iL --data-urlencode user=BLAHBLAH --data-urlencode password=BLAH http://foof.com/signin
curl: (6) Could not resolve host: foof.com
$ echo $?
6
Even when the login fails, the HTTP server will still return a page with an error message. As curl is able to retrieve this page it will finish successfully.
In order to get curl to fail on server errors, you need the --fail parameter. Although this may not be fail-safe according to the curl man page, it is worth a try.
If --fail does not work, you could parse the header in the output of your curl request or have a look at the --write-out parameter.
I need to get the final URL after a page redirect preferably with curl or wget.
For example http://google.com may redirect to http://www.google.com.
The contents are easy to get(ex. curl --max-redirs 10 http://google.com -L), but I'm only interested in the final url (in the former case http://www.google.com).
Is there any way of doing this by using only Linux built-in tools? (command line only)
curl's -w option and the sub variable url_effective is what you are
looking for.
Something like
curl -Ls -o /dev/null -w %{url_effective} http://google.com
More info
-L Follow redirects
-s Silent mode. Don't output anything
-o FILE Write output to <file> instead of stdout
-w FORMAT What to output after completion
More
You might want to add -I (that is an uppercase i) as well, which will make the command not download any "body", but it then also uses the HEAD method, which is not what the question included and risk changing what the server does. Sometimes servers don't respond well to HEAD even when they respond fine to GET.
Thanks, that helped me. I made some improvements and wrapped that in a helper script "finalurl":
#!/bin/bash
curl $1 -s -L -I -o /dev/null -w '%{url_effective}'
-o output to /dev/null
-I don't actually download, just discover the final URL
-s silent mode, no progressbars
This made it possible to call the command from other scripts like this:
echo `finalurl http://someurl/`
as another option:
$ curl -i http://google.com
HTTP/1.1 301 Moved Permanently
Location: http://www.google.com/
Content-Type: text/html; charset=UTF-8
Date: Sat, 19 Jun 2010 04:15:10 GMT
Expires: Mon, 19 Jul 2010 04:15:10 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 219
X-XSS-Protection: 1; mode=block
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
here.
</BODY></HTML>
But it doesn't go past the first one.
Thank you. I ended up implementing your suggestions: curl -i + grep
curl -i http://google.com -L | egrep -A 10 '301 Moved Permanently|302 Found' | grep 'Location' | awk -F': ' '{print $2}' | tail -1
Returns blank if the website doesn't redirect, but that's good enough for me as it works on consecutive redirections.
Could be buggy, but at a glance it works ok.
You can do this with wget usually. wget --content-disposition "url" additionally if you add -O /dev/null you will not be actually saving the file.
wget -O /dev/null --content-disposition example.com
The parameters -L (--location) and -I (--head) still doing unnecessary HEAD-request to the location-url.
If you are sure that you will have no more than one redirect, it is better to disable follow location and use a curl-variable %{redirect_url}.
This code do only one HEAD-request to the specified URL and takes redirect_url from location-header:
curl --head --silent --write-out "%{redirect_url}\n" --output /dev/null "https://""goo.gl/QeJeQ4"
Speed test
all_videos_link.txt - 50 links of goo.gl+bit.ly which redirect to youtube
1. With follow location
time while read -r line; do
curl -kIsL -w "%{url_effective}\n" -o /dev/null $line
done < all_videos_link.txt
Results:
real 1m40.832s
user 0m9.266s
sys 0m15.375s
2. Without follow location
time while read -r line; do
curl -kIs -w "%{redirect_url}\n" -o /dev/null $line
done < all_videos_link.txt
Results:
real 0m51.037s
user 0m5.297s
sys 0m8.094s
curl can only follow http redirects. To also follow meta refresh directives and javascript redirects, you need a full-blown browser like headless chrome:
#!/bin/bash
real_url () {
printf 'location.href\nquit\n' | \
chromium-browser --headless --disable-gpu --disable-software-rasterizer \
--disable-dev-shm-usage --no-sandbox --repl "$#" 2> /dev/null \
| tr -d '>>> ' | jq -r '.result.value'
}
If you don't have chrome installed, you can use it from a docker container:
#!/bin/bash
real_url () {
printf 'location.href\nquit\n' | \
docker run -i --rm --user "$(id -u "$USER")" --volume "$(pwd)":/usr/src/app \
zenika/alpine-chrome --no-sandbox --repl "$#" 2> /dev/null \
| tr -d '>>> ' | jq -r '.result.value'
}
Like so:
$ real_url http://dx.doi.org/10.1016/j.pgeola.2020.06.005
https://www.sciencedirect.com/science/article/abs/pii/S0016787820300638?via%3Dihub
This would work:
curl -I somesite.com | perl -n -e '/^Location: (.*)$/ && print "$1\n"'
I'm not sure how to do it with curl, but libwww-perl installs the GET alias.
$ GET -S -d -e http://google.com
GET http://google.com --> 301 Moved Permanently
GET http://www.google.com/ --> 302 Found
GET http://www.google.ca/ --> 200 OK
Cache-Control: private, max-age=0
Connection: close
Date: Sat, 19 Jun 2010 04:11:01 GMT
Server: gws
Content-Type: text/html; charset=ISO-8859-1
Expires: -1
Client-Date: Sat, 19 Jun 2010 04:11:01 GMT
Client-Peer: 74.125.155.105:80
Client-Response-Num: 1
Set-Cookie: PREF=ID=a1925ca9f8af11b9:TM=1276920661:LM=1276920661:S=ULFrHqOiFDDzDVFB; expires=Mon, 18-Jun-2012 04:11:01 GMT; path=/; domain=.google.ca
Title: Google
X-XSS-Protection: 1; mode=block
Can you try with it?
#!/bin/bash
LOCATION=`curl -I 'http://your-domain.com/url/redirect?r=something&a=values-VALUES_FILES&e=zip' | perl -n -e '/^Location: (.*)$/ && print "$1\n"'`
echo "$LOCATION"
Note: when you execute the command curl -I http://your-domain.com have to use single quotes in the command like curl -I 'http://your-domain.com'
You could use grep. doesn't wget tell you where it's redirecting too? Just grep that out.