This works & writes to the file with no issues:
curl -x myproxy.baseurl.com:80 -H "Authorization: Token mytoken" https://app.api.com/?status=Approved > output.txt
This sends the output to the command line & hangs until I hit ctl-c:
curl -x myproxy.baseurl.com:80 -H "Authorization: Token mytoken" https://app.api.com/?status=Approved&page_size=100000 > output.txt
Are there some kind of payload size limitations? The length of the JSON response in the first call is 270,928 characters. The length of the second, two parameter, call is 622,133.
Could it be a time-out issue? The first payload takes ~5 seconds, and the second payload takes ~11 seconds when testing with SoapUI.
UPDATE: If I don't ctl-c on the second call, eventually curl does time out: curl: (28) connect() timed out!
UPDATE 2: FWIW, this is part of a shell script I am writing, so yeah, it is a programming problem. :P
Make sure to quote your URL because the shell is interpreting the ampersand to put curl in the background.
curl -x myproxy.baseurl.com:80 -H "Authorization: Token mytoken" "https://app.api.com/?status=Approved&page_size=100000" > output.txt
Related
I was coding a simple rest service in Apache Camel, and testing it using the curl command by invoking the endpoint of my service.
The service receives in plain text a simple String like "ABC123-D-FE", but as I made multiple tests, the data was always received as "{ABC123-D-FE=}", always adding the "{ =}".
At first I thought it was my service catching the data like that, but every other method I tried (i.e. rest clients, postman, invoking the service by other services) never reproduced that results, and the service always received just the plain text data.
It was only formatted like that by using curl.
The command was:
curl -X POST -d ABC123-F-DE http://host/service
Can't find any reference to this behaviour, and the only conclusion is something curl does by default (and don't understand why or how to remove it).
I was using the curl command in Ubuntu Mate 20.04.
Edit: per Nick ODell's comment below, it almost certainly means it's parsed to a map with the key "ABC123-F-DE" having an empty value, like this json:
{
"ABC123-F-DE": ""
}
i guess it's your parsed-object-stringify function adding the { to signify start of map, and adding the = to specify "value of this key" and adding } to specify end of map?
lets check what curl actually sends with a little netcat server:
$ nc -l 1111
followed by
$ curl -X POST -d ABC123-F-DE http://localhost:1111
yields:
$ nc -l 1111
POST / HTTP/1.1
Host: localhost:1111
User-Agent: curl/7.84.0
Accept: */*
Content-Length: 11
Content-Type: application/x-www-form-urlencoded
ABC123-F-DE
Conclusion: it's definitely not curl.
My best guess: it's something weird with your server's application/x-www-form-urlencoded-parser?
I have been trying to run a BASH script which posts a request to an SMS server and on successful execution a message is received on the mentioned mobile number. Script is as shown below:
curl -k -X POST "http://192.168.10.3/u=admin&h=452ba065ebd1723598a51c7eca11d362&op=pv&to=1234567891&msg=Hello+to+all"
The above script is working fine. The message "Hello to all" is being received on the mobile number 1234567891. This number is however hard coded in the URL. In the actual scenario the mobile number would be available in a variable and the SMS would be sent to the mobile number available in this variable.
I have tried scripts like:
mobile_number="1234567891"
curl -k -X POST "http://192.168.10.3/u=admin&h=452ba065ebd1723598a51c7eca11d362&op=pv&to=$mobile_number&msg=Message+From+world"
and
x="http://192.168.10.3/u=admin&h=452ba065ebd1723598a51c7eca11d362&op=pv&to="
x+="1234567891
x+=&msg=Hello+to+all"
curl -k -X POST $x
However, i have been unsuccessful in executing them successfully. It would be of great help if someone could help me with the syntax.
Try out this principle, bash is different language than c++ or so :-):
#!/bin/bash
to="1234567891"
msg="Hello+to+all"
u="admin"
hash="452ba065ebd1723598a51c7eca11d362"
op="pv"
ip="192.168.10.3"
url="http://${ip}/u=admin&h=${hash}&op=${op}&to=${to}&msg=${msg}"
echo ${url}
Than : curl -k -X POST $url should work fine.
Why does this not work:
curl -X HEAD http://www.google.com
But these both work just fine:
curl -I http://www.google.com
curl -X GET http://www.google.com
You need to add the -i flag to the first command, to include the HTTP header in the output. This is required to print headers.
curl -X HEAD -i http://www.google.com
More here: https://serverfault.com/questions/140149/difference-between-curl-i-and-curl-x-head
curl --head https://www.example.net
I was pointed to this by curl itself; when I issued the command with -X HEAD, it printed:
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
google.com is not responding to HTTP HEAD requests, which is why you are seeing a hang for the first command.
It does respond to GET requests, which is why the third command works.
As for the second, curl just prints the headers from a standard request.
I am trying to get more familiar to curl, so I am trying to send post request to a webpage from a site I am already logged into :
curl --data "name=value" http://www.mysite.com >> post_request.txt
I had an output consisting of all the hmtl page with a message telling me I was not logged in so I retrieved the site's cookie (called PHPSESSID, is that of importance) stored it in /temp/cookies.txt and then
curl --cookie /tmp/cookies.txt --cookie-jar /tmp/cookies.txt --data "name=value" http://www.mysite.com > post_request.txt
but I still have the same message. Someone could help ?
I suggest you take a network dumping tool (wireshark or tcpdump) and sniff the communication between your shell and the server. Check if the cookie is sent and if it has the right content.
Whats the syntax for a linux command that hits a URL repeatedly, x number of times. I don't need to do anything with the data, I just need to replicate hitting refresh 20 times in a browser.
You could use URL sequence substitution with a dummy query string (if you want to use CURL and save a few keystrokes):
curl http://www.myurl.com/?[1-20]
If you have other query strings in your URL, assign the sequence to a throwaway variable:
curl http://www.myurl.com/?myVar=111&fakeVar=[1-20]
Check out the URL section on the man page: https://curl.haxx.se/docs/manpage.html
for i in `seq 1 20`; do curl http://url; done
Or if you want to get timing information back, use ab:
ab -n 20 http://url/
You might be interested in Apache Bench tool which is basically used to do simple load testing.
example :
ab -n 500 -c 20 http://www.example.com/
n = total number of request, c = number of concurrent request
You can use any bash looping constructs like FOR, with is compatible with Linux and Mac.
https://tiswww.case.edu/php/chet/bash/bashref.html#Looping-Constructs
In your specific case you can define N iterations, with N is a number defining how many curl executions you want.
for n in {1..N}; do curl <arguments>; done
ex:
for n in {1..20}; do curl -d #notification.json -H 'Content-Type: application/json' localhost:3000/dispatcher/notify; done
If you want to add an interval before executing the cron the next time you can add a sleep
for i in {1..100}; do echo $i && curl "http://URL" >> /tmp/output.log && sleep 120; done
If you want to add a bit of delay before each request you could use the watchcommand in Linux:
watch curl https://yourdomain.com/page
This will call your url every other second. Alter the interval by adding the ´-n´ parameter with a delay containing the number of seconds. For instance:
watch -n0.5 curl https://yourdomain.com/page
This will now call the url every half second.
CTRL+C to exit watch