Simple GET request with curl returns empty body (Content-Length: 0):
curl -v https://www.flyorientthai.com/booking/en/index.php
On the other hand wget can handle that url just fine:
wget https://www.flyorientthai.com/booking/en/index.php
What's wrong with curl?
Turned out 'Connection: Keep-Alive' header is required. It's added by default to request with wget but not with curl.
For anyone using "Copy as cURL" in the Chrome developer tools network tab to generate a curl command — one of the lines in the generated curl command is a header that looks something like...
-H 'If-None-Match: W/"18f6a6-6p8AL7X/p71IhN/WztZm60Aue4k"'
That causes the server to return an empty 304 response if nothing's changed. Just whip it out.
Let's try "Content-Type: application/x-www-form-urlencodeds"
Related
How can I send POST request with a csv or a text file to the server running on a localhost using cURL.
I have tried curl -X POST -d #file.csv http://localhost:5000/upload but I get
{
"message": "The browser (or proxy) sent a request that this server could not understand."
}
My server is flask_restful API. Thanks a lot in advance.
There are many alternate ways to accomplish this. One way is
I have used the following:
curl -F ‘data=#<file_location>’ <URL>
Eg. curl -F data=#data.csv localhost:5000/h
Your command can also be changed slightly like this
curl -X POST -H 'Content-Type: text/csv' -d #file.csv http://localhost:5000/upload
The above is one of the many ways.It can be sent either as a part of form or data, or multipart, etc. You can refer Medium Post
Curl's default Content-Type is application/x-www-form-urlencoded so your problem is probably that the data you are POSTing is not actually form data. It might work if you set the content type header properly:
-H "Content-Type: text/csv"
Though it does depend on the server.
I am new to using CURL. I am trying to connect to a SharePoint URL and pull the data in the form of json. My URL looks like this:
.../_api/web/lists/GetByTitle('titles list')/items
If I give the URL as it is without any encoding, it fails with a HTTP/1.1 400 Bad Request error.
I tried using -G and --data-urlencode like below:
curl -v -G -L --ntlm --user user:password -H 'Accept: application/json;odata=verbose' ".../_api/web/lists/GetByTitle" --data-urlencode "('titles list')" -d "/items"
doing this will convert my URL to .../_api/web/lists/GetByTitle?%28%27titles%20list%27%29&/items
But it fails with HTTP/1.1 404 Not Found since using -G will append to the URL with a ? and & . Putting ? and & to append to the URL will give me a different URL and hence the 404 not found error.
I have no issues accessing other end points like ../_api/web/lists since there is no need to encode it I guess.
How do I properly encode my URL and get the data without any errors?
Any help would be appreciated. Thank you.
I was able to solve the issue by directly giving URL with the encoded characters to cURL command. Like this:
curl -v -L --ntlm --user user:password -H 'Accept: application/json;odata=verbose' ".../_api/web/lists/GetByTitle%28%27titles%20List%27%29/items"
I hope this helps someone. I know I am no expert with linux or cURL and any better answers to this are welcome.
Before you post this as a duplicate; I've tried many of the suggestions I found around SO.
So far I've been using postman to post data to a Java web service. That works great as follows:
I now want to do the same using curl, so I tried it using the following ways:
$ curl -X POST --data "this is raw data" http://78.41.xx.xx:7778/
$ curl -X POST --data-binary "this is raw data" http://78.41.xx.xx:7778/
$ curl -X POST --data "#/home/kramer65/afile.txt" http://78.41.xx.xx:7778/
$ curl -X POST --data-binary "#/home/kramer65/afile.txt" http://78.41.xx.xx:7778/
Unfortunately, all of those show an empty raw body on the receiving side.
Does anybody know what I'm doing wrong here? How is my curl request different from my postman request? All tips are welcome!
curl's --data will by default send Content-Type: application/x-www-form-urlencoded in the request header. However, when using Postman's raw body mode, Postman sends Content-Type: text/plain in the request header.
So to achieve the same thing as Postman, specify -H "Content-Type: text/plain" for curl:
curl -X POST -H "Content-Type: text/plain" --data "this is raw data" http://78.41.xx.xx:7778/
Note that if you want to watch the full request sent by Postman, you can enable debugging for packed app. Check this link for all instructions. Then you can inspect the app (right-click in Postman) and view all requests sent from Postman in the network tab :
Is there a way to tell cURL command not to use server's side cache?
e.g; I have this curl command:
curl -v www.example.com
How can I ask curl to send a fresh request to not use the cache?
Note: I am looking for an executable command in the terminal.
I know this is an older question, but I wanted to post an answer for users with the same question:
curl -H 'Cache-Control: no-cache' http://www.example.com
This curl command servers in its header request to return non-cached data from the web server.
The -H 'Cache-Control: no-cache' argument is not guaranteed to work because the remote server or any proxy layers in between can ignore it. If it doesn't work, you can do it the old-fashioned way, by adding a unique querystring parameter. Usually, the servers/proxies will think it's a unique URL and not use the cache.
curl "http://www.example.com?foo123"
You have to use a different querystring value every time, though. Otherwise, the server/proxies will match the cache again. To automatically generate a different querystring parameter every time, you can use date +%s, which will return the seconds since epoch.
curl "http://www.example.com?$(date +%s)"
Neither -H 'Pragma: no-cache' nor -H 'Cache-Control: no-cache' helped me. In browser with "cmd+shift+r" (full reload) I was seeing a new version than the output of curl in terminal.
How to debug for yourself
To get the exact same result, I went to browser > F12 (Dev Tools) > Network/Requests > Right-click on the request > "Copy as cURL" and got the equivalent cURL command to the browser's call.
Then, I pasted that in the terminal and started removing the params one by one, until I found that surprisingly --compressed was making a difference in my case. (Calling CloudFront AWS)
You could try following ways to force not to keep Cache when curl.
Note: The server may or may not be configured to respect the Cache-Control header. Therefore, whether this method will work is dependent on the server or website we’re sending the HTTP request to.
curl command with the Cache-Control header
$ curl -H 'Cache-Control: no-cache, no-store' http://www.example.com
Adding the Pragma HTTP Header
$ curl -H 'Pragma: no-cache' http://www.example.com
Finally, the most common way: bypass the cache by changing the URL
curl -H 'Cache-Control: no-cache, no-store' http://www.example.com?$(date +%s)
My problem is I should use single quotes in a query string.
My code
if (event.queryStringParameters && event.queryStringParameters['Name']) {
responseMessage = 'Hello, ' + event.queryStringParameters['Name'] + '!';
}
My requests
curl https://cssrq1srud.execute-api.us-east-1.amazonaws.com/serverless_lambda_stage/hello?Name=Terraform
returns zsh: no matches found
curl 'https://cssrq1srud.execute-api.us-east-1.amazonaws.com/serverless_lambda_stage/hello?Name=Terraform'
returns {"message":"Hello, Terraform!"}
This didn't work for me:
curl -H 'Cache-Control: no-cache' http://www.example.com
but this ended up doing the trick:
curl -H 'Cache-Control: no-cache' http://www.example.com&someFakeParam=$RANDOM
the &someFakeParam=$RANDOM makes the URL unique each time and bypasses caching.
Is there a way to use curl to do a multipart/form-data post and specify that the data is application/json? It doesn't have to be curl.
This is what I use, and it works fine
curl -v -H "Content-Type:multipart/form-data"
-F "someName={\"your\":\"json\"};type=application/json"
Note: I'm on Windows, hence the requirement for double quotes
Or
-F "someName=#someJsonFile.json;type=application/json"