I was wondering if it is possible to use curl to only show the content-type of the response header.
I want to check if the content-type is text/html for example before downloading instate of downloading the file and then find out it is application/pdf.
I used the example below in the hope that it would return the document if it is valid for me and else do nothing or something! The sample below just prints the full content of the page.
curl -F "type=text/html" www.google.nl
But If i do something like the example below it still downloads the whole thing, and I don't think that is right...
curl -F "type=text/html" http://www.axmag.com/download/pdfurl-guide.pdf
Many thanks :D
Option -F is for forms. Instead you want to send a HEAD request for retrieving only the response header without the response body by using option -I.
To display an URL's content type:
curl -s -I www.google.nl | grep -i "^Content-Type:"
Here option -s is added for silent mode for excluding the progress meter and error messages.
You can also specify the Accept header in your HTTP request. This header is used to accept only specific content types:
curl -s -H "Accept: text/html" http://www.axmag.com/download/pdfurl-guide.pdf
But the disadvantage is that most webservers will serve you an error page which also has the content type text/html. Hence you will still get a HTML file.
You can use the "-w" option too, with the "content-type" parameter:
curl -s -o /dev/null -w '%{content_type}' 'google.com'
Where:
-s: Silent mode, dont send any more to screen
-o: Output to file, and in this case, sends to /dev/null
-w: Where you show only with you want, in this case, content type
Reference: https://curl.haxx.se/docs/manpage.html
Related
I need to make a POST request via cURL from the command line. Data for this request is located in a file. I know that via PUT this could be done with the --upload-file option.
curl host:port/post-file -H "Content-Type: text/xml" --data "contents_of_file"
You're looking for the --data-binary argument:
curl -i -X POST host:port/post-file \
-H "Content-Type: text/xml" \
--data-binary "#path/to/file"
In the example above, -i prints out all the headers so that you can see what's going on, and -X POST makes it explicit that this is a post. Both of these can be safely omitted without changing the behaviour on the wire. The path to the file needs to be preceded by an # symbol, so curl knows to read from a file.
I need to make a POST request via Curl from the command line. Data for this request is located in a file...
All you need to do is have the --data argument start with a #:
curl -H "Content-Type: text/xml" --data "#path_of_file" host:port/post-file-path
For example, if you have the data in a file called stuff.xml then you would do something like:
curl -H "Content-Type: text/xml" --data "#stuff.xml" host:port/post-file-path
The stuff.xml filename can be replaced with a relative or full path to the file: #../xml/stuff.xml, #/var/tmp/stuff.xml, ...
If you are using form data to upload file,in which a parameter name must be specified , you can use:
curl -X POST -i -F "parametername=#filename" -F "additional_parm=param2" host:port/xxx
Most of answers are perfect here, but when I landed here for my particular problem, I have to upload binary file (XLSX spread sheet) using POST method, I see one thing missing, i.e. usually its not just file you load, you may have more form data elements, like comment to file or tags to file etc as was my case. Hence, I would like to add it here as it was my use case, so that it could help others.
curl -POST -F comment=mycomment -F file_type=XLSX -F file_data=#/your/path/to/file.XLSX http://yourhost.example.com/api/example_url
I was having a similar issue in passing the file as a param. Using -F allowed the file to be passed as form data, but the content type of the file was application/octet-stream. My endpoint was expecting text/csv.
You are able to set the MIME type of the file with the following syntax:
-F 'file=#path/to/file;type=<MIME_TYPE>
So the full cURL command would look like this for a CSV file:
curl -X POST -F 'file=#path/to/file.csv;type=text/csv' https://test.com
There is good documentation on this and other options here: https://catonmat.net/cookbooks/curl/make-post-request#post-form-data
I had to use a HTTP connection, because on HTTPS there is default file size limit.
https://techcommunity.microsoft.com/t5/IIS-Support-Blog/Solution-for-Request-Entity-Too-Large-error/ba-p/501134
curl -i -X 'POST' -F 'file=#/home/testeincremental.xlsx' 'http://example.com/upload.aspx?user=example&password=example123&type=XLSX'
So for example, say you have:
curl -d#- http://localhost:8000
what does each part of -d#- mean? This type of thing is difficult to google and I couldn't find any info about it in the docs or curl --help.
Thanks guys.
The -d#- in this command line is explained as this.
-d is the Data flag defining what is sent in the POST request.
# means send the file after this character as the body of the POST request.
- means stdin.
So -d#- means to read from stdin to send to the URL as the body of the POST request.
I work on Linux and try to use curl to send requests to Google and save its reply as a html file.
When I use Google to search something, such as a string "abc", I find that the link of Google is: https://www.google.lu/#q=abc
So I try like this:
curl https://www.google.lu/#q=abc -o res.html
But the res.html is just the main page of Google, instead of the result of searching "abc".
How to do it?
Anything after the # is handled client side with JavaScript, which is why it doesn't work with curl.
You can instead use the traditional, non-AJAX interface on https://www.google.com/search?q=abc
It appears to block you unless you also spoof the user agent, so all in all:
curl \
-A 'Mozilla/5.0 (MSIE; Windows 10)' \
-o res.html \
"https://www.google.com/search?q=abc"
I'm having a problem with the YouTube API. I am trying to make a bash application that will make watching YouTube videos easy on command line in Linux. I'm trying to take some video search results through cURL, but it returns an error: curl: (16) HTTP/2 stream 1 was not closed cleanly: error_code = 1
the cURL command that I use is:
curl "https://ww.googleapis.com/youtube/v3/search" -d part="snippet" -d q="kde" -d key="~~~~~~~~~~~~~~~~"
And of course I add my YouTube data API key where the ~~~~~~~~ are.
What am I doing wrong?
How can I make it work and return the search attributes?
I can see two things that are incorrect in your request:
First, you mistyped "www" and said "ww". That is not a valid URL
Then, curl's "-d" options are for POSTing only, not GETting ,at least not by default. You have two options:
Add the -G switch to url, which lets curl re-interpret -d options as query options:
curl -G https://www.googleapis.com/youtube/v3/search -d part="snippet" -d q="kde" -d key="xxxx"
Rework your url to a typical GET request:
curl "https://www.googleapis.com/youtube/v3/search?part=snippet&q=kde&key=XX"
As a tip, using bash to interpret the resulting json might not be the best way to go. You might want to look into using python, javascript, etc. to run your query and interpret the resulting json.
I have a json file with the structure seen below:
{
url: "https://mysite.com/myjsonfile",
version_number: 69,
}
This json file is accessed from mysite.com/myrootjsonfile
I want to run a load data script to access mysite.com/myrootjsonfile and load the json content from the url field using curl and save the resulting content to local storage.
This is my attempt so far.
curl -o assets/content.json 'https://mysite.com/myrootjsonfile' | grep -Po '(?<="url": ")[^"]*'
unfortunately, instead of saving the content from mysite.com/myjsonfile its saving the content from above: mysite.com/myrootjsonfile. Can anyone point out what i might be doing wrong? Bear in mind in a completely new to curl. Thanks!
It is saving the content from myrootjsonfile because that is what you are telling curl to do - to save that file to the location assets/content.json, and then greping stdin, which is empty. You need to use two curl commands, one to download the root file (and process it to find the URL of the second), and the second to download the actual content you want. You can use command substitution for this:
my_url=$(curl https://mysite.com/myrootjsonfile | grep -Po '(?<=url: )[^,]*')
curl -o assets/content.json "$my_url"
I also changed the grep regex - this one matches a string of non-comma characters which follow after "url: ".
Assuming you wished to save the file to assets/content.json, note that flags are case sensitive.
Use -o instead of -O to redirect the output to assets/content.json.