curl --location --request GET 'https://sampleurl/sample.log.gz' \
--header 'Authorization: Bearer XXXTokenXXX' \
--data-raw '{
"enabled": true
}' | gunzip -c
My requirement is to download a gz log file. The above is the sample curl which works as expected. How can I add this to python code?
Edit: Turns out there's a library dedicated to this exact problem. Here's the gzip library, and a stackoverflow post about this exact topic.
Old answer: You may be able to do this using the requests library by passing in custom headers as a dictionary, and then writing the content of your response as a file.
import requests
url = 'https://sampleurl/sample.log.gz'
headers = {'Authorization': 'Bearer XXXTokenXXX'}
response = requests.get(url, headers=headers) # if the URL will redirect, include allow_redirects=True
with open('C:\\path\\to\\save\\to', 'wb') as file:
file.write(response.content)
You'll probably need to tinker around with that to get it working for your use-case (especially if that --data-raw bit is important), but that's the general idea. If you need to download a particularly large file or want to see some other alternative solutions, you can check out this older question.
Related
I need to make a POST request via cURL from the command line. Data for this request is located in a file. I know that via PUT this could be done with the --upload-file option.
curl host:port/post-file -H "Content-Type: text/xml" --data "contents_of_file"
You're looking for the --data-binary argument:
curl -i -X POST host:port/post-file \
-H "Content-Type: text/xml" \
--data-binary "#path/to/file"
In the example above, -i prints out all the headers so that you can see what's going on, and -X POST makes it explicit that this is a post. Both of these can be safely omitted without changing the behaviour on the wire. The path to the file needs to be preceded by an # symbol, so curl knows to read from a file.
I need to make a POST request via Curl from the command line. Data for this request is located in a file...
All you need to do is have the --data argument start with a #:
curl -H "Content-Type: text/xml" --data "#path_of_file" host:port/post-file-path
For example, if you have the data in a file called stuff.xml then you would do something like:
curl -H "Content-Type: text/xml" --data "#stuff.xml" host:port/post-file-path
The stuff.xml filename can be replaced with a relative or full path to the file: #../xml/stuff.xml, #/var/tmp/stuff.xml, ...
If you are using form data to upload file,in which a parameter name must be specified , you can use:
curl -X POST -i -F "parametername=#filename" -F "additional_parm=param2" host:port/xxx
Most of answers are perfect here, but when I landed here for my particular problem, I have to upload binary file (XLSX spread sheet) using POST method, I see one thing missing, i.e. usually its not just file you load, you may have more form data elements, like comment to file or tags to file etc as was my case. Hence, I would like to add it here as it was my use case, so that it could help others.
curl -POST -F comment=mycomment -F file_type=XLSX -F file_data=#/your/path/to/file.XLSX http://yourhost.example.com/api/example_url
I was having a similar issue in passing the file as a param. Using -F allowed the file to be passed as form data, but the content type of the file was application/octet-stream. My endpoint was expecting text/csv.
You are able to set the MIME type of the file with the following syntax:
-F 'file=#path/to/file;type=<MIME_TYPE>
So the full cURL command would look like this for a CSV file:
curl -X POST -F 'file=#path/to/file.csv;type=text/csv' https://test.com
There is good documentation on this and other options here: https://catonmat.net/cookbooks/curl/make-post-request#post-form-data
I had to use a HTTP connection, because on HTTPS there is default file size limit.
https://techcommunity.microsoft.com/t5/IIS-Support-Blog/Solution-for-Request-Entity-Too-Large-error/ba-p/501134
curl -i -X 'POST' -F 'file=#/home/testeincremental.xlsx' 'http://example.com/upload.aspx?user=example&password=example123&type=XLSX'
I have a scenario in which I call multiple API requests with the same headers.
Headers would usually have:
Authorization: Bearer ASADFW&YUWFIANDJDDAjd8sa87sad7sa8dhsahdusabhdw
Content-Type: text/html;charset=UTF-8
etc...
I need to set these headers key-value pair as a default so whenever I curl the APIs these get picked.
Is there a way to do this?
I usually just keep a text file open with the Curl command, something like:
curl -X POST -H "Authorization: Bearer ASADFW&YUWFIANDJDDAjd8sa87sad7sa8dhsahdusabhdw"
-H "Content-Type: text/html;charset=UTF-8" -d 'blah'
http://some_url.com:8080
Then, just copy and paste the above into a terminal and make the changes you want. Note that I have formatted the above on multiple lines for reading purposes only. In practice, the entire Curl command should be on a single line.
EDIT 2:
It's working I have added raw parameter Thanks a lot to Arty-chan.
curl --insecure --request GET --header 'PRIVATE-TOKEN: Y_F8YP3nUnFbzhxkQvgo' https://gitlabxxxxxxxxxxxxxxx/api/v4/projects/98/repository/files/src%2Fmain%2Fscript%2Fdeploiement%2Fsettings%2Frci%2Fsesameweb2.sh/raw?ref=master >> setting.sh
EDIT:
Thanks to Arty-chan.
I cannot download the file but I have long output. How can I download my sesameweb2.sh file please ?
The sh file should contain some export variable and if I check the content it's not the content of my actual file :(
{"file_name":"sesameweb2.sh","file_path":"src/main/script/deploiement/settings/rci/sesameweb2.sh","size":792,"encoding":"base64","content_sha256":"0ef870c1ebf9e0be5a5a976a4d3e5424a0b5752a75c67fe4b7744eb8fd6279d3","ref":"master","blob_id":"e3937fb8a25edd7a73bbbe5d6b2e267a4e404ef5","commit_id":"577c6c9ba661eb2726b5cd2095bc93bbd7eb3302","last_commit_id":"6921e0acd61d24066a3d680a5d77bd68f880e4d1","content":"IyEvdXNyL2Jpbi9lbnYgYmFzaApleHBvcnQgQU5UX0hPTUU9L3Vzci9hcGFjaGUvYXBhY2hlLWFudC0xLjkuNwpleHBvcnQgTUFWRU5fSE9NRT0vdXNyL2FwYWNoZS9hcGFjaGUtbWF2ZW4tMy4xLjEKZXhwb3J0IEpBVkFfSE9NRT0vdXNyL2phdmEvamRrMS43LjBfNzkKZXhwb3J0IFRPTUNBVF9JTlNUPWFwYWNoZS10b21jYXQtMzEKZXhwb3J0IE5PTV9XQVI9Y3J5c2FsaWQKZXhwb3J0IE5PTV9XQVJfMj1jcnlzYWxpZHdlYgpleHBvcnQgQ09NUFVURVJOQU1FPVMwMFNMUjIwMQpleHBvcnQgUkVGRVJFTlRJRUxfTkFNRT1zZXNhbWV3ZWIKZXhwb3J0IGdyb3VwaWQ9cGFjaWZpY2EuYXBwbGljYXRpb25zLiR7UkVGRVJFTlRJRUxfTkFNRX12MTIKZXhwb3J0IGFydGVmYWN0aWQ9JHtOT01fV0FSfQpleHBvcnQgYXJ0ZWZhY3RpZF8yPSR7Tk9NX1dBUl8yfQpleHBvcnQgYXBwX3ZlcnNpb249TEFURVNUCmV4cG9ydCBuZXh1c191cmw9aHR0cDovL3JlcG9zaXRvcnktbWFuYWdlci5wYWNpZmljYS5jcmVkaXQtYWdyaWNvbGUuZnIvbmV4dXMKZXhwb3J0IG5leHVzX3JlcG89cGFjaWZpY2Etc25hcHNob3RzLXJlY2V0dGUtaW5mby1sb2NhbApleHBvcnQgZW52aXJvbm5lbWVudD1yZWNldHRlSW5mb3JtYXRpcXVlCmV4cG9ydCBtYXZlbl9zZXR0aW5nc19wYXRoPS9tbnQvbmFzX3VzaWwvYXBwcy9CdWlsZFRvb2xzL21hdmVuX3NldHRpbmdzL3NldHRpbmdzX3JlY2V0dGVfaW5mb3JtYXRpcXVlX2xpbnV4LnhtbApleHBvcnQgYnVpbGRfYW50X25hbWU9YnVpbGRfJHtSRUZFUkVOVElFTF9OQU1FfS54bWwK"}
Cannot understand how using api to download with curl my file on gitlab.
here my url:
https://gitlabxxxxxxx/administration/gitlab/raw/master/src/main/script/deploiement/settings/rci/setting.sh
I have checked official website but cannot understand how it's works and which part I should replace:
curl --request GET --header 'PRIVATE-TOKEN: <your_access_token>' 'https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb/raw?ref=master'
I have tried but it's not working
curl --insecure --request GET --header 'PRIVATE-TOKEN: Y_F8YP3nUnFbzhxkQvgo' 'https://gitlabxxxxxxxxxx/api/v4/administration/gitlab/98/src/main/script/deploiement/settings/rci/sesameweb2.sh/app%2Fmodels%2Fkey%2Erb/raw?ref=master'
My project ID is : 98
I would like to download this setting.sh, I'am using gitlab-ce 12.6.4.
Thanks for help
When following the docs, you need to follow the /projects/:id/repository/files/:file_path format for the link, so don't replace everything, just the pieces with : in front.
You also need to make sure that the file path is using URI encoding.
With the information that you have, it would look something like this:
https://gitlabxxxxxxxxxx/api/v4/projects/98/repository/files/src%2Fmain%2Fscript%2Fdeploiement%2Fsettings%2Frci%2Fsetting.sh?ref=master
curl -k --request GET --header PRIVATE-TOKEN:"<PRIVATE-TOKEN>" https://<gitlab_url>/api/v4/projects/<id>/repository/files/<path_url_encoded>/raw?ref=master > <destination_path>
OR
refer official example:
curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb?ref=master"
Here:
PRIVATE-TOKEN - enter your private token
path_url_encoded = Check here
destination_path - path where you want to download the file with filename
Check the Official Doc
I am new to bash scripting. I have spent hours searching for a solution..
#!/bin/bash
# EMAIL_TO, FROM_EMAIL, etc variables are initialized here
# ...
# ...
maildata='{"personalizations": [{"to": [{"email": "'${EMAIL_TO}'"}]}],"from": {"email": "'${FROM_EMAIL}'",
"name": "'${FROM_NAME}'"},"subject": "'${SUBJECT}'","content": [{"type": "text/html", "value": "'${bodyHTML}'"}]}'
curl --url https://api.sendgrid.com/v3/mail/send \
--header 'Authorization: Bearer '$SENDGRID_API_KEY \
--header 'Content-Type: application/json' \
--data "'$maildata'"
The snippet above works fine whenever bodyHTML is set here, inside the script.
But I want an external html file to be sent in this manner.
So, the question:
How can I send, for example, "mail.html" using the above scheme?
(How to set the bodyHTML variable? What command(s) to use?
bodyHTML=$(cat "mail.html") # results in an error message on executing the curl line)
The issue seems to have to do with the bodyHTML (initialized either from file or within script) containing double quotation marks. As in , for example. But what to do with it...? Perhaps I need to first serialize HTML as JSON within the script..? How?
i guess bodyHTML needs proper json-encoding, try
maildata='{"personalizations": [{"to": [{"email": "'${EMAIL_TO}'"}]}],"from": {"email": "'${FROM_EMAIL}'",
"name": "'${FROM_NAME}'"},"subject": "'${SUBJECT}'","content": [{"type": "text/html", "value": '$(printf "%s" "${bodyHTML}" | php -r 'echo json_encode(stream_get_contents(STDIN));')'}]}'
... ofc they should probably ALL be properly json-encoded, not just the HTML, but the html seems like the thing most likely to break your json here..
I work on Linux and try to use curl to send requests to Google and save its reply as a html file.
When I use Google to search something, such as a string "abc", I find that the link of Google is: https://www.google.lu/#q=abc
So I try like this:
curl https://www.google.lu/#q=abc -o res.html
But the res.html is just the main page of Google, instead of the result of searching "abc".
How to do it?
Anything after the # is handled client side with JavaScript, which is why it doesn't work with curl.
You can instead use the traditional, non-AJAX interface on https://www.google.com/search?q=abc
It appears to block you unless you also spoof the user agent, so all in all:
curl \
-A 'Mozilla/5.0 (MSIE; Windows 10)' \
-o res.html \
"https://www.google.com/search?q=abc"