I am trying to get more familiar to curl, so I am trying to send post request to a webpage from a site I am already logged into :
curl --data "name=value" http://www.mysite.com >> post_request.txt
I had an output consisting of all the hmtl page with a message telling me I was not logged in so I retrieved the site's cookie (called PHPSESSID, is that of importance) stored it in /temp/cookies.txt and then
curl --cookie /tmp/cookies.txt --cookie-jar /tmp/cookies.txt --data "name=value" http://www.mysite.com > post_request.txt
but I still have the same message. Someone could help ?
I suggest you take a network dumping tool (wireshark or tcpdump) and sniff the communication between your shell and the server. Check if the cookie is sent and if it has the right content.
Related
I was coding a simple rest service in Apache Camel, and testing it using the curl command by invoking the endpoint of my service.
The service receives in plain text a simple String like "ABC123-D-FE", but as I made multiple tests, the data was always received as "{ABC123-D-FE=}", always adding the "{ =}".
At first I thought it was my service catching the data like that, but every other method I tried (i.e. rest clients, postman, invoking the service by other services) never reproduced that results, and the service always received just the plain text data.
It was only formatted like that by using curl.
The command was:
curl -X POST -d ABC123-F-DE http://host/service
Can't find any reference to this behaviour, and the only conclusion is something curl does by default (and don't understand why or how to remove it).
I was using the curl command in Ubuntu Mate 20.04.
Edit: per Nick ODell's comment below, it almost certainly means it's parsed to a map with the key "ABC123-F-DE" having an empty value, like this json:
{
"ABC123-F-DE": ""
}
i guess it's your parsed-object-stringify function adding the { to signify start of map, and adding the = to specify "value of this key" and adding } to specify end of map?
lets check what curl actually sends with a little netcat server:
$ nc -l 1111
followed by
$ curl -X POST -d ABC123-F-DE http://localhost:1111
yields:
$ nc -l 1111
POST / HTTP/1.1
Host: localhost:1111
User-Agent: curl/7.84.0
Accept: */*
Content-Length: 11
Content-Type: application/x-www-form-urlencoded
ABC123-F-DE
Conclusion: it's definitely not curl.
My best guess: it's something weird with your server's application/x-www-form-urlencoded-parser?
I try to read/fetch this file:
https://blockchain-office.com/file.txt with a bash script over dev/tcp without using curl,wget, etc..
I found this example:
exec 3<>/dev/tcp/www.google.com/80
echo -e "GET / HTTP/1.1\r\nhost: http://www.google.com\r\nConnection: close\r\n\r\n" >&3
cat <&3
I change this to my needs like:
exec 3<>/dev/tcp/www.blockchain-office.com/80
echo -e "GET / HTTP/1.1\r\nhost: http://www.blockchain-office.com\r\nConnection: close\r\n\r\n" >&3
cat <&3
When i try to run i receive:
400 Bad Request
Your browser sent a request that this server could not understand
I think this is because strict ssl/only https connections is on.
So i change it to :
exec 3<>/dev/tcp/www.blockchain-office.com/443
echo -e "GET / HTTP/1.1\r\nhost: https://www.blockchain-office.com\r\nConnection: close\r\n\r\n" >&3
cat <&3
When i try to run i receive:
400 Bad Request
Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.
So i even can't get a normal connection without get the file!
All this post's does not fit, looks like ssl/tls is the problem only http/80 works, if i don't use curl, wget, lynx, openssl, etc...:
how to download a file using just bash and nothing else (no curl, wget, perl, etc.)
Using /dev/tcp instead of wget
How to get a response from any URL?
Read file over HTTP in Shell
I need a solution to get/read/fetch a normal txt file from a domain over https only with /dev/tcp no other tools like curl, and output in my terminal or save in a variable without wget, etc.., is it possible and how, or is it there an other solution over the terminal with the standard terminal utilities?
You can use openssl s_client to perform the equivalent operation but delegate the SSL part:
#!/bin/sh
host='blockchain-office.com'
port=443
path='/file.txt'
crlf="$(printf '\r\n_')"
crlf="${crlf%?}"
{
printf '%s\r\n' \
"GET ${path} HTTP/1.1" \
"host: ${host}" \
'Connection: close' \
''
} |
openssl s_client -quiet -connect "${host}:${port}" 2 >/dev/null | {
# Skip headers by reading up until encountering a blank line
while IFS="${crlf}" read -r line && [ -n "$line" ]; do :; done
# Output the raw body content
cat
}
Instead of cat to output the raw body, you may want to check some headers like Content-Type, Content-Transfer-Encoding and even maybe navigate and handle recursive MIME chunks, then decode the raw content to something.
After all the comments and research, the answer is no, we can't get/fetch files using only the standard tools with the shell like /dev/tcp because we can't handle ssl/tls without handle the complete handshake.
It is only possbile with the http/80.
i dont think bash's /dev/tcp supports ssl/tls
If you use /dev/tcp for a http/https connection you have to manage the complete handshake including ssl/tls, http headers, chunks and more. Or you use curl/wget that manage it for you.
then shell is the wrong tool because it is not capable of performing any of the SSL handshake without using external resources/commands. Now relieve and use what you want and can from what I show you here as the cleanest and most portable POSIX-shell grammar implementation of a minimal HTTP session through SSL. And then maybe it is time to consider alternative options (not using HTTPS, using languages with built-in or standard library SSL support).
We will use curl, wget and openssl on seperate docker containers now.
I think there are still some requirements in the future to see if we keep only one of them or all of them.
We will use the script from #Léa Gris in a docker container too.
I have been trying to run a BASH script which posts a request to an SMS server and on successful execution a message is received on the mentioned mobile number. Script is as shown below:
curl -k -X POST "http://192.168.10.3/u=admin&h=452ba065ebd1723598a51c7eca11d362&op=pv&to=1234567891&msg=Hello+to+all"
The above script is working fine. The message "Hello to all" is being received on the mobile number 1234567891. This number is however hard coded in the URL. In the actual scenario the mobile number would be available in a variable and the SMS would be sent to the mobile number available in this variable.
I have tried scripts like:
mobile_number="1234567891"
curl -k -X POST "http://192.168.10.3/u=admin&h=452ba065ebd1723598a51c7eca11d362&op=pv&to=$mobile_number&msg=Message+From+world"
and
x="http://192.168.10.3/u=admin&h=452ba065ebd1723598a51c7eca11d362&op=pv&to="
x+="1234567891
x+=&msg=Hello+to+all"
curl -k -X POST $x
However, i have been unsuccessful in executing them successfully. It would be of great help if someone could help me with the syntax.
Try out this principle, bash is different language than c++ or so :-):
#!/bin/bash
to="1234567891"
msg="Hello+to+all"
u="admin"
hash="452ba065ebd1723598a51c7eca11d362"
op="pv"
ip="192.168.10.3"
url="http://${ip}/u=admin&h=${hash}&op=${op}&to=${to}&msg=${msg}"
echo ${url}
Than : curl -k -X POST $url should work fine.
I am trying to find the instances of the word (pattern) "Zardoz" in the output of this command:
curl http://imdb.com/title/tt0070948
I tried using: curl http://imdb.com/title/tt0070948 | grep "Zardoz"
but it just returned "file not found".
Any suggestions? I would like to use grep to do this.
You need to tell curl use to -L (--location) option:
curl -L http://imdb.com/title/tt0070948 | grep "Zardoz"
(HTTP/HTTPS) If the server reports that the requested page has
moved to a different location (indicated with a Location: header
and a 3XX response code), this option will make curl redo the
request on the new place
When curl follows a redirect and the request is not a plain GET
(for example POST or PUT), it will do the following request with
a GET if the HTTP response was 301, 302, or 303. If the response
code was any other 3xx code, curl will re-send the following
request using the same unmodified method
.
I am trying to do a script to log on a certain site and get the page infos as I would be logged.
I've searched on stacks and it appears that I must do it with 3 wgets:
one to get the hidden token , one for the cookies and post datas and the last one to get what I want. Here's the code:
#!/bin/bash
# get the login page to get the hidden field data
wget -a log.txt -O loginpage.html --user-agent="Mozilla/5.0" site/login
hiddendata=$(cat loginpage.html | grep __Req | cut -d'"' -f6,6 | head -n1 | sed s/\"//g)
echo "Logging with user $1 and pass $2"
wget --secure-protocol=auto --save-cookies cookies.txt --post-data="LoginDataModel.LoginName=$1&LoginDataModel.Password=$2&__RequestVerificationToken=${hidden_data}" --user-agent="Mozilla/5.0" site/login/login
where site/login is the login page and site/login/login is the post action and the post-data values are the
Logging with user x and pass y
--2015-02-07 12:29:07-- site/Login/Login
Resolving site (site)... 91.208.180.39
Connecting to site (site)|91.208.180.39|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: site/Login/Login [following]
--2015-02-07 12:29:18-- site/Login/Login
Resolving site (site)... 91.208.180.39
Connecting to site (site)|91.208.180.39|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-02-07 12:29:23 ERROR 404: Not Found.
when I check , the site/login/login exists. what am I doing wrong? Thank you.
I haven't done yet the third wget to get what I want since I can't connect properly.
I still can't comment so I will post it as answer. I would recommend to use curl instead of wget.
An example how to do it with curl: Appnexus Authentication Service. By your description its exactly what you need.
http://wiki.appnexus.com/display/sdk/Authentication+Service