I want to send some HTTP POST requests to a website. There's no problem with doing it by my browser, since I'm logged in and I have my authenticated session.
I don't know how to make POST requests by Python while using that authenticated session (so importing data from the browser), and I don't even know if it's possibile, so I thought that I could trigger these POST requests in browser from the Python script.
I know there exists a library called "webbrowser" that allows to make requests by opening new tabs in browser. My problem is that I need a POST request, not a GET one.
So I ask you for 2 suggestions: is there a way to import authenticated session such that Pyhon can make requests for websites that require auth?
If not, is there a way of doing something like this:
webbrowser.open(url, new=0)
but with POST requests?
Thanks!
I agree with Mr. C, but alternatively...
Checkout Selenium. You can't make a direct POST request, but you can do anything that you would usually do as a normal user.
Check out the requests library. Seems like you need more functionality than the webbrowser module provides.
Related
I am trying to login to a website and based on the REST API guide provided, I will be able to receive some data for an application.
The two steps required for me to enact are:
1. Send a HTTP post command for authentication.
2. Send a GET command for receiving the data.
When I send the post command using python requests, i receive the required json response showing my login rights. e.g. role admin.
However, when I perform the get command after, it doesn't retrieve the data but sends an HTML form showing that I require authentication even though I have authenticated already.
Has anyone encountered this and how will I be able to solve it?
I am working on this for a customer and as a result, cannot post the actual login I am using and url and will hence replace this with my name with the code I will display.
Thanks
enter image description here
You can take a look at the session object in the requests library. It's used to keep a session across multiple requests.
https://3.python-requests.org/user/advanced/#session-objects
import requests
s = requests.Session()
s.get('http://httpbin.org/cookies/set?authcookie=123')
r = s.get('http://httpbin.org/cookies')
print(r.text)
This is a sample that uses a request session object to call a url that sets a cookie and then does another call with the same session (including the cookie). Just some basic example using http://httpbin.org/.
can any explain me about how the server responds if we click the button in browser using PHP script then which situation we use JAVASCRIPT and which situation we have to use PHP .
If you're asking about how it works, I'll suggest to google about HTTP protocol, POST and GET methods.
Usually, you use JavaScript to send request to a server when you want it to be AJAX, ie when you want to send request to server and receive response without reloading page. And you use PHP when you have form in your HTML (like login or registration form), and you send it to server in one request to go to another page
now I am learning how to build single page web applications with node.js. When I come to the communication part between server and client, I am confusing about the difference between http post request and web socket communication
Here is my understanding:
Web socket like socket.io is asynchronous communication while POST is synchronous communication method.
socket.io is mainly used in dealing with real-time task like html games or dynamic online chatting, or boradcasting some informations.
for validating forms when some user sign up for certain website, it is really good to check instantly if the user name is already been taken or not, I think it is using socket.io to achieve that.
for example in a login page, after I press the username and password, then press 'login' button, I could just use emit things in socket.io to transfer a JSON file with these information. when we checked these information and send back a flag string like 'true' or 'false ' to indicate if the login is success or not.
So my question is:
Why we still use HTTP POST for login and signup ?
Is socket.io better than HTTP POST ?
Why not replace HTTP POST with socket.io ?
P.S for some large files, I have not go into these parts, so I have no idea.
Here is some additions:
After reading a similar question I find I seem to assume that WebSocket is a replacement for HTTP. It is not. It's an extension.
I think this asnwered my third question Why not replace HTTP POST with socket.io.
I'm trying to automate some datascraping from a website. However, because the user has to go through a login screen a wget cronjob won't work, and because I need to make an HTTPS request, a simple Perl script won't work either. I've tried looking at the "DejaClick" addon for Firefox to simply replay a series of browser events (logging into the website, navigating to where the interesting data is, downloading the page, etc.), but the addon's developers for some reason didn't include saving pages as a feature.
Is there any quick way of accomplishing what I'm trying to do here?
A while back I used mechanize wwwsearch.sourceforge.net/mechanize and found it very helpful. It supports urllib2 so it should also work with HTTPS requests as I read now. So my comment above could hopefully prove wrong.
You can record your action with IRobotSoft web scraper. See demo here: http://irobotsoft.com/help/
Then use saveFile(filename, TargetPage) function to save the target page.
I need to login to a website with username and password, and then download a file. The url of the file is static. How do I automate the above process with Linux/Unix scripts? Thanks a lot.
Jiangzhe
well, it's not that simple.
what you need to do is the following:
send an HTTP POST request containing your username and password to the login form's URL.
you will get a cookie (probably containing a session ID).
send an HTTP GET request for the file, sending your cookie details in the HTTP headers.
you probably should use some scripting language with an HTTP library (python's httplib and urllib2 are great options).
Just use CURL to send POST or GET request with login data to site and then do second request to download file.