s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("privnote.com", 80))
#s = ssl.wrap_socket(s, keyfile=None, certfile=None, server_side=False, cert_reqs=ssl.CERT_NONE, ssl_version=ssl.PROTOCOL_SSLv23)
def claim_note(note_url):
s.sendall(b'DELETE /'+note_url.encode()+b'HTTP/1.1\r\nX-Requested-With: XMLHttpRequest\r\nHost: privnote.com\r\n')
print(s.recv(4096))
This is my code, now let me first start by saying that I have tried so many different things apart from this. I’ve tried https port and http port, 443, 80. I’ve commented and uncommented the statement that wraps the socket with ssl. All with the same outcome. Either the api returning absolutely nothing or the api telling me the request couldn’t be understood by the server. I was looking at a GitHub repo and only one header was used and it was because it was for an Ajax call which was x-requested-with. I tried adding user agent content type and now I’m just using host and x requested with. It’s a DELETE request and the url is the first 8 chars after the link. I’ve also tried adding \r\n\r\n at the end and even tried content-length. I don’t know what else to do. I want to know why the server is saying that.
There are multiple problems with your code. If you actually print out the request you are trying to sent it will look like this:
b'DELETE /node_urlHTTP/1.1\r\nX-Requested-With: XMLHttpRequest\r\nHost: privnote.com\r\n'
There are two problems with this line: a missing space between /node_url and HTTP/1.1 and a missing final \r\n als end-of-header marker at the end. Once these are fixed you get a successful response - a 302 redirect to the HTTPS version:
b'HTTP/1.1 302 Found\r\nDate:...\r\nLocation: https://privnote.com/node_url ...
When repeating the request with HTTPS and a valid node_url (with an invalid node_url you get an error that DELETE is not an allowed method):
s.connect(("privnote.com", 443))
s = ssl.wrap_socket(s)
...
b'HTTP/1.1 200 OK\r\n ...
Related
Client
Python Version - 3.9,
Python Requests module version - 2.25
Server
Java 13,
Tomcat 9.
I have a Tomcat+Java based server exposing REST APIs. I am writing a client in python to consume those APIs. Everything is fine until I send empty body in POST request. It is a valid use case for us. If I send empty body I get 400 bad request error - Invalid character found in method name [{}POST]. HTTP method names must be tokens. If I send empty request from POSTMAN or Java or CURL it works fine, problem is only when I used python as a client.
Following is python snippet -
json_object={}
header = {'alias': 'A', 'Content-Type' : 'application/json', 'Content-Length' : '0'}
resp = requests.post(url, auth=(username, password), headers=header, json=json_object)
I tried using data as well instead of json param to send payload with not much of success.
I captured the wireshark dumps to undertand it further and found that, the request tomcat received is not as per RFC2616 (https://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html). Especially the part -
Request-Line = Method SP Request-URI SP HTTP-Version CRLF
Because I could see in from wireshark dumps it looked like - {}POST MY-APP-URI HTTP/1.1
As we can see the empty body is getting prefixed with http-method, hence tomcat reports that as an error.
I then looked at python http library code -client.py. Following are relevant details -
File - client.py
Method - _send_output (starting at line # 1001) - It first sends the header at line #1010 and then the body somewhere down in the code. I thought(I could be wrong here) perhaps in this case header is way longer 310 bytes than body 2 bytes, so by the time complete header is sent on wire body is pushed and hence TCP frames are order in such a way that body appears first. To corroborate this I added a delay of 1 second just after sending header line#1011 and bingo, the error disappeared and it started working fine. Not sure if this is completely correct analysis, but can someone in the know can confirm or let me know how to fix this.
I have studied the code as follow in Cherrypy web development,
if returnpage != '':
raise cherrypy.InternalRedirect(returnpage)
else:
raise cherrypy.HTTPRedirect("/hqc")
Google doesn't help much in this case after I did some research.
I've checked from cherrypy's __doc__, but the documentation there is very terse.
>>>print(cherrypy.InternalRedirect.__doc__)
Exception raised to switch to the handler for a different URL.
This exception will redirect processing to another path within the site
(without informing the client). Provide the new path as an argument when
raising the exception. Provide any params in the querystring for the new URL.
>>> print(cherrypy.HTTPRedirect.__doc__)
Exception raised when the request should be redirected.
This exception will force a HTTP redirect to the URL or URL's you give it.
The new URL must be passed as the first argument to the Exception,
e.g., HTTPRedirect(newUrl). Multiple URLs are allowed in a list.
If a URL is absolute, it will be used as-is. If it is relative, it is
assumed to be relative to the current cherrypy.request.path_info.
If one of the provided URL is a unicode object, it will be encoded
using the default encoding or the one passed in parameter.
There are multiple types of redirect, from which you can select via the
``status`` argument. If you do not provide a ``status`` arg, it defaults to
303 (or 302 if responding with HTTP/1.0).
Examples::
raise cherrypy.HTTPRedirect("")
raise cherrypy.HTTPRedirect("/abs/path", 307)
raise cherrypy.HTTPRedirect(["path1", "path2?a=1&b=2"], 301)
See :ref:`redirectingpost` for additional caveats.
My questions are:
- Why bother with redirect when you can simply invoke another handler?
- What are some practical senarios for the two redirect exception respectively?
InternalRedirect is only handled in the server side, this means that the client would not be aware of that redirection, because in terms of the HTTP protocol that is mediating the session between the client and the server, nothing changed. By server side I mean ONLY CherryPy will be aware of the rediction, if you have some intermediate server (like an nginx reverse proxy) it would not see anything different.
For example if the client visited a url /page_one and then you used raise InternalRedirect('/page_two'), the client (browser) will receive the content from the /page_two handler in the /page_one url. If you raised a regular HTTPRedirect the server would end the first request with an HTTP status code of 303 (or any other status that you passed to the exception) and a Location header to /page_two. Then is the client who will initiate another request to /page_two, basically everybody will be aware of the redirection (more info about HTTP redirection). Most of the time this is the better alternative.
Additionally you could detect if the request came from a previous InternalRedirect by verifying the cherrypy.request.prev property. It will have the previous cherrypy.request object as its value or None.
For the sake of a possible (maybe not the best example) use of an InternalRedirect, checkout this production/beta example page, in addition I added a tool to prohibit the client to reach to handlers directly.
The client will see a different content in the same page /. Note that the access log that CherryPy generates will log the url of the handler that end up handling the request, in this case you will see /_beta or /_production.
import random
import cherrypy
#cherrypy.tools.register('before_handler')
def private_handler():
"""End the request with HTTP 404 not found if the client
tries to reach the handler directly instead of being
internally redirected from other handler.
"""
if cherrypy.request.prev is None:
raise cherrypy.NotFound()
class MainApp:
#cherrypy.expose
def index(self):
# 50/50 change of receiving production or the new SHINY beta page
use_beta = random.randint(0, 1)
if use_beta:
raise cherrypy.InternalRedirect('/_beta')
else:
raise cherrypy.InternalRedirect('/_production')
#cherrypy.tools.private_handler()
#cherrypy.expose
def _production(self):
return (
"<html>"
"<h2>{}</h2>"
"</html>"
).format(
"Welcome to our awesome site!"
)
#cherrypy.tools.private_handler()
#cherrypy.expose
def _beta(self):
return (
"<html>"
'<h1 style="color: blue">{}</h1>'
"<p>{}</p>"
"</html>"
).format(
"Welcome to our awesome site!",
"Here is our new beta content..."
)
cherrypy.quickstart(MainApp())
I had a code snippet that I tried a couple of weeks ago, that was getting the HTML of the page no problem (I was learning about NLP and wanted to try a couple of things with the titles). However, now when I perform a request with proxies (proxies=session.proxies) I get404. When I omit using the proxies everything is fine (that means that I am using my own IP, headers in both cases are identical)... Can someone help me with what is going on here? I am positive that the IPs are blocked. When I use some free proxies from the Internet everything is fine. But they are super unstable, and so it's not possible to do anything. I have also looked at the ips that this code snippet produces:
with Controller.from_port(port=9051) as controller:
controller.authenticate('my_pass')
controller.signal(Signal.NEWNYM)
time.sleep(controller.get_newnym_wait())
2 ips worked (Netherlands and France) from like 20 (Austria, UK, Netherlands (but starts with different two leading digits), Liberia, France (also different two leading digis)) that I have tried so far. Is it possible to tell tor to use proxies from specific countries? I guess that wouldn't help me much. If only I could cycle through the working IPs, but I have read somewhere that that is not possible.
Here is the code that preceeds the above code snippet:
import requests
global session
session = requests.session()
session.proxies = {}
session.proxies['http'] = 'socks5h://localhost:9050'
session.proxies['https'] = 'socks5h://localhost:9050'
url = 'https://www.reuters.com/finance/stocks/company-news/AAPL.OQ?date=12222016'
html = requests.get(url, timeout=(120, 120), proxies=session.proxies)
print(html)
Tried changing user agents to no avail.
Also note, that for the above to work, you need this.
I've been scratching my head the whole day yesterday about this and to my surprise, can't seem to find an easy way to check this.
I am using Python's Requests library to pass my proxy such as:
def make_request(url):
with requests.Session() as s:
s.mount("http://", HTTPAdapter(max_retries=3))
s.mount("https://", HTTPAdapter(max_retries=3))
page = None
d.rotate(-1) #d contains a dict of my proxies. this allows to rotate through the proxies everytime make_request is called.
s.proxies = d[0]
page = s.get(url, timeout=3)
print('proxy used: ' + str(d[0]))
return page.content
Problem is, I can't seem to make the request fail when the proxy is not expected to work. It seems there is always a fallback on my internet ip if the proxy is not working.
For example: I tried passing a random proxy ip like 101.101.101.101:8800 or removing the ip authentication that is needed on my proxies, the request is still passed, even though it should'nt.
I thought adding the timeout parameters when passing the request would do the trick, but obviously it didn't.
So
Why does this happen?
How can I check from which ip a request is being made?
From what I have seen so far, you should use the form
s.get(url, proxies = d)
This should use the proxies in the dict d to make a connection.
This form allowed me to check with working proxies and non-working proxies the status_code
print(s.status_code)
I will update once I find out whether it just circulates over the proxies in the dict to match a working one, or one is able to actually select which one to be used.
[UPDATE]
Tried to work around the dict in proxies, to use different proxy if I wanted to. However, proxies must be a dict to work. So I used a dict in the form of:
d = {"https" : 'https://' + str(proxy_ips[n].strip('\n'))}
This seems to work and allow me to use an ip I want to. Although it seems quite dull, I hope someone might come and help!
The proxies used can be seen through:
requests.utils.getproxies()
or
requests.utils.get_environ_proxies(url)
I hope that helps, obviously quite old question, but still!
I want to test whether some URLs are broken or not.
Now I just assert some words I know are on these pages. I feel this isn't the best that i can do. Any help?
I figured it out,
You Can detect "500 Internal Server Error" using poltergeist
Inspecting network traffic
You can inspect the network traffic (i.e. what resources have been loaded) on the current page by calling page.driver.network_traffic. This returns an array of request objects. A request object has a response_parts method containing data about the response chunks.
so, this will work :
page.driver.network_traffic.each do |request|
request.response_parts.uniq(&:url).each do |response|
puts "Error : #{response.url}" if response.status == 500
end
end