I am learning how websocket works in python 3.
I add print(sock) to def handshake of _handshake.py in websocket source to learn what is the message inside sock
And the result is sth like this:
Print out sock:<ssl.SSLSocket fd=508, family=AddressFamily.AF_INET, type=0, proto=0, laddr=('192.168.1.2', 58730), raddr=('202.160.125.211', 443)>
I wonder what laddr and raddr is?
I know that is too basic but without solid background as me it appears complicated to understand
I have searched gg for those keywords but there is no explaination.
def handshake(sock, hostname, port, resource, **options):
headers, key = _get_handshake_headers(resource, hostname, port, options)
header_str = "\r\n".join(headers)
send(sock, header_str)
dump("request header", header_str)
print("Print out sock:{}".format(sock))
status, resp = _get_resp_headers(sock)
if status in SUPPORTED_REDIRECT_STATUSES:
return handshake_response(status, resp, None)
success, subproto = _validate(resp, key, options.get("subprotocols"))
if not success:
raise WebSocketException("Invalid WebSocket Header")
return handshake_response(status, resp, subproto)
laddr means local address and raddr means remote address of the socket. Depending on the context of the process or application, one address becomes the remote to the other socket.
Related
I am new to sockets and don't really know how to receive multiple messages from the same client. I only receive the first message and not the rest.
Server code:
import socket
IP = "127.0.0.1"
PORT = 65432
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((IP, PORT))
server.listen()
while True:
communication_socket, address = server.accept()
msg = communication_socket.recv(1024).decode("utf-8")
print(msg)
Client code:
import socket
import time
HOST = "127.0.0.1"
PORT = 65432
socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
socket.connect((HOST, PORT))
socket.send("success".encode("utf-8"))
time.sleep(2)
socket.send("?".encode("utf-8"))
socket.close()
communication_socket, address = server.accept()
msg = communication_socket.recv(1024).decode("utf-8")
The current code accepts the new connection and then does a single recv. That's why it gets only the first data. What you need are multiple recv here after the accept.
... multiple messages from the same client
TCP has no concept of messages. It is only a byte stream. There is no guaranteed 1:1 relation between send and recv, even if it might look like this in many situations. Thus message semantics must be an explicit part of the application protocol, like delimiting messages with new lines, having only fixed size messages or similar.
I have a very basic server written in Python as follows:
import socket
from time import sleep
import requests
c = None #Client socket1
addr = None #Client address1
server_socket1 = socket.socket() #by default it is SOCK_STREAM (TCP) and has porotocal AF_INET (IPv4)
server_socket1.bind(('127.0.0.1',9999)) #server machine's ip and port on which it will send and recieve connections from
server_socket1.listen(2) #We will only accept two connections as of now , one for each client
print("Server started successfully!!!")
print("Waiting for connections...\n\n")
while (((c is None)and(addr is None))):
if((c is None) and (addr is None)):
c,addr = server_socket1.accept()
print("Intrusion detected at address 127.0.0.1:9999 ")
print("Client connected with ip address "+str(addr))
client_ip=str(addr)
while True:
msg = c.recv(4096)
if(msg!=None):
#print(msg)
headers, sep, body = msg.partition(b'\r\n\r\n')
headers = headers.decode('utf-8')
print(headers)
html_body = "<html><body><h1>You are not authorized to acces this Page!</p><br><p>3 more attemps and your ip will be jailed!</p></body></html>"
response_headers = {
'Content-Type': 'text/html; encoding=utf8',
'Content-Length': len(html_body),
'Connection': 'close',
}
response_headers_raw = ''.join('%s: %s\r\n' % (k, v) for k, v in response_headers.items())
response_proto = 'HTTP/1.1'
response_status = '200'
response_status_text = 'OK' # this can be random
# sending all this stuff
r = '%s %s %s\r\n' % (response_proto, response_status, response_status_text)
c.sendall(r.encode())
c.sendall(response_headers_raw.encode())
c.sendall(b'\r\n') # to separate headers from body
c.send(html_body.encode(encoding="utf-8"))
I have then used ngrok to forward my port 9999 on the web. Then I execute the server.
Now, when I connect to the ngrok's provided link via my mobile phone, I get the response from my server, that is a single lined HTML content, as seen in the code itself.
But, the c,addr = socket.accept() should return the IP of the connected client. In my case, I have used my phone to connect to ngrok, which should use my phone's public IP to connect to it, still on my server side, it shows something like this:
Can someone please tell me what am I doing wrong here?
What you are seeing makes perfect sense, as the phone is not directly connected to your server (it can't be, since your server is listening on 127.0.0.1 aka localhost, so it can only accept connections that originate from the same machine).
The phone is connected to ngrok, and then ngrok is connected to your server. So you are seeing the IP that ngrok is connecting to your server from. There is simply no way for your server to get the IP of the phone, unless ngrok includes the phone's IP in the HTTP request it sends to your server, such as in an X-Forwarded-For, X-Original-Forwarded-For, X-Real-IP, etc kind of request header, which are common for proxies to send (but which I don't see in your screenshot, but it is incomplete).
--------- --------- ----------
| phone | <-> | ngrok | <-> | server |
--------- --------- ----------
^ ^
| |
desired IP is here but you are getting IP from here
The udp server and client on my local pc.
cat server.py
import socket
MAX_BYTES =65535
def server():
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.bind(('127.0.0.1',10000))
print('Listening at {}'.format(sock.getsockname()))
while True:
data,address = sock.recvfrom(MAX_BYTES)
text = data.decode('ascii')
print('The client at {} says {!r} '.format(address,text))
if __name__ == "__main__":
server()
Bind port 10000 with localhost-127.0.0.1,and listening to the message send from client.
cat client.py
import socket
import time
from datetime import datetime
MAX_BYTES =65535
def client():
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.bind(('127.0.0.1',10001))
text = 'The time is {}'.format(datetime.now())
data = text.encode('ascii')
while True:
time.sleep(10)
sock.sendto(data,('127.0.0.1',10000))
print('The OS assinged me the address {}'.format(sock.getsockname()))
if __name__ == "__main__":
client()
Run the server.py and client.py on my local pc,server can receive message send from client.
Now i change 127.0.0.1 in the line in client.py with my remote vps_ip.
sock.sendto(data,('127.0.0.1',10000))
into
sock.sendto(data,('remote_ip',10000))
Push server.py into my vps.Start client.py on my local pc,server.py on remote vps,start them all.
In my client,an error info occurs:
File "client.py", line 13, in client
sock.sendto(data,('remote_ip',10000))
OSError: [Errno 22] Invalid argument
How to make remote ip receive message send from my local client pc?
Two things that could be happening:
You're not passing the remote IP correctly. Make sure that your not passing literally 'remote_ip' and replace it with a valid IPv4 IP address string (IE: '192.168.0.100') for the server. (FYI technically on the server you can just put '0.0.0.0' to listen on all IP addresses)
You could still be binding the client to the local address to (127.0.0.1), but setting the destination to a valid external address (192.168.0.100). Remove the socket.bind line of code in the client to test this, you shouldn't need it.
If these both don't work, then add the results of a ping command running on the client and targeting the server.
I'd like to manually (using the socket and ssl modules) make an HTTPS request through a proxy which itself uses HTTPS.
I can perform the initial CONNECT exchange just fine:
import ssl, socket
PROXY_ADDR = ("proxy-addr", 443)
CONNECT = "CONNECT example.com:443 HTTP/1.1\r\n\r\n"
sock = socket.create_connection(PROXY_ADDR)
sock = ssl.wrap_socket(sock)
sock.sendall(CONNECT)
s = ""
while s[-4:] != "\r\n\r\n":
s += sock.recv(1)
print repr(s)
The above code prints HTTP/1.1 200 Connection established plus some headers, which is what I expect. So now I should be ready to make the request, e.g.
sock.sendall("GET / HTTP/1.1\r\n\r\n")
but the above code returns
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
Reason: You're speaking plain HTTP to an SSL-enabled server port.<br />
Instead use the HTTPS scheme to access this URL, please.<br />
</body></html>
This makes sense too, since I still need to do an SSL handshake with the example.com server to which I'm tunneling. However, if instead of immediately sending the GET request I say
sock = ssl.wrap_socket(sock)
to do the handshake with the remote server, then I get an exception:
Traceback (most recent call last):
File "so_test.py", line 18, in <module>
ssl.wrap_socket(sock)
File "/usr/lib/python2.6/ssl.py", line 350, in wrap_socket
suppress_ragged_eofs=suppress_ragged_eofs)
File "/usr/lib/python2.6/ssl.py", line 118, in __init__
self.do_handshake()
File "/usr/lib/python2.6/ssl.py", line 293, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [Errno 1] _ssl.c:480: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
So how can I do the SSL handshake with the remote example.com server?
EDIT: I'm pretty sure that no additional data is available before my second call to wrap_socket because calling sock.recv(1) blocks indefinitely.
This should work if the CONNECT string is rewritten as follows:
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)
Not sure why this works, but maybe it has something to do with the proxy I'm using. Here's an example code:
from OpenSSL import SSL
import socket
def verify_cb(conn, cert, errun, depth, ok):
return True
server = 'mail.google.com'
port = 443
PROXY_ADDR = ("proxy.example.com", 3128)
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(PROXY_ADDR)
s.send(CONNECT)
print s.recv(4096)
ctx = SSL.Context(SSL.SSLv23_METHOD)
ctx.set_verify(SSL.VERIFY_PEER, verify_cb)
ss = SSL.Connection(ctx, s)
ss.set_connect_state()
ss.do_handshake()
cert = ss.get_peer_certificate()
print cert.get_subject()
ss.shutdown()
ss.close()
Note how the socket is first opened and then open socket placed in SSL context. Then I manually initialize SSL handshake. And output:
HTTP/1.1 200 Connection established
<X509Name object '/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com'>
It's based on pyOpenSSL because I needed to fetch invalid certificates too and Python built-in ssl module will always try to verify the certificate if it's received.
Judging from the API of the OpenSSL and GnuTLS library, stacking a SSLSocket onto a SSLSocket is actually not straightforwardly possible as they provide special read/write functions to implement the encryption, which they are not able to use themselves when wrapping a pre-existing SSLSocket.
The error is therefore caused by the inner SSLSocket directly reading from the system socket and not from the outer SSLSocket. This ends in sending data not belonging to the outer SSL session, which ends badly and for sure never returns a valid ServerHello.
Concluding from that, I would say there is no simple way to implement what you (and actually myself) would like to accomplish.
Finally I got somewhere expanding on #kravietz and #02strich answers.
Here's the code
import threading
import select
import socket
import ssl
server = 'mail.google.com'
port = 443
PROXY = ("localhost", 4433)
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)
class ForwardedSocket(threading.Thread):
def __init__(self, s, **kwargs):
threading.Thread.__init__(self)
self.dest = s
self.oursraw, self.theirsraw = socket.socketpair(socket.AF_UNIX, socket.SOCK_STREAM)
self.theirs = socket.socket(_sock=self.theirsraw)
self.start()
self.ours = ssl.wrap_socket(socket.socket(_sock=self.oursraw), **kwargs)
def run(self):
rl, wl, xl = select.select([self.dest, self.theirs], [], [], 1)
print rl, wl, xl
# FIXME write may block
if self.theirs in rl:
self.dest.send(self.theirs.recv(4096))
if self.dest in rl:
self.theirs.send(self.dest.recv(4096))
def recv(self, *args):
return self.ours.recv(*args)
def send(self, *args):
return self.outs.recv(*args)
def test():
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(PROXY)
s = ssl.wrap_socket(s, ciphers="ALL:aNULL:eNULL")
s.send(CONNECT)
resp = s.read(4096)
print (resp, )
fs = ForwardedSocket(s, ciphers="ALL:aNULL:eNULL")
fs.send("foobar")
Don't mind custom cihpers=, that only because I didn't want to deal with certificates.
And there's depth-1 ssl output, showing CONNECT, my response to it ssagd and depth-2 ssl negotiation and binary rubbish:
[dima#bmg ~]$ openssl s_server -nocert -cipher "ALL:aNULL:eNULL"
Using default temp DH parameters
Using default temp ECDH parameters
ACCEPT
-----BEGIN SSL SESSION PARAMETERS-----
MHUCAQECAgMDBALAGQQgmn6XfJt8ru+edj6BXljltJf43Sz6AmacYM/dSmrhgl4E
MOztEauhPoixCwS84DL29MD/OxuxuvG5tnkN59ikoqtfrnCKsk8Y9JtUU9zuaDFV
ZaEGAgRSnJ81ogQCAgEspAYEBAEAAAA=
-----END SSL SESSION PARAMETERS-----
Shared ciphers: [snipped]
CIPHER is AECDH-AES256-SHA
Secure Renegotiation IS supported
CONNECT mail.google.com:443 HTTP/1.0
Connection: close
sagq
�u\�0�,�(�$��
�"�!��kj98���� �m:��2�.�*�&���=5�����
��/�+�'�#�� ����g#32��ED���l4�F�1�-�)�%���</�A������
�� ������
�;��A��q�J&O��y�l
It doesn't sound like there's anything wrong with what you're doing; it's certainly possible to call wrap_socket() on an existing SSLSocket.
The 'unknown protocol' error can occur (amongst other reasons) if there's extra data waiting to be read on the socket at the point you call wrap_socket(), for instance an extra \r\n or an HTTP error (due to a missing cert on the server end, for instance). Are you certain you've read everything available at that point?
If you can force the first SSL channel to use a "plain" RSA cipher (i.e. non-Diffie-Hellman) then you may be able to use Wireshark to decrypt the stream to see what's going on.
Building on #kravietz answer. Here is a version that works in Python3 through a Squid proxy:
from OpenSSL import SSL
import socket
def verify_cb(conn, cert, errun, depth, ok):
return True
server = 'mail.google.com'
port = 443
PROXY_ADDR = ("<proxy_server>", 3128)
CONNECT = "CONNECT %s:%s HTTP/1.0\r\nConnection: close\r\n\r\n" % (server, port)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(PROXY_ADDR)
s.send(str.encode(CONNECT))
s.recv(4096)
ctx = SSL.Context(SSL.SSLv23_METHOD)
ctx.set_verify(SSL.VERIFY_PEER, verify_cb)
ss = SSL.Connection(ctx, s)
ss.set_connect_state()
ss.do_handshake()
cert = ss.get_peer_certificate()
print(cert.get_subject())
ss.shutdown()
ss.close()
This works in Python 2 also.
I'm working on an app which needs to have a WebSockets API, and will also integrate Jupyter (former IPython) notebooks as a relatively minor feature. Since Jupyter already uses WebSockets for communication, how difficult it would be to integrate it as a general library for serving other WebSockets APIs apart to its own? Or am I better off using another library such as aiohttp? I'm looking for any advice and hints abut the best practices for this. Thanks!
You can proxy WebSockets from your main application to Jupyter.
It really doesn't matter what technology you use to serve WebSockets, the proxy loop will be very similar (wait for message, push message forward). However, it will be web server dependent as Python does not have standard to WebSockets akin WSGI.
I did one in pyramid_notebook project. Running Jupyter in its own process is must as, at least by the time of writing the code, embedding Jupyter directly to your application was not feasible. I am not sure though if the latest version have changed this. Jupyter itself was using Tornado.
"""UWSGI websocket proxy."""
from urllib.parse import urlparse, urlunparse
import logging
import time
import uwsgi
from pyramid import httpexceptions
from ws4py import WS_VERSION
from ws4py.client import WebSocketBaseClient
#: HTTP headers we need to proxy to upstream websocket server when the Connect: upgrade is performed
CAPTURE_CONNECT_HEADERS = ["sec-websocket-extensions", "sec-websocket-key", "origin"]
logger = logging.getLogger(__name__)
class ProxyClient(WebSocketBaseClient):
"""Proxy between upstream WebSocket server and downstream UWSGI."""
#property
def handshake_headers(self):
"""
List of headers appropriate for the upgrade
handshake.
"""
headers = [
('Host', self.host),
('Connection', 'Upgrade'),
('Upgrade', 'WebSocket'),
('Sec-WebSocket-Key', self.key.decode('utf-8')),
# Origin is proxyed from the downstream server, don't set it twice
# ('Origin', self.url),
('Sec-WebSocket-Version', str(max(WS_VERSION)))
]
if self.protocols:
headers.append(('Sec-WebSocket-Protocol', ','.join(self.protocols)))
if self.extra_headers:
headers.extend(self.extra_headers)
logger.info("Handshake headers: %s", headers)
return headers
def received_message(self, m):
"""Push upstream messages to downstream."""
# TODO: No support for binary messages
m = str(m)
logger.debug("Incoming upstream WS: %s", m)
uwsgi.websocket_send(m)
logger.debug("Send ok")
def handshake_ok(self):
"""
Called when the upgrade handshake has completed
successfully.
Starts the client's thread.
"""
self.run()
def terminate(self):
super(ProxyClient, self).terminate()
def run(self):
"""Combine async uwsgi message loop with ws4py message loop.
TODO: This could do some serious optimizations and behave asynchronously correct instead of just sleep().
"""
self.sock.setblocking(False)
try:
while not self.terminated:
logger.debug("Doing nothing")
time.sleep(0.050)
logger.debug("Asking for downstream msg")
msg = uwsgi.websocket_recv_nb()
if msg:
logger.debug("Incoming downstream WS: %s", msg)
self.send(msg)
s = self.stream
self.opened()
logger.debug("Asking for upstream msg")
try:
bytes = self.sock.recv(self.reading_buffer_size)
if bytes:
self.process(bytes)
except BlockingIOError:
pass
except Exception as e:
logger.exception(e)
finally:
logger.info("Terminating WS proxy loop")
self.terminate()
def serve_websocket(request, port):
"""Start UWSGI websocket loop and proxy."""
env = request.environ
# Send HTTP response 101 Switch Protocol downstream
uwsgi.websocket_handshake(env['HTTP_SEC_WEBSOCKET_KEY'], env.get('HTTP_ORIGIN', ''))
# Map the websocket URL to the upstream localhost:4000x Notebook instance
parts = urlparse(request.url)
parts = parts._replace(scheme="ws", netloc="localhost:{}".format(port))
url = urlunparse(parts)
# Proxy initial connection headers
headers = [(header, value) for header, value in request.headers.items() if header.lower() in CAPTURE_CONNECT_HEADERS]
logger.info("Connecting to upstream websockets: %s, headers: %s", url, headers)
ws = ProxyClient(url, headers=headers)
ws.connect()
# TODO: Will complain loudly about already send headers - how to abort?
return httpexceptions.HTTPOk()