How to make a TLS connection using python? - python-3.x

I would like to create a TLS connection to a server. Then, I want to send some encrypted data to the server. I know the hostname and port and I have the certificate. Surprisingly, I also received the private key of the server. However, I think it is not normal that I received the private key.
The first question is that, do I really need the private key to make a TLS connection?
By the way, I am using this python script
import socket
import ssl
server_addr = '**.**.**.**'
server_port = ****
server_cert = 'server.crt'
server_key = 'server.key' # I use the private key
context = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
context.verify_mode = ssl.CERT_REQUIRED
context.load_cert_chain(certfile=server_cert, keyfile=server_key)
bindsocket = socket.socket()
bindsocket.connect((server_addr, server_port))
I am using the private key in the above script. It works without any error. However, when I try to bind() instead of connect(), i.e.,
bindsocket.bind((server_addr, server_port))
I get the following error:
OSError: [Errno 99] Cannot assign requested address
I have read many related questions about the above error, however, I still do not understand why this happens. Since, I have the hostname, port, certificate, and the key, I expect to create a TLS connection successfully.
The second question is that how can I establish a TLS connection? Is my script correct?
I highly appreciate any comment to improve the script.

So, first of all, you should absolutely not have the private key! As the name says, it is private and not necessary to establish a connection.
You could have the public key, but even that is not necessary as long as you use standard SSL and you trust the CA that signed the servers certificate.
Are you sure, it is the private key? Does the file begin with -----BEGIN PRIVATE KEY-----? Check with openssl rsa -noout -text -in server.key.
Refer to the wikipedia article and this post for more on asymmetric cryptography.
Further along the way:
With socket.bind() you bind a socket to a port on your local machine. This is not possible, as your machine does not have the address (you provide a server address).
From your code, it looks like you are trying to open the socket as a server. You will need the private key for that, but then you will be accepting connections and not connect to other machines yourself. I have a feeling, that you are mixing up two things here.
Refer to the python documentation of socket.bind() and to this question as this seems to be closely related.
Also check out the python documentation on ssl. I took the example, that does what you are asking for from said documentation:
import socket, ssl, pprint
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# require a certificate from the server
ssl_sock = ssl.wrap_socket(s,
ca_certs="/etc/ca_certs_file",
cert_reqs=ssl.CERT_REQUIRED)
ssl_sock.connect(('www.verisign.com', 443))
pprint.pprint(ssl_sock.getpeercert())
# note that closing the SSLSocket will also close the underlying socket
ssl_sock.close()
Also have a look on the example on how to open a SSL socket in server mode.
Further thoughts:
Do you really need to do all that TLS stuff yourself? If the server, for example, uses HTTPS (SSL encrypted HTTP), you can just use the http.client library.
Feel free to ask, if you need me to clarify something. I'll update my answer accordingly.
EDIT:
As you indicated, you want to open a port in server mode, I made an example for you (it heavily leans on the python documentation example):
import socket, ssl
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
context.load_cert_chain(certfile="cert.pem", keyfile="key.pem")
bindsocket = socket.socket()
bindsocket.bind(('127.0.0.1', 10023))
bindsocket.listen(5)
def deal_with_client(connstream):
data = connstream.recv(1024)
# empty data means the client is finished with us
while data:
print(data)
data = connstream.recv(1024)
while True:
newsocket, fromaddr = bindsocket.accept()
connstream = context.wrap_socket(newsocket, server_side=True)
try:
deal_with_client(connstream)
finally:
connstream.shutdown(socket.SHUT_RDWR)
connstream.close()
Running it:
% python3 ssltest.py
b'hello server!\n'
b'this is data\n'
The client side:
% openssl s_client -connect 127.0.0.1:10023
CONNECTED(00000005)
depth=0 C = SE, ST = Some-State, O = Internet Widgits Pty Ltd
verify error:num=18:self signed certificate
verify return:1
depth=0 C = SE, ST = Some-State, O = Internet Widgits Pty Ltd
verify return:1
---
Certificate chain
0 s:C = SE, ST = Some-State, O = Internet Widgits Pty Ltd
i:C = SE, ST = Some-State, O = Internet Widgits Pty Ltd
---
Server certificate
-----BEGIN CERTIFICATE-----
... certificate ...
-----END CERTIFICATE-----
subject=C = SE, ST = Some-State, O = Internet Widgits Pty Ltd
issuer=C = SE, ST = Some-State, O = Internet Widgits Pty Ltd
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2272 bytes and written 404 bytes
Verification error: self signed certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
... session stuff ...
Extended master secret: yes
---
hello server!
this is data
^C
If you use some usual protocol like HTTP, you should use a library though. Here is a example with flask:
>>> from flask import Flask
>>> app = Flask(__name__)
>>>
>>> #app.route("/")
... def hello():
... return "Hello World!"
...
>>> if __name__ == "__main__":
... app.run(ssl_context=('cert.pem', 'key.pem'))
...
* Serving Flask app "__main__" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on https://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [06/Aug/2020 11:45:50] "GET / HTTP/1.1" 200 -

Related

Specify SNI server_hostname when performing request with asyncio/aiohttp

Hello fellow developers ! I'm stuck in a corner case and I'm starting to be out of hairs to pull... Here is the plot :
load-balancer.example.com:443 (TCP passthrough)
/\
/ \
/ \
/ \
s1.example.com:443 s2.example.com:443
(SSL/SNI) (SSL/SNI)
The goal is to stress-test the upstreams s1 and s2 directly using aiohttp with certificate-validation enable. Since the load-balancer does not belong to me I don't want to do the stress-test over it.
the code is not supposed to run on other platforms than GNU Linux with at least Python-v3.7 (but I can use any recent version if needed)
all servers serve a valid certificate for load-balancer.example.com
openssl validates the certificate from the upstreams when using openssl s_connect s1.example.com:443 -servername load-balancer.example.com
cURL needs curl 'https://load-balancer.example.com/' --resolve s1.example.com:443:load-balancer.example.com and also validates successfully
I am able to launch a huge batch of async ClientSession.get requests on both upstreams in parallel but for each request I need to somehow tell asyncio or aiohttp to use load-balancer.example.com as server_hostname, otherwise the SSL handshake fails.
Is there an easy way to setup the ClientSession to use a specific server_hostname when setting up the SSL socket ?
Does someone have already done something like that ?
EDIT : here is the most simple snippet with just a single request :
import aiohttp
import asyncio
async def main_async(host, port, uri, params=[], headers={}, sni_hostname=None):
if sni_hostname is not None:
print('Setting SNI server_name field ')
#
# THIS IS WHERE I DON'T KNOW HOW TO TELL aiohttp
# TO SET THE server_name FIELD TO sni_hostname
# IN THE SSL SOCKET BEFORE PERFORMING THE SSL HANDSHAKE
#
try:
async with aiohttp.ClientSession(raise_for_status=True) as session:
async with session.get(f'https://{host}:{port}/{uri}', params=params, headers=headers) as r:
body = await r.read()
print(body)
except Exception as e:
print(f'Exception while requesting ({e}) ')
if __name__ == "__main__":
asyncio.run(main_async(host='s1.example.com', port=443,
uri='/api/some/endpoint',
params={'apikey': '0123456789'},
headers={'Host': 'load-balancer.example.com'},
sni_hostname='load-balancer.example.com'))
When running it with real hosts, it throws
Cannot connect to host s1.example.com:443 ssl:True
[SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] '
certificate verify failed: certificate has expired (_ssl.c:1131)')])
Note that the error certificate has expired indicates that the certificate proposed to the client is the default certificate since the SNI hostname is s1.example.com which is unknow by the webserver running there.
When running it against the load-balancer it works just fine, the SSL handshake happens with the upstreams which serve the certificate and everything is valid.
Also note that
sni_callback does not help since it is called after the handshake has started and the certificate was received (and at this point server_hostname is a read-only property anyway)
it does not seem to be possible to set server_hostname when creating an SSLContext allthough SSLContext.wrap_socket does support server_hostname but I was not able to make that work
I hope someone knows how to fill the comment block in that snippet ;-]

When we run this code , we see additional ciphers other than the ones mentioned.How do we force enable specific ciphers only?

pprint.pprint(context.get_ciphers()) prints additional ciphers too. How do we force the usage of ciphers and extensions ?
import socket, ssl
import pprint
import ssl
context = ssl.create_default_context()
CIPHERS ="ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305-SHA256:ECDHE-RSA-AES128-CBC-SHA:ECDHE-ECDSA-AES128-CBC-SHA:ECDHE-RSA-AES256-CBC-SHA:ECDHE-ECDSA-AES256-CBC-SHA:RSA-AES128-GCM-SHA256:RSA-AES256-GCM-SHA384:RSA-AES128-CBC-SHA:RSA-AES256-CBC-SHA:ECDHE-RSA-3DES-EDE-CBC-SHA:RSA-3DES-EDE-CBC-SHA:AES128-GCM-SHA256:CHACHA20-POLY1305-SHA256:AES:256-GCM-SHA384"
context.set_ciphers(CIPHERS)
pprint.pprint(context.get_ciphers())
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
domain = 'google.com'
sslSocket = context.wrap_socket(s, server_hostname = domain)
sslSocket.connect((domain, 443))
#context.options |= ssl.OP_NO_SSLv2
#context.options |= ssl.OP_NO_SSLv3
print(sslSocket.cipher())
print(sslSocket.version())
print(ssl.OPENSSL_VERSION)
sslSocket.close()
While it is unclear what ciphers you exactly mean with "we see additional ciphers other than the ones mentioned" you likely mean TLS 1.3 ciphers like TLS_AES_128_GCM_SHA256. From the documentation of set_ciphers:
OpenSSL 1.1.1 has TLS 1.3 cipher suites enabled by default. The suites cannot be disabled with set_ciphers().
If you want to disable TLS 1.3 ciphers you have to disable TLS 1.3:
context.options |= ssl.OP_NO_TLSv1_3
Note that this does not actually remove the ciphers from the output of get_ciphers but it makes these unusable. To actually set these ciphers one would need to call the OpenSSL API SSL_CTX_set_ciphersuites. This API is not yet available from Python.

TLS session resumption in python TLS socket

I have a simple TLS client in python that connects to TLS servers. I do not have control over the servers. I need a fresh TLS handshake with each server even if I visited it recently.
1) Do non-browser TLS clients such as the following python client perform session resumption by default?
2) How can I know if they do or do not? How can I disable session resumption if it is performed in the background?
Please note that I create a new socket for each new domain that I connect to.
import socket, ssl
context = ssl.SSLContext()
context.verify_mode = ssl.CERT_NONE
context.check_hostname = False
mycipher = "DHE-RSA-AES128-SHA"
context.set_ciphers(mycipher)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
domain = "google.com"
mySocket = context.wrap_socket(sock, server_hostname = domain)
mySocket.connect((domain, 443))
mySocket.close()
This is a difficult but interesting question. And you did use the appropriate term TLS which brings me joy ;-)
First, see that session resumption has changed in TLS1.3, so this may impact things in the future for you (at least the naming, pre TLS1.3 this feature speaks more about sessions and tickets while TLS1.3 prefers to speak about a pre shared key): https://timtaubert.de/blog/2017/02/the-future-of-session-resumption/
Now about your questions:
1) Do non-browser TLS clients such as the following python client perform session resumption by default?
Python documentation at https://docs.python.org/3.7/library/ssl.html does not say anything about "resumption"; I would posit if it says nothing about it it is not doing it at all. But it does speak up about "cache" which could be a synonym.
In fact you can note at https://docs.python.org/3.7/library/ssl.html#ssl.SSLSession that the session has the following attributes: ticket_lifetime_hint and has_ticket; this should be related to resumption.
Now look at https://docs.python.org/3.7/library/ssl.html#ssl.SSLContext.options and if you browse all possible values, you get:
ssl.OP_NO_TICKET
Prevent client side from requesting a session ticket.
So my assumption in 1) may be wrong and you have session resumption by default. You should disable it using the above option
(On the contrary) it seems to exist in PyOpenSSL (vs just stock ssl) because there is even one question here on how to disable this feature for that library: How to disable session resumption in pyOpenSSL?
Empirically you could try to discover that by running your application in such a way that it connects twice in close time to same endpoint and see what kind of TLS messages are exchanged. For example in TLS 1.3 (should be similar with other versions, but some name may change), a fresh handshake starts with ClientHello/ServerHello where a resumption starts with the same messages but with a pre_shared_key extension. Its presence will show TLS resumption. In TLS 1.2 the client would send a SessionTicketextension during resumption handshake.
2) How can I know if they do or do not? How can I disable session resumption if it is performed in the background?
If I am right about the above, make sure to use ssl.OP_NO_TICKET in ths SSL context object. Otherwise it is OP_ALL by default which is a bag of various options designed to maximize interoperability but the content may change depending on your Python version and the underlying OpenSSL library used.
If your session has the has_ticket attribute filled I guess it uses TLS resumption or is set up to use it.
1) Do non-browser TLS clients such as the following python client perform session resumption by default?
By default the python tls/ssl libraries do not perform session resumption.
2) How can I know if they do or do not? How can I disable session resumption if it is performed in the background?
You can verify that session resumption has not happened with the following code:
import socket
import ssl
hostname = 'www.stackoverflow.com'
context = ssl.create_default_context()
# Create a new socket, then create a secure socket
sock = socket.create_connection((hostname, 443))
ssock = context.wrap_socket(sock, server_hostname=hostname)
# Was the TLS Session re-used?
print(ssock.session_reused) # False
TLS session resumption can be enabled by using last ssl session.
hostname = 'google.com'
port = 443
resource = '/'
context = ssl.SSLContext(ssl.PROTOCOL_TLS)
sock = socket.create_connection((hostname, port))
ssock = context.wrap_socket(sock, server_hostname=hostname)
#send - receive
ssr = ssock.session
print(ssock.session_reused) # False
ssock.close()
sock = socket.create_connection((hostname, port))
ssock = context.wrap_socket(sock, server_hostname=hostname, session=ssr)
#send - receive
print(ssock.session_reused) # True , if server support it
ssock.close()

Decryption of openssl packets with static certificates

I am working on an ethical hacking project to monitor all the encrypted packets through OpenSSL.
I do have both the public and private keys (cert files). My application code snippet for regular packet decryption is as follows:
SSL_library_init();
ctx = InitCTX();
server = OpenConnection(hostname, atoi(portnum));
ssl = SSL_new(ctx); /* create new SSL connection state */
SSL_set_fd(ssl, server); /* attach the socket descriptor */
ShowCerts(ssl); /* get any certs */
SSL_write(ssl,acClientRequest, strlen(acClientRequest)); /* encrypt & send message */
bytes = SSL_read(ssl, buf, sizeof(buf)); /* get reply & decrypt */
SSL_free(ssl); /* release connection state */
SSL_read basically gets the certificate at the time of handshaking and utilizes it for decrypting the data. Is there any way to provide the same certificate offline for decryption of data.
Any help/pointers would be highly appreciable.
Generally TLS is gravitating to ephemeral key exchange, DHE or ECDHE. With ephemeral key exchange the session key (pre-master secret and master secret) are calculated using key agreement with temporary Diffie Hellman keys rather than the RSA or ECDSA key pair that is part of the certificate. So often you cannot do this.
You can however explicitly select one of the older RSA_ ciphersuites. In this case the pre-master secret is encrypted on the client side using the server's public key. The private key of the server can then decrypt this pre-master secret, calculate the session keys using the PRF (HMAC based key derivation) and then verify / decrypt all the packets.
It should be possible to do this using Wireshark, yes.
Note that TLS 1.3 will not support the RSA_ ciphersuites anymore. You would have to capture a public key of the client and private key of the server, the public key of the server and a private key of the client, or indeed the session keys directly to decrypt the traffic. Actually, that was one of the common complaints for TLS 1.3; that decrypting the traffic afterwards is not possible. That's however by design; the NSA cannot do this either.

boost asio with ECDSA certificate issue

I am implementing SSL server using boost::asio.
The context initialization is shown in below code
boost::asio::ssl::context_base::method SSL_version =
static_cast<boost::asio::ssl::context_base::method>(param_values[ID_PROTOCOL_VERSION].int32_value);
// load certificate files
boost::shared_ptr<boost::asio::ssl::context> context_ = boost::shared_ptr<boost::asio::ssl::context>(
new boost::asio::ssl::context(SSL_version));
p_ctx = boost::static_pointer_cast<void>(context_);
context_->set_options(boost::asio::ssl::context::default_workarounds);
context_->use_certificate_chain_file(cert_chain_file);
context_->use_certificate_file(cert_file, boost::asio::ssl::context::pem);
context_->use_private_key_file(cert_file, boost::asio::ssl::context::pem);
context_->set_verify_mode(boost::asio::ssl::verify_peer | boost::asio::ssl::verify_fail_if_no_peer_cert);
context_->set_verify_callback(boost::bind(&verify_certificate_cb, _1, _2));
if (param_values[ID_CIPHER_LIST].int32_value != 0)
{
std::string cipher_list = "";
generate_cipher_list(param_values[ID_CIPHER_LIST].int32_value, cipher_list);
MA5G_logger::log(PRIORITY_INFO, "Supported cipher list %s", cipher_list.c_str());
SSL_CTX_set_cipher_list((reinterpret_cast<boost::asio::ssl::context*>(p_ctx.get()))->native_handle(),
cipher_list.c_str());
}
in the cipher_list, I am supporting below list
AES128-SHA:AES256-SHA:AES128-SHA256:AES256-SHA256:AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA
With ECDSA certificates if I use cipher_list given above then client can not connect to the server and gives error "No shared cipher". But if I do not give cipher_list then the client can successfully connect to the server. The same cipher list works fine with RSA certificates.
The same ECDSA certificates work fine if I use openssl s_server with -cipher option to provide supported cipher_list
Can anyone help with this issue?
No sorry buddy I found the answer after lot of research.
The problem is with the cipher list and not with the code / certificate.
The same certificate uses ECDHE-ECDSA-AES256-SHA cipher with openssl client-server while used ECDH-ECDSA-AES256-SHA cipher for boost asio SSL client-server.
Anyways thanks #rkyser for your help!
I found this buried in the FAQ of the openssl-1.0.1 source code:
Why can't I make an SSL connection to a server using a DSA certificate?
Typically you'll see a message saying there are no shared ciphers when
the same setup works fine with an RSA certificate. There are two
possible causes. The client may not support connections to DSA servers
most web browsers (including Netscape and MSIE) only support
connections to servers supporting RSA cipher suites. The other cause
is that a set of DH parameters has not been supplied to the server. DH
parameters can be created with the dhparam(1) command and loaded using
the SSL_CTX_set_tmp_dh() for example: check the source to s_server in
apps/s_server.c for an example.
So based on this, make sure you are setting your DH parameters using SSL_CTX_set_tmp_dh().

Resources