This question explains how to read the TLS section of a packet using scapy.
However, my program is not able to read it. All it returns is a bunch of hexadecimal characters
>>> from scapy.all import *
>>> load_layer('tls')
>>> cap = rdpcap('tls.pcap')
>>> p1=cap[0]
>>> p1
<Ether dst=14:cc:20:51:33:ea src=f4:f2:6d:93:51:f1 type=0x800 |<IP version=4 ihl=5 tos=0x0 len=146 id=62970 flags=DF frag=0 ttl=64 proto=tcp chksum=0x50a0 src=192.168.1.143 dst=54.254.250.149 |<TCP sport=49335 dport=50443 seq=549695462 ack=200962336 dataofs=5 reserved=0 flags=PA window=4380 chksum=0xb0ac urgptr=0 |<Raw load="\x17\x03\x01\x00 \xf2\x10\xfd\x95N'\xf2\xaf\x99tp\x93\xbc\xe9\x81w\x91\x1b\xe0\xc9M:\x9a!]\xb0!\xae\xd2\x86\xb0>\x17\x03\x01\x00#d>\x0b\xee\xf0\xab\xded\x02E)\x0e0\xbb\xe6\x82uU\xb22\x87\xd6\xe4n[\x1d\x18\xe8\xd6\x1c\x00N_C\xe6\xdd\xbe\x89#6p\xd9\xaf\x19\xb3s\x07H\xdeF\x88\xdar\x0f\x8a\n!4\xeb\xd3F\xefgH" |>>>>
I want to get the tls record version, tls record length and the tls record content type.
This is screenshot of the packet opened in wireshark.
Can somepne please show me what I am doing wrong and how to read the tls content properly?
I am using Python3.6, and thus am not able to use stable scapy-ssl_tls, which is currently limited to Python 2.
You are so close. You just need to use TLS(pkt.load).
Download a TLS Capture
For this example, use this tls capture from Wireshark's Bugzilla.
We can see that packet 4 is the TLS Client Hello:
tshark -r DNS-over-TLS.pcapng -Y "frame.number==4"
4 0.122267 133.93.28.45 → li280-151.members.linode.com TLSv1 384 Client
Hello 00:00:5e:00:01:18 ← 48:d7:05:df:86:0b
Load with Scapy
Make sure that you have the cryptography library installed, as it's required for loading TLS captures.
>>> import cryptography
>>> # No errors
Reproducing what you have so far with this capture:
>>> from scapy.all import *
>>> load_layer('tls')
>>> cap = rdpcap('DNS-over-TLS.pcapng')
>>> tls_client_hello=cap[3] # Wireshark numbers packets starting at 1, scapy at 0
>>> tls_client_hello
<Ether dst=14:cc:20:51:33:ea src=f4:f2:6d:93:51:f1 type=0x800 |<IP version=4
ihl=5 tos=0x0 len=146 id=62970 flags=DF frag=0 ttl=64 proto=tcp chksum=0x50a0
src=192.168.1.143 dst=54.254.250.149 |<TCP sport=49335 dport=50443 seq=549695462
ack=200962336 dataofs=5 reserved=0 flags=PA window=4380 chksum=0xb0ac urgptr=0 |
<Raw load="\x17\x03\x01\x00
\xf2\x10\xfd\x95N'\xf2\xaf\x99tp\x93\xbc\xe9\x81w\x91\x1b\xe0\xc9M:\x9a!]\xb0!\xa
e\xd2\x86\xb0>\x17\x03\x01\x00#d>\x0b\xee\xf0\xab\xded\x02E)\x0e0\xbb\xe6\x82uU\x
b22\x87\xd6\xe4n[\x1d\x18\xe8\xd6\x1c\x00N_C\xe6\xdd\xbe\x89#6p\xd9\xaf\x19\xb3s\
x07H\xdeF\x88\xdar\x0f\x8a\n!4\xeb\xd3F\xefgH" |>>>>
Note that the part that we want to view is called Raw load. To access this part of the packet, you use tls_client_hello.load. Keep in mind that TLS will take a bytes object that contains the data, but not an entire packet.
>>> TLS(tls_client_hello.load)
<TLS type=handshake version=TLS 1.0 len=313 iv=b'' msg=[<TLSClientHello
msgtype=client_hello msglen=309 version=TLS 1.2 gmt_unix_time=Tue, 18 May 2077
23:20:52 +0000 (3388605652)
random_bytes=d6d533aca04dca42db8b123b0a143dcd580079147122e4de095c15cf sidlen=0
sid='' cipherslen=182 ciphers=[TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
< TLS output truncated ... >
Further Reading
I highly recommend looking at Scapy TLS Notebooks that do a good job of documenting scapy+TLS usage.
Related
I am trying to have regular expression only print the result one time. Is there any suggestions? Since I want the code to read the entire text file, but there are many dates that are the same, but I just want the code to return that date one time only.
code:
import re
filename = set(open('wireshark.txt', 'r'))
pattern_object = re.compile(r'(\d\d\d\d-\d\d-\d\d)')
for line in filename:
match_object = pattern_object.search(line)
if match_object:
regex = match_object.group(1)
print(regex)
Text file:
No. Time Source Destination Protocol Length Info
2 2021-02-12 13:33:12.206424 192.168.1.151 172.217.10.46 QUIC 1392 Initial, DCID=e4267bae554f387d, PKN: 1, CRYPTO, PADDING
Frame 2: 1392 bytes on wire (11136 bits), 1392 bytes captured (11136 bits) on interface \Device\NPF_{28AA034F-AC94-4D4A-9CA9-9AEA5D0EF2C1}, id 0
Ethernet II, Src: Micro-St_0e:cd:34 (00:d8:61:0e:cd:34), Dst: Verizon_fb:8b:82 (20:c0:47:fb:8b:82)
Internet Protocol Version 4, Src: 192.168.1.151, Dst: 172.217.10.46
User Datagram Protocol, Src Port: 57189, Dst Port: 443
QUIC IETF
No. Time Source Destination Protocol Length Info
3 2021-02-12 13:33:12.225610 172.217.10.46 192.168.1.151 QUIC 1392 Initial, SCID=e4267bae554f387d, PKN: 1, ACK, CRYPTO, PADDING
Frame 3: 1392 bytes on wire (11136 bits), 1392 bytes captured (11136 bits) on interface \Device\NPF_{28AA034F-AC94-4D4A-9CA9-9AEA5D0EF2C1}, id 0
Ethernet II, Src: Verizon_fb:8b:82 (20:c0:47:fb:8b:82), Dst: Micro-St_0e:cd:34 (00:d8:61:0e:cd:34)
Internet Protocol Version 4, Src: 172.217.10.46, Dst: 192.168.1.151
User Datagram Protocol, Src Port: 443, Dst Port: 57189
QUIC IETF
No. Time Source Destination Protocol Length Info
4 2021-02-12 13:33:12.225989 192.168.1.151 172.217.10.46 TLSv1.2 146 Application Data
No. Time Source Destination Protocol Length Info
4 2021-04-12 13:33:12.225989 192.168.1.151 172.217.10.46 TLSv1.2 146 Application Data
No. Time Source Destination Protocol Length Info
4 2021-06-12 13:33:12.225989 192.168.1.151 172.217.10.46 TLSv1.2 146 Application Data
No. Time Source Destination Protocol Length Info
4 2021-06-12 13:33:12.225989 192.168.1.151 172.217.10.46 TLSv1.2 146 Application Data
Code execute output:
2021-02-12
2021-02-12
2021-02-12
2021-02-12
2021-02-12
2021-02-12
2021-04-12
2021-06-12
2021-06-12
desire code execute output:
2021-02-12
2021-04-12
2021-06-12
Here's a minimal example for how to get all the unique dates in the file.
Essentially, it's a 4 stage process:
Store the pattern to search for as a string
Open the file and get all the text
Use re.findall() to get all of the text matching the pattern
Use set() to keep only the unique matches
import re
# Make the pattern
pattern = '(\d\d\d\d-\d\d-\d\d)'
# Open the file and read all the text into a variable
with open('wireshark.txt') as file:
text = file.read()
# Search the text for anything matching the pattern
matches = re.findall(pattern, text)
# Print the unique matches
print(set(matches))
The key thing here is the combination of re.findall() (search for multiple matches at once) and set() (to get rid of duplicates.
The below csv is only a snippet of my main data file.
customer.csv
customer_id,order_id,number_of_items
10,4736,9
5,3049,1
1,4689,3
6,4114,9
1,4524,15
2,3727,16
3,3507,7
7,3988,3
5,4993,16
6,1945,4
7,3081,7
3,3707,2
5,1739,12
9,4167,17
7,3242,12
2,3109,10
10,2197,20
10,3528,13
8,4917,2
5,1713,19
8,4224,4
7,2160,2
10,2044,19
10,2956,8
3,3906,2
5,2288,16
7,1854,20
7,4404,2
9,1622,2
7,3685,2
10,2755,10
3,3390,10
6,1424,6
3,2127,15
4,1221,15
9,2994,14
1,1413,13
7,2771,7
3,4579,13
10,2208,4
CURRENTLY ALL I HAVE
import os
os.path.getsize("customer.csv") # outputs, 424 bytes
HOW I THINK I NEED TO PROCEED
I think I need to do something with open csv and read bytes? Then look at each row bit wise?
Please note, I am not looking specifically for someone to just give me an answer on how to do this (although that would be appreciated). Therefore, if someone could just point me in the right direction or give me some topics to look into that would be great. Side note, I know I am supposed to use encoding and decoding somewhere for this task.
This script will use the csv to load the data from customer.csv and compute the average using the builtin statistics module:
import csv
from statistics import mean
with open('customer.csv', newline='') as csvfile:
data = csv.DictReader(csvfile)
# group the customers by customer_id
customers = {}
for order in data:
customers.setdefault(order['customer_id'], []).append(int(order['number_of_items']))
# print the `average`:
print('{:<15} {}'.format('customer_id', 'average'))
for k, v in customers.items():
print('{:<15} {:.2f}'.format(k, mean(v)))
Prints:
customer_id average
10 11.86
5 12.80
1 10.33
6 6.33
2 13.00
3 8.17
7 6.88
9 11.00
8 3.00
4 15.00
Aim: Get the arrival time from the pcap files
Language: python3.7
Tools: Scapy.sniff
Above all ,i want get the arrival time data,in the .pcap ,when i use wireshark ,i saw the data in the Ethernet Frame,but when i use
#Scapy.sniff(offline='.pcap') ,i just get the Ether,TCP,IP and others ,so how can i get that data?
Thanx alot!
>>from scapy.all import *
>>a = sniff(offline = '***.pcap')
>>a[0]
[out]:
<Ether dst=*:*:*:*:*:* src=*:*:*:*:*:* type=** |<IP version=4 ihl=5 tos=0x20 len=52 id=14144 flags=DF frag=0 ttl=109 proto=tcp chksum=0x5e3b src=*.*.*.* dst=*.*.*.* |<TCP sport=gcsp dport=http seq=1619409885 ack=1905830025 dataofs=8 reserved=0 flags=A window=65535 chksum=0xfdb5 urgptr=0 options=[('NOP', None), ('NOP', None), ('SAck', (1905831477, 1905831485))] |>>>
[ ]:
The packet time from the pcap is available in the time member:
print(a[0].time)
It's kept as a floating point value (the standard python "timestamp" format). To get it in a form more easily understandable, you may want to use the datetime module:
>>> from datetime import datetime
>>> dt = datetime.fromtimestamp(a[0].time)
>>> print(dt)
2018-11-12 03:03:00.259780
The scapy documentation isn't great. It can be very instructive to use the interactive help facility. For example, in the interpreter:
$ python
>>> from scapy.all import *
>>> a = sniff(offline='mypcap.pcap')
>>> help(a[0])
This will show you all the methods and attributes of the object represented by a[0]. In your case, that is an instance of class Ether(scapy.packet.Packet).
I have an micropython client on esp32 board, and Python on linux server. I am trying send 5.5kb text file from Python Server to MicroPython client. It sends successfully but MicroPython client does not receive all data. Codes as follows;
Python Server:
with open('downloads/%s' % (request_path), 'rb') as f:
data = f.read()
self.wfile.write(data) #data is 5.5kb
MicroPython Client
recvData = sock.read(4096).decode('utf-8').split("\r\n")
print("Response_Received:: %s" % recvData)
sock.close()
Response_Received:: ['HTTP/1.0 200 OK', 'Server: SimpleHTTP/0.6 Python/3.5.3', 'Date: Sat, 09 Jun 2018 09:29:41 GMT', '', '# Ity: asdasd\n# ksduygfkhsgdkjfksjdhfg\n kjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy98\n 47y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhs\n gdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349rio\n t34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r3\n 49riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkv\n nvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogijiksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweu\n oiruy9847y397r349riot34jt;o\n giji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbs\n djkvbjcxbvhweioufhoiweuoiruy9847y397\n r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufh\n oiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduyg\n fkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhwei\n oufhoiweuoiruy9847y397r349riot\n 34jt;ogiji4vuijo4vjlkvnvl;kksduyg\n fkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhw\n eioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjd\n hfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiwe\n uoiruy9847y397r349riot34jt;ogiji4vuij\n o4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhwe\n ioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfk\n hsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;o\n giji4vuijo4vjlkvnvl;kksduygfkhsgdk\n jfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvh\n weioufhoiweuoiruy9847y397r349riot34jt;ogiji\n 4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiru\n y9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;k4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduyg\n fkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcx\n bvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhs\n gdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdj\n nvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogijiksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9bfkjcbsdjkvbjcxbvhweioufhoi847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweu\nnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogijiksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweu\nnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogijiksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufhoiweuoiruy9847y397r349riot34jt;ogiji4vuijo4vjlkvnvl;kksduygfkhsgdkjfksjdhfgkjdhsbfkjdhsbfkjcbsdjkvbjcxbvhweioufnvl;k']
Client receives only 4140 bytes of the array data in due to buffer size(4096), 4th element of the recvData is lost. MicroPython does not accept over this Buffer size. How can i receive all my data (5.5kb) in 4th element of recvData array without any loss?
I have tried to fragment the received data, but it was not successful.
while True:
chunck = s.recv(4096)
if not chunck:
break
fragments.append(chunck)
Since your goal is to write the file to the filesystem, the simplest solution is to stop trying to hold the entire file in memory. Instead of building up your fragments array, just write the received chunks to a file:
with open('datafile', 'w') as fd:
while True:
chunk = s.recv(4096)
if not chunk:
break
fd.write(chunk)
This requires a constant amount of memory and can be used to receive
files of arbitrary size.
If I generate an Ethernet frame without any upper layers payload and send it at layer two with sendp(), then I receive the "Mac address to reach destination not found. Using broadcast." warning and frame put to wire indeed uses ff:ff:ff:ff:ff:ff as a destination MAC address. Why is this so? Shouldn't the Scapy send exactly the frame I constructed?
My crafted package can be seen below:
>>> ls(x)
dst : DestMACField = '01:00:0c:cc:cc:cc' (None)
src : SourceMACField = '00:11:22:33:44:55' (None)
type : XShortEnumField = 0 (0)
>>> sendp(x, iface="eth0")
WARNING: Mac address to reach destination not found. Using broadcast.
.
Sent 1 packets.
>>>
Most people encountering this issue are incorrectly using send() (or sr(), sr1(), srloop()) instead of sendp() (or srp(), srp1(), srploop()). For the record, the "without-p" functions like send() are for sending layer 3 packets (send(IP())) while the "with-p" variants are for sending layer 2 packets (sendp(Ether() / IP())).
If you define x like I do below and use sendp() (and not send()) and you still have this issue, you should probably try with the latest version from the project's git repository (see https://github.com/secdev/scapy).
I've tried:
>>> x = Ether(src='01:00:0c:cc:cc:cc', dst='00:11:22:33:44:55')
>>> ls(x)
dst : DestMACField = '00:11:22:33:44:55' (None)
src : SourceMACField = '01:00:0c:cc:cc:cc' (None)
type : XShortEnumField = 0 (0)
>>> sendp(x, iface='eth0')
.
Sent 1 packets.
At the same time I was running tcpdump:
# tcpdump -eni eth0 ether host 00:11:22:33:44:55
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
12:33:47.774570 01:00:0c:cc:cc:cc > 00:11:22:33:44:55, 802.3, length 14: [|llc]