A Question with using scapy.sniff for get the 'Ethernet Frame' in pcap files - scapy

Aim: Get the arrival time from the pcap files
Language: python3.7
Tools: Scapy.sniff
Above all ,i want get the arrival time data,in the .pcap ,when i use wireshark ,i saw the data in the Ethernet Frame,but when i use
#Scapy.sniff(offline='.pcap') ,i just get the Ether,TCP,IP and others ,so how can i get that data?
Thanx alot!
>>from scapy.all import *
>>a = sniff(offline = '***.pcap')
>>a[0]
[out]:
<Ether dst=*:*:*:*:*:* src=*:*:*:*:*:* type=** |<IP version=4 ihl=5 tos=0x20 len=52 id=14144 flags=DF frag=0 ttl=109 proto=tcp chksum=0x5e3b src=*.*.*.* dst=*.*.*.* |<TCP sport=gcsp dport=http seq=1619409885 ack=1905830025 dataofs=8 reserved=0 flags=A window=65535 chksum=0xfdb5 urgptr=0 options=[('NOP', None), ('NOP', None), ('SAck', (1905831477, 1905831485))] |>>>
[ ]:

The packet time from the pcap is available in the time member:
print(a[0].time)
It's kept as a floating point value (the standard python "timestamp" format). To get it in a form more easily understandable, you may want to use the datetime module:
>>> from datetime import datetime
>>> dt = datetime.fromtimestamp(a[0].time)
>>> print(dt)
2018-11-12 03:03:00.259780
The scapy documentation isn't great. It can be very instructive to use the interactive help facility. For example, in the interpreter:
$ python
>>> from scapy.all import *
>>> a = sniff(offline='mypcap.pcap')
>>> help(a[0])
This will show you all the methods and attributes of the object represented by a[0]. In your case, that is an instance of class Ether(scapy.packet.Packet).

Related

SyntaxError when using print(""" with a lst of numbers to populate a file with GROMACS patched with PLUMED

I am using GROMACS with PLUMED to run MD simulations. In setting up my plumed file for collecting the S2/SH CV from Omar(https://www.plumed.org/doc-v2.8/user-doc/html/_s2_c_m.html) I am having difficulties with the line:
File "makingplumed.py", line 25
""" % (x,i)file=f)
^
SyntaxError: invalid syntax
Here is the code I am trying to run:
# here we create the PLUMED input file with python
with open("plumed.dat","w") as f:
# print initial stuff
#K# from __future__ import print_function
# Define Atoms which are Oxygen hydrogen bond acceptors
ATOMS=[21,35,45,62,76,97,109,133,152,174,188,202,213,227,239,253,269,280,292,311,323,339,353,377,401,416,426,447,466,477,488,503,518,538,560,575,597,617,624,641,655,677,692,702,722,743,765,784,798,820,844,866,883,897,919,939,961,978,988,1004,1021,1040]
#Define heavy atoms for S2CM CV (protein and backbone and not hydrogen)
heavy_atoms_nh: GROUP ATOMS=1,5,7,10,12,16,20,21,22,23,26,29,32,34,35,36,38,40,44,45,46,48,50,53,54,55,57,59,61,62,63,64,67,70,73,75,76,79,81,84,85,87,89,90,92,94,96,97,98,100,102,105,106,107,108,109,110,112,114,117,120,123,124,125,126,129,132,133,134,136,138,141,143,147,151,152,153,155,157,160,163,166,169,173,174,175,177,179,181,185,187,188,189,191,193,195,199,201,202,203,205,207,210,212,213,214,216,217,218,220,224,226,227,228,230,232,235,236,237,238,239,240,241,244,247,250,252,253,254,256,258,260,264,268,269,270,272,274,277,279,280,281,283,285,288,291,392,293,295,297,299,303,306,310,311,312,314,316,319,320,321,322,323,326,328,330,334,338,339,340,342,344,346,350,352,353,354,356,358,361,364,367,369,370,373,376,377,378,380,382,385,388,391,393,397,400,401,402,404,406,409,412,415,416,417,419,421,425,426,427,429,431,434,345,437,439,442,444,446,447,448,450,452,455,457,461,465,466,467,469,471,474,476,477,478,480,482,485,487,489,491,493,496,499,500,501,502,503,504,506,508,511,514,515,516,517,518,519,521,523,526,527,529,531,533,535,537,538,539,541,543,546,549,552,555,559,560,561,563,265,268,571,572,573,574,575,576,578,580,583,586,589,592,596,597,598,600,602,605,606,608,610,612,614,616,617,618,620,623,624,625,627,629,632,635,636,640,641,642,644,646,648,652,654,655,656,658,660,663,666,669,672,676,677,678,680,682,685,688,691,692,693,695,697,701,702,703,705,707,710,711,713,715,717,719,721,722,723,725,727,730,731,733,735,738,740,742,743,744,746,748,751,754,757,760,764,765,766,768,770,773,775,779,783,784,785,786,789,792,795,797,798,799,801,803,806,809,812,815,819,820,821,823,825,828,829,831,833,834,836,838,840,842,843,844,845,847,849,852,855,858,861,865,866,867,869,871,874,877,878,879,882,883,884,886,888,891,892,893,896,867,898,900,902,905,908,911,914,918,919,920,922,924,927,928,930,932,934,936,938,939,940,942,944,947,950,953,956,960,961,962,964,966,969,975,973,977,978,979,981,983,987,988,989,991,993,995,999,1003,1004,1005,1007,1009,1012,1015,1016,1017,1020,1021,1022,1024,1026,1029,1031,1035,1039,1040,1041,1043,1045,1048,1049,1051,1053,1055,1057,1059,1060,1061
for x in range(len(ATOMS)):
for i in range(1, 60):
print("""
S2CM ...
NH_ATOMS=x,x+2
HEAVY_ATOMS=heavy_atoms_nh
LABEL=S2nh-%d
R_EFF=0.10
PREFACTOR_A=0.80
EXPONENT_B=1.0
OFFSET_C=0.10
N_I=1
NOPBC
... S2CM
""" % (x,i)file=f)
I am just learning python and Linux this summer as I am getting involved with computational biochemistry research, so if there is a simple fix I am very sorry for the waste of time, and I appreciate any and all time and attention to this matter.
python3 --version
Python 3.6.13
Thank You,
David Cummins
Masters Student at Western Washington University

Python error upon exif data extraction via Pillow module: invalid continuation byte

I am writing a piece of code to extract exif data from images using Python. I downloaded the Pillow module using pip3 and am using some code I found online:
from PIL import Image
from PIL.ExifTags import TAGS
imagename = "path to file"
image = Image.open(imagename)
exifdata = image.getexif()
for tagid in exifdata:
tagname = TAGS.get(tagid, tagid)
data = exifdata.get(tagid)
if isinstance(data, bytes):
data = data.decode()
print(f"{tagname:25}: {data}")
On some images this code works. However, for images I took on my Olympus camera I get the following error:
GPSInfo : 734
Traceback (most recent call last):
File "_pathname redacted_", line 14, in <module>
data = data.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 30: invalid continuation byte
When I remove the data = data.decode() part, I get the following:
GPSInfo : 734
PrintImageMatching : b"PrintIM\x000300\x00\x00%\x00\x01\x00\x14\x00\x14\x00\x02\x00\x01\x00\x00\x00\x03\x00\xf0\x00\x00\x00\x07\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x00\t\x00\x00\x00\x00\x00\n\x00\x00\x00\x00\x00\x0b\x008\x01\x00\x00\x0c\x00\x00\x00\x00\x00\r\x00\x00\x00\x00\x00\x0e\x00P\x01\x00\x00\x10\x00`\x01\x00\x00 \x00\xb4\x01\x00\x00\x00\x01\x03\x00\x00\x00\x01\x01\xff\x00\x00\x00\x02\x01\x83\x00\x00\x00\x03\x01\x83\x00\x00\x00\x04\x01\x83\x00\x00\x00\x05\x01\x83\x00\x00\x00\x06\x01\x83\x00\x00\x00\x07\x01\x80\x80\x80\x00\x10\x01\x83\x00\x00\x00\x00\x02\x00\x00\x00\x00\x07\x02\x00\x00\x00\x00\x08\x02\x00\x00\x00\x00\t\x02\x00\x00\x00\x00\n\x02\x00\x00\x00\x00\x0b\x02\xf8\x01\x00\x00\r\x02\x00\x00\x00\x00 \x02\xd6\x01\x00\x00\x00\x03\x03\x00\x00\x00\x01\x03\xff\x00\x00\x00\x02\x03\x83\x00\x00\x00\x03\x03\x83\x00\x00\x00\x06\x03\x83\x00\x00\x00\x10\x03\x83\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\t\x11\x00\x00\x10'\x00\x00\x0b\x0f\x00\x00\x10'\x00\x00\x97\x05\x00\x00\x10'\x00\x00\xb0\x08\x00\x00\x10'\x00\x00\x01\x1c\x00\x00\x10'\x00\x00^\x02\x00\x00\x10'\x00\x00\x8b\x00\x00\x00\x10'\x00\x00\xcb\x03\x00\x00\x10'\x00\x00\xe5\x1b\x00\x00\x10'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x05\x05\x00\x00\x00##\x80\x80\xc0\xc0\xff\xff\x00\x00##\x80\x80\xc0\xc0\xff\xff\x00\x00##\x80\x80\xc0\xc0\xff\xff\x05\x05\x05\x00\x00\x00##\x80\x80\xc0\xc0\xff\xff\x00\x00##\x80\x80\xc0\xc0\xff\xff\x00\x00##\x80\x80\xc0\xc0\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
ResolutionUnit : 2
ExifOffset : 230
ImageDescription : OLYMPUS DIGITAL CAMERA
Make : OLYMPUS CORPORATION
Model : E-M10MarkII
Software : Version 1.2
Orientation : 1
DateTime : 2020:02:13 15:02:57
YCbCrPositioning : 2
YResolution : 350.0
Copyright :
XResolution : 350.0
Artist :
How should I fix this problem? Should I use a different Python module?
I did some digging and figured out the answer to the problem I posted about. I originally postulated that the rest of the metadata was in the byte data:
b"PrintIM\x000300\x00\x00%\x00\x01\x00\x14\x00\x14\x00\x02\x00\x01\x00\x00\x00\x03\x00\xf0\x00\x00\x00\x07\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x00\t\x00\x00\x00\x00\x00\n\x00\x00\x00\x00\x00\x0b\x008\x01\x00\x00\x0c\x00\x00\x00\x00\x00\r\x00\x00\x00\x00\x00\x0e\x00P\x01\x00\x00\x10\x00`\x01\x00\x00 \x00\xb4\x01\x00\x00\x00\x01\x03\x00\x00\x00\x01\x01\xff\x00\x00\x00\x02\x01\x83\x00\x00\x00\x03\x01\x83\x00\x00\x00\x04\x01\x83\x00\x00\x00\x05\x01\x83\x00\x00\x00\x06\x01\x83\x00\x00\x00\x07\x01\x80\x80\x80\x00\x10\x01\x83\x00\x00\x00\x00\x02\x00\x00\x00\x00\x07\x02\x00\x00\x00\x00\x08\x02\x00\x00\x00\x00\t\x02\x00\x00\x00\x00\n\x02\x00\x00\x00\x00\x0b\x02\xf8\x01\x00\x00\r\x02\x00\x00\x00\x00 \x02\xd6\x01\x00\x00\x00\x03\x03\x00\x00\x00\x01\x03\xff\x00\x00\x00\x02\x03\x83\x00\x00\x00\x03\x03\x83\x00\x00\x00\x06\x03\x83\x00\x00\x00\x10\x03\x83\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\t\x11\x00\x00\x10'\x00\x00\x0b\x0f\x00\x00\x10'\x00\x00\x97\x05\x00\x00\x10'\x00\x00\xb0\x08\x00\x00\x10'\x00\x00\x01\x1c\x00\x00\x10'\x00\x00^\x02\x00\x00\x10'\x00\x00\x8b\x00\x00\x00\x10'\x00\x00\xcb\x03\x00\x00\x10'\x00\x00\xe5\x1b\x00\x00\x10'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x05\x05\x05\x00\x00\x00##\x80\x80\xc0\xc0\xff\xff\x00\x00##\x80\x80\xc0\xc0\xff\xff\x00\x00##\x80\x80\xc0\xc0\xff\xff\x05\x05\x05\x00\x00\x00##\x80\x80\xc0\xc0\xff\xff\x00\x00##\x80\x80\xc0\xc0\xff\xff\x00\x00##\x80\x80\xc0\xc0\xff\xff\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
That assumption wasn't correct. Although the above is metadata, it simply isn't the metadata I am looking for (in my case the FocalLength attribute). Rather it appears to be Olympus specific metadata. The answer to my solution was to find all the metadata. I found a piece of code that worked very well in Stack Overflow: In Python, how do I read the exif data for an image?.
I used the following code by Nicolas Gervais:
import os,sys
from PIL import Image
from PIL.ExifTags import TAGS
for (k,v) in Image.open(sys.argv[1])._getexif().items():
print('%s = %s' % (TAGS.get(k), v))
I replaced sys.argv[1] with the path name to the image file.
Alternate Solution
As MattDMo mentioned, there are also specific libraries for reading EXIF data in Python. One that I found that look promising is ExifRead which can be download by typing the following in the terminal:
pip install ExifRead

Python 3: time.tzset() alternative for Windows?

I am new to Python. I am reading about dates and times from the lovely book 'Python 3 Standard Library by Example' by Doug Hellmann and I stumbled upon this code snippet:
import time
import os
def show_zone_info():
print(f'''\
TZ : {os.environ.get('TZ', '(not set)')}
tzname: {time.tzname}
Zone : {time.timezone} ({time.timezone / 3600})
DST : {time.daylight}
Time : {time.ctime()}
''')
if __name__ == '__main__':
print('Default: ')
show_zone_info()
ZONES = [
'GMT',
'Europe/Amsterdam'
]
for zone in ZONES:
os.environ['TZ'] = zone
# time.tzset() # Only available on Unix
print(zone, ':')
show_zone_info()
Problem is, time.tzset() is only available on Unix and without it on Windows machine, timezone doesn't change during the run time of the code. What is the alternative to time.tzset() on Windows? (I am running Python 3.8.3 on Windows 10 at the time of asking this question.)

How can I read and process 100 bytes at a time from a large CSV file?

The below csv is only a snippet of my main data file.
customer.csv
customer_id,order_id,number_of_items
10,4736,9
5,3049,1
1,4689,3
6,4114,9
1,4524,15
2,3727,16
3,3507,7
7,3988,3
5,4993,16
6,1945,4
7,3081,7
3,3707,2
5,1739,12
9,4167,17
7,3242,12
2,3109,10
10,2197,20
10,3528,13
8,4917,2
5,1713,19
8,4224,4
7,2160,2
10,2044,19
10,2956,8
3,3906,2
5,2288,16
7,1854,20
7,4404,2
9,1622,2
7,3685,2
10,2755,10
3,3390,10
6,1424,6
3,2127,15
4,1221,15
9,2994,14
1,1413,13
7,2771,7
3,4579,13
10,2208,4
CURRENTLY ALL I HAVE
import os
os.path.getsize("customer.csv") # outputs, 424 bytes
HOW I THINK I NEED TO PROCEED
I think I need to do something with open csv and read bytes? Then look at each row bit wise?
Please note, I am not looking specifically for someone to just give me an answer on how to do this (although that would be appreciated). Therefore, if someone could just point me in the right direction or give me some topics to look into that would be great. Side note, I know I am supposed to use encoding and decoding somewhere for this task.
This script will use the csv to load the data from customer.csv and compute the average using the builtin statistics module:
import csv
from statistics import mean
with open('customer.csv', newline='') as csvfile:
data = csv.DictReader(csvfile)
# group the customers by customer_id
customers = {}
for order in data:
customers.setdefault(order['customer_id'], []).append(int(order['number_of_items']))
# print the `average`:
print('{:<15} {}'.format('customer_id', 'average'))
for k, v in customers.items():
print('{:<15} {:.2f}'.format(k, mean(v)))
Prints:
customer_id average
10 11.86
5 12.80
1 10.33
6 6.33
2 13.00
3 8.17
7 6.88
9 11.00
8 3.00
4 15.00

Cannot read tls section even after calling load_layer('tls') in scapy

This question explains how to read the TLS section of a packet using scapy.
However, my program is not able to read it. All it returns is a bunch of hexadecimal characters
>>> from scapy.all import *
>>> load_layer('tls')
>>> cap = rdpcap('tls.pcap')
>>> p1=cap[0]
>>> p1
<Ether dst=14:cc:20:51:33:ea src=f4:f2:6d:93:51:f1 type=0x800 |<IP version=4 ihl=5 tos=0x0 len=146 id=62970 flags=DF frag=0 ttl=64 proto=tcp chksum=0x50a0 src=192.168.1.143 dst=54.254.250.149 |<TCP sport=49335 dport=50443 seq=549695462 ack=200962336 dataofs=5 reserved=0 flags=PA window=4380 chksum=0xb0ac urgptr=0 |<Raw load="\x17\x03\x01\x00 \xf2\x10\xfd\x95N'\xf2\xaf\x99tp\x93\xbc\xe9\x81w\x91\x1b\xe0\xc9M:\x9a!]\xb0!\xae\xd2\x86\xb0>\x17\x03\x01\x00#d>\x0b\xee\xf0\xab\xded\x02E)\x0e0\xbb\xe6\x82uU\xb22\x87\xd6\xe4n[\x1d\x18\xe8\xd6\x1c\x00N_C\xe6\xdd\xbe\x89#6p\xd9\xaf\x19\xb3s\x07H\xdeF\x88\xdar\x0f\x8a\n!4\xeb\xd3F\xefgH" |>>>>
I want to get the tls record version, tls record length and the tls record content type.
This is screenshot of the packet opened in wireshark.
Can somepne please show me what I am doing wrong and how to read the tls content properly?
I am using Python3.6, and thus am not able to use stable scapy-ssl_tls, which is currently limited to Python 2.
You are so close. You just need to use TLS(pkt.load).
Download a TLS Capture
For this example, use this tls capture from Wireshark's Bugzilla.
We can see that packet 4 is the TLS Client Hello:
tshark -r DNS-over-TLS.pcapng -Y "frame.number==4"
4 0.122267 133.93.28.45 → li280-151.members.linode.com TLSv1 384 Client
Hello 00:00:5e:00:01:18 ← 48:d7:05:df:86:0b
Load with Scapy
Make sure that you have the cryptography library installed, as it's required for loading TLS captures.
>>> import cryptography
>>> # No errors
Reproducing what you have so far with this capture:
>>> from scapy.all import *
>>> load_layer('tls')
>>> cap = rdpcap('DNS-over-TLS.pcapng')
>>> tls_client_hello=cap[3] # Wireshark numbers packets starting at 1, scapy at 0
>>> tls_client_hello
<Ether dst=14:cc:20:51:33:ea src=f4:f2:6d:93:51:f1 type=0x800 |<IP version=4
ihl=5 tos=0x0 len=146 id=62970 flags=DF frag=0 ttl=64 proto=tcp chksum=0x50a0
src=192.168.1.143 dst=54.254.250.149 |<TCP sport=49335 dport=50443 seq=549695462
ack=200962336 dataofs=5 reserved=0 flags=PA window=4380 chksum=0xb0ac urgptr=0 |
<Raw load="\x17\x03\x01\x00
\xf2\x10\xfd\x95N'\xf2\xaf\x99tp\x93\xbc\xe9\x81w\x91\x1b\xe0\xc9M:\x9a!]\xb0!\xa
e\xd2\x86\xb0>\x17\x03\x01\x00#d>\x0b\xee\xf0\xab\xded\x02E)\x0e0\xbb\xe6\x82uU\x
b22\x87\xd6\xe4n[\x1d\x18\xe8\xd6\x1c\x00N_C\xe6\xdd\xbe\x89#6p\xd9\xaf\x19\xb3s\
x07H\xdeF\x88\xdar\x0f\x8a\n!4\xeb\xd3F\xefgH" |>>>>
Note that the part that we want to view is called Raw load. To access this part of the packet, you use tls_client_hello.load. Keep in mind that TLS will take a bytes object that contains the data, but not an entire packet.
>>> TLS(tls_client_hello.load)
<TLS type=handshake version=TLS 1.0 len=313 iv=b'' msg=[<TLSClientHello
msgtype=client_hello msglen=309 version=TLS 1.2 gmt_unix_time=Tue, 18 May 2077
23:20:52 +0000 (3388605652)
random_bytes=d6d533aca04dca42db8b123b0a143dcd580079147122e4de095c15cf sidlen=0
sid='' cipherslen=182 ciphers=[TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
< TLS output truncated ... >
Further Reading
I highly recommend looking at Scapy TLS Notebooks that do a good job of documenting scapy+TLS usage.

Resources