I am using scapy 2.4.5 with Python 3.9.5 on Windows 11. I have npcap version 1.55 install.
I have some Wireshark packet captures that I am trying to use scapy's sniff function on the file and filter out various packets.
However, when I use filter="udp" with sniff I get an exception about tcpdump not being available.
Below is the script I am currently trying to use.
from scapy.all import *
conf.use_pcap = True
pcap_file_path = r"C:\8OCT21_DDL_00001_20211008214804"
packets = sniff(offline=pcap_file_path,
count=10,
filter="udp")
packets.summary()
However I get this exception:
File "C:\Python39\lib\site-packages\scapy\sendrecv.py", line 1263, in sniff
sniffer._run(*args, **kwargs)
File "C:\Python39\lib\site-packages\scapy\sendrecv.py", line 1072, in _run
sniff_sockets.update((PcapReader(
File "C:\Python39\lib\site-packages\scapy\sendrecv.py", line 1074, in <genexpr>
tcpdump(fname,
File "C:\Python39\lib\site-packages\scapy\utils.py", line 2095, in tcpdump
raise Scapy_Exception(
scapy.error.Scapy_Exception: tcpdump is not available
Any idea on how to use scapy sniff on Windows with npcap instead of tcpdump?
The problem is not your filter, rather it's the "offline" option in sniff function. You can perform live sniff of packets, or use rdpcap() function to first load pcap in RAM, and then do what you want to do.
from scapy.all import rdpcap
from scapy.layers.inet import UDP
scapy_cap = rdpcap("responses.pcap")
for pck in PCAP:
if pck[UDP]:
print(pck.summary())
This is not a solution to why it throws tcpdump exception, but more of a workaround to get something out of your pcap.
Keep in mind that large pcap files will eat RAM like nothing else if you try to load them with rdpcap().
I am attempting to change this line to become acceptable by python3 from a python2 set of source:
Here is the error:
TypeError: unicode strings are not supported, please encode to bytes:
'$PMTK251,9600*17\r\n'
Can anyone tell my why this is this way or how I can change it to suit Python3 methods?
It is a GPS set of source in Python2 that still works but I see that all ideas relating to Python2 will be gone from availability and/or is already pretty much done and gone.
So, my ideas were to update that line and others.
In python3, I receive errors relating to bytes and I have currently read about the idea of (arg, newline='') in source when attempting to make .csv files in Python3.
I am still at a loss w/ how to incorporate Python3 in this specific line.
I can offer more about the line or the rest of the source if necessary. I received this source from toptechboy.com. I do not think that fellow ever updated the source to work w/ Python3.
class GPS:
def __init__(self):
#This sets up variables for useful commands.
#This set is used to set the rate the GPS reports
UPDATE_10_sec = "$PMTK220,10000*2F\r\n" #Update Every 10 Seconds
UPDATE_5_sec = "$PMTK220,5000*1B\r\n" #Update Every 5 Seconds
UPDATE_1_sec = "$PMTK220,1000*1F\r\n" #Update Every One Second
UPDATE_200_msec = "$PMTK220,200*2C\r\n" #Update Every 200 Milliseconds
#This set is used to set the rate the GPS takes measurements
MEAS_10_sec = "$PMTK300,10000,0,0,0,0*2C\r\n" #Measure every 10 seconds
MEAS_5_sec = "$PMTK300,5000,0,0,0,0*18\r\n" #Measure every 5 seconds
MEAS_1_sec = "$PMTK300,1000,0,0,0,0*1C\r\n" #Measure once a second
MEAS_200_msec= "$PMTK300,200,0,0,0,0*2F\r\n" #Meaure 5 times a second
#Set the Baud Rate of GPS
BAUD_57600 = "$PMTK251,57600*2C\r\n" #Set Baud Rate at 57600
BAUD_9600 ="$PMTK251,9600*17\r\n" #Set 9600 Baud Rate
#Commands for which NMEA Sentences are sent
ser.write(BAUD_57600)
sleep(1)
ser.baudrate = 57600
GPRMC_ONLY = "$PMTK314,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0*29\r\n" #Send only the GPRMC Sentence
GPRMC_GPGGA = "$PMTK314,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0*28\r\n"#Send GPRMC AND GPGGA Sentences
SEND_ALL = "$PMTK314,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0*28\r\n" #Send All Sentences
SEND_NOTHING = "$PMTK314,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0*28\r\n" #Send Nothing
...
That is the GPS Class Mr. McWhorter wrote for a GPS Module in python2. I am trying to configure this python2 source into a workable python3 class.
I am receiving errors like "needs to be bytes" and/or "cannot use bytes here".
Anyway, if you are handy w/ Python3 and know where I am making mistakes on this source to transfer it over to Python3, please let me know. I have tried changing the source many times to accept bytes and to be read as a utf-string.
Here: Best way to convert string to bytes in Python 3? <<< This seems like the most popular topic on this subject but it does not answer my question so far (I think).
This line simply works when adding a b for bytes in front of the string...like so.
(b'$PMTK251,9600*17\r\n')
That should rid you of that error of TypeError: unicode strings are not supported, please encode to bytes:
I am developing a Python app which iterates through an Elasticsearch instance and stores all the documents into an HTML file. I am using Apache 2 as the server and I get always the same unicode error: UnicodeEncodeError: 'ascii' codec can't encode character '\xed'.
I've tried encoding the string with encode('utf-8') and it seems to avoid the error, but I get the output in the wrong format:
'b'Assidue As\xc3\xadduamente.
This is the Python code:
for row in resp["hits"]["hits"]:
post = row['_source']['text']
fichero_html.write(post)
The elasticsearch documents store words in spanish, so the graphic accents need to be correctly displayed.
Thanks in advance for your help.
EDIT:
This code works fine on the Python3 console, but it does not work on the server side.
for row in resp["hits"]["hits"]:
post = (row['_source']['text'])+"</br></br>"
post = post.encode('utf8')
post = post.decode('utf8')
fichero_html.write(post)
May it be something related to the apache server configuration?
I am trying to read the payload of all packets in a .pcap file using Pyshark. I am able to open and read the file, access the packets and their other information but I am not able to find the correct attribute/method to use to access the payload of a packet. Any suggestions ? Is there any other way to read packet payloads in .pcap files using python for windows 10 ?
(I tried using Scapy instead of Pyshark, but apparently there is some issue with running Scapy on Windows, it does not work on my system as well)
I found these lines in different code snippets of pyshark projects on the Internet and on StackOverflow. I tried them but none of them work :
import pyshark
cap = pyshark.FileCapture('file.pcap')
pkt = cap[1]
#for other information
print(pkt.tcp.flags_ack) #this works
print(pkt.tcp.flags_syn) #this works
print(pkt.tcp.flags_fin) #this works
#for payload
print(pkt.tcp.data) #does not work, AttributeError
print(pkt.tcp.payload) #does not work, AttributeError
print(pkt.data.data) #does not work, AttributeError
This code will print the value associated with the field name tcp.payload.
capture = pyshark.FileCapture(pcap_file, display_filter='tcp')
for packet in capture:
field_names = packet.tcp._all_fields
field_values = packet.tcp._all_fields.values()
for field_name in field_names:
for field_value in field_values:
if field_name == 'tcp.payload':
print(f'{field_name} -- {field_value}')
# outputs
tcp.payload -- \xc2\xb7\xc2\xb7\xc2\xb7\xc2\xb7\xc2\xb7\xc2\xb7\xc2\xb7AP\xc2\xb7\xc2\xb7\xc2\xb7
tcp.payload -- 0x00001e2c
tcp.payload -- 113977858
...
In order to use that API you have to pass appropriate parameter into constructor of 'FileCapture' class:
import pyshark
cap = pyshark.FileCapture('file.pcap', include_raw=True, use_json=True)
pkt = cap[1]
print(pkt.data.data) # Will work
'include_raw' is the key here, but 'use_json' is needed when when 'include_raw' is used.
dir cap[].
This one will give you all accessible attributes related to your capture., look there if there is the payload option.
I'm looking at trying to read pcap files from various CTF events.
Ideally, I would like something that can do the breakdown of information such as wireshark, but just being able to read the timestamp and return the packet as a bytestring of some kind would be welcome.
The problem is that there is little or no python 3 support with all the commonly cited libraries: dpkt, pylibpcap, pcapy, etc.
Does anyone know of a pcap library that works with python 3?
to my knowledge, there are at least 2 packages that seems to work with Python 3: pure-pcapfile and dpkt:
pure-pcapfile is easy to install in python 3 using pip. It's very easy to use but still limited to decoding Ethernet and IP data. The rest is left to you. But it works right out of the box.
dpkt doesn't work right out of the box and needs some manipulation before. They are porting it to Python 3 and plan to have a Python 2 and 3 compatible version for version 2.0. Unfortunately, it's not there yet. However, it is way more complete than pure-pcapfile and can decode many protocols. If your packet embeds several layers of protocols, it will decode them automatically for you. The only problem is that you need to make a few corrections here and there to make it work (as the time of writing this comment).
pure-pcapfile
the only one that I found working for Python 3 so far is pcapfile. You can find it at https://pypi.python.org/pypi/pypcapfile/ or install it by doing pip3 install pypcapfile.
There are just basic functionalities but it works very well for me and has been updated quite recently (at the time of writing this message):
from pcapfile import savefile
file = open('mypcapfile.pcp' , 'rb')
pcapfile = savefile.load_savefile(file,verbose=True)
If everything goes well, you should see something like this:
[+] attempting to load mypcapfile.pcap
[+] found valid header
[+] loaded 1234 packets
[+] finished loading savefile.
A few remarks now. I'm using Python 3.4.3. And doing import pcapfile will not import anything from it (I'm still a beginner with Python) but the only basic information and functions from the package. Next, you have to explicitly open your file in read binary mode by passing 'rb' as the mode in the open() function. In the documentation they don't say it explicitly.
The rest is like in the documentation:
packet = pcapfile.packets[12]
to access the packet number 12 (the 13th packet then, the first one being at 0). And you have basic functionalities like
packet.timestamp
to get a timestamp or
packet.raw()
to get raw data.
The documentation mentions functions to do packet decoding of some standard formats like Ethernet and IP.
dpkt
dpkt is not available for Python 3 so you need to do the following, assuming you have access to a command line. The code is available on https://github.com/kbandla/dpkt.git and you must download it before:
git clone https://github.com/kbandla/dpkt.git
cd dpkt
git checkout --track origin/migrate_py3
git pull
This 4 commands do the following:
clone (download) the code from its git repository on github
go into the newly created directory named dpkt
switch to the branch name migrate_py3 which contains the Python 3 code. As you can see from the name of this branch, it's still experimental. So far it works for me.
(just in case) download again the code
then copy the directory named dpkt in your project or wherever Python 3 can find it.
Later on, in Python 3 here is what you have to do to get started:
import dpkt
file = open('mypcapfile.pcap','rb')
will open your file. Don't forget the 'rb' binary mode in Python 3 (same thing as in pure-pcapfile).
pcap = dpkt.pcap.Reader(file)
will read and decode your file
for ts, buf in pcap:
eth = dpkt.ethernet.Ethernet(buf)
print(eth)
will, for example, decode Ethernet packet and print them. Then read the documentation on how to use dpkt. If your packets contain IP or TCP layer, then dpkt.ethernet.Ethernet(buf) will decode them as well. Also note that in the for loop, we have access to the timestamps in ts.
You may want to iterate it in a less constrained form and doing as follows will help:
(ts,buf) = next(pcap)
eth = dpkt.ethernet.Ethernet(buf)
where the first line get the next tuple from the pcap file. If pcap is False then you read everything.