I'm beginner in Python and I would like to build simple port sniffer.
For this purpose I'm using code from this site, as example: Simple packege snffer using python
And I would like to unpack bites from socket to exctract http header, using function struct.unpack()
What format of string should I use in unpack HTTP header, (e.g '!BBH', "!BBHHHBBH4s4s",'!HHLLBBHHH')
the HTTP header is not fixed-length, so you'll need to parse it other way, for example:
import logging
log = logging.getLogger(__name__)
def parse_http(data):
lines = data.split(b'\r\n')
if len(lines) < 1:
log.error('Invalid http header: %s', lines)
return
request = lines[0]
header = {}
rest = []
in_header = True
for line in lines[1:]:
if line == b'':
in_header = False
continue
if in_header:
try:
key, val = line.split(b': ')
except ValueError:
log.error('Invalid header line: %s', line)
continue
header[key] = val
else:
rest.append(line)
return request, header, b'\r\n'.join(rest)
In order to detect a HTTP packet, you could check if the payload starts with POST, GET, HTTP ... etc
Related
I created a function to send data to a graphite server. It sends the metricname, value and the timestamp to the graphite server at execution:
def collect_metric(metricname, value, timestamp):
sock = socket.socket()
sock.connect( ("localhost", 2003) )
sock.send("%s %s %s\n" % (metricname, value, timestamp))
sock.close()
This function above worked fine in Python2. I had to rewrite this function for Python3. Now no data will be send to graphite. No log entries in the graphite/carbon logs or something else ...:
def collect_metric(metricname, value, timestamp):
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect( ("localhost", 2003) )
metricname = metricname.encode()
if type(value) == "str":
value = value.encode()
timestamp = timestamp.encode()
message = bytearray()
message = bytes(metricname+b" "+value+b" "+timestamp)
sock.sendall(message)
print(message.decode())
sock.close()
I receive no errors. Also on terminal I get the right format/output (see "print(message.decode())")
Has anybody some ideas why it doesn't work?
Thanks.
The bytearray is without any encoding. Try this:
message = (metricname+" "+value+" "+timestamp).encode("UTF-8")
sock.send(messages)
It seems like you're missing '\n' at the end of the message you're sending
message = bytes(metricname+b" "+value+b" "+timestamp)
should be:
message = bytes(metricname+b" "+value+b" "+timestamp + '\n')
This code acts as an early warning system for ADFS failures, which works fine when run locally. Problem is that when I run it in Lambda, it loops non stop.
In short:
lambda_handler() runs pagecheck()
pagecheck() produces the info needed then passes 2 lists (msgdet_list, error_list) and an int (error_count) to notification().
notification() collates and prints the output. The output is two key variables (notificationheader and notificationbody).
I've #commentedOut the SNS piece which would usually email the info, and am using print() to instead send the info to CloudWatch logs until I can get the loop sorted. Logs:
CloudWatch logs
If I run this locally, it produces a clean single output. In Lambda, the function will loop until it times out. It's almost like every time the lists are updated, they're passed to the notification() module and it's run. I can limit the function time, but would rather fix the code!
Cheers,
tac
# This python/boto3/lambda script sends a request to an Office 365 landing page, parses return details to confirm a successful redirect to /
# the organisation ADFS homepage, authenticates homepge is correct, raises any errors, and sends a consolodated report to /
# an AWS SNS topic.
# Run once to produce pageserver and htmlchar values for global variables.
# Import required modules
import boto3
import urllib.request
from urllib.request import Request, urlopen
from datetime import datetime
import time
import re
import sys
# Global variables to be set
url = "https://outlook.com/CONTOSSO.com"
adfslink = "https://sts.CONTOSSO.com/adfs/ls/?client-request-id="
# Input after first run
pageserver = "Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0"
htmlchar = 18600
# Input AWS SNS ARN
snsarn = 'arn:aws:sns:ap-southeast-2:XXXXXXXXXXXXX:Daily_Check_Notifications_CONTOSSO'
sns = boto3.client('sns')
def pagecheck():
# Present the request to the webpage as if coming from a user in a browser
user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)'
values = {'name' : 'user'}
headers = { 'User-Agent' : user_agent }
data = urllib.parse.urlencode(values)
data = data.encode('ascii')
# "Null" the Message Detail and Error lists
msgdet_list = []
error_list = []
request = Request(url)
req = urllib.request.Request(url, data, headers)
response = urlopen(request)
with urllib.request.urlopen(request) as response:
# Get the URL. This gets the real URL.
acturl = response.geturl()
msgdet_list.append("\nThe Actual URL is:")
msgdet_list.append(str(acturl))
if adfslink not in acturl:
error_list.append(str("Redirect Fail"))
# Get the HTTP resonse code
httpcode = response.code
msgdet_list.append("\nThe HTTP code is: ")
msgdet_list.append(str(httpcode))
if httpcode//200 != 1:
error_list.append(str("No HTTP 2XX Code"))
# Get the Headers as a dictionary-like object
headers = response.info()
msgdet_list.append("\nThe Headers are:")
msgdet_list.append(str(headers))
if response.info() == "":
error_list.append(str("Header Error"))
# Get the date of request and compare to UTC (DD MMM YYYY HH MM)
date = response.info()['date']
msgdet_list.append("The Date is: ")
msgdet_list.append(str(date))
returndate = str(date.split( )[1:5])
returndate = re.sub(r'[^\w\s]','',returndate)
returndate = returndate[:-2]
currentdate = datetime.utcnow()
currentdate = currentdate.strftime("%d %b %Y %H%M")
if returndate != currentdate:
date_error = ("Date Error. Returned Date: ", returndate, "Expected Date: ", currentdate, "Times in UTC (DD MMM YYYY HH MM)")
date_error = str(date_error)
date_error = re.sub(r'[^\w\s]','',date_error)
error_list.append(str(date_error))
# Get the server
headerserver = response.info()['server']
msgdet_list.append("\nThe Server is: ")
msgdet_list.append(str(headerserver))
if pageserver not in headerserver:
error_list.append(str("Server Error"))
# Get all HTML data and confirm no major change to content size by character lenth (global var: htmlchar).
html = response.read()
htmllength = len(html)
msgdet_list.append("\nHTML Length is: ")
msgdet_list.append(str(htmllength))
msgdet_list.append("\nThe Full HTML is: ")
msgdet_list.append(str(html))
msgdet_list.append("\n")
if htmllength // htmlchar != 1:
error_list.append(str("Page HTML Error - incorrect # of characters"))
if adfslink not in str(acturl):
error_list.append(str("ADFS Link Error"))
error_list.append("\n")
error_count = len(error_list)
if error_count == 1:
error_list.insert(0, 'No Errors Found.')
elif error_count == 2:
error_list.insert(0, 'Error Found:')
else:
error_list.insert(0, 'Multiple Errors Found:')
# Pass completed results and data to the notification() module
notification(msgdet_list, error_list, error_count)
# Use AWS SNS to create a notification email with the additional data generated
def notification(msgdet_list, error_list, errors):
datacheck = str("\n".join(msgdet_list))
errorcheck = str("\n".join(error_list))
notificationbody = str(errorcheck + datacheck)
if errors >1:
result = 'FAILED!'
else:
result = 'passed.'
notificationheader = ('The daily ADFS check has been marked as ' + result + ' ' + str(errors) + ' ' + str(error_list))
if result != 'passed.':
# message = sns.publish(
# TopicArn = snsarn,
# Subject = notificationheader,
# Message = notificationbody
# )
# Output result to CloudWatch logstream
print('Response: ' + notificationheader)
else:
print('passed')
sys.exit()
# Trigger the Lambda handler
def lambda_handler(event, context):
aws_account_ids = [context.invoked_function_arn.split(":")[4]]
pagecheck()
return "Successful"
sys.exit()
Your CloudWatch logs contain the following error message:
Process exited before completing request
This is caused by invoking sys.exit() in your code. Locally your Python interpreter will just terminate when encountering such a sys.exit().
AWS Lambda on the other hand expects a Python function to just return and handles sys.exit() as an error. As your function probably got invoked asynchronously AWS Lambda retries to execute it twice.
To solve your problem, you can replace the occurences of sys.exit() with return or even better, just remove the sys.exit() calls, as there would be already implicit returns in the places where you use sys.exit().
I'm starting to play around with Python sockets and I found the following example online for sending and receiving data to a web URL:
import socket
import sys
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error:
print("Failed to create socket")
sys.exit()
print("Socket created")
host = input("Enter URL here: ")
port = 80
try:
remoteIP = socket.gethostbyname(host)
except socket.gaierror:
print("Host name could not be resolved")
sys.exit()
s.connect((remoteIP, port))
print("Socket connected to " + host + " on IP " + remoteIP)
message = "GET / HTTP/1.1\r\n\r\n"
messagebytes = str.encode(message)
try:
s.sendall(messagebytes)
except socket.error:
print("Send failed")
sys.exit()
print("Message sent successfully")
reply = s.recv(4096)
print(reply)
Now the issue is whenever I run this code, I get the following output:
b'HTTP/1.1 400 Bad Request\r\nCache-Control: no-cache\r\nPragma: no-cache\r\nContent-Type: text/html; charset=utf-8\r\nProxy-Connection: close\r\nConnection: close\r\nContent-Length: 1229\r\n\r\n\r\n\r\nRequest Error\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nMy Internet Access\r\n\r\n--\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nRequest Error\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nYour request could not be processed. Request could not be handled\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nThis could be caused by a misconfiguration, or possibly a malformed request.\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nFor assistance, please log a call with the helpdesk.\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n'
My only guess for this outcome is that the message being sent is sent as b" therefore GET / HTTP/1.1\r\n\r\n" and that b c, therefore, making it an invalid request. I tried decoding it as well and it didn't help as I get the exact same outcome as when I do not decode it.
If that is the case how can I fix this?
I'v tried decode the response and I made it.
>>> a=b'HTTP/1.1 400 Bad Request\r\nCache-Control: no-cache\r\nPragma: no-cache\r\nContent-Type: text/html; charset=utf-
8\r\nProxy-Connection: close\r\nConnection: close\r\nContent-Length: 1229\r\n\r\n\r\n\r\nRequest Error\r\n\r\n\r\n\r\n\r
\n\r\n\r\n\r\nMy Internet Access\r\n\r\n--\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r
\n\r\n\r\n\r\nRequest Error\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nYour request could no
t be processed. Request could not be handled\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nThis could be caused by a mi
sconfiguration, or possibly a malformed request.\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nFor assistance,
please log a call with the helpdesk.\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n'
>>> a.decode('UTF-8')
'HTTP/1.1 400 Bad Request\r\nCache-Control: no-cache\r\nPragma: no-cache\r\nContent-Type: text/html; charset=utf-8\r\nPr
oxy-Connection: close\r\nConnection: close\r\nContent-Length: 1229\r\n\r\n\r\n\r\nRequest Error\r\n\r\n\r\n\r\n\r\n\r\n\
r\n\r\nMy Internet Access\r\n\r\n--\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\
r\n\r\nRequest Error\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nYour request could not be pr
ocessed. Request could not be handled\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nThis could be caused by a misconfig
uration, or possibly a malformed request.\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\nFor assistance, please
log a call with the helpdesk.\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n'
But problem with your code is in request headers. If you want to get some content of page, you can use urllib.request package.
First you need some setup:
import urllib.request as req
class Fetch:
"""
Class responsible for getting code of page at specified url
"""
def __init__(self, url):
self.url = url
self.data = None
self.obj = None
self.headers = None
self.encoding = None
def fetch_file(self):
"""
:return: str contents of a website
Returns string data form an url
"""
self.obj = req.urlopen(self.url)
self.encoding = self.obj.headers['content-type'].split('charset=')[-1]
self.data = self.obj.read().decode(self.encoding)
return self.data
Then you import Fetch class and pass desired url to init object. At the end you call fetch_file() it returns encoded response if in response headers is charset specified.
Try this when sending:
message = "GET / HTTP/1.1\r\n\r\n"
s.sendall(messagebytes.encode())
And while receiving:
reply = s.recv(4096)
print(reply.decode())
So I'm connected to an IRC chat using socket.connec
I log in by passing my variables via socket.send
The log in is successful and then I sit in a while true loop using
Socket.recv(1024)
If I just continually print the response everything looks fine, but let's say I want to add to the end of the string... I noticed that socket.recv doesn't always get the full message (only grabs up to 1024 as expected) and the remainder of the message is in the next iteration of the loop.
This makes its it impossible to process the feedback line by line.
Is there a better way to constantly read the data without it getting trunked? Is it possible to figure out the size of the response before receiving it so the buffer can be set dynamically?
TCP is a stream-based protocol. Buffer the bytes received and only extract complete messages from the stream.
For complete lines, look for newline characters in the buffer.
Example server:
import socket
class Client:
def __init__(self,socket):
self.socket = socket
self.buffer = b''
def getline(self):
# if there is no complete line in buffer,
# add to buffer until there is one.
while b'\n' not in self.buffer:
data = self.socket.recv(1024)
if not data:
# socket was closed
return ''
self.buffer += data
# break the buffer on the first newline.
# note: partition(n) return "left of n","n","right of n"
line,newline,self.buffer = self.buffer.partition(b'\n')
return line + newline
srv = socket.socket()
srv.bind(('',5000))
srv.listen(1)
conn,where = srv.accept()
client = Client(conn)
print(f'Client connected on {where}')
while True:
line = client.getline()
if not line:
break
print(line)
Example client:
s=socket()
s.connect(('127.0.0.1',5000))
s.sendall(b'line one\nline two\nline three\nincomplete')
s.close()
Output on server:
Client connected on ('127.0.0.1', 2667)
b'line one\n'
b'line two\n'
b'line three\n'
I have a problem with Burpsuite API that I can't find a proper function to print out the response for edited requests . I'm developing a new plugin for burpsuite with python . myscript is simply takes requests from proxy then it edit headers and send it again .
from burp import IBurpExtender
from burp import IHttpListener
import re,urllib2
class BurpExtender(IBurpExtender, IHttpListener):
def registerExtenderCallbacks(self, callbacks):
self._callbacks = callbacks
self._helpers = callbacks.getHelpers()
callbacks.setExtensionName("Burp Plugin Python Demo")
callbacks.registerHttpListener(self)
return
def processHttpMessage(self, toolFlag, messageIsRequest, currentRequest):
# only process requests
if messageIsRequest:
requestInfo = self._helpers.analyzeRequest(currentRequest)
#timestamp = datetime.now()
#print "Intercepting message at:", timestamp.isoformat()
headers = requestInfo.getHeaders()
#print url
if(requestInfo.getMethod() == "GET"):
print "GET"
print requestInfo.getUrl()
response = urllib2.urlopen(requestInfo.getUrl())
print response
elif(requestInfo.getMethod() == "POST"):
print "POST"
print requestInfo.getUrl()
#for header in headers:
#print header
bodyBytes = currentRequest.getRequest()[requestInfo.getBodyOffset():]
bodyStr = self._helpers.bytesToString(bodyBytes)
bodyStr = re.sub(r'=(\w+)','=<xss>',bodyStr)
newMsgBody = bodyStr
newMessage = self._helpers.buildHttpMessage(headers, newMsgBody)
print "Sending modified message:"
print "----------------------------------------------"
print self._helpers.bytesToString(newMessage)
print "----------------------------------------------\n\n"
currentRequest.setRequest(newMessage)
return
You need to print the response but you don't do anything in case messageIsRequest is false. When messageIsRequest is false it means that the currentRequest is a response and you can print out the response as you did for the request. I did it in Java like this:
def processHttpMessage(self, toolFlag, messageIsRequest, httpRequestResponse):
if messageIsRequest:
....
else
HTTPMessage = httpRequestResponse.getResponse()
print HTTPMessage
There is even a method that lets you bind request and response together when using a proxy. It can be found in IInterceptedProxyMessage:
/**
* This method retrieves a unique reference number for this
* request/response.
*
* #return An identifier that is unique to a single request/response pair.
* Extensions can use this to correlate details of requests and responses
* and perform processing on the response message accordingly.
*/
int getMessageReference();
I don't think it is supported for HTTPListeners.
I am writing the extensions in Java and tried to translate to Python for this anwser. I haven't tested this code and some bugs might be introduced due to translation.