I am trying to figure out a way to launch Mitmproxy from a python script (which I have done) and save any traffic to a dump file (which i need help with).
By googling, looking at mitmproxy git issues and reading example code, this is what I have so far:
from mitmproxy import proxy, options
from mitmproxy.tools.dump import DumpMaster
from mitmproxy.addons import core
class AddHeader:
def __init__(self):
self.num = 0
def response(self, flow):
self.num = self.num + 1
print(self.num)
flow.response.headers["count"] = str(self.num)
addons = [
AddHeader()
]
opts = options.Options(listen_host='127.0.0.1', listen_port=8080)
pconf = proxy.config.ProxyConfig(opts)
m = DumpMaster(None)
m.server = proxy.server.ProxyServer(pconf)
# print(m.addons)
m.addons.add(addons)
print(m.addons)
# m.addons.add(core.Core())
try:
m.run()
except KeyboardInterrupt:
m.shutdown()
Issue is, this creates an error AttributeError: No such option: body_size_limit which seems to be mitigated with master.addons.add(core.Core) but this core addon already exists in DumpMaster so that fires a different error.
Inspecting the addons that are currently loaded by DumpMaster i do see the save to file addon is loaded, but I am not clear how to access that so that any traffic that is going through the proxy, regardless if it is request, response, ws or tcp can be written to a dump file
Thanks!
Here is a redacted list of the addons that are loaded
mitmproxy.addons.streambodies.StreamBodies object at 0x111542da0>
mitmproxy.addons.save.Save object at 0x111542dd8>
mitmproxy.addons.upstream_auth.UpstreamAuth object at 0x111542e10>
just add those 2 lines after opts = options.Options(listen_host='127.0.0.1', listen_port=8080)
opts.add_option("body_size_limit", int, 0, "")
opts.add_option("keep_host_header", bool, True, "")
your code snippet already runs a working proxy. However, the option to dump the recorded traffic into a file during runtime (save_stream_file) is part of the Save-Addon which is loaded by default after the DumpMaster instance is created. Therefore, you need to set the save_stream_file option after creating the DumpMaster instance. Took me a while to figure it out as well but this worked for me, saving your output stream to a file named traffic_stream:
from mitmproxy import proxy, options
from mitmproxy.tools.dump import DumpMaster
opts = options.Options(listen_port=8081)
opts.add_option("body_size_limit", int, 0, "")
pconf = proxy.config.ProxyConfig(opts)
m = DumpMaster(None)
m.server = proxy.server.ProxyServer(pconf)
m.options.set('save_stream_file=traffic_stream')
try:
m.run()
except KeyboardInterrupt:
m.shutdown()
Hope it works for you as well!
Related
I need to find out for a script I'm writing who is the true owner of a file in an smb share (mounted using mount -t cifs of course on my server and using net use through windows machines).
Turns out it is a real challenge finding this information out using python on a linux server.
I tried using many many smb libraries (such as smbprotocol, smbclient and others), nothing worked.
I find few solutions for windows, they all use pywin32 or another windows specific package.
And I also managed to do it from bash using smbcalcs but couldn't do it cleanly but using subprocess.popen('smbcacls')..
Any idea on how to solve it?
This was unbelievably not a trivial task, and unfortunately the answer isn't simple as I hoped it would be..
I'm posting this answer if someone will be stuck with this same problem in the future, but hope maybe someone would post a better solution earlier
In order to find the owner I used this library with its examples:
from smb.SMBConnection import SMBConnection
conn = SMBConnection(username='<username>', password='<password>', domain=<domain>', my_name='<some pc name>', remote_name='<server name>')
conn.connect('<server name>')
sec_att = conn.getSecurity('<share name>', r'\some\file\path')
owner_sid = sec_att.owner
The problem is that pysmb package will only give you the owner's SID and not his name.
In order to get his name you need to make an ldap query like in this answer (reposting the code):
from ldap3 import Server, Connection, ALL
from ldap3.utils.conv import escape_bytes
s = Server('my_server', get_info=ALL)
c = Connection(s, 'my_user', 'my_password')
c.bind()
binary_sid = b'....' # your sid must be in binary format
c.search('my_base', '(objectsid=' + escape_bytes(binary_sid) + ')', attributes=['objectsid', 'samaccountname'])
print(c.entries)
But of course nothing will be easy, it took me hours to find a way to convert a string SID to binary SID in python, and in the end this solved it:
# posting the needed functions and omitting the class part
def byte(strsid):
'''
Convert a SID into bytes
strdsid - SID to convert into bytes
'''
sid = str.split(strsid, '-')
ret = bytearray()
sid.remove('S')
for i in range(len(sid)):
sid[i] = int(sid[i])
sid.insert(1, len(sid)-2)
ret += longToByte(sid[0], size=1)
ret += longToByte(sid[1], size=1)
ret += longToByte(sid[2], False, 6)
for i in range(3, len(sid)):
ret += cls.longToByte(sid[i])
return ret
def byteToLong(byte, little_endian=True):
'''
Convert bytes into a Python integer
byte - bytes to convert
little_endian - True (default) or False for little or big endian
'''
if len(byte) > 8:
raise Exception('Bytes too long. Needs to be <= 8 or 64bit')
else:
if little_endian:
a = byte.ljust(8, b'\x00')
return struct.unpack('<q', a)[0]
else:
a = byte.rjust(8, b'\x00')
return struct.unpack('>q', a)[0]
... AND finally you have the full solution! enjoy :(
I'm adding this answer to let you know of the option of using smbprotocol; as well as expand in case of misunderstood terminology.
SMBProtocol Owner Info
It is possible to get the SID using the smbprotocol library as well (just like with the pysmb library).
This was brought up in the github issues section of the smbprotocol repo, along with an example of how to do it. The example provided is fantastic and works perfectly. An extremely stripped down version
However, this also just retrieves a SID and will need a secondary library to perform a lookup.
Here's a function to get the owner SID (just wrapped what's in the gist in a function. Including here in case the gist is deleted or lost for any reason).
import smbclient
from ldap3 import Server, Connection, ALL,NTLM,SUBTREE
def getFileOwner(smb: smbclient, conn: Connection, filePath: str):
from smbprotocol.file_info import InfoType
from smbprotocol.open import FilePipePrinterAccessMask,SMB2QueryInfoRequest, SMB2QueryInfoResponse
from smbprotocol.security_descriptor import SMB2CreateSDBuffer
class SecurityInfo:
# 100% just pulled from gist example
Owner = 0x00000001
Group = 0x00000002
Dacl = 0x00000004
Sacl = 0x00000008
Label = 0x00000010
Attribute = 0x00000020
Scope = 0x00000040
Backup = 0x00010000
def guid2hex(text_sid):
"""convert the text string SID to a hex encoded string"""
s = ['\\{:02X}'.format(ord(x)) for x in text_sid]
return ''.join(s)
def get_sd(fd, info):
""" Get the Security Descriptor for the opened file. """
query_req = SMB2QueryInfoRequest()
query_req['info_type'] = InfoType.SMB2_0_INFO_SECURITY
query_req['output_buffer_length'] = 65535
query_req['additional_information'] = info
query_req['file_id'] = fd.file_id
req = fd.connection.send(query_req, sid=fd.tree_connect.session.session_id, tid=fd.tree_connect.tree_connect_id)
resp = fd.connection.receive(req)
query_resp = SMB2QueryInfoResponse()
query_resp.unpack(resp['data'].get_value())
security_descriptor = SMB2CreateSDBuffer()
security_descriptor.unpack(query_resp['buffer'].get_value())
return security_descriptor
with smbclient.open_file(filePath, mode='rb', buffering=0,
desired_access=FilePipePrinterAccessMask.READ_CONTROL) as fd:
sd = get_sd(fd.fd, SecurityInfo.Owner | SecurityInfo.Dacl)
# returns SID
_sid = sd.get_owner()
try:
# Don't forget to convert the SID string-like object to a string
# or you get an error related to "0" not existing
sid = guid2hex(str(_sid))
except:
print(f"Failed to convert SID {_sid} to HEX")
raise
conn.search('DC=dell,DC=com',f"(&(objectSid={sid}))",SUBTREE)
# Will return an empty array if no results are found
return [res['dn'].split(",")[0].replace("CN=","") for res in conn.response if 'dn' in res]
to use:
# Client config is required if on linux, not if running on windows
smbclient.ClientConfig(username=username, password=password)
# Setup LDAP session
server = Server('mydomain.com',get_info=ALL,use_ssl = True)
# you can turn off raise_exceptions, or leave it out of the ldap connection
# but I prefer to know when there are issues vs. silently failing
conn = Connection(server, user="domain\username", password=password, raise_exceptions=True,authentication=NTLM)
conn.start_tls()
conn.open()
conn.bind()
# Run the check
fileCheck = r"\\shareserver.server.com\someNetworkShare\someFile.txt"
owner = getFileOwner(smbclient, conn, fileCheck)
# Unbind ldap session
# I'm not clear if this is 100% required, I don't THINK so
# but better safe than sorry
conn.unbind()
# Print results
print(owner)
Now, this isn't super efficient. It takes 6 seconds for me to run this one a SINGLE file. So if you wanted to run some kind of ownership scan, then you probably want to just write the program in C++ or some other low-level language instead of trying to use python. But for something quick and dirty this does work. You could also setup a threading pool and run batches. The piece that takes longest is connecting to the file itself, not running the ldap query, so if you can find a more efficient way to do that you'll be golden.
Terminology Warning, Owner != Creator/Author
Last note on this. Owner != File Author. Many domain environments, and in particular SMB shares, automatically alter ownership from the creator to a group. In my case the results of the above is:
What I was actually looking for was the creator of the file. File creator and modifier aren't attributes which windows keeps track of by default. An administrator can enable policies to audit file changes in a share, or auditing can be enabled on a file-by-file basis using the Security->Advanced->Auditing functionality for an individual file (which does nothing to help you determine the creator).
That being said, some applications store that information for themselves. For example, if you're looking for Excel this answer provides a method for which to get the creator of any xls or xlsx files (doesn't work for xlsb due to the binary nature of the files). Unfortunately few files store this kind of information. In my case I was hoping to get that info for tblu, pbix, and other reporting type files. However, they don't contain this information (which is good from a privacy perspective).
So in case anyone finds this answer trying to solve the same kind of thing I did - Your best bet (to get actual authorship information) is to work with your domain IT administrators to get auditing setup.
I'm attempting to use Splinter (a package for Selenium) in multiple instances. However, instances aren't starting until the first thread completely finishes. So, the browser instance opens up, loads the page, sleeps, and only then does the 2nd thread start.
I need to run instances at the same time.
import threading
import time
from splinter import Browser
def worker(proxy, port):
proxy_settings = {"network.proxy.type": 1,
"network.proxy.ssl": proxy,
"network.proxy.ssl_port": port,
"network.proxy.socks": proxy,
"network.proxy.socks_port": port,
"network.proxy.socks_remote_dns": True,
"network.proxy.ftp": proxy,
"network.proxy.ftp_port": port
}
browser = Browser('firefox',
profile_preferences=proxy_settings,
capabilities={'pageLoadStrategy': 'eager'}) #eager or normal
print("Proxy: ", proxy, ":", proxy)
browser.visit("https://mxtoolbox.com/whatismyip/?" + proxy)
time.sleep(2)
ip1 = '22.22.222.222'
ip2 = '222.222.22.222'
p1 = int(2222)
p2 = int(2222)
p = []
p.append((ip1,p1))
p.append((ip2,p2))
x = 0
for pp in p:
threading.Thread(target=worker(pp[0], pp[1])).start()
In my longer code (the above is my attempt to figure out why I can't multi-thread) Im also getting an error in my editor that
Local variable 'browser' value is not used
That's because you're not starting a thread with the worker function, actually the last line should rather look like:
threading.Thread(target=worker, args=(pp[0], pp[1])).start()
As for your editor issue, I'd say it is editor dependent and without any information, it's hard to say (I'd mention that running pylint does not indicate such warning)
I would like to query Windows using a file extension as a parameter (e.g. ".jpg") and be returned the path of whatever app windows has configured as the default application for this file type.
Ideally the solution would look something like this:
from stackoverflow import get_default_windows_app
default_app = get_default_windows_app(".jpg")
print(default_app)
"c:\path\to\default\application\application.exe"
I have been investigating the winreg builtin library which holds the registry infomation for windows but I'm having trouble understanding its structure and the documentation is quite complex.
I'm running Windows 10 and Python 3.6.
Does anyone have any ideas to help?
The registry isn't a simple well-structured database. The Windows
shell executor has some pretty complex logic to it. But for the simple cases, this should do the trick:
import shlex
import winreg
def get_default_windows_app(suffix):
class_root = winreg.QueryValue(winreg.HKEY_CLASSES_ROOT, suffix)
with winreg.OpenKey(winreg.HKEY_CLASSES_ROOT, r'{}\shell\open\command'.format(class_root)) as key:
command = winreg.QueryValueEx(key, '')[0]
return shlex.split(command)[0]
>>> get_default_windows_app('.pptx')
'C:\\Program Files\\Microsoft Office 15\\Root\\Office15\\POWERPNT.EXE'
Though some error handling should definitely be added too.
Added some improvements to the nice code by Hetzroni, in order to handle more cases:
import os
import shlex
import winreg
def get_default_windows_app(ext):
try: # UserChoice\ProgId lookup initial
with winreg.OpenKey(winreg.HKEY_CURRENT_USER, r'SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\FileExts\{}\UserChoice'.format(ext)) as key:
progid = winreg.QueryValueEx(key, 'ProgId')[0]
with winreg.OpenKey(winreg.HKEY_CURRENT_USER, r'SOFTWARE\Classes\{}\shell\open\command'.format(progid)) as key:
path = winreg.QueryValueEx(key, '')[0]
except: # UserChoice\ProgId not found
try:
class_root = winreg.QueryValue(winreg.HKEY_CLASSES_ROOT, ext)
if not class_root: # No reference from ext
class_root = ext # Try direct lookup from ext
with winreg.OpenKey(winreg.HKEY_CLASSES_ROOT, r'{}\shell\open\command'.format(class_root)) as key:
path = winreg.QueryValueEx(key, '')[0]
except: # Ext not found
path = None
# Path clean up, if any
if path: # Path found
path = os.path.expandvars(path) # Expand env vars, e.g. %SystemRoot% for ext .txt
path = shlex.split(path, posix=False)[0] # posix False for Windows operation
path = path.strip('"') # Strip quotes
# Return
return path
I'm trying to make a python script wich reports me the port on 0.tcp.ngrok.io is started when I run the code on terminal (after moving ngrok executable file to /usr/local/bin)
ngrok tcp 22
I get ths kind of output
ngrok by #inconshreveable (Ctrl+C to quit)
Session Status connecting
Version 2.2.4
Region United States (us)
Web Interface http://127.0.0.1:4041
Forwarding tcp://0.tcp.ngrok.io:13014 -> localhost:22
Connections ttl opn rt1 rt5 p50 p90
0 0 0.00 0.00 0.00 0.00
My first attempt is to log the subprocess stdout to a variable , but as the stdout is cyclic the stdout.read() never ends this is the code
import subprocess
ngrok = subprocess.Popen(['ngrok','tcp','22'],stdout = subprocess.PIPE)
output_text = ngrok.stdout.read() # script stops here forever
[**code for getting domain:port from output_text**]
how can I get a "snapshot" of stdout to a variable , without stoping ngrok?
Is there another way of doing this (next try would be a webscraper on localhost , but it would be nice to have this knowledge for other commands , such as "top")
thanks in advance
I had the same issue when I was working with ngrok http and all the alternatives weren't working, resulting in deadlocks and I even could print the child process response created with ngrok. Thus, reading the ngrok docs I notice that there is a way to get the ngrok public url with requests.
Adding the code below:
localhost_url = "http://localhost:4041/api/tunnels" #Url with tunnel details
tunnel_url = requests.get(localhost_url).text #Get the tunnel information
j = json.loads(tunnel_url)
tunnel_url = j['Tunnels'][0]['PublicUrl'] #Do the parsing of the get
So, tunnel_url will return what you need. Adding the imports the full code would be like this:
import subprocess
import requests
import json
ngrok = subprocess.Popen(['ngrok','tcp','22'],stdout = subprocess.PIPE)
localhost_url = "http://localhost:4041/api/tunnels" #Url with tunnel details
tunnel_url = requests.get(localhost_url).text #Get the tunnel information
j = json.loads(tunnel_url)
tunnel_url = j['Tunnels'][0]['PublicUrl'] #Do the parsing of the get
not enough reputation to comment, feel free to update or comment on #Rodolfo great answer and then delete this
Perhaps they've changed the api slightly, this is what worked for me:
(ngrok executable next to the script, serving http on port 5000 and picking the https tunneling url)
import subprocess
import requests
import json
import time
if __name__ == '__main__':
ngrok = subprocess.Popen(['./ngrok','http','5000'],
stdout=subprocess.PIPE)
time.sleep(3) # to allow the ngrok to fetch the url from the server
localhost_url = "http://localhost:4040/api/tunnels" #Url with tunnel details
tunnel_url = requests.get(localhost_url).text #Get the tunnel information
j = json.loads(tunnel_url)
tunnel_url = j['tunnels'][1]['public_url'] #Do the parsing of the get
print(tunnel_url)
This is working for me:
def open_tunnel():
process = subprocess.Popen(f'/snap/bin/ngrok tcp {PORT} --log "stdout"', shell=True, stdout=PIPE, stderr=PIPE) # You can also use a list and put shell = False
while True:
output = process.stdout.readline()
if not output and process.poll() is not None:
break
elif b'url=' in output:
output = output.decode()
output = output[output.index('url=tcp://') + 10 : -1]
return output.split(':')
I use /snap/bin/ngrok because my pycharm does not recognize path, sorry for that. You can replace it by saying only ngrok
.
I am trying to change the directory of the couch database. I am using a python script to import a csv file to the CouchDB. Script is running ok. Here it is just in case:
from couchdbkit import Server, Database
from couchdbkit.loaders import FileSystemDocsLoader
from csv import DictReader
import sys, subprocess, math, os
def parseDoc(doc):
for k,v in doc.items():
if (isinstance(v,str)):
#print k, v, v.isdigit()
# #see if this string is really an int or a float
if v.isdigit()==True: #int
doc[k] = int(v)
else: #try a float
try:
if math.isnan(float(v))==False:
doc[k] = float(v)
except:
pass
return doc
def upload(db, docs):
db.bulk_save(docs)
del docs
return list()
def uploadFile(fname, dbname):
#connect to the db
theServer = Server()
db = theServer.get_or_create_db(dbname)
#loop on file for upload
reader = DictReader(open(fname, 'rU'), dialect = 'excel')
docs = list()
checkpoint = 100
i = 0
for doc in reader:
newdoc = parseDoc(doc)
docs.append(newdoc)
if len(docs)%checkpoint==0:
docs = upload(db,docs)
i += 1
print 'Number : %d' %i
#don't forget the last batch
docs = upload(db,docs)
if __name__=='__main__':
x = '/media/volume1/Crimes_-_2001_to_present.csv'
filename = x
dbname = 'test'
uploadFile(filename, dbname)
I saw plenty posts on how to change the directory for appending the database. If I leave the /etc/couchdb/local.ini as it is (original after installation) the script is appending data to the default directory /var/lib/couchdb/1.0.1/. When I modify the local.ini to store the database to another disk:
database_dir = /media/volume1
view_index_dir = /media/volume1
and after the reboot of the CouchDB service I get this error :
restkit.errors.RequestError: socket.error: [Errno 111] Connection refused
I have checked the open sockets (couchdb uses 5984 as default) and it is not opened. But I get no errors when I start CouchDB service.
Any ideas how to fix it ?
I think the error may be due to you have changed the directory location in Local.ini but when you are trying to make new connection to existing database, it cannot find it there.
So move the database_name.couch file to new location which you can put in local.ini and then try to make a connection. I think this should work.