I'm aware of the SO post How do I override file.write() in Python 3? but after looking it over and trying whats suggested I'm still stuck.
I want to override the file.write method in Python 3 so that I can "REDACT" certain words (Usernames, Passwords...etc).
I found a great example of overriding the print and general stdout and stderr http://code.activestate.com/recipes/119404/
The issue is that it doesn't work for file.write. How can I override the file.write?
My code for redacting when printing is:
def write(self, text):
for word in self.redacted_list:
text = text.replace(word, "REDACTED")
self.origOut.write(text)
return text
thanks
From the self.origOut.write(text) I assume you are trying to write an in-between-class that pretends to be a file but provides a different .write() method.
I don't see any problems in the code you posted (assuming it's a method of a class you use). Possibly you wrote a class but forgot to create instances of it?
Did you try to write something like this?:
class IAmNoARealFile:
def __init__(self, real_file):
self.origOut = real_file
def __getattr__(self, attr_name): # provide everything a file has
return getattr(self.origOut, attr_name)
def write(self, ...):
...
with open('test.txt', 'w') as f:
f = IAmNotARealFile(f) # did you forget this?
f.write('some text SECRET blah SECRET') # calls IAMNotARealFile.write with your extra code
with open('test.txt') as f:
f = IAmNotARealFile(f)
print(f.read()) # this "falls through" to the actual file object
you will also probably want to return self.origOut.write() in your own .write(), if you don't have a specific reason not to.
Note that if you rewrite open() to directly return IAMNotARealFile:
def open(*args, **kwargs):
return IAMNotARealFile(open(*args, **kwargs))
you will have to manually supply (some) "magic methods" because
This method may still be bypassed when looking up special methods as the result of implicit invocation via language syntax or built-in functions. See Special method lookup.
--docs for .__getattribute__(), but it also applies to .__getattr__()
Why?
Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).
-- On special ("magic") method lookup [code style and emphasis mine]
Related
I am writing a python program that I want to have a command line interface that behaves in a particular way
The command line interface should accept the following invocations:
my_prog test.svg foo
my_prog --font=Sans test.svg foo
(it will generate an svg with the word foo written in the specified or default font)
Now I want to be able to also have this command accept the following invocation...
my_prog --list-fonts
which will list all of the valid options to --font as determined by the fonts available on the system.
I am using argparse, and I have something like this:
parser = argparse.ArgumentParser()
parser.add_argument('output_file')
parser.add_argument('text')
parser.add_argument('--font', help='list options with --list-fonts')
parser.add_argument('--list-fonts', action='store_true')
args = parser.parse_args()
however this does not make the --list-fonts option behave as I would like as the two positional arguments are still required.
I have also tried using subparsers, but these still need a workaround to prevent the other options being required every time.
How do I get the desired behaviour with argparse.
argparse allows you to define arbitrary actions to take when encountering an argument, based on the action keyword argument to add_argument (see the docs)
You can define an action to list your fonts and then abort argument parsing, which will avoid checking for the other required arguments.
this could look like this:
class ListFonts(argparse.Action):
def __call__(self, parser, namespace, values, option_string):
print("list of fonts here")
parser.exit() # exits the program with no more arg parsing and checking
Then you can add it to your argument like so:
parser.add_argument('--list-fonts', nargs=0, action=ListFonts)
Note nargs=0 has been added so that this argument doesn't require a value (the code in the question achieved this with action='store_true')
This solution has a side-effect of enabling the invocations like the following to also list the fonts and exits without running the main program:
my_prog --font Sans test.svg text --list-fonts
This is likely not a problem as it's not a typical use case, especially if the help text explains this behaviour.
If defining a new class for each such option feels too heavyweight, or perhaps you have more than one option that has this behaviour, then you could consider having a function that implements the desired action for each argument and then have a kind of factory function that returns a class that wraps the function. A complete example of this is shown below.
def list_fonts():
print("list of fonts here")
def override(func):
""" returns an argparse action that stops parsing and calls a function
whenever a particular argument is encountered. The program is then exited """
class OverrideAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string):
func()
parser.exit()
return OverrideAction
parser = argparse.ArgumentParser()
parser.add_argument('output_file')
parser.add_argument('text')
parser.add_argument('--font', help='list options with --list-fonts')
parser.add_argument('--list-fonts', nargs=0, action=override(list_fonts),
help='list the font options then stop, don\'t generate output')
args = parser.parse_args()
I am trying to create a file writer based on Pandas' ExcelWriter. I proceeded as I usually do with classes in Python (3) with inheritance:
import pandas as pd
class Writer(pd.ExcelWriter):
def __init__(self, fname, engine='openpyxl'):
pd.ExcelWriter.__init__(self, fname, engine=engine)
self.newvar = 0
However, when I try to use it, I cannot access newvar:
test = Writer('test.xlsx')
test.newvar
returns:
AttributeError: '_XlsxWriter' object has no attribute 'nmax'
And when I check the type of test, it returns:
pandas.io.excel._XlsxWriter
I don't understand what I am missing since I used this kind of inheritance in many other cases. Any idea would be appreciated!
This is because pandas.ExcelWriter.__new__ returns a different class than itself (actually it is an abc.ABCMeta). The class is chosen based on the extension of the file path and the engine which is used - you could observe that when you checked the type of the newly created instance. That means the __init__ method of whatever class is returned gets called. You can think of ExcelWriter as some kind of proxy for the specific writers for each format and engine (though it also defines the API which such a writer must provide).
In order to make your writer available (for the given engine), you need to register it.
But before you can do that you need to make your class compatible by following the instructions which you'll find via help(pandas.ExcelWriter). For the sake of completeness I cite them here:
# Defining an ExcelWriter implementation (see abstract methods for more...)
# - Mandatory
# - ``write_cells(self, cells, sheet_name=None, startrow=0, startcol=0)``
# --> called to write additional DataFrames to disk
# - ``supported_extensions`` (tuple of supported extensions), used to
# check that engine supports the given extension.
# - ``engine`` - string that gives the engine name. Necessary to
# instantiate class directly and bypass ``ExcelWriterMeta`` engine
# lookup.
# - ``save(self)`` --> called to save file to disk
# - Mostly mandatory (i.e. should at least exist)
# - book, cur_sheet, path
# - Optional:
# - ``__init__(self, path, engine=None, **kwargs)`` --> always called
# with path as first argument.
So with that in mind we can extend your class:
class Writer(pd.ExcelWriter):
engine = 'openpyxl'
supported_extensions = ('xlsx',)
def write_cells(self, cells, sheet_name=None, startrow=0, startcol=0):
# Implement something useful here.
pass
def save(self):
# Implement something useful here.
pass
def __init__(self, fname, engine='openpyxl', **kwargs):
super().__init__(self, fname, engine=engine, **kwargs)
Now you can use pd.io.excel.register_writer(Writer) to register the writer. But you need to make sure the engine which you've specified matches your version of openpyxl. You can check the process of how a specific writer is chosen here; the writers which are currently registered for each version can be checked via print(pd.io.excel._writers).
As a side note: You can also subclass one of the already available specific writers and reuse their write_cells and save methods for example (however you'll need to register your writer also in that case):
_Openpyxl1Writer
_Openpyxl20Writer
_Openpyxl22Writer
_XlwtWriter
_XlsxWriter
I have several classes imported on a code but I need to instantiate only those classes that are listed on a text file. So I have something like this
from module1 import c1
from module2 import c2
...
and in the text file I have a list of only those classes I want to instantiate like
c1()
c2(True)
...
so I want to read the file lines to a list (classes) and do something like
for i in classes:
classes_list.append(i)
so that each element of the list is an instantiated class. I tried doing this based on other solutions I found here
for i in classes:
classes_list.append(globals()[i])
but I always get this error
KeyError: 'c1()'
or
KeyError: 'c2(True)'
Any ideas how something like this could be possible?
You are implementing a mini-language that expresses how to call certain functions. This can get difficult, but it turns out python already implements its own mini language with the eval function. With eval, python will compile and execute python expressions.
This is considered risky for stuff coming from anonymous and potentially malicious users on the network but may be a reasonable solution for people who have some level of trust. For instance, if the people writing these files are in your organization and could mess with you a thousand ways anyway, you may be able to trust them with this. I implemented a system were people could write fragments of test code and my system would wrap it all up and turn it into a test suite. No problem because those folks already had complete access to the systems under test.
module1.py
def c1(p=1):
return p
def c2(p=1):
return p
def c3(p=1):
return p
test.py
import module1
my_globals = {
'c1': module1.c1,
'c2': module1.c2,
'c3': module1.c3,
}
test = ["c1()",
"c2(p=c1())",
"c3('i am a string')",
"c1(100)"]
for line in test:
print(line.strip() + ':', eval(line, my_globals))
result
c1(): 1
c2(p=c1()): 1
c3('i am a string'): i am a string
c1(100): 100
I'm doing an extended project as one of my qualifications in my current College and I chose to write a python Strategy/RPG game. As a result, I ended up with the highest level of Python knowledge (Surpassing my Computing Teacher who only ever uses the basics... and used Tkinter only once a few years ago. Every one else who has decided to make a program, are either coding in Lua, Java, C++, HTML/CSS/Java-Script or, those who are coding in python, they are only using the basics learned from our teacher.)
I say "Highest level of Python knowledge" but really it isn't that high... I only know a little beyond the basics.
As a result, a forum post is the best place I can turn to for help.
So in my game I defined this function:
#"Given_String" is the question that one would want to ask. (With the answer being an integer between 1 and "Choice_Range" (inclusive)
def Value_Error(Given_String,Error_Message,Choice_Range):
while True:
try:
Temp=int(input(Given_String))
if Temp<1 or Temp>Choice_Range:
print(Error_Message)
else:
break
except ValueError:
print(Error_Message)
return Temp
I then wanted to add tkinter to my code, because the game would have to be in a separate window, and not in the console. As a result, I had to change this code so that it displays the "Given_Message" and the "Error_Message" in a tkinter window, and uses the value that has been typed into an entry box when defining "Temp".
I wrote this code to make this work: (Or at least most of it)
#This code is stored in a different file for neatness and hence I had to import "sys" to avoid circular imports.
#This code is made to be flexible so that I can later re-use it when necessary.
#This code starts with the function all the way at the bottom. The rest are made to add flexibility and to structure the algorithm.
#This code hasn't been fully run (Because of the Error crashing the Python Shell) so it can contain other Run-time Errors that I'm not aware of yet.
import sys
def Generate_Window(Window_Name,X_Parameter=5,Y_Parameter=50):
Temp=sys.modules['tkinter'].Tk()
Temp.title(Window_Name)
Temp.geometry(str(X_Parameter)+"x"+str(Y_Parameter))
return Temp
def Generate_Button(Master,Text="Submit"):
Temp=sys.modules["tkinter"].Button(Master,text=Text)
return Temp
def Generate_Entry(Master):
Temp=sys.modules["tkinter"].Entry(Master)
return Temp
def Generate_Label(Master,Given_String):
Temp=sys.modules["tkinter"].Label(Master,text=Given_String)
return Temp
def Com_Get_Entry(Given_Window,Given_Entry):
Temp=Given_Entry.get()
Given_Window.destroy()
return Temp
def Com_Confirm(Given_Window):
Given_Window.destroy()
def Generate_Entry_Box(Given_String):
Entry_Window=Generate_Window("Entry",X_Parameter=300)
Entry_Label=Generate_Label(Entry_Window,Given_String)
Entry_Entry=Generate_Entry(Entry_Window)
Entry_Button=Generate_Button(Entry_Window)
Entry_Button.configure(command=lambda:Com_Get_Entry(Entry_Window,Entry_Entry))
Entry_Label.grid(row=0,columnspan=2)
Entry_Entry.grid(row=1,column=0)
Entry_Button.grid(row=1,column=1)
def Generate_Alert_Message(Given_String):
Alert_Window=Generate_Window("Alert",X_Parameter=300)
Alert_Label=Generate_Label(Alert_Window,Given_String)
Alert_Button=Generate_Button(Alert_Window,Text="OK")
Alert_Button.configure(command=lambda:Com_Confirm(Alert_Window))
Alert_Label.grid(row=0,columnspan=2)
Alert_Button.grid(row=1,column=1)
def Get_Interger_Input_In_Range(Given_String,Error_Message,Choice_Range):
while True:
try:
Returned_Value=int(Generate_Entry_Box(Given_String))
if Returned_Value<1 or Returned_Value>Choice_Range:
Generate_Alert_Message(Error_Message)
else:
break
except ValueError:
Generate_Alert_Message(Error_Message)
return Temp
I already included in my code all that I was struggling with and that I could find an answer to.
I.E: On-click, do a certain action with given parameters.
One thing I could not find, is how to return the entered value to the original (Get_Interger_Input_In_Range()) function after the button has been clicked.
What I mean is something like this:
def Function1(GivenParameter1,GivenParameter2):
Temp=Function2(GivenParameter1)
Temp+=GiverParameter2 #random action
return Temp
def Function2(GivenParameter):
Button=Button(Master,command=Function3).grid()
Entry=Entry(Master).grid()
def Function3():
Temp=Entry.get()
return Temp
In Function1 I want Temp to equal the entered value from Function2.
Is there any way to do this without using classes? (I'm not too familiar with classes yet)
Is there any way to do this at all?
I haven't seen anyone give the answer I was looking for...
Because even if they said to use classes... I still didn't know how to return it (Explanation just below)
#The following code was written quickly for purposes of explaining what I mean. It doesn't actually work... (It seems that the button command is being called automatically...)
from tkinter import *
class Return_Value_In_Entry():
def __init__(self):
self.Master=Tk()
self.Entry=Entry(self.Master)
self.Button=Button(self.Master,text="Submit",command=self.Return())
def Return(self):
self.TempVar=self.Entry.get()
return self.TempVar
The way I see it, the Return() function would return the value to the button and not the function/assignment that called the class ... Which is the same problem I'm having with my code.
If you read this all then I really appreciate it. I hope someone can answer my question and tell me (if it's impossible otherwise) how to use classes to solve my "Little" yet large problem.
I fixed your example code (I think). The main problem is that this:
command=self.Return()
does not do what you think it does. It just assigns return value from Return() to command. This is incorrect. It should be
command=self.Return
This assigns function Return to command. Subsequently, when ever button is pressed, self.Return() is executed.
The full example is here:
from tkinter import *
class Return_Value_In_Entry():
def __init__(self):
self.Master=Tk()
self.Entry=Entry(self.Master)
self.Entry.pack()
self.Button=Button(self.Master,text="Submit",command=self.Return)
self.Button.pack()
self.Master.mainloop()
def Return(self):
self.TempVar=self.Entry.get()
print(self.TempVar)
Return_Value_In_Entry()
Now, whenever you press the Button, the value from the Entry widget is saved into self.TempVar and printed out, just to check if its working. Hope this helps.
Gif showing how the example program works:
I've got a class that wraps some file handling functionality I need. Another class creates an instance of the filehandler and uses it for an indeterminate amount of time. Eventually, the caller is destroyed, which destroys the only reference to the filehandler.
What is the best way to have the filehandler close the file?
I currently use __del__(self) but after seeing several different questions and articles, I'm under the impression this is considered a bad thing.
class fileHandler:
def __init__(self, dbf):
self.logger = logging.getLogger('fileHandler')
self.thefile = open(dbf, 'rb')
def __del__(self):
self.thefile.close()
That's the relevent bit of the handler. The whole point of the class is to abstract away details of working with the underlying file object, and also to avoid reading the entire file into memory unnecessarily. However, part of handling the underlying file is closing it when the object falls out of scope.
The caller is not supposed to know or care about the details involved in the filehandler. It is the filehandler's job to release any necessary resources involved when it falls out of scope. That's one of the reasons it was abstracted in the first place. So, I seem to be faced with moving the filehandler code into the calling object, or dealing with a leaky abstraction.
Thoughts?
__del__ is not, by itself, a bad thing. You just have to be extra careful to not create reference cycles in objects that have __del__ defined. If you do find yourself needing to create cycles (parent refers to child which refers back to parent) then you will want to use the weakref module.
So, __del__ is okay, just be wary of cylic references.
Garbage collection: The important point here is that when an object goes out of scope, it can be garbage collected, and in fact, it will be garbage collected... but when? There is no guarantee on the when, and different Python implementations have different characteristics in this area. So for managing resources, you are better off being explicit and either adding .close() on your filehandler or, if your usage is compatible, adding __enter__ and __exit__ methods.
The __enter__ and __exit__ methods are described here. One really nice thing about them is that __exit__ is called even when exceptions occur, so you can count or your resources being closed gracefully.
Your code, enhanced for __enter__/__exit__:
class fileHandler:
def __init__(self, dbf):
self.logger = logging.getLogger('fileHandler')
self.thefilename = dbf
def __enter__(self):
self.thefile = open(self.thefilename, 'rb')
return self
def __exit__(self, *args):
self.thefile.close()
Note that the file is being opened in __enter__ instead of __init__ -- this allows you to create the filehandler object once, and then use it whenever you need to in a with without recreating it:
fh = filehandler('some_dbf')
with fh:
#file is now opened
#do some stuff
#file is now closed
#blah blah
#need the file again, so
with fh:
# file is open again, do some stuff with it
#etc, etc
As you've written it the class doesn't make the file close any more reliably. If you simple drop the filehandler instance on the floor then the file won't close until the object is destroyed. This might be immediately or might not be until the object is garbage collected, but just dropping a plain file object on the floor would close it just as quickly. If the only reference to thefile is from inside your class object then when filehandler is garbage collected thefile will be also be garbage collected and therefore closed at the same time.
The correct way to use files is to use the with statement:
with open(dbf, 'rb') as thefile:
do_something_with(thefile)
that will guarantee that thefile is always closed whenever the with clause exits. If you want to wrap your file inside another object you can do that too by defining __enter__ and __exit__ methods:
class FileHandler:
def __init__(self, dbf):
self.logger = logging.getLogger('fileHandler')
self.thefile = open(dbf, 'rb')
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
self.thefile.close()
and then you can do:
with FileHandler(dbf) as fh:
do_something_with(fh)
and be sure the file gets closed promptly.