Working with python3, I had a requirement:
Perform some pre-work
Do the core work
Cleanup the pre-work
Taking inspiration from fixtures in pytest I came across this post and wrote some crazy code.
Though this crazy code works, I wish to understand the yield sorcery that makes it working :)
def db_connect_n_clean():
db_connectors = []
def _inner(db_obj):
db_connectors.append(db_obj)
print("Connect : ", db_obj)
yield _inner
for conn in db_connectors:
print("Dispose : ", conn)
This is the driver code:
pre_worker = db_connect_n_clean()
freaky_function = next(pre_worker)
freaky_function("1")
freaky_function("2")
try:
next(pre_worker)
except:
pass
It produces this output:
Connect : 1
Connect : 2
Dispose : 1
Dispose : 2
Traceback (most recent call last):
File "junk.py", line 81, in <module>
next(pre_worker)
StopIteration
What confuses me in this code is, that all the calls to the same generator freaky_func is maintaining a single list of db_connectors
After the first yield, all the objects are disposed and I hit StopIteration
I was thinking that calling freaky_func twice would maintain 2 separate lists and there would be 2 separate yields
Update: The goal of this question is not to understand how to achieve this. As it is evident from the comments, context-manager is the way to go. But my question is to understand how this piece of code is working. Basically, the python side of it.
One of my favorite tools to visualize Python with is PythonTutor.
Basically, you can see that on the first run next(pre_worker) returns the _inner function. Since _inner is inside db_connect_n_clean, it can access all of its variables.
Internally, in Python, _inner contains a reference to db_connectors. You can see the reference under __closure__:
>>> gen = db_connect_n_clean()
>>> inner = next(gen)
>>> inner.__closure__
(<cell at 0x000001B73FE6A3E0: list object at 0x000001B73FE87240>,)
>>> inner.__closure__[0].cell_contents
[]
The name of the reference is the same as the variable:
>>> inner.__code__.co_freevars
('db_connectors',)
Every time this specific function, with this specific __closure__ tries to access the db_connectors, it goes to the same list.
>>> inner(1)
Connect : 1
>>> inner(2)
Connect : 2
>>> inner.__closure__[0].cell_contents
[1, 2]
The original generator gen() is still paused at the first yield:
>>> gen.gi_frame.f_lineno
6 # Gen is stopped at line #6
>>> gen.gi_frame.f_locals["db_connectors"]
[1, 2]
When you advance it again using next() it continues on from the yield and closes everything:
>>> next(gen)
Dispose : 1
Dispose : 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
If you wish to understand how do generators work in general, there are plenty of answers and articles on the subject. I wrote this one for example.
If I didn't fully explain the situation, feel free to ask for clarification in the comments!
Related
First, let me disclaim that I am extremely new to the coding world and work requires me to use Python going forward. My experience is limited to having just completed SANS 573.
I'm trying to obtain the Date and Time from image files in their various formats. Excuse the example, I'm just using this to try and get it working.
This what I currently have:
from pathlib import Path
from PIL import Image
L88 = Path("L88.jpg")
def corvette(L88):
imgobj=Image.open(L88)
info=imgobj._getexif()
return info[36867]
>>> corvette(L88)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in corvette
KeyError: 36867
>>>
I am running the function from the desktop which is where I have the image currently saved.
I don't understand the error messages.
KeyError: 36867 also has me confused too as when I looked up the tag, that is what I found and what worked when I just did my course.
The _getexif() method returns a Python datastructury called a dictionary.
The values in a dictionary are accessed by their key.
In this case, the keys are numbers.
Not all keys are mandatory. You can use the keys() method to see which ones exist. Try:
print(info.keys())
to see which keys exist.
You can also test if a key exists:
if 36867 in info:
return info[36867]
else:
return None
Note that there is a mapping between these numbers and their exif tags names available.
You can use that to create a readable dictionary.
Note that not all JPEG images have EXIF information. In that case, the _getexif() method returns None, so you should take that into account and test for that:
from PIL import Image, ExifTags
def image_info(path):
imgobj = Image.open(path)
info = imgobj._getexif()
if info is None:
return None
rv = {
ExifTags.TAGS[key]: info[key]
for key in info.keys()
}
return rv
Assume I delete all my builtins using __builtins__={} Is there a way to force the reimport of all base modules? I thought about using _frozen_importlib.BuiltinImporter, could that be done using that method?
The goal is to see whether it can be done and how to extricate yourself from such a situation. It has no use beyond simply learning esoteric parts of python. This is python3 btw.
Your question was proposed as a puzzle in the Python chat room, back in March 2018.
The following solution, given by user Aran-Fey, restores the builtins module afresh in the REPL:
__builtins__ = __loader__.create_module(__loader__.find_spec('builtins'))
A more difficult variant, additionally clearing the globals() namespace and thus denying access to __loader__, was also proposed (and solved) in the chat room.
Note that setting __builtins__ = {} only disables access to the builtin namespace within an interactive REPL, not within a script, and the lookup in __builtins__ is considered a CPython implementation detail.
One highly impractical method of accessing the builtins module when its name is inaccessible is described at the bottom of https://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html.
Update: from a thread on Reddit about recovering cleared globals, a similar snippet that will get you the original builtins:
[
c for c in ().__class__.__base__.__subclasses__()
if c.__name__ == 'catch_warnings'
][0]()._module.__builtins__
By reassigning this object to __builtins__, the built-in objects become accessible again.
>>> __builtins__ = {}
>>> max
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'max' is not defined
>>> __builtins__ = [c for c in ().__class__.__base__.__subclasses__() if c.__name__ == 'catch_warnings'][0]()._module.__builtins__
>>> max
<built-in function max>
Quick explanation of the components of the statement:
() #tuple instance
().__class__ #the tuple class
().__class__.__base__ #tuple's base class, i.e. `object`
().__class__.__base__.__subclasses__() #every class that inherits from `object`
[c for c in ().__class__.__base__.__subclasses__() if c.__name__ == 'catch_warnings'][0] #the `warnings.catch_warnings` class
[c for c in ().__class__.__base__.__subclasses__() if c.__name__ == 'catch_warnings'][0]()._module #create a `catch_warnings` instance and access its `_module` attribute, which is the `warnings` module.
[c for c in ().__class__.__base__.__subclasses__() if c.__name__ == 'catch_warnings'][0]()._module.__builtins__ #`warnings.__builtins__` is the builtins module
__builtins__ = [c for c in ().__class__.__base__.__subclasses__() if c.__name__ == 'catch_warnings'][0]()._module.__builtins__ #assign result to `__builtins__`
As #Kevin noted in the comments, this seems to be different between the REPL and a script. In a script, reassigning __builtins__ seems to have no effect (Python 3.7.3):
# test.py
__builtins__ = {}
print(max([1,2,3]))
def really_weird():
__builtins__ = {"max": lambda x: min(x)}
print(max([1,2,3]))
really_weird()
# Output
3
3
max didn't change, even when __builtins__ reassigned the name (sort of).
In the REPL, you are correct that _frozen_importlib.BuiltinImporter can be used.
>>> import _frozen_importlib # This must be before reassigning __builtins___, since __builtins__ includes __import__
>>> __builtins__ = {}
>>> max([1,2,3])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'max' is not defined
>>> __builtins__ = _frozen_importlib.BuiltinImporter().load_module("builtins")
>>> max([1,2,3])
3
One thing to note: The module name is builtins, not __builtins__.
Also, as is noted in the comments, doing import builtins as b; ... __builtins__ = b or something similar also works.
If you don't want to do setup work, #Kevin's answer does the same without previous imports.
Why the following code produces an error
>>> object().foo = 'bar'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'object' object has no attribute 'foo'
while the next one works fine?
class A():
pass
A().foo = 'bar'
What is the exact difference between A and object? I believe o().foo = 'bar' leads to setattr('o', 'foo', 'bar') and this in turn results in o.__setattr__('foo', 'bar'). One would expect them to have identic __setattr__ methods since no overriding happens. Yet the outputs are different. Please explain why. What happens behind the scenes?
A similar pattern can be noticed for built-in functions and user-defined ones. I can't set my own attributes for let's say dict but it's perfectly ok to write (lambda:None).foo = 'bar'. What's going on here?
I've been having some issues with my code, you see I am a beginner at python programming and so I don't understand all the errors, So I would be quite happy for assistance down below is the code, which I have checked intensively just to find the error:
class Animal(object):
def __init__(self,legs,name):
def sleep(self,hours):
print("%s is sleeping for %d hours!" % (self.name,hours))
self.legs = legs
self.name = name
roscoe = Animal(4, "Canis Lupus Familiaris")
roscoe.name = ("Roscoe")
roscoe.sleep(4)
This is the Error:
Traceback (most recent call last):
File "class.py", line 9, in <module>
roscoe.sleep(4)
AttributeError: 'Animal' object has no attribute 'sleep'
You have a syntax error in the last line.
It should be:
roscoe.sleep(4)
instead of
roscue.sleep(4)
Giving more context since I see you're a begginner at Python. The traceback of Python interpreter (the "program" that runs Python code) tells you what happened. In this case, it says "name 'roscue' is not defined". This is usually a syntax error. Sometimes it can mean that you haven't defined that function. But in this case, it's the former.
Also, going a little bit further, you're probably going to get an error of indentation. In Python, you have to indent every block that you want to put together, either with tabs or with spaces.
Finally, think about your functions, you have to put them in order. Init is a function, and sleep is another function, so after each one you have a block. Different blocks should be indented separately. Here's how the code should look, but revise it instead of running it blindly.
class Animal(object):
def __init__(self,legs,name):
self.legs = legs
self.name = name
def sleep(self,hours):
print("%s is sleeping for %d hours!" % (self.name,hours))
roscoe = Animal(4, "Canis Lupus Familiaris")
roscoe.name = ("Roscoe")
roscoe.sleep(4)
Not very new to programming or to python but incredibly green to using pyunit. Need to use it for my new job and I keep getting this error but only sometimes when it is run. My code below.
import unittest
from nose_parameterized import parameterized
from CheckFromFile import listFileCheck, RepresentsFloat
testParams = listFileCheck()
class TestSequence(unittest.TestCase):
#parameterized.expand(testParams)
def test_sequence(self, name, a, b):
if RepresentsFloat(a):
self.assertAlmostEqual(a,b,2)
else:
self.assertEqual(a,b)
if __name__ == '__main__':
unittest.main()
What is happening here is that my test case is pulling a method listFileCheck from another class. What it does is it reads values from the serial port communicating with the control board and compares them with a calibration file. It puts the control board values in an MD array along with the calibration file values. These values can be either str, int, or float.
I used the test case to compare the values to one another however I keep getting this error but only sometimes. After every 3rd or so run it fails with this error.
Error
Traceback (most recent call last):
File "C:\Python34\lib\unittest\case.py", line 57, in testPartExecutor
yield
File "C:\Python34\lib\unittest\case.py", line 574, in run
testMethod()
TypeError: 'NoneType' object is not callable
Process finished with exit code 0
Anyone know why I might be getting this error on occasion?