Assume I delete all my builtins using __builtins__={} Is there a way to force the reimport of all base modules? I thought about using _frozen_importlib.BuiltinImporter, could that be done using that method?
The goal is to see whether it can be done and how to extricate yourself from such a situation. It has no use beyond simply learning esoteric parts of python. This is python3 btw.
Your question was proposed as a puzzle in the Python chat room, back in March 2018.
The following solution, given by user Aran-Fey, restores the builtins module afresh in the REPL:
__builtins__ = __loader__.create_module(__loader__.find_spec('builtins'))
A more difficult variant, additionally clearing the globals() namespace and thus denying access to __loader__, was also proposed (and solved) in the chat room.
Note that setting __builtins__ = {} only disables access to the builtin namespace within an interactive REPL, not within a script, and the lookup in __builtins__ is considered a CPython implementation detail.
One highly impractical method of accessing the builtins module when its name is inaccessible is described at the bottom of https://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html.
Update: from a thread on Reddit about recovering cleared globals, a similar snippet that will get you the original builtins:
[
c for c in ().__class__.__base__.__subclasses__()
if c.__name__ == 'catch_warnings'
][0]()._module.__builtins__
By reassigning this object to __builtins__, the built-in objects become accessible again.
>>> __builtins__ = {}
>>> max
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'max' is not defined
>>> __builtins__ = [c for c in ().__class__.__base__.__subclasses__() if c.__name__ == 'catch_warnings'][0]()._module.__builtins__
>>> max
<built-in function max>
Quick explanation of the components of the statement:
() #tuple instance
().__class__ #the tuple class
().__class__.__base__ #tuple's base class, i.e. `object`
().__class__.__base__.__subclasses__() #every class that inherits from `object`
[c for c in ().__class__.__base__.__subclasses__() if c.__name__ == 'catch_warnings'][0] #the `warnings.catch_warnings` class
[c for c in ().__class__.__base__.__subclasses__() if c.__name__ == 'catch_warnings'][0]()._module #create a `catch_warnings` instance and access its `_module` attribute, which is the `warnings` module.
[c for c in ().__class__.__base__.__subclasses__() if c.__name__ == 'catch_warnings'][0]()._module.__builtins__ #`warnings.__builtins__` is the builtins module
__builtins__ = [c for c in ().__class__.__base__.__subclasses__() if c.__name__ == 'catch_warnings'][0]()._module.__builtins__ #assign result to `__builtins__`
As #Kevin noted in the comments, this seems to be different between the REPL and a script. In a script, reassigning __builtins__ seems to have no effect (Python 3.7.3):
# test.py
__builtins__ = {}
print(max([1,2,3]))
def really_weird():
__builtins__ = {"max": lambda x: min(x)}
print(max([1,2,3]))
really_weird()
# Output
3
3
max didn't change, even when __builtins__ reassigned the name (sort of).
In the REPL, you are correct that _frozen_importlib.BuiltinImporter can be used.
>>> import _frozen_importlib # This must be before reassigning __builtins___, since __builtins__ includes __import__
>>> __builtins__ = {}
>>> max([1,2,3])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'max' is not defined
>>> __builtins__ = _frozen_importlib.BuiltinImporter().load_module("builtins")
>>> max([1,2,3])
3
One thing to note: The module name is builtins, not __builtins__.
Also, as is noted in the comments, doing import builtins as b; ... __builtins__ = b or something similar also works.
If you don't want to do setup work, #Kevin's answer does the same without previous imports.
Related
Working with python3, I had a requirement:
Perform some pre-work
Do the core work
Cleanup the pre-work
Taking inspiration from fixtures in pytest I came across this post and wrote some crazy code.
Though this crazy code works, I wish to understand the yield sorcery that makes it working :)
def db_connect_n_clean():
db_connectors = []
def _inner(db_obj):
db_connectors.append(db_obj)
print("Connect : ", db_obj)
yield _inner
for conn in db_connectors:
print("Dispose : ", conn)
This is the driver code:
pre_worker = db_connect_n_clean()
freaky_function = next(pre_worker)
freaky_function("1")
freaky_function("2")
try:
next(pre_worker)
except:
pass
It produces this output:
Connect : 1
Connect : 2
Dispose : 1
Dispose : 2
Traceback (most recent call last):
File "junk.py", line 81, in <module>
next(pre_worker)
StopIteration
What confuses me in this code is, that all the calls to the same generator freaky_func is maintaining a single list of db_connectors
After the first yield, all the objects are disposed and I hit StopIteration
I was thinking that calling freaky_func twice would maintain 2 separate lists and there would be 2 separate yields
Update: The goal of this question is not to understand how to achieve this. As it is evident from the comments, context-manager is the way to go. But my question is to understand how this piece of code is working. Basically, the python side of it.
One of my favorite tools to visualize Python with is PythonTutor.
Basically, you can see that on the first run next(pre_worker) returns the _inner function. Since _inner is inside db_connect_n_clean, it can access all of its variables.
Internally, in Python, _inner contains a reference to db_connectors. You can see the reference under __closure__:
>>> gen = db_connect_n_clean()
>>> inner = next(gen)
>>> inner.__closure__
(<cell at 0x000001B73FE6A3E0: list object at 0x000001B73FE87240>,)
>>> inner.__closure__[0].cell_contents
[]
The name of the reference is the same as the variable:
>>> inner.__code__.co_freevars
('db_connectors',)
Every time this specific function, with this specific __closure__ tries to access the db_connectors, it goes to the same list.
>>> inner(1)
Connect : 1
>>> inner(2)
Connect : 2
>>> inner.__closure__[0].cell_contents
[1, 2]
The original generator gen() is still paused at the first yield:
>>> gen.gi_frame.f_lineno
6 # Gen is stopped at line #6
>>> gen.gi_frame.f_locals["db_connectors"]
[1, 2]
When you advance it again using next() it continues on from the yield and closes everything:
>>> next(gen)
Dispose : 1
Dispose : 2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
If you wish to understand how do generators work in general, there are plenty of answers and articles on the subject. I wrote this one for example.
If I didn't fully explain the situation, feel free to ask for clarification in the comments!
The following code
class Foo:
def bar(self) -> None:
pass
foo = Foo()
if hasattr(foo.bar, '__annotations__'):
foo.bar.__annotations__ = 'hi'
crashes with
AttributeError: 'method' object has no attribute '__annotations__'
How can this happen?
The attribute error here is raised because you can't set any attribute on a method object:
>>> foo.bar.baz = 42
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'method' object has no attribute 'baz'
The exception here is perhaps confusing because method objects wrap a function object and proxy attribute read access to that underlying function object. So when attributes on the function exist, then hasattr() on the method will return True:
>>> hasattr(foo.bar, 'baz')
False
>>> foo.bar.__func__.baz = 42
>>> hasattr(foo.bar, 'baz')
True
>>> foo.bar.baz
42
However, you still can't set those attributes via the method, regardless:
>>> hasattr(foo.bar, 'baz')
True
>>> foo.bar.baz = 42
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'method' object has no attribute 'baz'
So, just because the attribute can be read doesn't mean you can set it. hasattr() is speaking the truth, you just interpreted it to mean something different.
Now, if you tried to set the __annotations__ attribute directly on the underlying function object you'd get another error message:
>>> foo.bar.__func__.__annotations__ = 'hi'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __annotations__ must be set to a dict object
You would want to use a dictionary object here:
>>> foo.bar.__func__.__annotations__ = {'return': 'hi'}
>>> foo.bar.__annotations__
{'return': 'hi'}
However, because __annotations__ is a mutable dictionary, it is just easier to directly manipulate the keys and values to that object, which is perfectly feasible to do via the method wrapper:
>>> foo.bar.__annotations__['return'] = 'int'
>>> foo.bar.__annotations__
{'return': 'int'}
Now, if you were hoping to set per instance annotations, you can't get away with setting attributes on method objects, because method objects are ephemeral, they are created just for the call, then usually discarded right after.
You would have to use custom method descriptor objects via a metaclass and re-create the __annotations__ attribute for those each time, or you could instead pre-bind methods with a new function object that would be given their own attributes. You then have to pay a larger memory price:
import functools
foo.bar = lambda *args, **kwargs: Foo.bar(foo, *args, **kwargs)
functools.update_wrapper(foo.bar, Foo.bar) # copy everything over to the new wrapper
foo.bar.__annotations__['return'] = 'hi'
Either way you completely kill important speed optimisations made in Python 3.7 this way.
And tools that operate on the most important use case for __annatotions__, type hints, do not actually execute code, they read code statically and would completely miss these runtime alterations.
You're getting an error. because __annotations__ is a dictionary. If you want to change values you'll have to do it like this:
if hasattr(foo.bar, '__annotations__'):
foo.bar.__annotations__['return'] = 'hi'
This will make the return value of your foo.bar be hi instead of None. The only thing I'm not sure about is how the __annotations__ are protected, not allowing you to change them from a dict to string, but I suppose it's some internal check in the source.
UPDATE
For more control over the signature you can use the inspect module and get the Signature object of your class(or method) and edit it from there. For example
import inspect
sig = inspect.signature(foo.bar)
sig.return_annotation # prints None (before modifying)
sig.replace(return_annotation="anything you want")
More on that here
OK, so I have a string, x = module.class.test_function(value), and I want to call it and get the response. I've tried to use getattr(module.class, test_function)(value) yet it gives the error:
AttributeError: module 'module' has no attribute 'test_function'
I'm new to these things in python, how would I do this?
Given a file my_module.py:
def my_func(greeting):
print(f'{greeting} from my_func!')
You can import your function and call it normally like this:
>>> from my_module import my_func
>>> my_func('hello')
hello from my_func!
Alternatively, if you want to import the function dynamically with getattr:
>>> import my_module
>>> getattr(my_module, 'my_func')
<function my_func at 0x1086aa8c8>
>>> a_func = getattr(my_module, 'my_func')
>>> a_func('bonjour')
bonjour from my_func!
I would only recommend this style if it's required by your use case, for instance, the method name to be called not being known until runtime, methods being generated dynamically, or something like that.
A good answer that explains getattr in more detail is - Why use setattr() and getattr() built-ins? and you can find a bit more at http://effbot.org/zone/python-getattr.htm.
I've been having some issues with my code, you see I am a beginner at python programming and so I don't understand all the errors, So I would be quite happy for assistance down below is the code, which I have checked intensively just to find the error:
class Animal(object):
def __init__(self,legs,name):
def sleep(self,hours):
print("%s is sleeping for %d hours!" % (self.name,hours))
self.legs = legs
self.name = name
roscoe = Animal(4, "Canis Lupus Familiaris")
roscoe.name = ("Roscoe")
roscoe.sleep(4)
This is the Error:
Traceback (most recent call last):
File "class.py", line 9, in <module>
roscoe.sleep(4)
AttributeError: 'Animal' object has no attribute 'sleep'
You have a syntax error in the last line.
It should be:
roscoe.sleep(4)
instead of
roscue.sleep(4)
Giving more context since I see you're a begginner at Python. The traceback of Python interpreter (the "program" that runs Python code) tells you what happened. In this case, it says "name 'roscue' is not defined". This is usually a syntax error. Sometimes it can mean that you haven't defined that function. But in this case, it's the former.
Also, going a little bit further, you're probably going to get an error of indentation. In Python, you have to indent every block that you want to put together, either with tabs or with spaces.
Finally, think about your functions, you have to put them in order. Init is a function, and sleep is another function, so after each one you have a block. Different blocks should be indented separately. Here's how the code should look, but revise it instead of running it blindly.
class Animal(object):
def __init__(self,legs,name):
self.legs = legs
self.name = name
def sleep(self,hours):
print("%s is sleeping for %d hours!" % (self.name,hours))
roscoe = Animal(4, "Canis Lupus Familiaris")
roscoe.name = ("Roscoe")
roscoe.sleep(4)
I wanted to use import with a variable name. For example I wanted to do something like this
from var import my_class
I went through pythons documentation, but seems thats a little confusing. Also I seen some other posting on stack overflow that give the example of something like this
import importlib
my_module = importlib.import_module("var, my_class)
This second example does work to a certain extent. The only issue I see here var is imported but I don't see the attributes of my_class in python's namespace. How would I equate this to my original example of
from var import my_class
Here's how to use importlib (there is no need for the second parameter):
var = importlib.import_module("var")
# Now, you can use the content of the module:
var.my_class()
There is no direct programmable equivalent for from var import my_class.
Note: As #DYZ points out in the comments, this way of solving this is not recommended in favor of importlib. Leaving it here for the sake of another working solution, but the Python docs advise "Direct use of import() is also discouraged in favor of importlib.import_module()."
Do you mean that you want to import a module whose name is defined by a variable? If so, you can use the __import__ method. For example:
>>> import os
>>> os.getcwd()
'/Users/christophershroba'
>>>
>>> name_to_import = "os"
>>> variable_module = __import__(name_to_import)
>>> variable_module.getcwd()
'/Users/christophershroba'
If you also want to call a variable method of that variable module you could use the __getattribute__ method on the module to get the function, and then call it with () as normal. The line below marked "See note" is not necessary, I just include it to show that the __getattribute__ method is returning a function.
>>> name_to_import = "os"
>>> method_to_call = "getcwd"
>>> variable_module = __import__(name_to_import)
>>> variable_module.__getattribute__(method_to_call) # See note
<built-in function getcwd>
>>> variable_module.__getattribute__(method_to_call)()
'/Users/christophershroba'
More documentation available for Python 3 here or Python2 here.