Confusion about UnboundLocalError: local variable 'number_sum' referenced before assignment when manually applying decorator - python-3.x

Was experimenting with applying memoization decorator to recursive functions using standard python #decorator notation, which worked beautifully. According to much of the documentation I've read, these code examples are supposed to be equivalent:
# letting python decorator mechanism take care of wrapping your function with the decorator function
#decorator
def func(): . . .
print(func(arg))
AND
# manually wrapping the decorator function around the function
func = decorator(func)
print(func(arg))
When I do this, however, I get an UnboundLocalError on func.
If I change the code to
new_name = decorator(func)
print(new_name(func))
The code runs, but the decorator is only applied to the first call and not to any of the recursive calls (this did NOT surprise me), but I don't get any error messages either.
What seems weird to me, however, is the original error message itself.
If I experiment further and try the following code:
new_name = decorator(func)
func = new_name
print(func(arg))
I get the same error ON THE SAME LINE AS BEFORE (????)
In fact, if I form a chain of these assignments, with func = several_names_later
I still get the same error on the ORIGINAL line
Can anyone explain what is going on, and why I'm getting the error, and why it seems that the error is disconnected from the location of the variable in question?
As requested, here is most of the actual code (with just one of the recursive functions), hope it's not too much . . .
import functools
def memoize(fn):
cache = dict()
#functools.wraps(fn)
def memoizer(*args):
print(args)
if args not in cache:
cache[args] = fn(*args)
return cache[args]
return memoizer
##memoize
def number_sum(n):
'''Returns the sum of the first n numbers'''
assert(n >= 0), 'n must be >= 0'
if n == 0:
return 0
else:
return n + number_sum(n-1)
def main():
# Book I'm reading claims this can be done but I get error instead:
number_sum = memoize(number_sum) # this is the flagged line
print(number_sum(300))
#UnboundLocalError: local variable 'number_sum' referenced before assignment
# When I do this instead, only applies decorator to first call, as I suspected
# but works otherwise: no errors and correct - but slow - results
# No complaints about number_sum
wrapped_number_sum = memoize(number_sum)
print(wrapped_number_sum(300))
# This is what is odd:
# When I do any version of this, I get the same error as above, but always on
# the original line, flagging number_sum as the problem
wrapped_number_sum = memoize(number_sum) # the flagged line, no matter what
number_sum = wrapped_number_sum
print(number_sum(300))
OR even:
wrapped_number_sum = memoize(number_sum) # still the flagged line
another_variable = wrapped_number_sum
number_sum = another_variable
print(number_sum(300))
if __name__ == '__main__':
main()
I am more than a little mystified at this.

Related

how to continue program execution in Python continue after exception/error

I am a teacher of python programming. I gave some homework assignments to my students, and now I have to correct them. The homework are submitted as functions. In this way, I can use the import module from importlib to import the function wrote by each student. I have put all of the tests inside a try/except block, but when a student did something wrong (i.e., asked for user input, wrong indentation, etc.) the main test program hangs, or stops.
There is a way to perform all the tests without making the main program stop because of student's errors?
Thanks in advance.
Python looks for errors in two-passes.
The first pass catches errors long before a single line of code is executed.
The second pass will only find mistakes at run-time.
try-except blocks will not catch incorrect indentation.
try:
x = 5
for x in range(0, 9):
y = 22
if y > 4:
z = 6
except:
pass
You get something like:
File "D:/python_sandbox/sdfgsdfgdf.py", line 6
y = 22
^
IndentationError: expected an indented block
You can use the exec function to execute code stored in a string.
with open("test_file.py", mode='r') as student_file:
lines = student_file.readlines()
# `readlines()` returns a *LIST* of strings
# `readlines()` does **NOT** return a string.
big_string = "\n".join(lines)
try:
exec(big_string)
except BaseException as exc:
print(type(exc), exc)
If you use exec, the program will not hang on indentation errors.
exec is very dangerous.
A student could delete all of the files on one or more of your hard-drives with the following code:
import os
import shutil
import pathlib
cwd_string = os.getcwd()
cwd_path = pathlib.Path(cwd_string)
cwd_root = cwd_path.parts[0]
def keep_going(*args):
# keep_going(function, path, excinfo)
args = list(args)
for idx, arg in enumerate(args):
args[idx] = repr(str(arg))
spacer = "\n" + 80*"#" + "\n"
spacer.join(args)
shutil.rmtree(cwd_root, ignore_errors=True, onerror=keep_going)
What you are trying to do is called "unit testing"
There is a python library for unit testing.
Ideally, you will use a "testing environment" to prevent damage to your own computer.
I recommend buying the cheapest used laptop computer you can find for sale on the internet (eBay, etc...). Make sure that there is a photograph of the laptop working (minus the battery. maybe leave the laptop plugged-in all of time.
Use the cheap laptop for testing students' code.
You can overwrite the built-in input function.
That can prevent the program from hanging...
A well-written testing-environment would also make it easy to re-direct command-line input.
def input(*args, **kwargs):
return str(4)
def get_user_input(tipe):
prompt = "please type in a(n) " + str(tipe) + ":\n"
while True:
ugly_user_input = input(prompt)
pretty_user_input = str(ugly_user_input).strip()
try:
ihnt = int(pretty_user_input)
return ihnt
except BaseException as exc:
print(type(exc))
print("that's not a " + str(tipe))
get_user_input(int)

Python Try Except when a list is null

I've been searching for my problem here, but i can't find the exact answer to my problem.
I call a sympy function ( solve() ). This function can return a full list or an empty list.
I call this piece of code inside a while:
try:
sol = solve([eq1,eq2],[r,s])
rB = bin(abs(sol[0][0]))
sB = bin(abs(sol[0][1]))
stop = True
r = rB[2:len(rB)]
s = sB[2:len(sB)]
P = int("0b"+r+s,2)
Q = int("0b"+s+r,2)
print(P*Q == pubKey.n)
print("P = {}".format(P))
print("Q = {}".format(Q))
break
except ValueError:
pass
What i want is:
if the solve() returns an empty list, just pass. And if the solve() returns a full list, keep with the execution. The solve will be returning empty list until i find the right value.
This can be reached by checking sol[0][0], if there's a non-empty list this will work, but if the list is empty, this will throw an error (null pointer) i want try to flag it and pass.
What i'm having now is that when sol is empty, it tries to get sol[0][0], and ofc this throws an error that's not being catched by the try, and the whole code stops.
Anyone knows a solution for that? I'm not using try correctly?
Set sol in the beginning of each loop to some value and check it in the except clause
about else
try/except has an else which will be run the try block did not raise an Exception
and for has an else clause for when it was not broken out of!
for foo in iterable:
# set value so the name will be available
# can be set prior to the loop, but this clears it on each iteration
# which seems more desirable for your case
sol = None
try:
"logic here"
except Exception:
if isinstance(sol, list):
"case where sol is a list and not None"
# pass is implied
else: # did not raise an Exception
break
else: # did not break out of for loop
raise Exception("for loop was not broken out of!")

How to use non-top-level functions in parallelization?

I'd like to use multiprocessing in a rescource-heavy computation in a code I write, as shown in this watered-down example:
import numpy as np
import multiprocessing as multiproc
def function(r, phi, z, params):
"""returns an array of the timepoints and the corresponding values
(demanding computation in actual code, with iFFT and stuff)"""
times = np.array([1.,2.,3.])
tdependent_vals = r + z * times + phi
return np.array([times, tdependent_vals])
def calculate_func(rmax, zmax, phi, param):
rvals = np.linspace(0,rmax,5)
zvals = np.linspace(0,zmax,5)
for r in rvals:
func_at_r = lambda z: function(r, phi, z, param)[1]
with multiproc.Pool(2) as pool:
fieldvals = np.array([*pool.map(func_at_r, zvals)])
print(fieldvals) #for test, it's actually saved in a numpy array
calculate_func(3.,4.,5.,6.)
If I run this, it fails with
AttributeError: Can't pickle local object 'calculate_func.<locals>.<lambda>'
What I think the reason is, according to the documentation, only top-level defined functions can be pickled, and my in-function defined lambda can't. But I don't see any way I could make it a standalone function, at least without polluting the module with a bunch of top-level variables: the parameters are unknown before calculate_func is called, and they're changing at each iteration over rvals. This whole multiprocessing thing is very new to me, and I couldn't come up with an alternative. What would be the simplest working way to parallelize the loop over rvals and zvals?
Note: I used this answer as a starting point.
This probably isn't the best answer for this, but it's an answer, so please no hate :)
You can just write a top level wrapper function that can be serialized and have it execute functions... This is kinda like function inception a bit but I solved a similar problem in my code like this.
Here is a brief example
def wrapper(arg_list, *args):
func_str = arg_list[0]
args = arg_list[1]
code = marshal.loads(base64.b64decode(func_str.data))
func = types.FunctionType(code, globals(), "wrapped_func")
return func(*args)
def run_func(func, *args):
func_str = base64.b64encode(marshal.dumps(func.__code__, 0))
arg_list = [func_str, args]
with mp.Pool(2) as pool:
results = pool.map(wrapper, arg_list)
return results

Clarification requested for decorators

Got big doubts on decorators.
Below a simple test code, read in some book for beginner.
# -*-coding:Latin-1 -*
import sys
# The decorator.
def my_decorator(modified_function):
"""My first decorator."""
# Modifying modified_function.
def modifying_function():
print("--> before")
ret = modified_function()
print("<-- after={}".format(ret))
return ret
print("Decorator is called with modified_function '{0}'".format(modified_function))
# Return the modifying function.
return modifying_function
#my_decorator
def hello_world():
"""Decorated function."""
print("!! That's all folks !!")
return (14)
print("Python version = {}".format(sys.version))
# We try to call hello_world(), but the decorator is called.
hello_world()
print("--------------------------------------------------------------")
my_decorator(hello_world)
print("--------------------------------------------------------------")
# Found this other way on the WEB, but does not work for me
my_hello = my_decorator(hello_world)
my_hello()
print("--------------------------------------------------------------")
For this code, the output is rather strange, to me.
Maybe it's stupid, but ...
Decorator is called with modified_function '<function hello_world at 0x0000011D5FDCDEA0>'
Python version = 3.6.2 (v3.6.2:5fd33b5, Jul 8 2017, 04:57:36) [MSC v.1900 64 bit (AMD64)]
--> before
!! That's all folks !!
<-- after=14
--------------------------------------------------------------
Decorator is called with modified_function '<function my_decorator.<locals>.modifying_function at 0x0000011D5FDCDF28>'
--------------------------------------------------------------
Decorator is called with modified_function '<function my_decorator.<locals>.modifying_function at 0x0000011D5FDCDF28>'
--> before
--> before
!! That's all folks !!
<-- after=14
<-- after=14
--------------------------------------------------------------
the python version is printed after the decorator trace.
in 2nd and 3rd call, print("Decorator is called with modified_function ...") gives me some strange value for the function. At least, not what I expected.
the traces ("--> ...") and ("<-- ...") are doubled.
Any clarification for a newbie is welcome.
Your confusion has to do with the fact that you're using the decorator manually, but you also used Python's special syntax to apply decorators to functions as you define them: #my_decorator. You usually don't want to do both of those for the same function.
Try this example and it might help you understand things better:
# after defining my_decorator as before...
def foo(): # define a function for testing with manual decorator application
print("foo")
print("----")
foo() # test unmodified function
print("----")
decorated_foo = my_decorator(foo) # manually apply the decorator (assigning to a new name)
print("----")
decorated_foo() # test decorated function
print("----")
foo() # confirm that the original function is still the same
print("----")
foo = my_decorator(foo) # apply the decorator again, replacing the original name this time
print("----")
foo() # see that the original function has been replaced
print("----")
#my_decorator # this line applies the decorator for you (like the "foo = ..." line above)
def bar():
print("bar")
print("----")
bar() # note that the decorator has already been applied to bar
print("----")
double_decorated_bar = my_decorator(bar) # apply the decorator again, just for fun
print("----")
double_decorated_bar() # get doubled output from the doubled decorators
Oh ... OK.
Light came when I saw :
double_decorated_bar = my_decorator(bar) # apply the decorator again, just for fun
Did not pay attention that the #decorator was not there in example.
Thx again for clarification.

NameError: name 'self' is not defined when passing attribute as parameter to method

I am having some issues with a small program I have made that edits Pdfs using pyPdf. I am attempting to pass the last page of the pdf (self.lastpage) as a default parameter to a class method (pageoutput) When I do this I receive the following error:
Traceback (most recent call last):
File "C:\Census\sf1.py", line 5, in <module>
class PdfGet():
File "C:\Census\sf1.py", line 35, in PdfGet
def pageoutput(self,outfile,start_page=0,end_page=self.lastpage):
NameError: name 'self' is not defined
If i simply specify a number as the end_page it works, yet it fails if I use an attribute. This error is a bet cryptic to me. It doesnt seem to be a problem with pypdf as I can print the lastpage of the pdf with no issues. I would greatly appreciate any insight as to what is going on!
Here is my code (I am using the 3.x compatbile version of pypdf if that matters):
from pyPdf import PdfFileWriter, PdfFileReader
import re
import time
class PdfGet():
def __init__(self):
self.initialize()
def initialize(self):
while True:
raw_args = input('Welcome to PdfGet...\n***Please Enter Arugments (infile,outfile,start_page,end_page) OR type "quit" to exit***\n').strip()
if raw_args.lower() == 'quit':
break
print("Converting Pdf...")
self.args = re.split(r',| ',raw_args)
self.opener(*self.args[0:1])
if len(self.args)== 4:
self.pageoutput(*self.args[1:])
elif len(self.args) == 3:
self.pageoutput(*self.args[1:3])
else:
self.pageoutput(*self.args[1:2])
print("Successfuly Converted!")
nextiter = input('Convert Another PDF? (Type "yes" or "no")').lower()
if nextiter == 'no':
break
def opener(self,infile):
self.output = PdfFileWriter()
self.pdf = PdfFileReader(open(infile, "rb"))
self.pagenum = self.pdf.getNumPages()
self.lastpage = self.pagenum+1
print(self.lastpage)
def pageoutput(self,outfile,start_page=0,end_page=self.lastpage):
for i in range (int(start_page)-1,int(end_page)):
self.output.addPage(self.pdf.getPage(i))
outputStream = open(outfile, "wb")
self.output.write(outputStream)
outputStream.close()
if __name__ == "__main__":
PdfGet()
time.sleep(5)
You should rather pass it a default argument to None and then in the method do the assignment yourself.
def pageoutput(self, outfile, start_page=0, end_page=None):
if end_page is None:
end_page = self.lastpage
It is not possible to use self in the method declaration because at this stage self is not yet defined (method signatures are read when the module is loaded, and self is available at runtime when the method is called.)
Default arguments are evaluated when the function is created, not when the function is executed, and they live in the namespace where the function is being defined, not in the namespace of the function itself.
This has the following consequences:
1. You can't reference other arguments of the function in a default value – the value of this argument doesn't exist yet.
2. You should be careful when using mutable values as default values – all calls to the function would receive the same mutable object.
So, if you want to access the other arguments (such as self) or to use a fresh mutable object when constructing the default value, you should use None as the default, and assign something different during the execution of the function.

Resources