I have the below code.
a = 10
def f(x):
return x + a
a = 3
f(1)
If we print(f1) the result is 4. It seems that the second assignment of a is used here (a=3). Could someone explain to me if there is a particular reason or rule for not using the first assignment (a=10)?
Thank you,
Dimitrios
a and x are resolved each time return x + a is executed. You reassign a before calling f. f knows that a is not a local variable so looks up "a" in its enclosing scope. That "a" has a 3 in it, and that is what is used.
As a general rule, it would be very bad for f to use the original assignment when its executed. That would mean that functions couldn't take advange of global variables changing. It would also require that global variables be assigned before the function is even defined, which is not particularly useful for a dynamic language.
You can hack it, though. If you define a parameter with a default value, it will be assigned when the function is defined and you get what you want.
a = 10
def f(x, a=a):
return x + a
a = 3
print(f(1))
You see, you've defined a before you defined f(x). But your call to f(1) was after you assigned the value 3 to "a".
f(x) will always use the current value of "a" when it its called, not when it is defined. If you want a constant value for "a" inside f(x), you should declare it inside the function, something like:
def f(x):
a = 10
return x + a
What exactly are the Python scoping rules?
If I have some code:
code1
class Foo:
code2
def spam.....
code3
for code4..:
code5
x()
Where is x found? Some possible choices include the list below:
In the enclosing source file
In the class namespace
In the function definition
In the for loop index variable
Inside the for loop
Also there is the context during execution, when the function spam is passed somewhere else. And maybe lambda functions pass a bit differently?
There must be a simple reference or algorithm somewhere. It's a confusing world for intermediate Python programmers.
Actually, a concise rule for Python Scope resolution, from Learning Python, 3rd. Ed.. (These rules are specific to variable names, not attributes. If you reference it without a period, these rules apply.)
LEGB Rule
Local — Names assigned in any way within a function (def or lambda), and not declared global in that function
Enclosing-function — Names assigned in the local scope of any and all statically enclosing functions (def or lambda), from inner to outer
Global (module) — Names assigned at the top-level of a module file, or by executing a global statement in a def within the file
Built-in (Python) — Names preassigned in the built-in names module: open, range, SyntaxError, etc
So, in the case of
code1
class Foo:
code2
def spam():
code3
for code4:
code5
x()
The for loop does not have its own namespace. In LEGB order, the scopes would be
L: Local in def spam (in code3, code4, and code5)
E: Any enclosing functions (if the whole example were in another def)
G: Were there any x declared globally in the module (in code1)?
B: Any builtin x in Python.
x will never be found in code2 (even in cases where you might expect it would, see Antti's answer or here).
Essentially, the only thing in Python that introduces a new scope is a function definition. Classes are a bit of a special case in that anything defined directly in the body is placed in the class's namespace, but they are not directly accessible from within the methods (or nested classes) they contain.
In your example there are only 3 scopes where x will be searched in:
spam's scope - containing everything defined in code3 and code5 (as well as code4, your loop variable)
The global scope - containing everything defined in code1, as well as Foo (and whatever changes after it)
The builtins namespace. A bit of a special case - this contains the various Python builtin functions and types such as len() and str(). Generally this shouldn't be modified by any user code, so expect it to contain the standard functions and nothing else.
More scopes only appear when you introduce a nested function (or lambda) into the picture.
These will behave pretty much as you'd expect however. The nested function can access everything in the local scope, as well as anything in the enclosing function's scope. eg.
def foo():
x=4
def bar():
print x # Accesses x from foo's scope
bar() # Prints 4
x=5
bar() # Prints 5
Restrictions:
Variables in scopes other than the local function's variables can be accessed, but can't be rebound to new parameters without further syntax. Instead, assignment will create a new local variable instead of affecting the variable in the parent scope. For example:
global_var1 = []
global_var2 = 1
def func():
# This is OK: It's just accessing, not rebinding
global_var1.append(4)
# This won't affect global_var2. Instead it creates a new variable
global_var2 = 2
local1 = 4
def embedded_func():
# Again, this doen't affect func's local1 variable. It creates a
# new local variable also called local1 instead.
local1 = 5
print local1
embedded_func() # Prints 5
print local1 # Prints 4
In order to actually modify the bindings of global variables from within a function scope, you need to specify that the variable is global with the global keyword. Eg:
global_var = 4
def change_global():
global global_var
global_var = global_var + 1
Currently there is no way to do the same for variables in enclosing function scopes, but Python 3 introduces a new keyword, "nonlocal" which will act in a similar way to global, but for nested function scopes.
There was no thorough answer concerning Python3 time, so I made an answer here. Most of what is described here is detailed in the 4.2.2 Resolution of names of the Python 3 documentation.
As provided in other answers, there are 4 basic scopes, the LEGB, for Local, Enclosing, Global and Builtin. In addition to those, there is a special scope, the class body, which does not comprise an enclosing scope for methods defined within the class; any assignments within the class body make the variable from there on be bound in the class body.
Especially, no block statement, besides def and class, create a variable scope. In Python 2 a list comprehension does not create a variable scope, however in Python 3 the loop variable within list comprehensions is created in a new scope.
To demonstrate the peculiarities of the class body
x = 0
class X(object):
y = x
x = x + 1 # x is now a variable
z = x
def method(self):
print(self.x) # -> 1
print(x) # -> 0, the global x
print(y) # -> NameError: global name 'y' is not defined
inst = X()
print(inst.x, inst.y, inst.z, x) # -> (1, 0, 1, 0)
Thus unlike in function body, you can reassign the variable to the same name in class body, to get a class variable with the same name; further lookups on this name resolve
to the class variable instead.
One of the greater surprises to many newcomers to Python is that a for loop does not create a variable scope. In Python 2 the list comprehensions do not create a scope either (while generators and dict comprehensions do!) Instead they leak the value in the function or the global scope:
>>> [ i for i in range(5) ]
>>> i
4
The comprehensions can be used as a cunning (or awful if you will) way to make modifiable variables within lambda expressions in Python 2 - a lambda expression does create a variable scope, like the def statement would, but within lambda no statements are allowed. Assignment being a statement in Python means that no variable assignments in lambda are allowed, but a list comprehension is an expression...
This behaviour has been fixed in Python 3 - no comprehension expressions or generators leak variables.
The global really means the module scope; the main python module is the __main__; all imported modules are accessible through the sys.modules variable; to get access to __main__ one can use sys.modules['__main__'], or import __main__; it is perfectly acceptable to access and assign attributes there; they will show up as variables in the global scope of the main module.
If a name is ever assigned to in the current scope (except in the class scope), it will be considered belonging to that scope, otherwise it will be considered to belonging to any enclosing scope that assigns to the variable (it might not be assigned yet, or not at all), or finally the global scope. If the variable is considered local, but it is not set yet, or has been deleted, reading the variable value will result in UnboundLocalError, which is a subclass of NameError.
x = 5
def foobar():
print(x) # causes UnboundLocalError!
x += 1 # because assignment here makes x a local variable within the function
# call the function
foobar()
The scope can declare that it explicitly wants to modify the global (module scope) variable, with the global keyword:
x = 5
def foobar():
global x
print(x)
x += 1
foobar() # -> 5
print(x) # -> 6
This also is possible even if it was shadowed in enclosing scope:
x = 5
y = 13
def make_closure():
x = 42
y = 911
def func():
global x # sees the global value
print(x, y)
x += 1
return func
func = make_closure()
func() # -> 5 911
print(x, y) # -> 6 13
In python 2 there is no easy way to modify the value in the enclosing scope; usually this is simulated by having a mutable value, such as a list with length of 1:
def make_closure():
value = [0]
def get_next_value():
value[0] += 1
return value[0]
return get_next_value
get_next = make_closure()
print(get_next()) # -> 1
print(get_next()) # -> 2
However in python 3, the nonlocal comes to rescue:
def make_closure():
value = 0
def get_next_value():
nonlocal value
value += 1
return value
return get_next_value
get_next = make_closure() # identical behavior to the previous example.
The nonlocal documentation says that
Names listed in a nonlocal statement, unlike those listed in a global statement, must refer to pre-existing bindings in an enclosing scope (the scope in which a new binding should be created cannot be determined unambiguously).
i.e. nonlocal always refers to the innermost outer non-global scope where the name has been bound (i.e. assigned to, including used as the for target variable, in the with clause, or as a function parameter).
Any variable that is not deemed to be local to the current scope, or any enclosing scope, is a global variable. A global name is looked up in the module global dictionary; if not found, the global is then looked up from the builtins module; the name of the module was changed from python 2 to python 3; in python 2 it was __builtin__ and in python 3 it is now called builtins. If you assign to an attribute of builtins module, it will be visible thereafter to any module as a readable global variable, unless that module shadows them with its own global variable with the same name.
Reading the builtin module can also be useful; suppose that you want the python 3 style print function in some parts of file, but other parts of file still use the print statement. In Python 2.6-2.7 you can get hold of the Python 3 print function with:
import __builtin__
print3 = __builtin__.__dict__['print']
The from __future__ import print_function actually does not import the print function anywhere in Python 2 - instead it just disables the parsing rules for print statement in the current module, handling print like any other variable identifier, and thus allowing the print the function be looked up in the builtins.
A slightly more complete example of scope:
from __future__ import print_function # for python 2 support
x = 100
print("1. Global x:", x)
class Test(object):
y = x
print("2. Enclosed y:", y)
x = x + 1
print("3. Enclosed x:", x)
def method(self):
print("4. Enclosed self.x", self.x)
print("5. Global x", x)
try:
print(y)
except NameError as e:
print("6.", e)
def method_local_ref(self):
try:
print(x)
except UnboundLocalError as e:
print("7.", e)
x = 200 # causing 7 because has same name
print("8. Local x", x)
inst = Test()
inst.method()
inst.method_local_ref()
output:
1. Global x: 100
2. Enclosed y: 100
3. Enclosed x: 101
4. Enclosed self.x 101
5. Global x 100
6. global name 'y' is not defined
7. local variable 'x' referenced before assignment
8. Local x 200
The scoping rules for Python 2.x have been outlined already in other answers. The only thing I would add is that in Python 3.0, there is also the concept of a non-local scope (indicated by the 'nonlocal' keyword). This allows you to access outer scopes directly, and opens up the ability to do some neat tricks, including lexical closures (without ugly hacks involving mutable objects).
EDIT: Here's the PEP with more information on this.
Python resolves your variables with -- generally -- three namespaces available.
At any time during execution, there
are at least three nested scopes whose
namespaces are directly accessible:
the innermost scope, which is searched
first, contains the local names; the
namespaces of any enclosing functions,
which are searched starting with the
nearest enclosing scope; the middle
scope, searched next, contains the
current module's global names; and the
outermost scope (searched last) is the
namespace containing built-in names.
There are two functions: globals and locals which show you the contents two of these namespaces.
Namespaces are created by packages, modules, classes, object construction and functions. There aren't any other flavors of namespaces.
In this case, the call to a function named x has to be resolved in the local name space or the global namespace.
Local in this case, is the body of the method function Foo.spam.
Global is -- well -- global.
The rule is to search the nested local spaces created by method functions (and nested function definitions), then search global. That's it.
There are no other scopes. The for statement (and other compound statements like if and try) don't create new nested scopes. Only definitions (packages, modules, functions, classes and object instances.)
Inside a class definition, the names are part of the class namespace. code2, for instance, must be qualified by the class name. Generally Foo.code2. However, self.code2 will also work because Python objects look at the containing class as a fall-back.
An object (an instance of a class) has instance variables. These names are in the object's namespace. They must be qualified by the object. (variable.instance.)
From within a class method, you have locals and globals. You say self.variable to pick the instance as the namespace. You'll note that self is an argument to every class member function, making it part of the local namespace.
See Python Scope Rules, Python Scope, Variable Scope.
Where is x found?
x is not found as you haven't defined it. :-) It could be found in code1 (global) or code3 (local) if you put it there.
code2 (class members) aren't visible to code inside methods of the same class — you would usually access them using self. code4/code5 (loops) live in the same scope as code3, so if you wrote to x in there you would be changing the x instance defined in code3, not making a new x.
Python is statically scoped, so if you pass ‘spam’ to another function spam will still have access to globals in the module it came from (defined in code1), and any other containing scopes (see below). code2 members would again be accessed through self.
lambda is no different to def. If you have a lambda used inside a function, it's the same as defining a nested function. In Python 2.2 onwards, nested scopes are available. In this case you can bind x at any level of function nesting and Python will pick up the innermost instance:
x= 0
def fun1():
x= 1
def fun2():
x= 2
def fun3():
return x
return fun3()
return fun2()
print fun1(), x
2 0
fun3 sees the instance x from the nearest containing scope, which is the function scope associated with fun2. But the other x instances, defined in fun1 and globally, are not affected.
Before nested_scopes — in Python pre-2.1, and in 2.1 unless you specifically ask for the feature using a from-future-import — fun1 and fun2's scopes are not visible to fun3, so S.Lott's answer holds and you would get the global x:
0 0
The Python name resolution only knows the following kinds of scope:
builtins scope which provides the Builtin Functions, such as print, int, or zip,
module global scope which is always the top-level of the current module,
three user-defined scopes that can be nested into each other, namely
function closure scope, from any enclosing def block, lambda expression or comprehension.
function local scope, inside a def block, lambda expression or comprehension,
class scope, inside a class block.
Notably, other constructs such as if, for, or with statements do not have their own scope.
The scoping TLDR: The lookup of a name begins at the scope in which the name is used, then any enclosing scopes (excluding class scopes), to the module globals, and finally the builtins – the first match in this search order is used.
The assignment to a scope is by default to the current scope – the special forms nonlocal and global must be used to assign to a name from an outer scope.
Finally, comprehensions and generator expressions as well as := asignment expressions have one special rule when combined.
Nested Scopes and Name Resolution
These different scopes build a hierarchy, with builtins then global always forming the base, and closures, locals and class scope being nested as lexically defined. That is, only the nesting in the source code matters, not for example the call stack.
print("builtins are available without definition")
some_global = "1" # global variables are at module scope
def outer_function():
some_closure = "3.1" # locals and closure are defined the same, at function scope
some_local = "3.2" # a variable becomes a closure if a nested scope uses it
class InnerClass:
some_classvar = "3.3" # class variables exist *only* at class scope
def inner_function(self):
some_local = "3.2" # locals can replace outer names
print(some_closure) # closures are always readable
return InnerClass
Even though class creates a scope and may have nested classes, functions and comprehensions, the names of the class scope are not visible to enclosed scopes. This creates the following hierarchy:
┎ builtins [print, ...]
┗━┱ globals [some_global]
┗━┱ outer_function [some_local, some_closure]
┣━╾ InnerClass [some_classvar]
┗━╾ inner_function [some_local]
Name resolution always starts at the current scope in which a name is accessed, then goes up the hierarchy until a match is found. For example, looking up some_local inside outer_function and inner_function starts at the respective function - and immediately finds the some_local defined in outer_function and inner_function, respectively. When a name is not local, it is fetched from the nearest enclosing scope that defines it – looking up some_closure and print inside inner_function searches until outer_function and builtins, respectively.
Scope Declarations and Name Binding
By default, a name belongs to any scope in which it is bound to a value. Binding the same name again in an inner scope creates a new variable with the same name - for example, some_local exists separately in both outer_function and inner_function. As far as scoping is concerned, binding includes any statement that sets the value of a name – assignment statements, but also the iteration variable of a for loop, or the name of a with context manager. Notably, del also counts as name binding.
When a name must refer to an outer variable and be bound in an inner scope, the name must be declared as not local. Separate declarations exists for the different kinds of enclosing scopes: nonlocal always refers to the nearest closure, and global always refers to a global name. Notably, nonlocal never refers to a global name and global ignores all closures of the same name. There is no declaration to refer to the builtin scope.
some_global = "1"
def outer_function():
some_closure = "3.2"
some_global = "this is ignored by a nested global declaration"
def inner_function():
global some_global # declare variable from global scope
nonlocal some_closure # declare variable from enclosing scope
message = " bound by an inner scope"
some_global = some_global + message
some_closure = some_closure + message
return inner_function
Of note is that function local and nonlocal are resolved at compile time. A nonlocal name must exist in some outer scope. In contrast, a global name can be defined dynamically and may be added or removed from the global scope at any time.
Comprehensions and Assignment Expressions
The scoping rules of list, set and dict comprehensions and generator expressions are almost the same as for functions. Likewise, the scoping rules for assignment expressions are almost the same as for regular name binding.
The scope of comprehensions and generator expressions is of the same kind as function scope. All names bound in the scope, namely the iteration variables, are locals or closures to the comprehensions/generator and nested scopes. All names, including iterables, are resolved using name resolution as applicable inside functions.
some_global = "global"
def outer_function():
some_closure = "closure"
return [ # new function-like scope started by comprehension
comp_local # names resolved using regular name resolution
for comp_local # iteration targets are local
in "iterable"
if comp_local in some_global and comp_local in some_global
]
An := assignment expression works on the nearest function, class or global scope. Notably, if the target of an assignment expression has been declared nonlocal or global in the nearest scope, the assignment expression honors this like a regular assignment.
print(some_global := "global")
def outer_function():
print(some_closure := "closure")
However, an assignment expression inside a comprehension/generator works on the nearest enclosing scope of the comprehension/generator, not the scope of the comprehension/generator itself. When several comprehensions/generators are nested, the nearest function or global scope is used. Since the comprehension/generator scope can read closures and global variables, the assignment variable is readable in the comprehension as well. Assigning from a comprehension to a class scope is not valid.
print(some_global := "global")
def outer_function():
print(some_closure := "closure")
steps = [
# v write to variable in containing scope
(some_closure := some_closure + comp_local)
# ^ read from variable in containing scope
for comp_local in some_global
]
return some_closure, steps
While the iteration variable is local to the comprehension in which it is bound, the target of the assignment expression does not create a local variable and is read from the outer scope:
┎ builtins [print, ...]
┗━┱ globals [some_global]
┗━┱ outer_function [some_closure]
┗━╾ <listcomp> [comp_local]
In Python,
any variable that is assigned a value is local to the block in which
the assignment appears.
If a variable can't be found in the current scope, please refer to the LEGB order.
I'm kind of newbie as programmer, but I wish to master Python and I'm developing open source application. This application has function to gather some information. This function takes 1 parameter. This parameter can be 0, 1 or 2. 0 = False, 1 = True, 2 = Multi. Also I have an if statement that does 2 actions. 1st - (when False) gathers single type value, 2nd - (when True) gathers multiple type values and when parameter is 2 (multi) then it will gather single type (1st) and multiple types (2nd). My if statement looks like this:
if False:
get_single_type = code.of.action
generators.generate_data(False, get_single_type)
elif True:
get_multiple_type = code.of.action
generators.generate_data(True, get_multiple_type)
else:
get_single_type = code.of.action
generators.generate_data(False, get_single_type)
get_multiple_type = code.of.action
generators.generate_data(True, get_multiple_type)
Is there maybe better way of avoiding this kind of coding, like in last else statement when I call both single and multiple.
Thank you in advance.
One thing I learned from Python is that although it lacks the Switch operator, you can use dictionary in a similar fashion to get things done since everything is an object:
def get_single():
# define your single function
get_single_type = code.of.action
generators.generate_data(False, get_single_type)
def get_multi():
# define your multi function
get_multiple_type = code.of.action
generators.generate_data(True, get_multiple_type)
actions = {
0: [get_single],
1: [get_multi],
2: [get_single, get_multi]
}
parameter = 0 # replace this line with however you are capturing the parameter
for action in actions[parameter]:
action()
This way you avoid c+p your code everywhere and have it referenced from the function, and your "actions" dictionary define the function to be used based on the parameter given.
In this case since you have multiple functions you want to call, I kept all dictionary items as a list so the structure is consistent and it can be iterated through to perform any number of actions.
Ensure you use leave out the () in the dictionary so that the functions aren't instantiated when the dictionary is defined. And remember to add () when you are actually calling the function from the dictionary to instantiate it.
This is something you will often encounter and it is pretty much always bad practice to be repeating code. Anyway, the way to do this is use two if-statements. This way, even if the first case passes, the second case can still pass. Oh, and assuming your variable that can be 0, 1 or 2 is called x, then we could either use or and two checks:
if x == 0 or x == 2:
but, personally, I prefer using in on a tuple:
if x in (0, 2):
get_single_type = code.of.action
generators.generate_data(False, get_single_type)
if x in (1, 2):
get_multiple_type = code.of.action
generators.generate_data(True, get_multiple_type)
I am interested in creating a list / array of functions "G" consisting of many small functions "g". This essentially should correspond to a series of functions 'evolving' in time.
Each "g" takes-in two variables and returns the product of these variables with an outside global variable indexed at the same time-step.
Assume obs_mat (T x 1) is a pre-defined global array, and t corresponds to the time-steps
G = []
for t in range(T):
# tried declaring obs here too.
def g(current_state, observation_noise):
obs = obs_mat[t]
return current_state * observation_noise * obs
G.append(g)
Unfortunately when I test the resultant functions, they do not seem to pick up on the difference in the obs time-varying constant i.e. (Got G[0](100,100) same as G[5](100,100)). I tried playing around with the scope of obs but without much luck. Would anyone be able to help guide me in the right direction?
This is a common "gotcha" to referencing variables from an outer scope when in an inner function. The outer variable is looked up when the inner function is run, not when the inner function is defined (so all versions of the function see the variable's last value). For each function to see a different value, you either need to make sure they're looking in separate namespaces, or you need to bind the value to a default parameter of the inner function.
Here's an approach that uses an extra namespace:
def make_func(x):
def func(a, b):
return a*b*x
return func
list_of_funcs = [make_func(i) for i in range(10)]
Each inner function func has access to the x parameter in the enclosing make_func function. Since they're all created by separate calls to make_func, they each see separate namespaces with different x values.
Here's the other approach that uses a default argument (with functions created by a lambda expression):
list_of_funcs = [lambda a, b, x=i: a*b*x for i in range(10)]
In this version, the i variable from the list comprehension is bound to the default value of the x parameter in the lambda expression. This binding means that the functions wont care about the value of i changing later on. The downside to this solution is that any code that accidentally calls one of the functions with three arguments instead of two may work without an exception (perhaps with odd results).
The problem you are running into is one of scoping. Function bodies aren't evaluated until the fuction is actually called, so the functions you have there will use whatever is the current value of the variable within their scope at time of evaluation (which means they'll have the same t if you call them all after the for-loop has ended)
In order to see the value that you would like, you'd need to immediately call the function and save the result.
I'm not really sure why you're using an array of functions. Perhaps what you're trying to do is map a partial function across the time series, something like the following?
from functools import partial
def g(current_state, observation_noise, t):
obs = obs_mat[t]
return current_state * observation_noise * obs
g_maker = partial(g, current, observation)
results = list(map(g_maker, range(T)))
What's happening here is that partial creates a partially-applied function, which is merely waiting for its final value to be evaluated. That final value is dynamic (but the first two are fixed in this example), so mapping that partially-applied function over a range of values gets you answers for each value.
Honestly, this is a guess because it's hard to see what else you are trying to do with this data and it's hard to see what you're trying to achieve with the array of functions (and there are certainly other ways to do this).
The issue (assuming that your G.append call is mis-indented) is simply that the name t is mutated when you loop over the iterator returned by range(T). Since every function g you create stores returns the same name t, they wind up all returning the same value, T - 1. The fix is to de-reference the name (the simplest way to do this is by sending t into your function as a default value for an argument in g's argument list):
G = []
for t in range(T):
def g(current_state, observation_noise, t_kw=t):
obs = obs_mat[t_kw]
return current_state * observation_noise * obs
G.append(g)
This works because it creates another name that points at the value that t references during that iteration of the loop (you could still use t rather than t_kw and it would still just work because tg is bound to the value that tf is bound to - the value never changes, but tf is bound to another value on the next iteration, while tg still points at the "original" value.