Is there a way to extract function parameters from a dictionary? - python-3.x

I am trying to create some more functional code in Python and I want to know if it is possible to transform dictionary (key,values) to pass as a function parameter.
I am currently doing this in a more imperative way, where I filter and then manually extract each key depending on the result of the filter. My current code:
def a(i: int, config: dict):
function_array = [function1, function2, function3]
selected = function_array[i]
if (i == "0"):
result = selected(x = config['x'])
elif (i == "1"):
result = selected(y = config['y'])
elif (i == "2"):
result = selected(z = config['z'])
return result
The current result is correct, but when I have many cases, I need to hardcode each parameter for the specified function. So, that is why I want to know if it is possible to pass the config object as I want (with an x when i is 0, for example) and then just do something like this:
def a(i: int, config: dict):
function_array = [function1, function2, function3]
result = function_array[i](config)
return result

The syntax for passing items from a dictionary as function parameters is simply selected(**config)
So for your example, it would look something like this:
def function1(x=0):
return x + 1
def function2(y=42):
return y * 2
def function3(z=100):
return z
def a(i, config):
function_array = [function1, function2, function3]
selected = function_array[i]
return selected(**config)
config = {x: 10}
a(0, config) # calls function1(x=10)
config = {y: 20}
a(1, config) # calls function2(y=20)
config = {}
a(2, config) # calls function3()

Every python function can be instructed to take a dictionary of keywords. See e.g. https://www.pythoncheatsheet.org/blog/python-easy-args-kwargs . (Official source at https://docs.python.org/3/reference/compound_stmts.html#function-definitions, but it's harder to read.)
You could do:
def a(i: int, keyword: str, **kwargs: dict):
if keyword in kwargs:
result = kwargs[keyword](i)
and you would run it with something like:
a(5, "func3", func1=print, func2=sum, func3=all)
Or, you could just pass a dictionary itself into the function:
def a(i: int, keyword: str, config: dict)
if keyword in config:
result = config[keyword](i)
This would be run with something like:
a(5, "func3", {"func1": print, "func2": sum, "func3": all})
The only difference is that the ** in the function declaration tells python to automatically make a dictionary out of explicit keywords. In the second example, you make the dictionary by yourself.
There's an important thing happening behind the scenes here. Functions are being passed around just like anything else. In python, functions are still objects. You can pass a function just as easily as you can pass an int. So if you wanted to have a list of lists, where each inner list is a function with some arguments, you easily could:
things_to_do = [[sum, 5, 7, 9], [any, 1, 0], [all, 1, 0]]
for thing_list in things_to_do:
function = thing_list[0]
args = thing_list[1:]
print(function(args))
And you'll get the following results:
21
True
False
(Note also that all of those functions take an iterable, such as a list. If you want to pass each argument separately, you would use *args instead of args.)
You can do it with defined functions, too. If you have
def foo1(arg1):
pass
def foo2(arg1, arg2):
pass
you can just as easily have
things_to_do = [[sum, 5, 7, 9], [foo1, 'a'], [foo2, 0, None]]

Related

Difference between lists direct assignment and slice assignment

I have:
def reverseString(self, s: List[str]) -> None:
s[:] = s[::-1] # Works
... and
def reverseString(self, s: List[str]) -> None:
s = s[::-1] # Doesn't work
Where s is a list of characters lets say s = ["k","a","k","a","s","h","i"]
While doing a question on leetcode it rejected when I used s = ... but accepted when I used s[:] = ... and also it was written that DO NOT RETURN ANYTHING but return s.reverse also worked.
This is actually a bit complex and requires two explanations.
First, a python function argument act as label on the passed object. For example, in the following code, arg is the local label (/name) attached to the initial list. When the label arg is re-used, for example by attaching it to a new object (17), the original list is not reachable anymore within function f.
On the other hand, outside of f, the list labeled L is still here, untouched:
def f(arg):
arg = 17
print(arg) # 17
L = ['a', 'list']
f(L)
print(L) # ['a', 'list']
That explains why the following function doesn't reverse your list in place:
def reverse_list(arg):
arg = arg[::-1]
print(arg) # ['list', 'a']
L = ['a', 'list']
reverse_list(L)
print(L) # ['a', 'list']
This function simply attach the label arg to a new list (that is indeed equal to the reversed list).
Secondly, the difference between arg[:] = ... and arg = ... is that the first will modify the content of the list (instead of attaching the label arg to a new object). This is the reason why the following works as expected:
def alt_reverse_list(arg):
arg[:] = arg[::-1]
L = ['a', 'list']
alt_reverse_list(L)
print(L) # ['list', 'a']
In this second example we say that the list has been mutated (modified in place). Here is a detailed explanation on slice assignments
For the same reason, calling arg.reverse() would have worked.
Identifying objects
Using the id() function can help figure out what is going on with the argument in the first example (where we don't mutate the list but affect a new value):
def reverse_list(arg):
print("List ID before: ", id(arg))
arg = arg[::-1]
print("List ID after: ", id(arg))
L = ['a', 'list']
print("Original list ID: ", id(L))
reverse_list(L)
print("Final list ID: ", id(L))
Which will print something like:
Original list ID: 140395368281088
List ID before: 140395368281088
List ID after: 140395368280447 <--- intruder spotted
Final list ID: 140395368281088
Here we can clearly see that after calling arg = arg[::-1] the object we are manipulating under the name arg is not the same. This shows why the function doesn't have any (side) effect.

What's the underlying implementation for most_common method of Counter?

I found a pyi file which has the following def
def most_common(self, n: Optional[int] = ...) -> List[Tuple[_T, int]]: ...
How could this happen? List is not defined, and no implementation?
Just highlight some valuable suggestions here for followers:
List is imported from the typing module; it's not the same thing as list. The .pyi file doesn't need to import it because stub files are never executed; they just have to be syntactically valid Python
If you use from future import annotations, you won't have to import typing to use List et al. in function annotations in .py files, either, since function annotations will be treated as string literals. (Starting in Python 4, that will be the default behavior. See PEP 563 for details.)
You are looking at the pyi file which is used solely for annotations. It is never executed by the Python interpreter. You can learn more about pyi files by reading PEP484.
Using a debugger, put a breakpoint on the line where you call most_commonand then step into the method.
Python 3.7 implementation.
...\Lib\collections\__init__.py:
def most_common(self, n=None):
'''List the n most common elements and their counts from the most
common to the least. If n is None, then list all element counts.
>>> Counter('abcdeabcdabcaba').most_common(3)
[('a', 5), ('b', 4), ('c', 3)]
'''
# Emulate Bag.sortedByCount from Smalltalk
if n is None:
return sorted(self.items(), key=_itemgetter(1), reverse=True)
return _heapq.nlargest(n, self.items(), key=_itemgetter(1))
_heapq.nlargest (in ...\Lib\heapq.py) implementation:
def nlargest(n, iterable, key=None):
"""Find the n largest elements in a dataset.
Equivalent to: sorted(iterable, key=key, reverse=True)[:n]
"""
# Short-cut for n==1 is to use max()
if n == 1:
it = iter(iterable)
sentinel = object()
if key is None:
result = max(it, default=sentinel)
else:
result = max(it, default=sentinel, key=key)
return [] if result is sentinel else [result]
# When n>=size, it's faster to use sorted()
try:
size = len(iterable)
except (TypeError, AttributeError):
pass
else:
if n >= size:
return sorted(iterable, key=key, reverse=True)[:n]
# When key is none, use simpler decoration
if key is None:
it = iter(iterable)
result = [(elem, i) for i, elem in zip(range(0, -n, -1), it)]
if not result:
return result
heapify(result)
top = result[0][0]
order = -n
_heapreplace = heapreplace
for elem in it:
if top < elem:
_heapreplace(result, (elem, order))
top, _order = result[0]
order -= 1
result.sort(reverse=True)
return [elem for (elem, order) in result]
# General case, slowest method
it = iter(iterable)
result = [(key(elem), i, elem) for i, elem in zip(range(0, -n, -1), it)]
if not result:
return result
heapify(result)
top = result[0][0]
order = -n
_heapreplace = heapreplace
for elem in it:
k = key(elem)
if top < k:
_heapreplace(result, (k, order, elem))
top, _order, _elem = result[0]
order -= 1
result.sort(reverse=True)
return [elem for (k, order, elem) in result]

Loop over Set of Functions for Variable Pipelining and Separate Usage

I have the following 10 functions:
def function1(data1,data2):
...
return value
def function2(data1,data2):
...
return value
...
def function10(data1,data2):
...
return value
I want to use these functions separately when needed but also
in a pipeline for calculating properties and appending to a list.
Like this:
collecting_list = []
for idx in range(10):
collecting_list.append(function1(data1[idx],data2[idx]))
collecting_list.append(function2(data1[idx],data2[idx]))
collecting_list.append(function3(data1[idx],data2[idx]))
collecting_list.append(function4(data1[idx],data2[idx]))
collecting_list.append(function5(data1[idx],data2[idx]))
collecting_list.append(function6(data1[idx],data2[idx]))
collecting_list.append(function7(data1[idx],data2[idx]))
collecting_list.append(function8(data1[idx],data2[idx]))
collecting_list.append(function9(data1[idx],data2[idx]))
collecting_list.append(function10(data1[idx],data2[idx])
Obviously I would need some property to loop over function names, but I never came across this problem before and was just wondering if I can call those functions in a loop without hard coding this and just adjusting the function-number (e.g. function1(), function2(), ... function10()).
Hints and ideas appreciated!
use lambda and exec.
you could have a string array of the function names, and lambda functions that return the data like something below. With lambda functions, you can reuse the same name dataX over and over again and with proper implementation get the right data needed. See below for a very basic, abstract example:
import random
def getData1():
return random.randint(1, 10)
def getData2():
return random.randint(11, 20)
def function1(data1):
print("f1, {}".format(data1))
def function2(data1, data2):
print("f2, {} and {}".format(data1, data2))
data1 = lambda: getData1() # these can be any function that serves as the
data2 = lambda: getData2() # source for your data. using lambda allows for
# anonymization and reuse
functionList = ["function1({})".format(data1()), "function2({},{})".format(data1(), data2())]
for f in functionList:
exec(f)
function1(data1())
You might ask why not just use getData1() in the function list instead of data1, and the answer has to do with parameters. If the getDataX functions required parameters, you wouldn't want to compute the functionList every time a parameter name changed. This is one of the benefits of using lambda and exec.
Um, sure?
import sys
import types
module_name = sys.modules[__name__]
def function1(data1, data2):
return ("func1", data1 + data2)
def function2(data1, data2):
return ("func2", data1 + data2)
def function3(data1, data2):
return ("func3", data1 + data2)
def function4(data1, data2):
return ("func4", data1 + data2)
def function5(data1, data2):
return ("func5", data1 + data2)
def get_functions():
func_list = list()
for k in sorted(module_name.__dict__.keys()):
if k.startswith('function'):
if isinstance(module_name.__dict__[k], types.FunctionType):
func_list.append(module_name.__dict__[k])
return func_list
def get_functions_2():
func_list = list()
for itr in range(1, 100):
try:
func_list.append(getattr(module_name, "function%s" % itr))
except:
break
return func_list
def run_pipeline(function_list):
collecting_list = list()
for idx, func in enumerate(function_list):
collecting_list.append(func(idx, idx))
return collecting_list
if __name__ == "__main__":
funcs = get_functions()
results = run_pipeline(funcs)
print(results)
Outputs:
[('func1', 0), ('func2', 2), ('func3', 4), ('func4', 6), ('func5', 8)]
Note: I probably wouldn't do it this way if I was trying to construct dynamic computational pipelines, but you can use this method. You could in theory create a file per pipeline and name them in order to use this method though?
Edit: Added get_functions_2 per request

Is it possible to adapt this approach of using dictionaries to find/evaluate one of many functions and its corresponding args?

Suppose one has several separate functions to evaluate some given data. Rather than use redundant if/else loops, one decides to use a dictionary key to find the particular function and its corresponding args. I feel like this is possible, but I can't figure out how to make this work. As a simplified example (that I hope to adapt for my case), consider the code below:
def func_one(x, a, b, c=0):
""" arbitrary function """
# c is initialized since it is necessary in func_two and has no effect in func_one
return a*x + b
def func_two(x, a, b, c):
""" arbitrary function """
return a*x**2 + b*x + c
def pick_function(key, x=5):
""" picks and evaluates arbitrary function by key """
if key != (1 or 2):
raise ValueError("key = 1 or 2")
## args = a, b, c
args_one = (1, 2, 3)
args_two = (4, 5, 3)
## function dictionary
func_dict = dict(zip([1, 2], [func_one, func_two]))
## args dictionary
args_dict = dict(zip([1, 2], [args_one, args_two]))
## apply function to args
func = func_dict[key]
args = args_dict[key]
## my original attempt >> return func(x, args)
return func(x, *args) ## << EDITED SOLUTION VIA COMMENTS BELOW
print(func_one(x=5, a=1, b=2, c=3)) # prints 7
But,
print(pick_function(1))
returns an error message
File "stack_overflow_example_question.py", line 17, in pick_function
return func(x, args)
TypeError: func_one() missing 1 required positional argument: 'b'
Clearly, not all of the args are being passed through with the dictionary. I've tried various combinations of adding/removing extra brackets and paranthesis from args_one and args_two (as defined in pick_function). Is this approach fruitful? Are there other convenient (in terms of readability and speed) approaches that do not require many if/else loops?
To fix your code with minimal changes, change return func(x, args) to return func(x, *args). I think this is what Anton vBR is suggesting in the comments.
However, I think your code could be further simplified by
using the * ("splat") and ** ("double-splat"?) positional/keyword argument unpacking operators like this:
def func_one(x, a, b, c=0):
""" arbitrary function """
# c is initialized since it is necessary in func_two and has no effect in func_one
return a*x + b
def func_two(x, a, b, c):
""" arbitrary function """
return a*x**2 + b*x + c
def func(key, *args, **kwargs):
funcmap = {1: func_one, 2: func_two}
return funcmap[key](*args, **kwargs)
def pick_function(key, x=5):
""" picks and evaluates arbitrary function by key """
argmap = {1: (1, 2, 3), 2: (4, 5, 3)}
return func(key, x, *argmap[key])
print(func_one(x=5, a=1, b=2, c=3))
# 7
print(pick_function(1))
# 7
print(func(1, 5, 1, 2, 3))
# 7
print(func(1, b=2, a=1, c=3, x=5))
# 7
If I understand what you are asking, I don't know that you want to use 1, 2... as your dictionary keys anyway. You can pass the name of the function and the list of the args you want to use straight to a function and use it that way like:
def use_function(func, argList):
return (func(*argList))
print(use_function(func_one, [5, 1, 2, 3]))
or:
def use_function_2(func, argDict):
return (func(**argDict))
print(use_function_2(func_one, {'a':1, 'b':2,'x':5, 'c':3}))
And if you like you could still use a dictionary to hold numbers which correspond to functions as well. That would look like this:
def pick_function_2(key, x=5):
""" picks and evaluates arbitrary function by key """
if key != (1 or 2):
raise ValueError("key = 1 or 2")
## args = a, b, c
args_one = [1, 2, 3]
args_two = [4, 5, 3]
## function dictionary
func_dict = dict(zip([1, 2], [func_one, func_two]))
## args dictionary
args_dict = dict(zip([1, 2], [args_one, args_two]))
## apply function to args
func = func_dict[key]
args = args_dict[key]
return func(*([x] + args))
print(pick_function_2(1))
However, this is starting to get a bit confusing. I would make sure you take some time and just double check that this is actually what you want to do.
You are trying to mixing named arguments and unnamed arguments!
As rule of thumb, if you pass a dict to a function using ** notation, you can access the arguments using **kwargs value. However, if you pass a tuple to the funcion using * notation, you can access the arguments using *args value.
I edited the method in the following way:
def func_one(*args, **kwargs):
""" arbitrary function """
# c is initialized since it is necessary in func_two and has no effect in func_one
if args:
print("args: {}".format(args))
x, other_args, *_ = args
a, b , c = other_args
elif kwargs:
print("kwargs: {}".format(kwargs))
x, a , b , c = kwargs['x'], kwargs['a'], kwargs['b'], kwargs['c']
return a*x + b
So, in the first call you have:
print(func_one(x=5, a=1, b=2, c=3)) # prints 7
kwargs: {'b': 2, 'a': 1, 'x': 5, 'c': 3}
7
Because you are passing named arguments.
In the second execution, you'll have:
print(pick_function(1))
args: (5, (1, 2, 3))
7
I know you wanted to find a solution without if/else, but you have to discriminate between theese two cases.

pytest test needs parametrization at collection phase and at setup time

I have some tests which I'd like to parametrize using some arguments which need the parametrization to happen during collection phase and some which need it to happen at setup time. I'm unable to use
metafunc.parametrize in a pytest_generate_test hook since I need some fixtures to have indirect=True to pass the argname as a request.param, but the other arguments need to have indirect=False.
Any ideas how to do this?
Here's an example of what my tests look like and what I want to do:
def pytest_generate_tests(metafunc):
if metafunc.function.__name__ == 'test_example':
argnames = []
argvalues = []
parameters = getattr(metafunc.function, 'paramlist', ())
for p in parameters:
if type(p) == list:
argnames = tuple(['myfixture'] + p)
else:
argvalues.append = tuple(['std'] + p['argvalues'])
argvalues.append = tuple(['pro'] + p['argvalues'])
# I want to do the following, but it won't work since some of the
# args need indirect set to true and some need indirect set to false.
metafunc.parametrize(argnames, argvalues, indirect=True)
elif 'myfixture' in metafunc.fixturenames:
# we have existing tests which use the fixture, but only with
standard
metafunc.parametrize("myfixture", "std")
else:
# we have existing tests which use older style parametrization,
non-fixture
for p in getattr(metafunc.function, 'paramlist', ()):
metafunc.addcall(funcargs=p)
def params(decolist):
def wrapper(function):
function.paramlist = decolist
return function
return wrapper
#pytest.fixture
def myfixture(request):
If request.param == 'std':
myfix = SomeObject()
elif request.param == 'pro':
myfix = SomeOtherObject()
def fin():
myfix.close()
request.addfinalizer(fin)
return myfix
#params([
['color', 'type'],
{ 'argvalues': [ 'blue', 'cat'] },
{ 'argvalues': ['pink', 'dog'] }
])
def test_example(myfixture, color, type):
# this is the new test we want to add
def test_something(myfixture):
# existing test which only uses std fixture
#params([
{'arg1': 1, 'arg2': 2},
{'arg1': 3, 'arg2': 5}
])
def test_old_style(arg1, arg2):
# existing tests which don't use fixtures
Thanks for reading through this! I know it's rather long.
per design all parametization happens at collection time

Resources