I would like to use this CLI template
https://mike.depalatis.net/blog/simplifying-argparse.html
for creating a tool for accessing the EMC Unity REST API.
It appears to be written with python3 in mind. Particularly the argument helper function.
def argument(*name_or_flags, **kwargs):
return ([*name_or_flags], kwargs)
I don't believe I understand exactly how the argument function is supposed to work and thus how I can modify it to work with python2.
e.g. If I had a function create_lun that had a few options, I think I need argument to return a list of arguments defined and thus I would decorate it as so:
#subcommand([argument('-o', '--pool', dest='pool', default="pool_1",
type=str, help='Name of Pool of str arg'),
argument('lun_name', metavar='lun_name', type=str,
help='Name of LUN to create in format: ldom-vol#')])
def create_lun(args):
and thus cli.py create_lun lun_name would create the lun and -h would show me this syntax.
If that assumption is correct I would need to translate python3's ability to
return ([*name_or_flags], kwargs)
into a python2.7 equivalent. Any thoughts on this greatly appreciated.
The line
return ([*name_or_flags], kwargs)
is the same as
return [*name_or_flags], kwargs
which is the same as
return name_or_flags, kwargs
In python 3, the syntax [*a_list] expands the elements of a list into a new list literal. It's intended purpose was for inserting an existing list into a new list, not for simply making a copy of a list.
In [1]: a = list(range(9))
In [2]: a
Out[2]: [0, 1, 2, 3, 4, 5, 6, 7, 8]
In [3]: [*a]
Out[3]: [0, 1, 2, 3, 4, 5, 6, 7, 8]
In [4]: ['hey', *a, 'there']
Out[4]: ['hey', 0, 1, 2, 3, 4, 5, 6, 7, 8, 'there']
To write [*name_or_flags] seems like an attempt at obfuscation. Possibly the author had wanted to create a new copy of name_or_flags, and in that case a slice would have been enough:
def argument(*name_or_flags, **kwargs):
return name_or_flags[:], kwargs
If no copy was needed, then the following will suffice.
def argument(*name_or_flags, **kwargs):
return name_or_flags, kwargs
This will work in both Python2 and Python3 and should yield the same results.
Related
I have a class that is a representation of a mathematical tensor. The tensor in the class, is stored as a single list, not lists inside another list. That means [[1, 2, 3], [4, 5, 6]] would be stored as [1, 2, 3, 4, 5, 6].
I've made a __setitem__() function and a function to handle taking slices of this tensor while it's in single list format. For example slice(1, None, None) would become slice(3, None, None) for the list mentioned above. However when I assign this slice a new value, the original tensor isn't updated.
Here is what the simplified code looks like
class Tensor:
def __init__(self, tensor):
self.tensor = tensor # Here I would flatten it, but for now imagine it's already flattened.
def __setitem__(self, slices, value):
slices = [slices]
temp_tensor = self.tensor # any changes to temp_tensor should also change self.tensor.
for s in slices: # Here I would call self.slices_to_index(), but this is to keep the code simple.
temp_tensor = temp_tensor[slice]
temp_tensor = value # In my mind, this should have also changed self.tensor, but it hasn't.
Maybe i'm just being stupid and can't see why this isn't working. Maybe my actual questions isn't just ' why doesn't this work?' but also 'is there a better way to do this?'. Thanks for any help you can give me.
NOTES:
Each 'dimension' of the list must have the same shape, so [[1, 2, 3], [4, 5]] isn't allowed.
This code is massively simplified as there are many other helper functions and stuff like that.
in __init__() I would flatten the list but as I just said to keep things simple I left that out, along with self.slice_to_index().
You should not think of python variables as in c++ or java. Think of them as labels you place on values. Check this example:
>>> l = []
>>> l.append
<built-in method append of list object at 0x7fbb0d40cf88>
>>> l.append(10)
>>> l
[10]
>>> ll = l
>>> ll.append(10)
>>> l
[10, 10]
>>> ll
[10, 10]
>>> ll = ["foo"]
>>> l
[10, 10]
As you can see, ll variable first points to the same l list but later we just make it point to another list. Modifying the later ll won't modify the original list pointed by l.
So, in your case if you want self.tensor to point to a new value, just do it:
class Tensor:
def __init__(self, tensor):
self.tensor = tensor # Here I would flatten it, but for now imagine it's already flattened.
def __setitem__(self, slices, value):
slices = [slices]
temp_tensor = self.tensor # any changes to the list pointed by temp_tensor will be reflected in self.tensor since it is the same list
for s in slices:
temp_tensor = temp_tensor[slice]
self.tensor = value
I'm migrating from python2 to python3 and I'm facing an issue which I have simplified to this:
import numpy as np
a = np.array([1, 2, None])
(a > 0).nonzero()
Traceback (most recent call last):
File "<input>", line 1, in <module>
TypeError: '>' not supported between instances of 'NoneType' and 'int'
In reality I'm processing np-arrays with millions of data and really need to keep the np-operation for performance. In python 2 this was working fine and returns what I expect, since python2 is not so keen on types. What is the best approach for migrating this?
One way to achieve the desired result is to use a lambda function with np.vectorize:
>>> a = np.array([1, 2, None, 4, -1])
>>> f = np.vectorize(lambda t: t and t>0)
>>> np.where(f(a))
(array([0, 1, 3], dtype=int64),)
Of course, if the array doesn't contain negative integers, you could just use np.where(a), as both None and 0 would evaluate to False:
>>> a = np.array([1, 2, None, 4, 0])
>>> np.where(a)
(array([0, 1, 3], dtype=int64),)
Another way this can be solved is by first converting the array to use the float dtype, which has the effect of converting None to np.nan. Then np.where(a>0) can be used as normal.
>>> a = np.array([1, 2, None, 4, -1])
>>> np.where(a.astype(float) > 0)
(array([0, 1, 3], dtype=int64),)
Time comparison:
So Bob's approach, while not as easy on the eyes, is about twice as fast as the np.vectorise approach, and slightly slower than the float conversion approach.
Code to reproduce:
import perfplot
import numpy as np
f = np.vectorize(lambda t: t and t>0)
choices = list(range(-10,11)) + [None]
def cdjb(arr):
return np.where(f(arr))
def cdjb2(arr):
return np.where(arr.astype(float) > 0)
def Bob(arr):
deep_copy = np.copy(arr)
deep_copy[deep_copy == None] = 0
return (deep_copy > 0).nonzero()[0]
perfplot.show(
setup=lambda n: np.random.choice(choices, size=n),
n_range=[2**k for k in range(25)],
kernels=[
cdjb, cdjb2, Bob
],
xlabel='len(a)',
)
To conclude, with the help of #CDJB and #DeepSpace, the best solution I found is to
replace the None values with a value suitable for the specific operation. Also included deep copy of array for not messing up the original data.
import numpy as np
a = np.array([1, None, 2, None])
deep_copy = np.copy(a)
deep_copy[deep_copy == None] = 0
result = (deep_copy > 0).nonzero()[0]
print(result)
[0 2]
I understand the process of passing the function as a parameter to a different function but, coming from the c# background, I don't understand the need of it.
Can someone please make me aware of some scenarios in which this is preferred?
One of the reasons why passing a function as a parameter is useful is the concept of lambda functions in python.
method2(lambda: method1('world'))
>>> hello world
The benefit of lambda functions are easily visible when used with python functions map(), filter(), and reduce().
Lambda functions with map()
>map(lambda x: x*2, my_list)
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38]
Lambda with reduce()
>reduce(lambda x, y: x+y, my_list)
190
Lambda with filter()
filter(lambda x: x >10, my_list)
[11, 12, 13, 14, 15, 16, 17, 18, 19]
Basically unlike c# your code gets reduced number of lines and becomes more efficient
since your function call and execution happens on the same line
Passing functions into functions allows to parameterise behaviour. This is not unlike passing values into functions allows to parameterise data.
def is_greater(what: int, base: int):
if what > base: # fixed behaviour, parameterised data
print(f'{what} is greater')
def is_valid(what: int, condition: 'Callable'):
if condition(what): # parameterised behaviour
print(f'{what} is valid')
Some common use-cases include:
map, filter and others that apply some behaviour to iterables. The functions itself merely implement the "apply to each element" part, but the behaviour can be swapped out:
>>> print(*map(float, ['1', '2', '3.0'])
1.0 2.0 3.0
In such situations, one often uses a lambda to define the behaviour on the fly.
>>> print(sorted(
... ['Bobby Tables', 'Brian Wayne', 'Charles Chapeau'],
... key=lambda name: name.split()[1]), # sort by last name
... )
['Charles Chapeau', 'Bobby Tables', 'Brian Wayne']
Function decorators that wrap a function with additional behaviour.
def print_call(func):
"""Decorator that prints the arguments its target is called with"""
def wrapped_func(*args, **kwargs):
print(f'call {func} with {args} and {kwargs}')
return func(*args, **kwargs)
return wrapped_func
#print_call
def rolling_sum(*numbers, initial=0):
totals = [initial]
for number in numbers:
totals.append(totals[-1] + number)
return totals
rolling_sum(1, 10, 27, 42, 5, initial=100)
# call <function rolling_sum at 0x10ed6fd08> with ([1, 10, 27, 42, 5],) and {'initial': 100}
Every time you see a decorator applied with # it is a higher order function.
Callbacks and payloads that are executed at another time, context, condition, thread or even process.
def call_after(delay: float, func: 'Callable', *args, **kwargs):
"""Call ``func(*args, **kwargs)`` after ``delay`` seconds"""
time.sleep(delay)
func(*args, **kwargs)
thread = threading.Thread(
target=call_after, # payload for the thread is a function
args=(1, print, 'Hello World'))
thread.start()
print("Let's see what happens...")
# Let's see what happens...
#
# Hello World
Passing functions instead of values allows to emulate lazy evaluation.
def as_needed(expensive_computation, default):
if random_condition():
return expensive_computation()
return default
See the following minimum code,
import numba
list_of_list = [[1, 2], [34, 100]]
#numba.njit()
def test(list_of_list):
if 1 in list_of_list[0]:
return 'haha'
test(list_of_list)
This won't work and it seems that list_of_list[0] is no longer behaves like a list during compile. However, the following code works:
list_of_list = [[1, 2], [34, 100]][0] # this is a list NOW!
#numba.njit()
def test(list_of_list):
if 1 in list_of_list:
return 'haha'
test(list_of_list)
This time, what I pass into is actually list, NOT list of list. Then it works. It seems for i in list works in numba, not for i in list_of_list.
In my use case, passing list of list or array like 2d data into numba function is common. Sometimes I only need one element in the list, which is dynamically determined in the program.
In order to make it work, I actually worked out a solution: making list_of_list flattened into a long list, then use linear index to extract one element in original list_of_list.
I am asking here, is there other alternative solutions?
The in method works on sets. Returning a string can also cause some problems.
Working example
import numba as nb
import numpy as np
array_2D = np.array([[1, 2], [34, 100]])
#nb.njit()
def test(array_2D):
if 1 in set(array_2D[0]):
#Strings also causes sometimes problems
#return 'haha'
return(1)
else:
return(-1)
You can return a string with my revised version. It passed test and worked successfully.
from numba import njit
import numpy as np
#njit
def test():
if 1 in set(np_list_of_list[0]):
return 'haha'
if __name__ == '__main__':
list_of_list = [[1, 2], [34, 100]]
np_list_of_list = np.array(list_of_list)
print(test())
filter, map, and reduce work perfectly in Python 2. Here is an example:
>>> def f(x):
return x % 2 != 0 and x % 3 != 0
>>> filter(f, range(2, 25))
[5, 7, 11, 13, 17, 19, 23]
>>> def cube(x):
return x*x*x
>>> map(cube, range(1, 11))
[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]
>>> def add(x,y):
return x+y
>>> reduce(add, range(1, 11))
55
But in Python 3, I receive the following outputs:
>>> filter(f, range(2, 25))
<filter object at 0x0000000002C14908>
>>> map(cube, range(1, 11))
<map object at 0x0000000002C82B70>
>>> reduce(add, range(1, 11))
Traceback (most recent call last):
File "<pyshell#8>", line 1, in <module>
reduce(add, range(1, 11))
NameError: name 'reduce' is not defined
I would appreciate if someone could explain to me why this is.
Screenshot of code for further clarity:
You can read about the changes in What's New In Python 3.0. You should read it thoroughly when you move from 2.x to 3.x since a lot has been changed.
The whole answer here are quotes from the documentation.
Views And Iterators Instead Of Lists
Some well-known APIs no longer return lists:
[...]
map() and filter() return iterators. If you really need a list, a quick fix is e.g. list(map(...)), but a better fix is often to use a list comprehension (especially when the original code uses lambda), or rewriting the code so it doesn’t need a list at all. Particularly tricky is map() invoked for the side effects of the function; the correct transformation is to use a regular for loop (since creating a list would just be wasteful).
[...]
Builtins
[...]
Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable.
[...]
The functionality of map and filter was intentionally changed to return iterators, and reduce was removed from being a built-in and placed in functools.reduce.
So, for filter and map, you can wrap them with list() to see the results like you did before.
>>> def f(x): return x % 2 != 0 and x % 3 != 0
...
>>> list(filter(f, range(2, 25)))
[5, 7, 11, 13, 17, 19, 23]
>>> def cube(x): return x*x*x
...
>>> list(map(cube, range(1, 11)))
[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]
>>> import functools
>>> def add(x,y): return x+y
...
>>> functools.reduce(add, range(1, 11))
55
>>>
The recommendation now is that you replace your usage of map and filter with generators expressions or list comprehensions. Example:
>>> def f(x): return x % 2 != 0 and x % 3 != 0
...
>>> [i for i in range(2, 25) if f(i)]
[5, 7, 11, 13, 17, 19, 23]
>>> def cube(x): return x*x*x
...
>>> [cube(i) for i in range(1, 11)]
[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]
>>>
They say that for loops are 99 percent of the time easier to read than reduce, but I'd just stick with functools.reduce.
Edit: The 99 percent figure is pulled directly from the What’s New In Python 3.0 page authored by Guido van Rossum.
As an addendum to the other answers, this sounds like a fine use-case for a context manager that will re-map the names of these functions to ones which return a list and introduce reduce in the global namespace.
A quick implementation might look like this:
from contextlib import contextmanager
#contextmanager
def noiters(*funcs):
if not funcs:
funcs = [map, filter, zip] # etc
from functools import reduce
globals()[reduce.__name__] = reduce
for func in funcs:
globals()[func.__name__] = lambda *ar, func = func, **kwar: list(func(*ar, **kwar))
try:
yield
finally:
del globals()[reduce.__name__]
for func in funcs: globals()[func.__name__] = func
With a usage that looks like this:
with noiters(map):
from operator import add
print(reduce(add, range(1, 20)))
print(map(int, ['1', '2']))
Which prints:
190
[1, 2]
Just my 2 cents :-)
Since the reduce method has been removed from the built in function from Python3, don't forget to import the functools in your code. Please look at the code snippet below.
import functools
my_list = [10,15,20,25,35]
sum_numbers = functools.reduce(lambda x ,y : x+y , my_list)
print(sum_numbers)
One of the advantages of map, filter and reduce is how legible they become when you "chain" them together to do something complex. However, the built-in syntax isn't legible and is all "backwards". So, I suggest using the PyFunctional package (https://pypi.org/project/PyFunctional/).
Here's a comparison of the two:
flight_destinations_dict = {'NY': {'London', 'Rome'}, 'Berlin': {'NY'}}
PyFunctional version
Very legible syntax. You can say:
"I have a sequence of flight destinations. Out of which I want to get
the dict key if city is in the dict values. Finally, filter out the
empty lists I created in the process."
from functional import seq # PyFunctional package to allow easier syntax
def find_return_flights_PYFUNCTIONAL_SYNTAX(city, flight_destinations_dict):
return seq(flight_destinations_dict.items()) \
.map(lambda x: x[0] if city in x[1] else []) \
.filter(lambda x: x != []) \
Default Python version
It's all backwards. You need to say:
"OK, so, there's a list. I want to filter empty lists out of it. Why?
Because I first got the dict key if the city was in the dict values.
Oh, the list I'm doing this to is flight_destinations_dict."
def find_return_flights_DEFAULT_SYNTAX(city, flight_destinations_dict):
return list(
filter(lambda x: x != [],
map(lambda x: x[0] if city in x[1] else [], flight_destinations_dict.items())
)
)
Here are the examples of Filter, map and reduce functions.
numbers = [10,11,12,22,34,43,54,34,67,87,88,98,99,87,44,66]
//Filter
oddNumbers = list(filter(lambda x: x%2 != 0, numbers))
print(oddNumbers)
//Map
multiplyOf2 = list(map(lambda x: x*2, numbers))
print(multiplyOf2)
//Reduce
The reduce function, since it is not commonly used, was removed from the built-in functions in Python 3. It is still available in the functools module, so you can do:
from functools import reduce
sumOfNumbers = reduce(lambda x,y: x+y, numbers)
print(sumOfNumbers)
from functools import reduce
def f(x):
return x % 2 != 0 and x % 3 != 0
print(*filter(f, range(2, 25)))
#[5, 7, 11, 13, 17, 19, 23]
def cube(x):
return x**3
print(*map(cube, range(1, 11)))
#[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000]
def add(x,y):
return x+y
reduce(add, range(1, 11))
#55
It works as is. To get the output of map use * or list