Related
I have a tensor t with dim b x 3 and a list of lengths len = [l_0, l_1, ..., l_n]. All entries in len sum to to b. I want to create a new tensor with dim n x 3, which stores the average of the entries in t. E.g. The first l_0 entries in t are averaged and build the first element in the new tensor. The following l_1 entries are averaged and build the second element, ...
Thanks for your help.
You can do so using a combination a cumulative list of indices as helper and a list comprehension to construct the new array:
>>> b, lens = 10, [2, 3, 1, 3, 1]
>>> t = torch.rand(b, 3)
tensor([[0.3567, 0.3998, 0.9396],
[0.4061, 0.6465, 0.6955],
[0.3500, 0.4135, 0.5288],
[0.0726, 0.9575, 0.3785],
[0.6216, 0.2975, 0.3293],
[0.3878, 0.0735, 0.8181],
[0.1694, 0.5446, 0.1179],
[0.7793, 0.6613, 0.1748],
[0.0964, 0.9825, 0.1651],
[0.1421, 0.0994, 0.8086]])
Build the cumulative list of indices:
>>> c = torch.cumsum(torch.tensor([0] + lens), 0)
tensor([ 0, 2, 5, 6, 9, 10])
Loop over c by twos, with an overlapping window. For example zip(c[:-1], c[1:]) works well. Each selection from i to j gets averaged on dim=0.
>>> [t[i:j].sum(0) for i, j in zip(c[:-1], c[1:])]
[tensor([0.7628, 1.0463, 1.6351]),
tensor([1.0442, 1.6685, 1.2367]),
tensor([0.3878, 0.0735, 0.8181]),
tensor([1.0451, 2.1885, 0.4578]),
tensor([0.1421, 0.0994, 0.8086])]
Then you can stack the list:
>>> torch.stack([t[i:j].sum(0) for i, j in zip(c[:-1], c[1:])])
tensor([[0.7628, 1.0463, 1.6351],
[1.0442, 1.6685, 1.2367],
[0.3878, 0.0735, 0.8181],
[1.0451, 2.1885, 0.4578],
[0.1421, 0.0994, 0.8086]])
I have been struggling with a programming problem lately, and I would appreciate any help that I could get.
Basically, the input is a list of numbers (both positive and negative, also, note that the numbers can repeat within the list), and I want to find the combinations of the numbers that lead upto a sum K.
For example,
List - [1,2,3,6,5,-2]
Required sum - 8
Output - Many combinations like : [5,3], [5,2,1], [6,3,1,-2]... and so on
I do understand that there are solutions available to this problem using module functions like “itertools.combinations” or using recursion - subset of sum (works efficiently only for positive numbers on the list), but I am looking for an efficient solution in python3 for lists upto a 100 numbers.
Any and every help would be appreciated.
You are trying to compute the sum over all subsets, not combinations.
This will never be efficient for 100 numbers (atleast in no way I know, either in C++ or in Python), because you need to actually compute all sums, and then filter.
This takes O(n*2^n) time using a method called Dynamic Programming - Sum over Subsets (DP-SOS). The following is the most efficient code possible:
a, k = [1, 2, 3, 6, 5, -2], 8
dp = [0] * (1 << len(a))
for idx, val in enumerate(a):
dp[1 << idx] = a[idx]
for i in range(len(a)):
for mask in range(len(dp)):
if mask & (1 << i):
dp[mask] += dp[mask^(1<<i)]
answer = []
for pos, sum in enumerate(dp):
if sum == k:
answer.append([val for idx, val in enumerate(a)
if (1 << idx) & pos])
Using itertools seems unnecessary to me, though you might get an answer in almost the same (or slightly longer) time, because the bitwise operators even in python are pretty fast.
Nevertheless, using itertools:
from itertools import chain, combinations
a, k = [1, 2, 3, 6, 5, -2], 8
def powerset(iterable):
"powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)"
s = list(iterable)
return chain.from_iterable(combinations(s, r) for r in range(len(s)+1))
answer = list(filter(lambda x: sum(x) == k, powerset(a)))
If you want to scale to 100 numbers, there might not be an exact solution, you would need to use heuristics and not compute the answer explicitly, because if the array is [8, 0, 0, ...(100 times)] then there are 2^99 subsets which you can anyways never compute explicitly or store.
I have a matrix (2d numpy ndarray, to be precise):
A = np.array([[4, 0, 0],
[1, 2, 3],
[0, 0, 5]])
And I want to roll each row of A independently, according to roll values in another array:
r = np.array([2, 0, -1])
That is, I want to do this:
print np.array([np.roll(row, x) for row,x in zip(A, r)])
[[0 0 4]
[1 2 3]
[0 5 0]]
Is there a way to do this efficiently? Perhaps using fancy indexing tricks?
Sure you can do it using advanced indexing, whether it is the fastest way probably depends on your array size (if your rows are large it may not be):
rows, column_indices = np.ogrid[:A.shape[0], :A.shape[1]]
# Use always a negative shift, so that column_indices are valid.
# (could also use module operation)
r[r < 0] += A.shape[1]
column_indices = column_indices - r[:, np.newaxis]
result = A[rows, column_indices]
numpy.lib.stride_tricks.as_strided stricks (abbrev pun intended) again!
Speaking of fancy indexing tricks, there's the infamous - np.lib.stride_tricks.as_strided. The idea/trick would be to get a sliced portion starting from the first column until the second last one and concatenate at the end. This ensures that we can stride in the forward direction as needed to leverage np.lib.stride_tricks.as_strided and thus avoid the need of actually rolling back. That's the whole idea!
Now, in terms of actual implementation we would use scikit-image's view_as_windows to elegantly use np.lib.stride_tricks.as_strided under the hoods. Thus, the final implementation would be -
from skimage.util.shape import view_as_windows as viewW
def strided_indexing_roll(a, r):
# Concatenate with sliced to cover all rolls
a_ext = np.concatenate((a,a[:,:-1]),axis=1)
# Get sliding windows; use advanced-indexing to select appropriate ones
n = a.shape[1]
return viewW(a_ext,(1,n))[np.arange(len(r)), (n-r)%n,0]
Here's a sample run -
In [327]: A = np.array([[4, 0, 0],
...: [1, 2, 3],
...: [0, 0, 5]])
In [328]: r = np.array([2, 0, -1])
In [329]: strided_indexing_roll(A, r)
Out[329]:
array([[0, 0, 4],
[1, 2, 3],
[0, 5, 0]])
Benchmarking
# #seberg's solution
def advindexing_roll(A, r):
rows, column_indices = np.ogrid[:A.shape[0], :A.shape[1]]
r[r < 0] += A.shape[1]
column_indices = column_indices - r[:,np.newaxis]
return A[rows, column_indices]
Let's do some benchmarking on an array with large number of rows and columns -
In [324]: np.random.seed(0)
...: a = np.random.rand(10000,1000)
...: r = np.random.randint(-1000,1000,(10000))
# #seberg's solution
In [325]: %timeit advindexing_roll(a, r)
10 loops, best of 3: 71.3 ms per loop
# Solution from this post
In [326]: %timeit strided_indexing_roll(a, r)
10 loops, best of 3: 44 ms per loop
In case you want more general solution (dealing with any shape and with any axis), I modified #seberg's solution:
def indep_roll(arr, shifts, axis=1):
"""Apply an independent roll for each dimensions of a single axis.
Parameters
----------
arr : np.ndarray
Array of any shape.
shifts : np.ndarray
How many shifting to use for each dimension. Shape: `(arr.shape[axis],)`.
axis : int
Axis along which elements are shifted.
"""
arr = np.swapaxes(arr,axis,-1)
all_idcs = np.ogrid[[slice(0,n) for n in arr.shape]]
# Convert to a positive shift
shifts[shifts < 0] += arr.shape[-1]
all_idcs[-1] = all_idcs[-1] - shifts[:, np.newaxis]
result = arr[tuple(all_idcs)]
arr = np.swapaxes(result,-1,axis)
return arr
I implement a pure numpy.lib.stride_tricks.as_strided solution as follows
from numpy.lib.stride_tricks import as_strided
def custom_roll(arr, r_tup):
m = np.asarray(r_tup)
arr_roll = arr[:, [*range(arr.shape[1]),*range(arr.shape[1]-1)]].copy() #need `copy`
strd_0, strd_1 = arr_roll.strides
n = arr.shape[1]
result = as_strided(arr_roll, (*arr.shape, n), (strd_0 ,strd_1, strd_1))
return result[np.arange(arr.shape[0]), (n-m)%n]
A = np.array([[4, 0, 0],
[1, 2, 3],
[0, 0, 5]])
r = np.array([2, 0, -1])
out = custom_roll(A, r)
Out[789]:
array([[0, 0, 4],
[1, 2, 3],
[0, 5, 0]])
By using a fast fourrier transform we can apply a transformation in the frequency domain and then use the inverse fast fourrier transform to obtain the row shift.
So this is a pure numpy solution that take only one line:
import numpy as np
from numpy.fft import fft, ifft
# The row shift function using the fast fourrier transform
# rshift(A,r) where A is a 2D array, r the row shift vector
def rshift(A,r):
return np.real(ifft(fft(A,axis=1)*np.exp(2*1j*np.pi/A.shape[1]*r[:,None]*np.r_[0:A.shape[1]][None,:]),axis=1).round())
This will apply a left shift, but we can simply negate the exponential exponant to turn the function into a right shift function:
ifft(fft(...)*np.exp(-2*1j...)
It can be used like that:
# Example:
A = np.array([[1,2,3,4],
[1,2,3,4],
[1,2,3,4]])
r = np.array([1,-1,3])
print(rshift(A,r))
Building on divakar's excellent answer, you can apply this logic to 3D array easily (which was the problematic that brought me here in the first place). Here's an example - basically flatten your data, roll it & reshape it after::
def applyroll_30(cube, threshold=25, offset=500):
flattened_cube = cube.copy().reshape(cube.shape[0]*cube.shape[1], cube.shape[2])
roll_matrix = calc_roll_matrix_flattened(flattened_cube, threshold, offset)
rolled_cube = strided_indexing_roll(flattened_cube, roll_matrix, cube_shape=cube.shape)
rolled_cube = triggered_cube.reshape(cube.shape[0], cube.shape[1], cube.shape[2])
return rolled_cube
def calc_roll_matrix_flattened(cube_flattened, threshold, offset):
""" Calculates the number of position along time axis we need to shift
elements in order to trig the data.
We return a 1D numpy array of shape (X*Y, time) elements
"""
# armax(...) finds the position in the cube (3d) where we are above threshold
roll_matrix = np.argmax(cube_flattened > threshold, axis=1) + offset
# ensure we don't have index out of bound
roll_matrix[roll_matrix>cube_flattened.shape[1]] = cube_flattened.shape[1]
return roll_matrix
def strided_indexing_roll(cube_flattened, roll_matrix_flattened, cube_shape):
# Concatenate with sliced to cover all rolls
# otherwise we shift in the wrong direction for my application
roll_matrix_flattened = -1 * roll_matrix_flattened
a_ext = np.concatenate((cube_flattened, cube_flattened[:, :-1]), axis=1)
# Get sliding windows; use advanced-indexing to select appropriate ones
n = cube_flattened.shape[1]
result = viewW(a_ext,(1,n))[np.arange(len(roll_matrix_flattened)), (n - roll_matrix_flattened) % n, 0]
result = result.reshape(cube_shape)
return result
Divakar's answer doesn't do justice to how much more efficient this is on large cube of data. I've timed it on a 400x400x2000 data formatted as int8. An equivalent for-loop does ~5.5seconds, Seberg's answer ~3.0seconds and strided_indexing.... ~0.5second.
I have a list containing n integers. The ith element of the list a, a[i], can be swapped into any integer x such that 0 ≤ x ≤ a[i]. For example if a[i] is 3, it can take values 0, 1, 2, 3.
The task is to find all permutations of such list. For example, if the list is
my_list = [2,1,4]
then the possible permutations are:
[0,0,0], [0,0,1], ... [0,0,4],
[0,1,0], [0,1,1], ... [0,1,4],
[1,0,0], [1,0,1], ... [1,0,4],
[1,1,0], [1,1,1], ... [1,1,4],
[2,0,0], [2,0,1], ... [2,0,4],
[2,1,0], [2,1,1], ... [2,1,4]
How to find all such permutations?
you could use a comibation of range to get all the 'valid' values for each element of the list and itertools.product:
import itertools
my_list = [2,1,4]
# get a list of lists with all the possible values
plist = [list(range(y+1)) for y in my_list]
#
permutations = sorted(list(itertools.product(*plist)))
more on itertools product see e.g. here on SO or the docs.
Here's a solution:
my_list=[2,1,4]
def premutation_list(p_list):
the_number=int("".join(map(str,p_list)))
total_len=len(str(the_number))
a=[i for i in range(the_number)]
r_list=[]
for i in a:
if len(str(i))<total_len:
add_rate=total_len - len(str(i))
b="0,"*add_rate
b=b.split(",")
b=b[0:len(b)-1]
b.append(str(i))
r_list.append([int(y) for x in b for y in x ])
else:
r_list.append([int(x) for x in str(i)])
return r_list
print(premutation_list(my_list))
Explanation:
The basic idea is just getting all the numbers till the given number. For example till 4 there are 0,1,2,3, number.
I have achieved this first by converting the list into a integer.
Then getting all the numbers till the_number.
Try this. Let me know if I misunderstood your question
def permute(l,cnt,n):
if cnt==n:
print(l)
return
limit = l[cnt]
for i in range(limit+1):
l[cnt]=i
permute(l[:n],cnt+1,n)
l =[2,1,4]
permute(l,0,3)
i‘ m using Python 3.7.
I have an Array like this:
L1 = [1,2,3,-10,8,12,300,17]
Now i want to filter the values(the -10 and the 300 is not okay)
The values in the array may be different but always counting up or counting down.
Has Python 3 a integrated function for that?
The result should look like this:
L1 = [1,2,3,8,12,17]
Thank you !
Edit from comments:
I want to keep each element if it is only a certain distance (toleranz: 10 f.e.) distance away from the one before.
Your array is a list. You can use built in functions:
L1 = [1,2,3,-10,8,12,300,17]
min_val = min(L1) # -10
max_val = max(L1) # 300
p = list(filter(lambda x: min_val < x < max_val, L1)) # all x not -10 or 300
print(p) # [1, 2, 3, 8, 12, 17]
Doku:
min()
max()
filter()
If you want instead an incremental filter you go through your list of datapoints and decide if to keep or not:
delta = 10
result = []
last = L1[0] # first one as last value .. check the remaining list L1[1:]
for elem in L1[1:]:
if last-delta < elem < last+delta:
result.append(last)
last = elem
if elem-delta < result[-1] < elem+delta :
result.append(elem)
print(result) # [1, 2, 3, 8, 12, 17]