I'm stuck in this problem and I would need some help:
Given an array arr, in each step, 1, 2 or 5 units have to be incremented to all but one item of the array (same amount of units to all of them). The goal is to find the minimum number of steps to all items to be equal.
First example
arr = [1, 1, 5]
1) [1 (+2), 1 (+2), 5] = [3, 3, 5]
2) [3 (+2), 3 (+2), 5] = [5, 5, 5]
Solution: 2 steps
Second example
arr = [2, 2, 3, 7]
1) [2 (+1), 2 (+1), 3, 7 (+1)] = [3, 3, 3, 8]
2) [3 (+5), 3 (+5), 3 (+5), 8] = [8, 8, 8, 8]
Solution: 2 steps
I have tried some things but I'm really stuck.
I consider a base case when all items are already equal. In another case, I try to find all the possible solutions by incrementing 1, 2 and 5 to every item but one in the array:
def equal(arr):
if (allElementsIdentical(arr)):
return 0
min = sys.maxsize
for i in [1, 2, 5]:
for j in range(len(arr)):
#Increment all items by "i", except item at index "j"
newArr = arr.copy()
for k in range(j):
newArr[k] += i
for k in range(j + 1, len(newArr)):
newArr[k] += i
movements = 1 + equal(newArr)
if movements < min:
min = movements
return min
This solution doesn't work because recursion never ends. E.g.
[1, 1, 5] -> [1, 2, 6] -> [1, 3, 7] -> [1, 4, 8] -> [1, 5, 9] -> ...
Is it my initial approach correct? How can I break it down in subproblems properly? How can I get the recurrence relation?
(I'm learning Python, so any comment about the syntax is also appreciated)
To me adding 1, 2 or 5 to all but one element seems a lot easier to think about as subtracting 1, 2 or 5 from just one element.
[1, 1, 5] -> 5 - 2 -> 3 - 2
[2, 2, 3, 7] -> 3 - 1 -> 7 - 5
To construct a recurrence, we can use a histogram and consider that to shift any value would cost its frequency in operations. Since we are allowed to reduce by 1, we can easily set a lower-bound for the lowest target we might need to shift all values to. Since the lowest could be reached by any other value, shifting all values down to (lowest-5) (as the HackerRank editorial notes), would involve n more operations than shifting all elements down to the lowest, as we first shift all elements to the lowest, then apply (-5) to each one.
Also noted by the editorial is that the smallest number of operations, k, to shift x to target 0, can be found in O(1) by the greedy
k = x / 5 + (x % 5) / 2 + (x % 5) % 2
Since you've asked to rather try and form a recurrence, under these circumstances, we would be left with solving the coin change problem (coins [5, 2, 1]) for each value in the histogram to reach the target. These are independent: it makes no difference the order by which we apply coin_change to each value to find the number of operations needed, which we then multiply by the value's frequency. Tallying the total operations for all values to reach each target, we choose the least.
We want to replace this problem with one that produces the same answer but will be easier to evaluate.
The trick to making it easier to evaluate with a dynamic programming approach is to have the same results show up in lots of places. And therefore we have to replace equivalent versions of this problem with normalized ones.
For a start, the answer doesn't depend on the order that the elements of the array are in. So we can replace our arrays with arrays sorted from smallest to largest. The operation is now we add x to everything but one, then reorder to canonical form.
Next, the answer doesn't depend on the value of the smallest element. So we can subtract that value from all entries. The operation is now we add x to everything but one, then reorder to canonical form, then subtract the smallest from everything.
This greatly reduces our problem. Enough that a breadth first search has a shot. But we have one more trick to pull. And that trick is that it doesn't matter what order we apply the operations in. And therefore we can apply all of our 5 operations before our 2 operations before our 1 operations. With this trick, we can replace each normalized node with (node, last_operation) and a starting last_operation of 5. The reason why this is a win is that now we have an upper bound for the rest of an A* search. That bound is the current number of steps + sum of ceil(node[i] / last_operation).
And now this can be solved with A* search. Let's do your examples by hand. Using the notation, (total cost, normalized, last_operation, steps).
Example 1: [1, 1, 5]
We normalize to [0, 0, 4] and have a last_operation of 5 and a cost of 0+0+1 = 1. No steps taken. So we start with:
(1, [0, 0, 4], 5)
We take that out, and consider our operations. We get the following for operation 5:
[0, 0, 4] + [5, 5, 0] = [5, 5, 4] => [0, 1, 1] # cost 1 + 0+1+1 = 3
[0, 0, 4] + [5, 0, 5] = [5, 0, 9] => [0, 5, 9] # cost 1 + 0+1+2 = 4
[0, 0, 4] + [0, 5, 5] = [0, 5, 9] => [0, 5, 9] # cost 1 + 0+1+2 = 4 DUP
And for operation 2 we get:
[0, 0, 4] + [2, 2, 0] = [2, 2, 4] => [0, 0, 2] # cost 1 + 0+0+1 = 2
[0, 0, 4] + [2, 0, 2] = [2, 0, 4] => [0, 2, 4] # cost 1 + 0+1+2 = 4
[0, 0, 4] + [0, 2, 2] = [0, 2, 4] => [0, 2, 4] # cost 1 + 0+1+2 = 4 DUP
And for operation 1 we get:
[0, 0, 4] + [1, 1, 0] = [1, 1, 4] => [0, 0, 3] # cost 1 + 0+0+3 = 4
[0, 0, 4] + [1, 0, 1] = [1, 0, 4] => [0, 1, 4] # cost 1 + 0+1+5 = 6
[0, 0, 4] + [0, 1, 1] = [0, 1, 4] => [0, 1, 4] # cost 1 + 0+1+5 = 6 DUP
We stick the 7 non-dups into our priority queue, and the best that comes out looks like this:
(total cost, normalized, last_operation, steps)
( 2, [0,0,2], 2, 1)
We then try operations 2 and 1 on this, and of course find that one of the outcomes is [0, 0, 0] after 2 steps.
Related
This is a pretty specific usage case, but I'm hoping someone out there is more familiar with PyTorch tensors than I am and can help me speed this up.
I'm working on implementing a custom similarity metric for a neural network and have successfully gotten it to work, but it is incredibly slow to calculate. Each epoch takes about a minute to run, which simply isn't going to work with how I wanted to compare it with other metrics. So, I've been trying to utilize PyTorch tensors more effectively to speed things up, but haven't had much success.
Basically, I need to sum up the integers in the 'counts' tensor between the min and max indices specified in the 'min' and 'max' tensors for each sample and cluster combination.
As mentioned, my original implementation using loops took about a minute per epoch to run, but I did manage to reduce that to about 18-20 seconds using list comprehensions:
# counts has size (16, 100), max and min have size (2708, 7, 16)
data_mass = torch.sum(torch.tensor([[[torch.pow(torch.sum(counts[k][min[i][j][k]:max[i][j][k]+1]) / divisor, 2) for k in range(len(counts))] for j in range(len(min[i]))] for i in range(len(min))]), 2)
This feels super janky, and I've seen some clever things done with PyTorch functions, but I haven't been able to find anything yet that addresses quite what I want to do. Thanks in advance! I'm happy to clarify anything that may not be clear, I understand the use case is a bit convoluted.
EDIT: I'll try and break down the code snippet above and provide a minimal example. Examples of minimal inputs might look like the following:
'min' and 'max' are both 3-dimensional tensors of shape (num_samples, num_clusters, num_features), such as this one of size (2, 3, 4)
min = tensor([[[1, 2, 3, 1],
[2, 1, 1, 2],
[1, 2, 2, 1]],
[[2, 3, 2, 1],
[3, 3, 1, 2],
[1, 0, 2, 1]]])
max = tensor([[[3, 3, 4, 4],
[3, 2, 3, 4],
[2, 4, 3, 2]],
[[4, 4, 3, 3],
[4, 4, 2, 3],
[2, 1, 3, 2]]])
'counts' is a 2-dimensional tensor of size(num_features, num_bins),
so for this example we'll say size (4, 5)
counts = tensor([[1, 2, 3, 4, 5],
[2, 5, 3, 1, 1],
[1, 2, 3, 4, 5],
[2, 5, 3, 1, 1]])
The core part of the code snippet given above is the summation of the counts tensor between the values given by the min and max tensors for each pair of indices given at each index in max/min. For the first sample/cluster combo above:
mins = [1, 2, 3, 1]
maxes = [3, 3, 4, 4]
#Starting with feature #1 (leftmost element of min/max, top row of counts),
we sum the values in counts between the indices specified by min and max:
min_value = mins[0] = 1
max_value = maxes[0] = 3
counts[0] = [1, 2, 3, 4, 5]
subset = counts[0][mins[0]:maxes[0]+1] = [2, 3, 4]
torch.sum(subset) = 9
#Second feature
min_value = mins[1] = 2
max_value = maxes[1] = 3
counts[1] = [2, 5, 3, 1, 1]
subset = counts[0][mins[0]:maxes[0]+1] = [3, 1]
torch.sum(subset) = 4
In my code snippet, I perform a few additional operations, but if we ignore those and just sum all the index pairs, the output will have the form
pre_sum_output = tensor([[[9, 4, 9, 10],
[7, 8, 9, 5]
[5, 5, 7, 8]],
[[12, 2, 7, 9],
[9, 2, 5, 4],
[5, 7, 7, 8]]])
Finally, I sum the output one final time along the third dimension:
data_mass = torch.sum(pre_sum_output, 2) = torch.tensor([[32, 39, 25],
[30, 20, 27]])
I then need to repeat this for every pair of mins and maxes in 'min' and 'max' (each [i][j][k]), hence the list comprehension above iterating through i and j to get each sample and cluster respectively.
By noticing that torch.sum(counts[0][mins[0]:maxes[0]+1]) is equal to cumsum[maxes[0]] - cumsum[mins[0]-1] where cumsum = torch.cumsum(counts[0]), you can get rid of the loops like so:
# Dim of sample, clusters, etc.
S, C, F, B = range(4)
# Copy min and max over bins
min = min.unsqueeze(B)
max = max.unsqueeze(B)
# Copy counts over samples and clusters
counts = counts.reshape(
1, # S
1, # C
*counts.shape # F x B
)
# Number of samples, clusters, etc.
ns, nc, nf, nb = min.size(S), min.size(C), min.size(F), counts.size(B)
# Calculate cumulative sum and copy over samples and clusters
cum_counts = counts.cumsum(dim=B).expand(ns, nc, nf, nb)
# Prevent index error when min index is 0
is_zero = min == 0
lo = (min - 1).masked_fill(is_zero, 0)
# Compute the contiguous sum from min to max (inclusive)
lo_sum = cum_counts.gather(dim=B, index=lo)
hi_sum = cum_counts.gather(dim=B, index=max)
sum_counts = torch.where(is_zero, hi_sum, hi_sum - lo_sum)
pre_sum_output = sum_counts.squeeze(B)
You can then sum over the 2nd dim to get data_mass.
I need a generator that yields every bit pattern of n bits in order of their sum (and more). For example for n=3,
1. sum([0, 0, 0]) = 0 ✓
2. sum([1, 0, 0]) = 1 ✓
3. sum([0, 1, 0]) = 1 ✓
4. sum([1, 1, 0]) = 2 ⨯ should be swapped with 5.
5. sum([0, 0, 1]) = 1 ⨯
6. sum([1, 0, 1]) = 2 ✓
7. sum([0, 1, 1]) = 2 ✓
8. sum([1, 1, 1]) = 3 ✓
Note that even though 3 and 5 have the same sum, 5 should be generated after 3. The correct order would have been, 000, 100, 010, 001, 110, 101, 011, 111. The idea here is that if this (2ⁿ, n) dimension matrix was multiplied by a (n, 1) vector that is sorted in ascending order than the product would be sorted as well. Eliminating the need to sort this product will help in optimising an experiment I am trying to run.
Here is my rather inefficient and incomplete attempt,
def forwards(n):
ls = [0]*n
for i in range(n):
for j in range(n-i):
copy = list(ls)
copy[j] = 1
yield copy
ls[-(i+1)] = 1
As you can see, this does get the order right but misses some patterns.
Here is another somewhat efficient but wrong order attempt.
def forwards(n):
for i in range(1 << n):
yield [(i >> k) & 1 for k in range(n)]
This one generates all patterns but in the wrong (shown in the example above) order.
I need this function to be efficient so solutions where a string is generated and then characters are converted to integers are discouraged.
Lastly, I am working in Python 3.9. You can use Numpy.
I believe this gives you the order you want:
First, sort in ascending order by number of 1's
Next, sort in descending order by lexicographical order
from itertools import combinations
def forwards(n):
for one_bits in range(n + 1):
for combination in combinations(range(n), one_bits):
yield [1 if x in combination else 0 for x in range(n)]
for x in forwards(4):
print("{}, one bits = {}, binary value = {}".format(x, sum(x), sum(x[i] * 2**(3 - i) for i in range(4))))
This works by handling the first condition in the first for loop for one_bits in range(n + 1):, and generating all possible N-sized combinations of indices between 0 and n - 1, and then those indices are set to 1, else 0.
This prints
[0, 0, 0, 0], one bits = 0, binary value = 0
[1, 0, 0, 0], one bits = 1, binary value = 8
[0, 1, 0, 0], one bits = 1, binary value = 4
[0, 0, 1, 0], one bits = 1, binary value = 2
[0, 0, 0, 1], one bits = 1, binary value = 1
[1, 1, 0, 0], one bits = 2, binary value = 12
[1, 0, 1, 0], one bits = 2, binary value = 10
[1, 0, 0, 1], one bits = 2, binary value = 9
[0, 1, 1, 0], one bits = 2, binary value = 6
[0, 1, 0, 1], one bits = 2, binary value = 5
[0, 0, 1, 1], one bits = 2, binary value = 3
[1, 1, 1, 0], one bits = 3, binary value = 14
[1, 1, 0, 1], one bits = 3, binary value = 13
[1, 0, 1, 1], one bits = 3, binary value = 11
[0, 1, 1, 1], one bits = 3, binary value = 7
[1, 1, 1, 1], one bits = 4, binary value = 15
So say I have a list sequences such as this.
I want to remove all sequences where its total sum = N and/or it has a contiguous subarray with sum = N.
For example, if N = 4, then (1,1,2) is not valid since its total is 4. (1,1,3) is also not valid since the (1,3) is also 4. (1,3,1) is also not valid for the same reason.
lst = [
(1,1,1), (1,1,2), (1,1,3),
(1,2,1), (1,2,2), (1,2,3),
(1,3,1), (1,3,2), (1,3,3),
(2,1,1), (2,1,2), (2,1,3),
(2,2,1), (2,2,2), (2,2,3),
(2,3,1), (2,3,2), (2,3,3),
(3,1,1), (3,1,2), (3,1,3),
(3,2,1), (3,2,2), (3,2,3),
(3,3,1), (3,3,2), (3,3,3)
]
E.g.
Input: 4 3
Output: 2 1 2
So what I have right now is
lst = [t for t in list(product(range(1,n),repeat=n-1)) if not any((sum(t[l:h+1]) % n == 0) for l, h in combinations(range(len(t)), 2))]
Currently it is in O(n2) if I'm not mistaken. What would be a better way to do this?
If you can use numpy, you can concatenate the total sum of each tuple with the contiguous value sums, then check if any of your resultign elements are equal to 4:
arr = np.array(lst)
arr[~(np.concatenate((np.sum(arr,axis=1).reshape(-1,1),
(arr[:,:-1]+ arr[:,1:])),axis=1) == 4).any(1)]
# or:
arr[(np.concatenate((np.sum(arr,axis=1).reshape(-1,1),
(arr[:,:-1]+ arr[:,1:])),axis=1) != 4).all(1)]
Returning:
array([[1, 1, 1],
[1, 2, 3],
[2, 1, 2],
[2, 3, 2],
[2, 3, 3],
[3, 2, 1],
[3, 2, 3],
[3, 3, 2],
[3, 3, 3]])
I have been working on finite field. Suppose I have a prime number p=7. So I get a list q=[0,1,2,3,4,5,6]. Now I want all the possible permutation of the elements of set q for 7 places. For example [1,1,1,4,6,3,1] is one of the possible permutation. Is there any inbuilt command in python for doing that? Actually I am working with bigger field where P is 127 (p=127).
Those aren't permutations because elements are repeated, this looks more like a product.
you can use itertools.product on repeated q lists (here for 3 elements):
import itertools
q=[0,1,2] # or q = list(range(3))
for z in itertools.product(*(q,)*len(q)): # using arg unpacking like if it was (q,q,q)
z = list(z) # to convert as list
print(z)
prints:
[0, 0, 0]
[0, 0, 1]
[0, 0, 2]
[0, 1, 0]
[0, 1, 1]
[0, 1, 2]
...snip...
[2, 2, 0]
[2, 2, 1]
[2, 2, 2]
for p=3 it prints 3**3 = 27 values. If p=127 well... sounds not reasonable.
I have a numpy array X, and I'd like to return another array Y whose entries are the indices of the n largest values of X i.e. suppose I have:
a =np.array[[1, 3, 5], [4, 5 ,6], [9, 1, 7]]
then say, if I want the first 5 "maxs"'s indices-here 9, 7 , 6 , 5, 5 are the maxs, and their indices are:
b=np.array[[2, 0], [2 2], [ 2 1], [1 1], [0 , 2])
I've been able to find some solutions and make this work for a one dimensional array like
c=np.array[1, 2, 3, 4, 5, 6]:
def f(a,N):
return np.argsort(a)[::-1][:N]
But have not been able to generate something that works in more than one dimension. Thanks!
Approach #1
Get the argsort indices on its flattened version and select the last N indices. Then, get the corresponding row and column indices -
N = 5
idx = np.argsort(a.ravel())[-N:][::-1] #single slicing: `[:N-2:-1]`
topN_val = a.ravel()[idx]
row_col = np.c_[np.unravel_index(idx, a.shape)]
Sample run -
# Input array
In [39]: a = np.array([[1,3,5],[4,5,6],[9,1,7]])
In [40]: N = 5
...: idx = np.argsort(a.ravel())[-N:][::-1]
...: topN_val = a.ravel()[idx]
...: row_col = np.c_[np.unravel_index(idx, a.shape)]
...:
In [41]: topN_val
Out[41]: array([9, 7, 6, 5, 5])
In [42]: row_col
Out[42]:
array([[2, 0],
[2, 2],
[1, 2],
[1, 1],
[0, 2]])
Approach #2
For performance, we can use np.argpartition to get top N indices without keeping sorted order, like so -
idx0 = np.argpartition(a.ravel(), -N)[-N:]
To get the sorted order, we need one more round of argsort -
idx = idx0[a.ravel()[idx0].argsort()][::-1]