I have three numpy arrays, let's say X, Y and Z.
X contains n arrays of dimension m, i.e. [[x11,x12,...,x1m],[x21,x22,...,x2m],...,[xn1,xn2,...,xnm]]
Y contains k (k > n) arrays of dimension m, i.e. [[y11,y12,...,y1m],[y21,y22,...,y2m],...,[yk1,yk2,...,ykm]]
Z contains p (p < k, p < n) arrays of dimension m, i.e. [[z11,z12,...,z1m],[z21,z22,...,z2m],...,[zp1,zp2,...,zpm]]
For each element Z[i] of the array Z, I need to compute the distance (euclidian) to every element of the array X and select the minimum distance, which will be denoted by dist_X[i]. I have to do the same but with the array Y and denote the minimum distance by dist_Y[i]. Then, for each element Z[i] of Z, I have to compute the value of dist_Y[i]/(dist_Y[i]+dist_X[i]).
I tried doing something like this:
import scipy
from scipy import spatial
def dist_sets(z):
tree_X = spatial.cKDTree(X)
tree_Y = spatial.cKDTree(Y)
dist_X, minid_X=tree_X.query(z)
dist_Y, minid_Y=tree_Y.query(z)
return dist_Y/(dist_Y+dist_X)
print(dist_sets(Z))
However, it takes A LOT of computing time for large n,k and p; for example (n,m)=(17727, 122), (k,m)=(542273, 122) and (p,m)=(140001, 122).
Is there a way to optimize the code in Python, in such a way that I could evaluate the function dist_sets(Z) for all the elements of Z?
The docs for KDTree mention that performance benefits deteriorate for larger dimensions. With 122 of them, you are probably better of with a naive vectorized solution. Here is one possibility:
from sklearn.metrics import pairwise_distances_argmin_min
def dist_sets2(Z):
iX, dX = pairwise_distances_argmin_min(Z, X)
iY, dY = pairwise_distances_argmin_min(Z, Y)
return dY / (dX + dY)
For k = p = 1000, this is 17 times faster on my machine than using cKDTree.
Related
Given are positive integers n, a, b, p[1], p[2], p[3] (All <= 10^9). Suppose we are standing at position x=0, and we want to reach point n. We can move only to right. For 1 move, we can move by 1 at cost p[1], move by a at cost p[2] and move by b at cost p[3]. The task is to find minimum cost if we can use this moves. There is no restrictions on number of moves.
The ideas I came up with are dynamic programming and linear equations with 3 variables. But still can't solve the problem.
UPD: Suppose also we are given with T (T <= 50) independent queries.
Think of the last step: it would be to move right by 1 unit, a units, or b units. If the last step is to move right 1 unit, then the total cost is p[1]+mincost(n-1). Similarly for the other two possibilities.
Thus, we get the recursive equation: mincost(n) = min{p[1]+mincost(n-1), p[2]+mincost(n-a), p[3]+mincost(n-b)}. This can be implemented using top-down recursion with memoization or bottom-up dynamic programming.
You mentioned that you had two ideas:
using a recurrence relation to write a dynamic programming algorithm;
solving a linear equation.
Both ideas are great. Since another answer already focuses on dynamic programming, I will focus on the linear equation.
Here is a reformulation of your problem:
MINIMIZE:
x * p[1] + y * p[2] + z * p[3]
UNDER CONSTRAINTS:
x + y * a + z * b = n
x, y, z ≥ 0
x, y, z are integers
This is an integer linear program. Formulating a problem as a linear program or as an integer linear program is a very useful skill. Congratulations!
If this is an interview question, the interviewer will probably be very happy with you simply formulating the problem as a linear program. Thus the "code" I wrote above would probably satisfy them.
There are many existing solvers for linear programs.
For instance, here is code in python, using library PuLP:
from pulp import LpProblem, LpVariable, LpMinimize
def min_cost_to_reach_point(n, a, b, p):
x = LpVariable('x', lowBound=0, cat='Integer')
y = LpVariable('y', lowBound=0, cat='Integer')
z = LpVariable('z', lowBound=0, cat='Integer')
P = LpProblem('min_cost_to_reach_point', LpMinimize)
P += x * p[0] + y * p[1] + z * p[2]
P += x + y * a + z * b == n
P.solve()
return {'cost': P.objective.value(), 'steps_1': x.value(), 'steps_a': y.value(), 'steps_b': z.value()}
n = 100
a, b = 11, 17
p = (2, 20, 20)
print( min_cost_to_reach_point(n, a, b, p) )
# {'cost': 128, 'steps_1': 4, 'steps_a': 1, 'steps_b': 5}
I have one hermitian matrix (specifically, a Hamiltonian). Though phase of a singe eigenvector can be arbitrary, the quantities I am calculating is physical (I reduced the code a bit keeping just the reproducible part). eig and eigh are giving very different results.
import numpy as np
import numpy.linalg as nlg
import matplotlib.pyplot as plt
def Ham(Ny, Nx, t, phi):
h = np.zeros((Ny,Ny), dtype=complex)
for ii in range(Ny-1):
h[ii+1,ii] = t
h[Ny-1,0] = t
h=h+np.transpose(np.conj(h))
u = np.zeros((Ny,Ny), dtype=complex)
for ii in range(Ny):
u[ii,ii] = -t*np.exp(-2*np.pi*1j*phi*ii)
u = u + 1e-10*np.eye(Ny)
H = np.kron(np.eye(Nx,dtype=int),h) + np.kron(np.diag(np.ones(Nx-1), 1),u) + np.kron(np.diag(np.ones(Nx-1), -1),np.transpose(np.conj(u)))
H[0:Ny,Ny*(Nx-1):Ny*Nx] = np.transpose(np.conj(u))
H[Ny*(Nx-1):Ny*Nx,0:Ny] = u
x=[]; y=[];
for jj in range (1,Nx+1):
for ii in range (1,Ny+1):
x.append(jj); y.append(ii)
x = np.asarray(x)
y = np.asarray(y)
return H, x, y
def C_num(Nx, Ny, E, t, phi):
H, x, y = Ham(Ny, Nx, t, phi)
ifhermitian = np.allclose(H, np.transpose(np.conj(H)), rtol=1e-5, atol=1e-8)
assert ifhermitian == True
Hp = H
V,wf = nlg.eigh(Hp) ##Check. eig gives different result
idx = np.argsort(np.real(V))
wf = wf[:, idx]
normmat = wf*np.conj(wf)
norm = np.sqrt(np.sum(normmat, axis=0))
wf = wf/(norm*np.sqrt(len(H)))
wf = wf[:, V<=E] ##Chose a subset of eigenvectors
V01 = wf*np.exp(1j*x)[:,None]; V12 = wf*np.exp(1j*y)[:,None]
V23 = wf*np.exp(1j*x)[:,None]; V30 = wf*np.exp(1j*y)[:,None]
wff = np.transpose(np.conj(wf))
C01 = np.dot(wff,V01); C12 = np.dot(wff,V12); C23 = np.dot(wff,V23); C30 = np.dot(wff,V30)
F = nlg.multi_dot([C01,C12,C23,C30])
ifhermitian = np.allclose(F, np.transpose(np.conj(F)), rtol=1e-5, atol=1e-8)
assert ifhermitian == True
evals, efuns = nlg.eig(F) ##Check eig gives different result
C = (1/(2*np.pi))*np.sum(np.angle(evals));
return C
C = C_num(16, 16, 0, 1, 1/8)
print(C)
Changing both nlg.eigh to nlg.eig, or even changing only the last one, giving very different results.
As I mentioned elsewhere, the eigenvalue and eigenvector are not unique.
The only thing that is true is that for each eigenvalue $A v = lambda v$, the two matrices returned by eig and eigh describe those solutions, it is natural that eig inexact but approximate results.
You can see that both the solutions will triangularize your matrix in different ways
H, x, y = Ham(16, 16, 1, 1./8)
D, V = nlg.eig(H)
Dh, Vh = nlg.eigh(H)
Then
import matplotlib.pyplot as plt
plt.figure(figsize=(14, 7))
plt.subplot(121);
plt.imshow(abs(np.conj(Vh.T) # H # Vh))
plt.title('diagonalized with eigh')
plt.subplot(122);
plt.imshow(abs(np.conj(V.T) # H # V))
plt.title('diagonalized with eig')
Plots this
That both diagonalizations were successfull, but the eigenvalues are indifferent order.
If you sort the eigenvalues you see they match
plt.plot(np.diag(np.real(np.conj(Vh.T) # H # Vh)))
plt.plot(np.diag(np.imag(np.conj(Vh.T) # H # Vh)))
plt.plot(np.sort(np.diag(np.real(np.conj(V.T) # H # V))))
plt.title('eigenvalues')
plt.legend(['real eigh', 'imag eigh', 'sorted real eig'], loc='upper left')
Since many eigenvalues are repeated, the eigenvector associated with a given eigenvalue is not unique as well, the only thing we can guarantee is that the eigenvectors for a given eigenvalue must span the same subspace.
The diagonalization test is the best in my opinion.
Is eigh always better than eig?
If you search for the eigenvalues in the lapack routines you will have many options. So it is I cannot discuss each possible implementation here. The common sense says that we can expect that the symmetric/hermitian routines to perform better, otherwise ther would be no reason to add one more routine that is more limited. But I never tested carefully the behavior of eig vs eigh.
To have an intuition compare the equation for tridiagonalization for symmetric matrices, and the equation for reduction of a general matrix to its Heisenberg form found here.
I'm not a mathematician so please excuse me if I use the wrong vocabulary. This is not for any assignment or homework.
From point (0,0) to (4, -1) there are five possible 'taxi-cab' paths. Let n = sum of absolute difference between points, 5. Number of paths from either one to the other = (n * n-1) / 2 = 10. From one point to the other, half of that = 5.
But when I hand-draw this I count 13 'edges' (correct term?) between all the nodes or vertices(?) used. It's this number I'd like to calculate for any n-dimensional graph (up to 6-dimensions). How does one arrive at 13 from the vector (4, -1)?
If x-difference is dx, and y-difference is dy, then number of edges is
NE = dx*(dy+1) + dy*(dx+1)
(number of horizontal edges + number of vertical edges)
For your case dx=4, dy=1, and NE =4*2+5*1=13
For dx=4, dy=2 NE= 4*3+5*2=22
The same logic works for n-dimensional grids. For example, in 3D there are
dy*(dx+1)*(dz+1) vertical edges (along OY)
dx*(dy+1)*(dz+1) horizontal edges (along OX)
dz*(dx+1)*(dy+1) edges in depth (along OZ)
that gives 144 edges for 3x3x3 cube
Simple Python (suitable for any language) program to calculate this quantity for any dimension:
def numedges(dims:list):
prod = 1
for dim in dims:
prod *= (dim + 1)
ne = 0
for dim in dims:
ne += prod * dim // (dim+1) #integer division
return ne
print(numedges([2,4]))
print(numedges([3,3,3]))
print(numedges([2,3,4,5,6,7]))
>>>
22
144
96408
Using reduce:
from functools import reduce
def numedges(dims:list):
product = reduce(lambda x,y: x*(y+1), dims, 1)
return(reduce(lambda x,y: x+product*y//(y+1), dims, 0))
I'd like some help with numpy and arrays. I want to calculate the gradient of a vector field.
Suppose I have a function foo that takes a tuple of coordinates (x,y,z) and returns a vector (u,v,w).
Then if I have an array of coordinates POS = [[x1,y1,z1],[x2,y2,z2],[x3,y3,z3],etc] I can generate an array of vectors with origin in pos and direction in DIR = [[u1,v1,w1],[u2,v2,w2],[u3,v3,w3],etc].
Now how could I calculate the gradient of this vector field in every point of POS ? What I need in the end would be something like another array GRAD = [grad1, grad2, grad3, etc] where every grad would be a 3x3 array of the partial derivatives of the vector field in that corresponding point in POS.
PS: I know I can derivate manually the function foo and then implement the derivatives in python but in my case the function foo is really complex and I want to do the derivatives this way :)
EDIT1: for now I'm coming up with POS this way:
parts = 100
limit = 10
xs = linspace(-limit, limit, parts)
ys = linspace(-limit, limit, parts)
zs = linspace(-limit, limit, parts)
POS = array([(x, y, z) for z in zs for y in ys for x in xs])
DIR = array([foo(pos) for pos in POS])
which allows me to do this if necessary:
POS = POS.reshape(parts,parts,parts,3)
DIR = DIR.reshape(parts,parts,parts,3)
You can use numpy.gradient for this in the following way:
import numpy as np
N = 100
limit = .1
def vec(x,y,z): # Example vector field
return np.array([x,x,z])
x = np.arange(-limit, limit, 2*limit/N) # np.arange takes the spacing as 3. arg
y = np.arange(-limit, limit, 2*limit/N)
z = np.arange(-limit, limit, 2*limit/N)
# Create 3D grid from 1D arrays, indexing is important!
X,Y,Z = np.meshgrid(x,y,z,indexing='ij')
V = vec(X,Y,Z) # Get vector field, shape: (3,N,N,N)
D = np.gradient(V, x, y, z, axis=(1,2,3)) # Get gradient, this is a list!
D = np.array(D).transpose((1,0,2,3,4)) # Make it an array and flip first axes.
The resulting array has shape (3,3,N,N,N) where the first axis indexes the component of the vector field and the second axis indexes the coordinate direction w.r.t. which the derivative was computed.
I want to do this sort of loop in Theano:
def add_multiply(a,b, k):
return a+b+k, a*b*k, k
x=1
y=2
k=1
tuples = []
for i in range(5):
x,y,k = add_multiply(x,y,k)
tuples.append((x,y,k))
However, when I do
x0 = T.dvector('x0')
i = T.iscalar('i')
results,updates=th.scan(fn=add_multiply,outputs_info=[{'initial':x0,'taps':[-1]}],n_steps=i)
I get TypeError: add_multiply() takes exactly 3 arguments (1 given). If I change it so that the function takes a single tuple instead, I get ValueError: length not known
In particular, I eventually want to differentiate the entire result with respect to k.
The first error is because your add_multiply function takes 3 arguments but, by having only one element in the outputs_info list, you're only providing a single argument. It's not clear if you intended the x0 vector to be the initial value for just a or were expecting it to be spread over a, b, and k. The latter isn't supported by Theano and, in general, tuples are not supported by Theano. In Theano, everything needs to be a tensor (e.g. scalars are just special types of tensors with zero dimensions).
You can achieve a replica of the Python implementation in Theano as follows.
import theano
import theano.tensor as tt
def add_multiply(a, b, k):
return a + b + k, a * b * k
def python_main():
x = 1
y = 2
k = 1
tuples = []
for i in range(5):
x, y = add_multiply(x, y, k)
tuples.append((x, y, k))
return tuples
def theano_main():
x = tt.constant(1, dtype='uint32')
y = tt.constant(2, dtype='uint32')
k = tt.scalar(dtype='uint32')
outputs, _ = theano.scan(add_multiply, outputs_info=[x, y], non_sequences=[k], n_steps=5)
g = theano.grad(tt.sum(outputs), k)
f = theano.function(inputs=[k], outputs=outputs + [g])
tuples = []
xvs, yvs, _ = f(1)
for xv, yv in zip(xvs, yvs):
tuples.append((xv, yv, 1))
return tuples
print 'Python:', python_main()
print 'Theano:', theano_main()
Note that in the Theano version, all the tuple handling happens outside Theano; Python has to convert from the three tensors returned by the Theano function into a list of tuples.
Update:
It's unclear what "the entire result" should refer to but the code has been updated to show how you might differentiate with respect to k. Note that in Theano the symbolic differentiation only works with scalar expressions, but can differentiate with respect to multi-dimensional tensors.
In this update the add_multiply method no longer returns k since that is constant. For similar reasons, the Theano version now accepts k as a non_sequence.