In the code supplied below I am trying to iterate over 2D numpy array [i][k]
Originally it is a code which was written in Fortran 77 which is older than my grandfather. I am trying to adapt it to python.
(for people interested whatabouts: it is a simple hydraulics transients event solver)
Bear in mind that all variables are introduced in my code which I don't paste here.
H = np.zeros((NS,50))
Q = np.zeros((NS,50))
Here I am assigning the first row values:
for i in range(NS):
H[0][i] = HR-i*R*Q0**2
Q[0][i] = Q0
CVP = .5*Q0**2/H[N]
T = 0
k = 0
TAU = 1
#Interior points:
HP = np.zeros((NS,50))
QP = np.zeros((NS,50))
while T<=Tmax:
T += dt
k += 1
for i in range(1,N):
CP = H[k][i-1]+Q[k][i-1]*(B-R*abs(Q[k][i-1]))
CM = H[k][i+1]-Q[k][i+1]*(B-R*abs(Q[k][i+1]))
HP[k][i-1] = 0.5*(CP+CM)
QP[k][i-1] = (HP[k][i-1]-CM)/B
#Boundary Conditions:
HP[k][0] = HR
QP[k][0] = Q[k][1]+(HP[k][0]-H[k][1]-R*Q[k][1]*abs(Q[k][1]))/B
if T == Tc:
TAU = 0
CV = 0
else:
TAU = (1.-T/Tc)**Em
CV = CVP*TAU**2
CP = H[k][N-1]+Q[k][N-1]*(B-R*abs(Q[k][N-1]))
QP[k][N] = -CV*B+np.sqrt(CV**2*(B**2)+2*CV*CP)
HP[k][N] = CP-B*QP[k][N]
for i in range(NS):
H[k][i] = HP[k][i]
Q[k][i] = QP[k][i]
Remember i is for rows and k is for columns
What I am expecting is that for all k number of columns the values should be calculated until T<=Tmax condition is met. I cannot figure out what my mistake is, I am getting the following errors:
RuntimeWarning: divide by zero encountered in true_divide
CVP = .5*Q0**2/H[N]
RuntimeWarning: invalid value encountered in multiply
QP[N][k] = -CV*B+np.sqrt(CV**2*(B**2)+2*CV*CP)
QP[N][k] = -CV*B+np.sqrt(CV**2*(B**2)+2*CV*CP)
ValueError: setting an array element with a sequence.
Looking at your first iteration:
H = np.zeros((NS,50))
Q = np.zeros((NS,50))
for i in range(NS):
H[0][i] = HR-i*R*Q0**2
Q[0][i] = Q0
The shape of H is (NS,50), but when you iterate over a range(NS) you apply that index to the 2nd dimension. Why? Shouldn't it apply to the dimension with size NS?
In numpy arrays have 'C' order by default. Last dimension is inner most. They can have a F (fortran) order, but let's not go there. Thinking of the 2d array as a table, we typically talk of rows and columns, though they don't have a formal definition in numpy.
Lets assume you want to set the first column to these values:
for i in range(NS):
H[i, 0] = HR - i*R*Q0**2
Q[i, 0] = Q0
But we can do the assignment whole rows or columns at a time. I believe new versions of Fortran also have these 'whole-array' functions.
Q[:, 0] = Q0
H[:, 0] = HR - np.arange(NS) * R * Q0**2
One point of caution when translating to Python. Indexing starts with 0; so does ranges and np.arange(...).
H[0][i] is functionally the same as H[0,i]. But when using slices you have to use the H[:,i] format.
I suspect your other iterations have similar problems, but I'll stop here for now.
Regarding the errors:
The first:
RuntimeWarning: divide by zero encountered in true_divide
CVP = .5*Q0**2/H[N]
You initialize H as zeros so it is normal that it complains of division by zero. Maybe you should add a conditional.
The third:
QP[N][k] = -CV*B+np.sqrt(CV**2*(B**2)+2*CV*CP)
ValueError: setting an array element with a sequence.
You define CVP = .5*Q0**2/H[N] and then CV = CVP*TAU**2 which is a sequence. And then you try to assign a derivate form it to QP[N][K] which is an element. You are trying to insert an array to a value.
For the second error I think it might be related to the third. If you could provide more information I would like to try to understand what happens.
Hope this has helped.
Related
I have a nested loop that has to loop through a huge amount of data.
Assuming a data frame with random values with a size of 1000,000 rows each has an X,Y location in 2D space. There is a window of 10 length that go through all the 1M data rows one by one till all the calculations are done.
Explaining what the code is supposed to do:
Each row represents a coordinates in X-Y plane.
r_test is containing the diameters of different circles of investigations in our 2D plane (X-Y plane).
For each 10 points/rows, for every single diameter in r_test, we compare the distance between every point with the remaining 9 points and if the value is less than R we add 2 to H. Then we calculate H/(N**5) and store it in c_10 with the index corresponding to that of the diameter of investigation.
For this first 10 points finally when the loop went through all those diameters in r_test, we read the slope of the fitted line and save it to S_wind[ii]. So the first 9 data points will have no value calculated for them thus giving them np.inf to be distinguished later.
Then the window moves one point down the rows and repeat this process till S_wind is completed.
What's a potentially better algorithm to solve this than the one I'm using? in python 3.x?
Many thanks in advance!
import numpy as np
import pandas as pd
####generating input data frame
df = pd.DataFrame(data = np.random.randint(2000, 6000, (1000000, 2)))
df.columns= ['X','Y']
####====creating upper and lower bound for the diameter of the investigation circles
x_range =max(df['X']) - min(df['X'])
y_range = max(df['Y']) - min(df['Y'])
R = max(x_range,y_range)/20
d = 2
N = 10 #### Number of points in each window
#r1 = 2*R*(1/N)**(1/d)
#r2 = (R)/(1+d)
#r_test = np.arange(r1, r2, 0.05)
##===avoiding generation of empty r_test
r1 = 80
r2= 800
r_test = np.arange(r1, r2, 5)
S_wind = np.zeros(len(df['X'])) + np.inf
for ii in range (10,len(df['X'])): #### maybe the code run slower because of using len() function instead of a number
c_10 = np.zeros(len(r_test)) +np.inf
H = 0
C = 0
N = 10 ##### maybe I should also remove this
for ind in range(len(r_test)):
for i in range (ii-10,ii):
for j in range(ii-10,ii):
dd = r_test[ind] - np.sqrt((df['X'][i] - df['X'][j])**2+ (df['Y'][i] - df['Y'][j])**2)
if dd > 0:
H += 1
c_10[ind] = (H/(N**2))
S_wind[ii] = np.polyfit(np.log10(r_test), np.log10(c_10), 1)[0]
You can use numpy broadcasting to eliminate all of the inner loops. I'm not sure if there's an easy way to get rid of the outermost loop, but the others are not too hard to avoid.
The inner loops are comparing ten 2D points against each other in pairs. That's just dying for using a 10x10x2 numpy array:
# replacing the `for ind` loop and its contents:
points = np.hstack((np.asarray(df['X'])[ii-10:ii, None], np.asarray(df['Y'])[ii-10:ii, None]))
differences = np.subtract(points[None, :, :], points[:, None, :]) # broadcast to 10x10x2
squared_distances = (differences * differences).sum(axis=2)
within_range = squared_distances[None,:,:] < (r_test*r_test)[:, None, None] # compare squares
c_10 = within_range.sum(axis=(1,2)).cumsum() * 2 / (N**2)
S_wind[ii] = np.polyfit(np.log10(r_test), np.log10(c_10), 1)[0] # this is unchanged...
I'm not very pandas savvy, so there's probably a better way to get the X and Y values into a single 2-dimensional numpy array. You generated the random data in the format that I'd find most useful, then converted into something less immediately useful for numeric operations!
Note that this code matches the output of your loop code. I'm not sure that's actually doing what you want it to do, as there are several slightly strange things in your current code. For example, you may not want the cumsum in my code, which corresponds to only re-initializing H to zero in the outermost loop. If you don't want the matches for smaller values of r_test to be counted again for the larger values, you can skip that sum (or equivalently, move the H = 0 line to in between the for ind and the for i loops in your original code).
I am trying to find the indices of the n smallest values in a list of tensors in pytorch. Since these tensors might contain many non-unique values, I cannot simply compute percentiles to obtain the indices. The ordering of non-unique values does not matter however.
I came up with the following solution but am wondering if there is a more elegant way of doing it:
import torch
n = 10
tensor_list = [torch.randn(10, 10), torch.zeros(20, 20), torch.ones(30, 10)]
all_sorted, all_sorted_idx = torch.sort(torch.cat([t.view(-1) for t in tensor_list]))
cum_num_elements = torch.cumsum(torch.tensor([t.numel() for t in tensor_list]), dim=0)
cum_num_elements = torch.cat([torch.tensor([0]), cum_num_elements])
split_indeces_lt = [all_sorted_idx[:n] < cum_num_elements[i + 1] for i, _ in enumerate(cum_num_elements[1:])]
split_indeces_ge = [all_sorted_idx[:n] >= cum_num_elements[i] for i, _ in enumerate(cum_num_elements[:-1])]
split_indeces = [all_sorted_idx[:n][torch.logical_and(lt, ge)] - c for lt, ge, c in zip(split_indeces_lt, split_indeces_ge, cum_num_elements[:-1])]
n_smallest = [t.view(-1)[idx] for t, idx in zip(tensor_list, split_indeces)]
Ideally a solution would pick a random subset of the non-unique values instead of picking the entries of the first tensor of the list.
Pytorch does provide a more elegant (I think) way to do it, with torch.unique_consecutive (see here)
I'm going to work on a tensor, not a list of tensors because as you did yourself, there's just a cat to do. Unraveling the indices afterward is not hard either.
# We want to find the n=3 min values and positions in t
n = 3
t = torch.tensor([1,2,3,2,0,1,4,3,2])
# To get a random occurrence, we create a random permutation
randomizer = torch.randperm(len(t))
# first, we sort t, and get the indices
sorted_t, idx_t = t[randomizer].sort()
# small util function to extract only the n smallest values and positions
head = lambda v,w : (v[:n], w[:n])
# use unique_consecutive to remove duplicates
uniques_t, counts_t = head(*torch.unique_consecutive(sorted_t, return_counts=True))
# counts_t.cumsum gives us the position of the unique values in sorted_t
uniq_idx_t = torch.cat([torch.tensor([0]), counts_t.cumsum(0)[:-1]], 0)
# And now, we have the positions of uniques_t values in t :
final_idx_t = randomizer[idx_t[uniq_idx_t]]
print(uniques_t, final_idx_t)
#>>> tensor([0,1,2]), tensor([4,0,1])
#>>> tensor([0,1,2]), tensor([4,5,8])
#>>> tensor([0,1,2]), tensor([4,0,8])
EDIT : I think the added permutation solves your need-random-occurrence problem
I am writing a function to evaluate and return a non linear system of equations, and give the jacobian. I then plan to call the function in a while loop to use the newton method to solve the system of equations.
I used the numpy package and read over its documentation, tried to limit the number of iterations, changed the dtype in the array and searched online to see if someone else had a similar problem.
This function is meant to solve a neoclassical growth model (a problem in macroeconomics) in finite time , T. The set of equations include T euler equations, T constraints, and one terminal condition. Thus the result should be an array of length 2T+1 containing the values of the equations, and a (2T+1)x(2T+1) jacobian matrix.
When I try to run the function for small array (arrays of length 1, and 3) it works perfectly. As soon as I try an array of length 5 or more, I start encountering RuntimeWarnings.
import numpy as np
def solver(args, params):
b,s,a,d = params[0], params[1], params[2], params[3]
guess = np.copy(args)
#Euler
euler = guess[:len(guess)//2]**(-sigma) - beta*guess[1:len(guess)//2+1]**(-sigma)*(1-delta+alpha*guess[len(guess)//2+1:]**(alpha-1))
#Budget Constraint
kzero_to_T = np.concatenate(([k0], guess[len(guess)//2+1:]))
bc_t = guess[:len(guess)//2] + guess[len(guess)//2+1:] - kzero_to_T[:-1]**alpha - (1-delta)*kzero_to_T[:-1]
bc_f = guess[len(guess)//2] -kzero_to_T[-1]**alpha - kzero_to_T[-1]*(1-delta)
bc = np.hstack((bc_t, bc_f))
Evals = np.concatenate((euler, bc))
# top half of the jacobian
jac_dot_5 = np.zeros((len(args)//2, len(args)))
for t in range(len(args)//2):
for i in range(len(args)):
if t == i and len(args)//2+(i+1)<=len(args):
jac_dot_5[t][t] = -sigma*args[t]**(-sigma-1)
jac_dot_5[t][t+1] = sigma*beta*args[t+1]*(1-delta+alpha*args[len(args)//2+(t+1)]**(alpha-1))
jac_dot_5[t][len(args)//2+(t+1)] = beta*args[t+1]**(-sigma)*alpha*(alpha-1)*args[len(args)//2+(t+1)]
# bottom half of the jacobian
jac_dot_1 = np.zeros((len(args)//2, len(args)))
for u in range(len(args)//2):
for v in range(len(args)):
if u==v and u>=1 and (len(args)//2 + u+1 < len(args)):
jac_dot_1[u][u] = 1
jac_dot_1[u][len(args)//2+(u)] = 1
jac_dot_1[u][len(args)//2+(u+1)] = -alpha*args[len(args)//2 + (u+1)]**(alpha-1) -(1-delta)
jac_dot_1[0][0] = 1
jac_dot_1[0][len(args)//2 +1] = 1
# last row of the jacobian
final_bc = np.zeros((1,len(args)))
final_bc[0][len(args)//2] = 1
final_bc[0][-1] = -alpha*args[-1]**(alpha-1) -(1-delta)
jac2Tn1 = np.concatenate((jac_dot_5, jac_dot_1, final_bc), axis=0)
point = coll.namedtuple('point', ['Output', 'Jacobian', 'Euler', 'BC'])
result = point(Output = Evals, Jacobian = jac2Tn1, Euler=euler, BC=bc )
return result
The code for implementing the algorithm:
p = (beta, sigma, alpha, delta)
for i in range(20):
k0 = np.linspace(2.49, 9.96, 20)[i]
vars0 = np.array([1,1,1,1,1], dtype=float)
vars1 = np.array([20,20,20,20,20], dtype=float)
Iter2= 0
while abs(solver(vars1,p).Output).max()>1e-8 and Iter2<300:
Iter2+=1
inv_jac1 = np.linalg.inv(solver(vars0,p).Jacobian)
vars1 = vars0 - inv_jac1#solver(vars0,p).Output
vars0=vars1
if Iter2 == 100:
break
I expect the output to be vars1 containing the updated values. The actual output is array([nan, nan, nan, nan, nan]). The way the function has been written, it should be able to give the output for inputs of arbitrary guesses of length 2T+1, where T is number of periods of time.
I get three error messages during the execution of the loop:
C:\Users\Peter\Anaconda3\lib\site-packages\ipykernel_launcher.py:19: RuntimeWarning: invalid value encountered in power
C:\Users\Peter\Anaconda3\lib\site-packages\ipykernel_launcher.py:23: RuntimeWarning: invalid value encountered in power
C:\Users\Peter\Anaconda3\lib\site-packages\ipykernel_launcher.py:41: RuntimeWarning: invalid value encountered in double_scalars
I tried to code my issue from scratch and I couldn't make it any shorter- I need both, the evaluations of the equations and the jacobian to implement the algorithm. From my testing it looks like at some point the equation results (the solver(vars0,p).Output entry) become nan, but I am not sure why that would happen, the array should get close to 0, per the condition abs(solver(vars1,p).Output).max()>1e-8 and then just break out of the loop.
I want to implement Karatsuba multiplication algorithm in python.But it is not working completely.
The code is not working for the values of x or y greater than 999.For inputs below 1000,the program is showing correct result.It is also showing correct results on base cases.
#Karatsuba method of multiplication.
f = int(input()) #Inputs
e = int(input())
def prod(x,y):
r = str(x)
t = str(y)
lx = len(r) #Calculation of Lengths
ly = len(t)
#Base Case
if(lx == 1 or ly == 1):
return x*y
#Other Case
else:
o = lx//2
p = ly//2
a = x//(10*o) #Calculation of a,b,c and d.
b = x-(a*10*o) #The Calculation is done by
c = y//(10*p) #calculating the length of x and y
d = y-(c*10*p) #and then dividing it by half.
#Then we just remove the half of the digits of the no.
return (10**o)*(10**p)*prod(a,c)+(10**o)*prod(a,d)+(10**p)*prod(b,c)+prod(b,d)
print(prod(f,e))
I think there are some bugs in the calculation of a,b,c and d.
a = x//(10**o)
b = x-(a*10**o)
c = y//(10**p)
d = y-(c*10**p)
You meant 10 to the power of, but wrote 10 multiplied with.
You should train to find those kinds of bugs yourself. There are multiple ways to do that:
Do the algorithm manually on paper for specific inputs, then step through your code and see if it matches
Reduce the code down to sub-portions and see if their expected value matches the produced value. In your case, check for every call of prod() what the expected output would be and what it produced, to find minimal input values that produce erroneous results.
Step through the code with the debugger. Before every line, think about what the result should be and then see if the line produces that result.
I'm trying to calculate histogram for an image. I'm using the following formula to calculate the bin
%bin = red*(N^2) + green*(N^1) + blue;
I have to implement the following Matlab functions.
[row, col, noChannels] = size(rgbImage);
hsvImage = rgb2hsv(rgbImage); % Ranges from 0 to 1.
H = zeros(4,4,4);
for col = 1 : columns
for row = 1 : rows
hBin = floor(hsvImage(row, column, 1) * 15);
sBin = floor(hsvImage(row, column, 2) * 4);
vBin = floor(hsvImage(row, column, 3) * 4);
F(hBin, sBin, vBin) = hBin, sBin, vBin + 1;
end
end
When I run the code I get the following error message "Subscript indices must either be real positive integers or logical."
As I am new to Matlab and Image processing, I'm not sure if the problem is with implementing the algorithm or a syntax error.
There are 3 problems with your code. (Four if you count that you changed from H to F your accumulator vector, but I'll assume that's a typo.)
First one, your variable bin can be zero at any moment if the values of a giving pixel are low. And F(0) is not a valid index for a vector or matrix. This is why you are getting that error.
You can solve easily by doing F(bin+1) and keep in mind that your F vector will have your values shifted one position over.
Second error, you are assigning the value bin + 1 to your accumulator vector F, which is not what you want, you want to add 1 every time a pixel in that range is found, what you should do is F(bin+1) = F(bin+1) + 1;. This way the values of F will be increasing all the time.
Third error is simpler, you forgot to implement your bin = red*(N^2) + green*(N^1) + blue; equation