Related
I am trying to optimize a funciton that is trying to maximize the correlation between two (pandas) time series arrays (X and Y). This is done by using three parameters (a, b, c) and a third time series array (Z). The Z array is used to reindex the values in the X array (based on the parameters a, b, c) in such a way as to maximize the correlation of the reindexed X array (Xnew) with the Y array.
Below is some pseudo-code to demonstrate what I amy trying to do. I have attempted this using LMfit and scipy optimize but I am not sure how to make this task work in those packages. For example in LMfit if I tried to minimize the MyOpt function (which passes back a single value of the correlation metric) then it complains that I have more parameters than outputs. However, if I pass back the time series of the corrlation metric (diff) the the parameter values remain fixed at their input values.
I know the reindexing function I am using works because using the rather crude methods similar to the code below give signifianct changes in the mean (diff) metric passed back.
My knowledge of these optimizaiton packages is not up to scratch for this job so if anyone has a suggestion on how to tackle this, I would be greatfull.
def GetNewIndex(Z, a, b ,c):
old_index = np.arange(0, len(Z))
index_adj = some_func(a,b,c)
new_index = old_index + index_adj
max_old = np.max(old_index)
new_index[new_index > max_old] = max_old
new_index[new_index < 0] = 0
return new_index
def MyOpt(params, X, Y ,Z):
a = params['A']
b = params['B']
c = params['C']
# estimate lag (in samples) based on ambient RH
new_index = GetNewIndex(Z, a, b, c)
# assign old values to new locations and convert back to pandas series
Xnew = np.take(X.values, new_index)
Xnew = pd.Series(Xnew, index=X.index)
cc = Y.rolling(1201, center=True).corr(Xnew)
cc = cc.interpolate(limit_direction='both', limit_area=None)
diff = 1-np.abs(cc)
return np.mean(diff)
#==================================================
X = some long pandas time series data
Y = some long pandas time series data
Z = some long pandas time series data
As = [1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2]
Bs = [0, 0 ,0, 1, 1, 1, 0, 0, 0, 1, 1, 1]
Cs = [5, 6, 5, 6, 5, 6, 5, 6, 5, 6, 5, 6]
outs = []
for A, B, C in zip(As, Bs, Cs):
params={'A':A, 'B':B, 'C':C}
out = MyOpt(params, X, Y, Z)
outs.append(out)
I have a tensor t with dim b x 3 and a list of lengths len = [l_0, l_1, ..., l_n]. All entries in len sum to to b. I want to create a new tensor with dim n x 3, which stores the average of the entries in t. E.g. The first l_0 entries in t are averaged and build the first element in the new tensor. The following l_1 entries are averaged and build the second element, ...
Thanks for your help.
You can do so using a combination a cumulative list of indices as helper and a list comprehension to construct the new array:
>>> b, lens = 10, [2, 3, 1, 3, 1]
>>> t = torch.rand(b, 3)
tensor([[0.3567, 0.3998, 0.9396],
[0.4061, 0.6465, 0.6955],
[0.3500, 0.4135, 0.5288],
[0.0726, 0.9575, 0.3785],
[0.6216, 0.2975, 0.3293],
[0.3878, 0.0735, 0.8181],
[0.1694, 0.5446, 0.1179],
[0.7793, 0.6613, 0.1748],
[0.0964, 0.9825, 0.1651],
[0.1421, 0.0994, 0.8086]])
Build the cumulative list of indices:
>>> c = torch.cumsum(torch.tensor([0] + lens), 0)
tensor([ 0, 2, 5, 6, 9, 10])
Loop over c by twos, with an overlapping window. For example zip(c[:-1], c[1:]) works well. Each selection from i to j gets averaged on dim=0.
>>> [t[i:j].sum(0) for i, j in zip(c[:-1], c[1:])]
[tensor([0.7628, 1.0463, 1.6351]),
tensor([1.0442, 1.6685, 1.2367]),
tensor([0.3878, 0.0735, 0.8181]),
tensor([1.0451, 2.1885, 0.4578]),
tensor([0.1421, 0.0994, 0.8086])]
Then you can stack the list:
>>> torch.stack([t[i:j].sum(0) for i, j in zip(c[:-1], c[1:])])
tensor([[0.7628, 1.0463, 1.6351],
[1.0442, 1.6685, 1.2367],
[0.3878, 0.0735, 0.8181],
[1.0451, 2.1885, 0.4578],
[0.1421, 0.0994, 0.8086]])
Please help me understand how to code the following task in Python using input
Programming challenge description:
Write a short Python program that takes two arrays a and b of length n
storing int values, and returns the dot product of a and b. That is, it returns
an array c of length n such that c[i] = a[i] · b[i], for i = 0,...,n−1.
Test Input:
List1's input ==> 1 2 3
List2's input ==> 2 3 4
Expected Output: 2 6 12
Note that the dot product is defined in mathematics to be the sum of the elements of the vector c you want to build.
That said, here is a possibiliy using zip:
c = [x * y for x, y in zip(a, b)]
And the mathematical dot product would be:
sum(x * y for x, y in zip(a, b))
If the lists are read from the keyboard, they will be read as string, you have to convert them before applying the code above.
For instance:
a = [int(s) for s in input().split(",")]
b = [int(s) for s in input().split(",")]
c = [x * y for x, y in zip(a, b)]
Using for loops and appending
list_c = []
for a, b in zip(list_a, list_b):
list_c.append(a*b)
And now the same, but in the more compact list comprehension syntax
list_c = [a*b for a, b in zip(list_a, list_b)]
From iPython
>>> list_a = [1, 2, 3]
>>> list_b = [2, 3, 4]
>>> list_c = [a*b for a, b in zip(list_a, list_b)]
>>> list_c
[2, 6, 12]
The zip function packs the lists together, element-by-element:
>>> list(zip(list_a, list_b))
[(1, 2), (2, 3), (3, 4)]
And we use tuple unpacking to access the elements of each tuple.
From fetching the input and using map & lambda functions to provide the result. If you may want to print the result with spaces between (not as list), use the last line
list1, list2 = [], []
list1 = list(map(int, input().rstrip().split()))
list2 = list(map(int, input().rstrip().split()))
result_list = list(map(lambda x,y : x*y, list1, list2))
print(*result_list)
I came out with two solutions. Both or them are the ones that are expected in a Python introductory course:
#OPTION 1: We use the concatenation operator between lists.
def dot_product_noappend(list_a, list_b):
list_c = []
for i in range(len(list_a)):
list_c = list_c + [list_a[i]*list_b[i]]
return list_c
print(dot_product_noappend([1,2,3],[4,5,6])) #FUNCTION CALL TO SEE RESULT ON SCREEN
#OPTION 2: we use the append method
def dot_product_append(list_a, list_b):
list_c = []
for i in range(len(list_a)):
list_c.append(list_a[i]*list_b[i])
return list_c
print(dot_product_append([1,2,3],[4,5,6])) #FUNCTION CALL TO SEE RESULT ON SCREEN
Just note that the first method requires that you cast the product of integers to be a list before you can concatenate it to list_c. You do that by using braces ([[list_a[i]*list_b[i]] instead of list_a[i]*list_b[i]). Also note that braces are not necessary in the last method, because the append method does not require to pass a list as parameter.
I have added the two function calls with the values you provided, for you to see that it returns the correct result. Choose whatever function you like the most.
I have a dictionary:
d = {'A-A': 1, 'A-B':2, 'A-C':3, 'B-A':5, 'B-B':1, 'B-C':5, 'C-A':3,
'C-B':4, 'C-C': 9}
and a list:
L = [A,B,C]
I have a DataFrame:
df =pd.DataFrame(columns = L, index=L)
I would like to fill each row in df by values in dictionary based on dictionary keys.For example:
A B C
A 1 2 3
B 5 1 5
C 3 4 9
I tried doing that by:
df.loc[L[0]]=[1,2,3]
df.loc[L[1]]=[5,1,5]
df.loc[L[2]] =[3,4,9]
Is there another way to do that especially when there is a huge data?
Thank you for help
Here is another way that I can think of:
import numpy as np
import pandas as pd
# given
d = {'A-A': 1, 'A-B':2, 'A-C':3, 'B-A':5, 'B-B':1, 'B-C':5, 'C-A':3,
'C-B':4, 'C-C': 9}
L = ['A', 'B', 'C']
# copy the key values into a numpy array
z = np.asarray(list(d.values()))
# reshape the array according to your DataFrame
z_new = np.reshape(z, (3, 3))
# copy it into your DataFrame
df = pd.DataFrame(z_new, columns = L, index=L)
This should do the trick, though it's probably not the best way:
for index in L:
prefix = index + "-"
df.loc[index] = [d.get(prefix + column, 0) for column in L]
Calculating the prefix separately beforehand is probably slower for a small list and probably faster for a large list.
Explanation
for index in L:
This iterates through all of the row names.
prefix = index + "-"
All of the keys for each row start with index + "-", e.g. "A-", "B-"… etc..
df.loc[index] =
Set the contents of the entire row.
[ for column in L]
The same as your comma thing ([1, 2, 3]) just for an arbitrary number of items. This is called a "list comprehension".
d.get( , 0)
This is the same as d[ ] but returns 0 if it can't find anything.
prefix + column
Sticks the column on the end, e.g. "A-" gives "A-A", "A-B"…
i‘ m using Python 3.7.
I have an Array like this:
L1 = [1,2,3,-10,8,12,300,17]
Now i want to filter the values(the -10 and the 300 is not okay)
The values in the array may be different but always counting up or counting down.
Has Python 3 a integrated function for that?
The result should look like this:
L1 = [1,2,3,8,12,17]
Thank you !
Edit from comments:
I want to keep each element if it is only a certain distance (toleranz: 10 f.e.) distance away from the one before.
Your array is a list. You can use built in functions:
L1 = [1,2,3,-10,8,12,300,17]
min_val = min(L1) # -10
max_val = max(L1) # 300
p = list(filter(lambda x: min_val < x < max_val, L1)) # all x not -10 or 300
print(p) # [1, 2, 3, 8, 12, 17]
Doku:
min()
max()
filter()
If you want instead an incremental filter you go through your list of datapoints and decide if to keep or not:
delta = 10
result = []
last = L1[0] # first one as last value .. check the remaining list L1[1:]
for elem in L1[1:]:
if last-delta < elem < last+delta:
result.append(last)
last = elem
if elem-delta < result[-1] < elem+delta :
result.append(elem)
print(result) # [1, 2, 3, 8, 12, 17]