How to select best kernel_size and max_pool_size in CNN1D - conv-neural-network

I have data with shape size 1,89. setup kernel_size = 3 and pool_size = 2 on the conv1d layer. However, the model is not able to predict the peak well. i think the problem is because the kernel_size and pool_size that i set are not suitable for my data. how to determine good kernel_size and pool_size. thank you.
the Data : 0.0, 0.0, 84.0, 81.0, 60.0, 22.0, 0.0, 0.0, 0.0, 0.0, 37.0,
121.0, 116.0, 1.0, 10.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 92.0, 90.0, 74.0, 46.0, 0.0, 0.0, 0.0, 0.0, 28.0, 121.0, 117.0, 90.0, 54.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 47.0, 62.0, 54.0, 57.0, 23.0, 63.0, 26.0, 62.0, 52.0, 138.0, 126.0, 98.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 19.0, 44.0, 74.0, 89.0, 119.0, 77.0, 141.0, 137.0, 119.0, 0.0, 0.0, 0.0, 0.0, 91.0, 115.0, 89.0, 143.0, 146.0, 45.0, 0.0, 0.0, 0.0, 65.0, 89.0, 1.0, 0.0, 0.0, 0.0
Data shape: 15000,89
model structure :
model = Sequential()
model.add(Conv1D(64,kernel_size=5,strides=1,padding='same',data_format='channels_last',input_shape=(X_train.shape[1],1)))
model.add(Dropout(0.1))
model.add(Activation('relu'))
model.add(MaxPooling1D(pool_size = 4, strides = 1))
model.add(Conv1D(32,kernel_size=5,strides=1,padding='same'))
model.add(Dropout(0.1))
model.add(Activation('relu'))
model.add(MaxPooling1D(pool_size = 4, strides = 1))
model.add(Conv1D(16,kernel_size=5,strides=1,padding='same'))
model.add(Dropout(0.1))
model.add(Activation('relu'))
model.add(MaxPooling1D(pool_size = 4, strides = 1))
model.add(Flatten())
model.add(Dense(64,activation='relu'))
model.add(Dense(32,activation='relu'))
model.add(Dense(16,activation='relu'))
model.add(Dense(1,activation='sigmoid'))
Prediction Result:

Related

Panda how to overwrite new value on previous value?

I have the following code :
# Append columns to an empty DataFrame.
self.df = pd.DataFrame(columns = ["ID", "Features"],index=['index1'])
tracking_id = output[4]
print(tracking_id in set(self.df['ID']))
if tracking_id in self.df['ID'] :
df2 = pd.DataFrame([[tracking_id], features])
self.df.update(df2)
print(self.df)
else :
self.df = self.df.append({'ID' : tracking_id, 'Features' : features}, ignore_index = True)
In this code, first I check whether an element with the same İD is available or not. Is it is available, previous feature value should be updated with the new one. Actually it works but not correctly.
My output is :
ID Features
0 NaN NaN
1 1.0 [[1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 50.0...
2 4.0 [[0.0, 0.0, 0.0, 1.0, 89.0, 15.0, 0.0, 0.0, 10...
3 4.0 [[70.0, 41.0, 17.0, 41.0, 4.0, 0.0, 0.0, 0.0, ...
4 4.0 [[42.0, 18.0, 16.0, 14.0, 2.0, 0.0, 0.0, 0.0, ...
5 6.0 [[3.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 8.0, 59.0...
6 6.0 [[0.0, 6.0, 7.0, 9.0, 12.0, 3.0, 0.0, 0.0, 51....
As you can see, the same value is appended like absent İD on the list. However sometimes it is not added to the list just is overwrited to previous ones. How can I solve this problem ?

How to write a custom loss function in LGBM?

I have a binary cross-entropy implementation in Keras. I would like to implement the same one in LGBM as a custom loss. Now I understand LGBM of course has 'binary' objective built-in but I would like to implement this one custom-made on my own as a starter for some future enhancements.
Here is the code,
def custom_binary_loss(y_true, y_pred):
"""
Keras version of binary cross-entropy (works like charm!)
"""
# https://github.com/tensorflow/tensorflow/blob/v2.3.1/tensorflow/python/keras/backend.py#L4826
y_pred = K.clip(y_pred, K.epsilon(), 1 - K.epsilon())
term_0 = (1 - y_true) * K.log(1 - y_pred + K.epsilon()) # Cancels out when target is 1
term_1 = y_true * K.log(y_pred + K.epsilon()) # Cancels out when target is 0
return -K.mean(term_0 + term_1, axis=1)
# --------------------
def custom_binary_loss_lgbm(y_pred, train_data):
"""
LGBM version of binary cross-entropy
"""
y_pred = 1.0 / (1.0 + np.exp(-y_pred))
y_true = train_data.get_label()
y_true = np.expand_dims(y_true, axis=1)
y_pred = np.expand_dims(y_pred, axis=1)
epsilon_ = 1e-7
y_pred = np.clip(y_pred, epsilon_, 1 - epsilon_)
term_0 = (1 - y_true) * np.log(1 - y_pred + epsilon_) # Cancels out when target is 1
term_1 = y_true * np.log(y_pred + epsilon_) # Cancels out when target is 0
grad = -np.mean(term_0 + term_1, axis=1)
hess = np.ones(grad.shape)
return grad, hess
But using the above my LGBM model only predicts zeros. Now my dataset is balanced and everything looks cool so what's the error here?
params = {
'objective': 'binary',
'num_iterations': 100,
'seed': 21
}
ds_train = lgb.Dataset(df_train[predictors], y, free_raw_data=False)
reg_lgbm = lgb.train(params=params, train_set=ds_train, fobj=custom_binary_loss_lgbm)
I also tried with a different hessian hess = (y_pred * (1. - y_pred)).flatten(). Although I don't know what hessian really means it didn't work either!
list(map(lambda x: 1.0 / (1.0 + np.exp(-x)), reg_lgbm.predict(df_train[predictors])))
[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, .............]

Try setting the metric parameter to the string "None" in params, like this:
params = {
'objective': 'binary',
'metric': 'None',
'num_iterations': 100,
'seed': 21
}
Otherwise, according to the documentation, the algorithm would choose a default evaluation method for objective set to 'binary'

can some one help to fit the array in kmeans clustering

when i try to fit it in kmeans clustering it throws error "ValueError: setting an array element with a sequence."
from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=5)
kmeans.fit(df)
Array decription.
Name: Vector, Length: 179, dtype: object
0 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
1 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
10 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
100 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
101 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...

Your column has a list in it. It needs to be opened up into multiple columns before passing it to KMeans.
df = pd.read_json('/Users/roshansk/Downloads/NewsArticles.json')
#Extracting the vectors into columns
vectors = df.Vector.apply(pd.Seriesies)
from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=5)
kmeans.fit(vectors)

Get content from nested dictionary more effiiciently

I am iterating through nested dictionaries and fetching all values in different list. I have already created the code that does the job. But I need it to be more efficient. Does anyone know a more efficient way?
Code:
import collections
dict = {
0.5: {u'Start': 0.0, u'Decision Logic#3': 5.0, u'Stop': 0.0, u'Decision Action#1': 5.0, u'Field Level DQ': 120.0},
1.0: {u'Start': 0.0, u'Decision Logic#3': 5.0, u'Stop': 0.0, u'Decision Action#1': 5.0, u'Field Level DQ': 115.0},
2.0: {u'Start': 0.0, u'Decision Logic#3': 5.0, u'Stop': 0.0, u'Decision Action#1': 5.0, u'Field Level DQ': 120.0},
4.0: {u'Start': 0.0, u'Decision Logic#3': 5.0, u'Stop': 0.0, u'Decision Action#1': 5.0, u'Field Level DQ': 120.0},
32.0: {u'Start': 3.0, u'Decision Logic#3': 5.0, u'Stop': 0.0, u'Decision Action#1': 5.0, u'Field Level DQ': 120.0},
8.0: {u'Start': 0.0, u'Decision Logic#3': 5.0, u'Stop': 0.0, u'Decision Action#1': 5.0, u'Field Level DQ': 115.0},
64.0: {u'Start': 2.0, u'Decision Logic#3': 5.0, u'Stop': 0.0, u'Decision Action#1': 5.0, u'Field Level DQ': 123.0},
128.0: {u'Start': 5.0, u'Decision Logic#3': 5.0, u'Stop': 0.0, u'Decision Action#1': 5.0, u'Field Level DQ': 122.0},
256.0: {u'Start': 3.0, u'Decision Logic#3': 5.0, u'Stop': 0.0, u'Decision Action#1': 5.0, u'Field Level DQ': 121.0},
16.0: {u'Start': 0.0, u'Decision Logic#3': 5.0, u'Stop': 0.0, u'Decision Action#1': 5.0, u'Field Level DQ': 120.0}
64.0: {u'Watermark': 100.0, u'Decision Action#1': 5.0, u'Stop': 0.0,
u'Decision Logic#3': 5.0, u'Start': 5.0,
u'Token Maskin Action#1': 425.0,
u'Field Level DQ': 122.0}
}
dict = collections.OrderedDict(sorted(dict.items()))
list_start, decision_logic, stop, decision_action, field_dq = [], [], [], [], []
for main_key, main_val in dict.items():
if "Start" in main_val:
list_start.append(main_val['Start'])
if "Decision Logic#3" in main_val:
decision_logic.append(main_val['Decision Logic#3'])
if "Stop" in main_val:
stop.append(main_val['Stop'])
if "Decision Action#1" in main_val:
decision_action.append(main_val['Decision Action#1'])
if "Field Level DQ" in main_val:
field_dq.append(main_val['Field Level DQ'])
print(list_start)
print(decision_logic)
print(stop)
print(decision_action)
print(field_dq)
The output i want is like it should create a list automatically(generic to be precise) if i add more key in nested dictionary in this way:
{
'list_start' : [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 2.0, 5.0, 3.0]
'decision_logic' : [5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0]
'stop' : [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
'decision_action' : [5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0]
'field_dq' : [120.0, 115.0, 120.0, 120.0, 115.0, 120.0, 120.0, 123.0, 122.0, 121.0]
}
Thanks in advance.

You can use List Comprehension:
import collections
dict1 = collections.OrderedDict(sorted(dict.items()))
list_start = [j['Start'] for i,j in dict1.items()]
decision_logic = [j['Decision Logic#3'] for i,j in dict1.items()]
stop = [j['Stop'] for i,j in dict1.items()]
decision_action = [j['Decision Action#1'] for i,j in dict1.items()]
field_dq = [j['Field Level DQ'] for i,j in dict1.items()]
# Output
# list start = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 2.0, 5.0, 3.0]
# decision_logic = [5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0]
# stop = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
# decision_action = [5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0]
# field_dq = [120.0, 115.0, 120.0, 120.0, 115.0, 120.0, 120.0, 123.0, 122.0, 121.0]

i think i modified my answer and i successfully created code in more generic way.As i said the key-value pair(like Start, Decision Action, Stop) of list of dictionaries can be exist or may not be and also there is a chance of adding new nested dictionaries like this so this cannot be made as static.Anyways here is the generic code :
dict = collections.OrderedDict(sorted(dict.items()))
get_total_list = []
for i, j in dict.items():
for item in j:
get_total_list.append(item) if item not in get_total_list else None
print("Total Legends :", get_total_list)
main_list = {}
for item in get_total_list:
sublist = []
for i, j in dict.items():
if item in j:
sublist.append(j[item])
main_list[item] = sublist
print(main_list)
# # Output:
{u'Watermark': [100.0],
u'Decision Action#1': [5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0],
u'Decision Logic#3': [5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0],
u'Stop': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
u'Start': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.0, 5.0, 5.0, 3.0],
u'Token Maskin Action#1': [425.0],
u'Field Level DQ': [120.0, 115.0, 120.0, 120.0, 115.0, 120.0, 120.0, 122.0,
122.0, 121.0]}

sympy solve() gives implicit/incorrect answer

I'm trying to solve an equation system with 16 equations and 16 unknowns using sympy but it doesn't seem to solve it well.
I want to solve the system [K][d]=[f] where [K] is the coefficients matrix, [d] the unknowns and [f] are constants. I know some unknowns "d" and some constants "f", so I have same number for both equations and unknowns, but when I substitute these values into the equations and try to solve it the results for all "dx" include "dx8". I checked the matrix determinant and is positive so I should get a unique answer.
Here is the code:
import sympy as sp
import numpy as np
K = np.array([[560000000.0, 0.0, -480000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0,-80000000.0, 120000000.0, 0.0, -200000000.0, 0.0, 0.0, 0.0, 0.0],
[0.0, 393333333.3, 120000000.0, -180000000.0, 0.0, 0.0, 0.0, 0.0,80000000.0, -213333333.3, -200000000.0, 0.0, 0.0, 0.0, 0.0, 0.0],
[-480000000.0, 120000000.0, 1120000000.0, -200000000.0,-480000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0, -160000000.0,200000000.0, 0.0, -200000000.0, 0.0, 0.0],
[80000000.0, -180000000.0, -200000000.0, 786666666.7, 120000000.0,-180000000.0, 0.0, 0.0, 0.0, 0.0, 200000000.0, -426666666.7,-200000000.0, 0.0, 0.0, 0.0],
[0.0, 0.0, -480000000.0, 120000000.0, 1120000000.0, -200000000.0,-480000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0, -160000000.0,200000000.0, 0.0, -200000000.0],
[0.0, 0.0, 80000000.0, -180000000.0, -200000000.0, 786666666.7,120000000.0, -180000000.0, 0.0, 0.0, 0.0, 0.0, 200000000.0,-426666666.7, -200000000.0, 0.0],
[0.0, 0.0, 0.0, 0.0, -480000000.0, 120000000.0, 560000000.0,-200000000.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -80000000.0, 80000000.0],
[0.0, 0.0, 0.0, 0.0, 80000000.0, -180000000.0, -200000000.0,393333333.3, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 120000000.0,-213333333.3],
[-80000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 560000000.0,-200000000.0, -480000000.0, 120000000.0, 0.0, 0.0, 0.0, 0.0],
[120000000.0, -213333333.3, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,-200000000.0, 393333333.3, 80000000.0, -180000000.0, 0.0, 0.0, 0.0,0.0],
[0.0, -200000000.0, -160000000.0, 200000000.0, 0.0, 0.0, 0.0, 0.0,-480000000.0, 80000000.0, 1120000000.0, -200000000.0, -480000000.0,120000000.0, 0.0, 0.0],
[-200000000.0, 0.0, 200000000.0, -426666666.7, 0.0, 0.0, 0.0, 0.0,120000000.0, -180000000.0, -200000000.0, 786666666.7, 80000000.0,-180000000.0, 0.0, 0.0],
[0.0, 0.0, 0.0, -200000000.0, -160000000.0, 200000000.0, 0.0, 0.0,0.0, 0.0, -480000000.0, 80000000.0, 1120000000.0, -200000000.0,-480000000.0, 120000000.0],
[0.0, 0.0, -200000000.0, 0.0, 200000000.0, -426666666.7, 0.0, 0.0,0.0, 0.0, 120000000.0, -180000000.0, -200000000.0, 786666666.7,80000000.0, -180000000.0],
[0.0, 0.0, 0.0, 0.0, 0.0, -200000000.0, -80000000.0, 120000000.0,0.0, 0.0, 0.0, 0.0, -480000000.0, 80000000.0, 560000000.0, 0.0],
[0.0, 0.0, 0.0, 0.0, -200000000.0, 0.0, 80000000.0, -213333333.3,0.0, 0.0, 0.0, 0.0, 120000000.0, -180000000.0, 0.0, 393333333.3]])
x = [sp.var('dx'+ str(i+1)) for i in range(8)]
y = [sp.var('dy'+ str(i+1)) for i in range(8)]
fx = [sp.var('fx'+ str(i+1)) for i in range(8)]
fy = [sp.var('fy'+ str(i+1)) for i in range(8)]
xy = list(sum(zip(x, y), ()))
fxy = list(sum(zip(fx, fy), ()))
M = sp.Matrix(K)*sp.Matrix(xy)
Ec = [sp.Eq(M[i], fxy[i]) for i in range(16)]
#known values
d_kwn = [(dy1, 0), (dy2, 0), (dy3, 0), (dy4, 0)]
f_kwn = [(fx5, 0), (fy5, 0), (fx6, 0), (fy6, -3000), (fx7, 0), (fy7, -3000),(fx8, 0), (fy8, 0), (fx1, 0), (fx2, 0), (fx3, 0), (fx4, 0)]
for var in d_kwn:
for i, eq in enumerate(Ec):
Ec[i] = eq.subs(var[0], var[1])
for var in f_kwn:
for i, eq in enumerate(Ec):
Ec[i] = eq.subs(var[0], var[1])
Sols = sp.solvers.solve(Ec)
sp.Matrix(sorted(Sols.items(), key=str))
And this is the output I'm getting:
{dx1: dx8−3.54468009860439⋅10−6,
dx2: dx8−1.8414987360977⋅10−6,
dx3: dx8−2.11496606381994⋅10−7,
dx4: dx8+2.05943267588118⋅10−7,
dx5: dx8−1.24937663359153⋅10−6,
dx6: dx8−1.55655946713284⋅10−6,
dx7: dx8−1.08797652070783⋅10−6,
dy5: −2.10639657360695⋅10−6,
dy6: −6.26959460018537⋅10−6,
dy7: −6.32191585665888⋅10−6,
dy8: −2.7105825114088⋅10−6,
fy1: 439.746516706791,
fy2: 2640.65618690176,
fy3: 2399.44807607611,
fy4: 520.14922031534}
I don't know why I'm not getting a result for dx8. I tried adding more equations because theoretically: dx1 = dx4, dx2 = dx3, dx5 = dx8, dx6 = dx7 and so on. But it gives me and empty list.
Any help will be appreciated.

If you need to use Sympy, then the following may work. First we can solve the reduced system of equations only for unknown d values. Then once we know all d values we can calculate the unknown f values by doing [K][d]=[f] for only the unknown f equation numbers (not implemented in the code below).
import sympy as sp
import numpy as np
K = np.array([[560000000.0, 0.0, -480000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0,-80000000.0, 120000000.0, 0.0, -200000000.0, 0.0, 0.0, 0.0, 0.0],
[0.0, 393333333.3, 120000000.0, -180000000.0, 0.0, 0.0, 0.0, 0.0,80000000.0, -213333333.3, -200000000.0, 0.0, 0.0, 0.0, 0.0, 0.0],
[-480000000.0, 120000000.0, 1120000000.0, -200000000.0,-480000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0, -160000000.0,200000000.0, 0.0, -200000000.0, 0.0, 0.0],
[80000000.0, -180000000.0, -200000000.0, 786666666.7, 120000000.0,-180000000.0, 0.0, 0.0, 0.0, 0.0, 200000000.0, -426666666.7,-200000000.0, 0.0, 0.0, 0.0],
[0.0, 0.0, -480000000.0, 120000000.0, 1120000000.0, -200000000.0,-480000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0, -160000000.0,200000000.0, 0.0, -200000000.0],
[0.0, 0.0, 80000000.0, -180000000.0, -200000000.0, 786666666.7,120000000.0, -180000000.0, 0.0, 0.0, 0.0, 0.0, 200000000.0,-426666666.7, -200000000.0, 0.0],
[0.0, 0.0, 0.0, 0.0, -480000000.0, 120000000.0, 560000000.0,-200000000.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -80000000.0, 80000000.0],
[0.0, 0.0, 0.0, 0.0, 80000000.0, -180000000.0, -200000000.0,393333333.3, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 120000000.0,-213333333.3],
[-80000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 560000000.0,-200000000.0, -480000000.0, 120000000.0, 0.0, 0.0, 0.0, 0.0],
[120000000.0, -213333333.3, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,-200000000.0, 393333333.3, 80000000.0, -180000000.0, 0.0, 0.0, 0.0,0.0],
[0.0, -200000000.0, -160000000.0, 200000000.0, 0.0, 0.0, 0.0, 0.0,-480000000.0, 80000000.0, 1120000000.0, -200000000.0, -480000000.0,120000000.0, 0.0, 0.0],
[-200000000.0, 0.0, 200000000.0, -426666666.7, 0.0, 0.0, 0.0, 0.0,120000000.0, -180000000.0, -200000000.0, 786666666.7, 80000000.0,-180000000.0, 0.0, 0.0],
[0.0, 0.0, 0.0, -200000000.0, -160000000.0, 200000000.0, 0.0, 0.0,0.0, 0.0, -480000000.0, 80000000.0, 1120000000.0, -200000000.0,-480000000.0, 120000000.0],
[0.0, 0.0, -200000000.0, 0.0, 200000000.0, -426666666.7, 0.0, 0.0,0.0, 0.0, 120000000.0, -180000000.0, -200000000.0, 786666666.7,80000000.0, -180000000.0],
[0.0, 0.0, 0.0, 0.0, 0.0, -200000000.0, -80000000.0, 120000000.0,0.0, 0.0, 0.0, 0.0, -480000000.0, 80000000.0, 560000000.0, 0.0],
[0.0, 0.0, 0.0, 0.0, -200000000.0, 0.0, 80000000.0, -213333333.3,0.0, 0.0, 0.0, 0.0, 120000000.0, -180000000.0, 0.0, 393333333.3]])
x = [sp.var('dx'+ str(i+1)) for i in range(8)]
y = [sp.var('dy'+ str(i+1)) for i in range(8)]
fx = [sp.var('fx'+ str(i+1)) for i in range(8)]
fy = [sp.var('fy'+ str(i+1)) for i in range(8)]
xy = list(sum(zip(x, y), ()))
fxy = list(sum(zip(fx, fy), ()))
M = sp.Matrix(K)*sp.Matrix(xy)
Ec = [sp.Eq(M[i], fxy[i]) for i in range(16)]
#known values
d_kwn = [(dy1, 0), (dy2, 0), (dy3, 0), (dy4, 0)]
f_kwn = [(fx5, 0), (fy5, 0), (fx6, 0), (fy6, -3000), (fx7, 0), (fy7, -3000),(fx8, 0), (fy8, 0), (fx1, 0), (fx2, 0), (fx3, 0), (fx4, 0)]
for var in d_kwn:
for i, eq in enumerate(Ec):
Ec[i] = eq.subs(var[0], var[1])
for var in f_kwn:
for i, eq in enumerate(Ec):
Ec[i] = eq.subs(var[0], var[1])
Ec_part = []
for i in [0,2,4,6,8,9,10,11,12,13,14,15]:
Ec_part.append(Ec[i])
unknwns = [*x, *y[4:8]]
Sols = sp.linsolve(Ec_part,unknwns)
Sols = next( iter(Sols) )
#sp.Matrix(sorted(Sols.items(), key=str))

It is convenient to solve system of linear equations in Numpy itself. The type of system you are solving appears in Finite Element Analysis often with boundary conditions. Is it fine if we only use Numpy? If yes, the following code will do the job. We already know which elements of f and d are known we can use Numpy array indexing to solve the reduced set of equations as follows:
import numpy as np
# The NxN Coefficients matrix
K = np.array([[560000000.0, 0.0, -480000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0,-80000000.0, 120000000.0, 0.0, -200000000.0, 0.0, 0.0, 0.0, 0.0],
[0.0, 393333333.3, 120000000.0, -180000000.0, 0.0, 0.0, 0.0, 0.0,80000000.0, -213333333.3, -200000000.0, 0.0, 0.0, 0.0, 0.0, 0.0],
[-480000000.0, 120000000.0, 1120000000.0, -200000000.0,-480000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0, -160000000.0,200000000.0, 0.0, -200000000.0, 0.0, 0.0],
[80000000.0, -180000000.0, -200000000.0, 786666666.7, 120000000.0,-180000000.0, 0.0, 0.0, 0.0, 0.0, 200000000.0, -426666666.7,-200000000.0, 0.0, 0.0, 0.0],
[0.0, 0.0, -480000000.0, 120000000.0, 1120000000.0, -200000000.0,-480000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0, -160000000.0,200000000.0, 0.0, -200000000.0],
[0.0, 0.0, 80000000.0, -180000000.0, -200000000.0, 786666666.7,120000000.0, -180000000.0, 0.0, 0.0, 0.0, 0.0, 200000000.0,-426666666.7, -200000000.0, 0.0],
[0.0, 0.0, 0.0, 0.0, -480000000.0, 120000000.0, 560000000.0,-200000000.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, -80000000.0, 80000000.0],
[0.0, 0.0, 0.0, 0.0, 80000000.0, -180000000.0, -200000000.0,393333333.3, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 120000000.0,-213333333.3],
[-80000000.0, 80000000.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 560000000.0,-200000000.0, -480000000.0, 120000000.0, 0.0, 0.0, 0.0, 0.0],
[120000000.0, -213333333.3, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,-200000000.0, 393333333.3, 80000000.0, -180000000.0, 0.0, 0.0, 0.0,0.0],
[0.0, -200000000.0, -160000000.0, 200000000.0, 0.0, 0.0, 0.0, 0.0,-480000000.0, 80000000.0, 1120000000.0, -200000000.0, -480000000.0,120000000.0, 0.0, 0.0],
[-200000000.0, 0.0, 200000000.0, -426666666.7, 0.0, 0.0, 0.0, 0.0,120000000.0, -180000000.0, -200000000.0, 786666666.7, 80000000.0,-180000000.0, 0.0, 0.0],
[0.0, 0.0, 0.0, -200000000.0, -160000000.0, 200000000.0, 0.0, 0.0,0.0, 0.0, -480000000.0, 80000000.0, 1120000000.0, -200000000.0,-480000000.0, 120000000.0],
[0.0, 0.0, -200000000.0, 0.0, 200000000.0, -426666666.7, 0.0, 0.0,0.0, 0.0, 120000000.0, -180000000.0, -200000000.0, 786666666.7,80000000.0, -180000000.0],
[0.0, 0.0, 0.0, 0.0, 0.0, -200000000.0, -80000000.0, 120000000.0,0.0, 0.0, 0.0, 0.0, -480000000.0, 80000000.0, 560000000.0, 0.0],
[0.0, 0.0, 0.0, 0.0, -200000000.0, 0.0, 80000000.0, -213333333.3,0.0, 0.0, 0.0, 0.0, 120000000.0, -180000000.0, 0.0, 393333333.3]])
# A logical array for indexing
N = K.shape[0] # The number of columns in K
N_2 = int(N/2);
# Prepare the 'f'
fx = np.zeros( N_2 );
fy = np.zeros( N_2 );
fx[ [0,1,2,3,4,5,6,7] ] = np.array([0]*N_2) # Known values of fx
fy[ [4,5,6,7] ] = np.array([0,-3000,-3000,0])
f = np.concatenate( (fx,fy) )
# Solve for the unknown equations only
d = np.zeros( N )
rows = np.array([0,1,2,3,4,5,6,7,12,13,14,15])
rows = rows[:, np.newaxis]
columns = np.array([0,1,2,3,4,5,6,7,12,13,14,15])
d[ columns ] = np.linalg.solve( K[ rows, columns ], f[ columns ] )
# Calculate unknown f values
f[ [8,9,10,11] ] = K[ [8,9,10,11], [8,9,10,11] ]*d[[8,9,10,11]]

Resources