Tensorflow: Custom loss function to encourage/restrict false positives/negatives - keras

Question

I am looking for a way to encourage/restrict false positives/negatives. I have not been able to find a solution, which I can get working - most likely due to my lack of experience.
I found this post: Custom loss function in Keras to penalize false negatives which pretty much has the same purpose, but I cannot get the answer to work.
First I kept on getting this error:
AttributeError: 'Tensor' object has no attribute '_numpy'
After some searching around I found that it could be solved with "model.run_eagerly = True", this though provides this error:
No gradients provided for any variable
I would like not to use "model.run_eagerly = True" given that I am not entirely sure what it does, but it quite obviously makes it so that I need to do something with calculating gradients. I have therefore altered the code, but keep getting the No gradients provided for any variable.
I have made an example of this:
import tensorflow as tf
import tensorflow.keras.backend as K
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
class Custom_loss_Class(tf.keras.losses.Loss):
def __init__(self, recall_weight = 0.5, spec_weight = 0.5, name="custom"):
super().__init__(name=name)
self.recall_weight = recall_weight
self.spec_weight = spec_weight
def call(self, y_true, y_pred):
y_pred = tf.math.round(y_pred)
y_true = tf.math.round(y_true)
TN = tf.dtypes.cast(tf.math.logical_and(tf.math.equal(y_true, 0), tf.math.equal(y_pred, 0)), tf.float32)
TP = tf.dtypes.cast(tf.math.logical_and(tf.math.equal(y_true, 1), tf.math.equal(y_pred, 1)), tf.float32)
FP = tf.dtypes.cast(tf.math.logical_and(tf.math.equal(y_true, 0), tf.math.equal(y_pred, 1)), tf.float32)
FN = tf.dtypes.cast(tf.math.logical_and(tf.math.equal(y_true, 1), tf.math.equal(y_pred, 0)), tf.float32)
TN = tf.reduce_sum(TN)
TP = tf.reduce_sum(TP)
FP = tf.reduce_sum(FP)
FN = tf.reduce_sum(FN)
specificity = TN / (TN + FP + K.epsilon())
recall = TP / (TP + FN + K.epsilon())
loss = tf.Variable(1- (self.recall_weight * recall + self.spec_weight * specificity))
return loss
data = load_breast_cancer()
X_train, X_test, Y_train, Y_test = train_test_split(data.data, data.target, test_size=0.3)
N, D = X_train.shape
scalar = StandardScaler()
X_train = scalar.fit_transform(X_train)
X_test = scalar.transform(X_test)
i = Input(shape=(D,))
x = Dense(64, activation="relu")(i)
x = Dense(1, activation="sigmoid")(x)
model = Model(i, x)
model.compile(optimizer="adam",
loss=Custom_loss_Class(recall_weight=0.1, spec_weight=0.9),
metrics="accuracy")
r = model.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=10)
I am aware that I can use class_weight to give one class more weight, but that is not really what I want. In this example I would like to heavily restrict false negatives so a patient with cancer do not get a negative prediction, but at the same time not get too many false positives. It seems like a common use case and I therefore hope someone made a good function for it.
Edit:
I am aware that this most likely is due to my loss function not being differentiable, but I do not know how to change that so I get the described functionality.